[00:01:12] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.20, 6.97, 6.55 [00:01:15] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 5.93, 6.87, 6.58 [00:03:07] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 2.68, 5.50, 6.07 [00:03:14] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 2.55, 5.32, 6.05 [00:27:51] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.21, 7.15, 6.22 [00:29:26] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 6.99, 6.61, 6.04 [00:29:50] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 6.26, 6.84, 6.23 [00:31:22] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 6.72, 6.48, 6.05 [00:33:47] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 4.23, 6.19, 6.15 [00:52:28] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.25, 6.64, 6.25 [00:54:27] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.37, 6.21, 6.15 [01:25:05] PROBLEM - os191 Current Load on os191 is WARNING: LOAD WARNING - total load average: 3.60, 2.74, 1.74 [01:25:59] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.11, 7.06, 6.36 [01:29:05] RECOVERY - os191 Current Load on os191 is OK: LOAD OK - total load average: 2.13, 2.56, 1.91 [01:31:09] `$wgDefaultSkin = "vector-2022"` on fivenightsatfreddyswiki [01:31:19] !log `$wgDefaultSkin = "vector-2022"` on fivenightsatfreddyswiki [01:31:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:31:54] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.54, 6.46, 6.37 [01:47:40] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.96, 6.53, 5.99 [01:49:52] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.32, 6.39, 5.74 [01:51:37] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 4.43, 5.96, 5.93 [01:51:48] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 3.96, 5.55, 5.52 [02:16:16] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.55, 5.99, 5.27 [02:16:44] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.42, 6.22, 5.36 [02:20:44] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 6.79, 6.49, 5.66 [02:28:07] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.04, 7.43, 6.44 [02:28:44] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.21, 6.96, 6.24 [02:32:04] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 6.01, 7.12, 6.58 [02:32:44] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 5.97, 6.54, 6.25 [02:34:03] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 6.13, 6.79, 6.52 [03:13:30] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.98, 6.73, 5.93 [03:13:30] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.66, 6.61, 5.89 [03:19:18] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 5.86, 6.46, 6.07 [03:19:25] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.46, 6.73, 6.26 [03:23:12] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.30, 6.80, 6.28 [03:25:08] PROBLEM - os202 Current Load on os202 is CRITICAL: LOAD CRITICAL - total load average: 8.20, 7.27, 6.52 [03:25:19] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.60, 7.36, 6.60 [03:26:21] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 60.22, 29.81, 13.39 [03:27:58] PROBLEM - db161 APT on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:28:16] PROBLEM - db161 Puppet on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:28:59] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 3.88, 6.59, 6.49 [03:29:16] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 2.61, 6.02, 6.34 [03:29:48] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [03:29:51] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [03:30:01] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [03:30:10] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [03:30:14] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.014 second response time [03:30:24] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [03:30:36] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 10 backends are down. mw151 mw152 mw161 mw171 mw172 mw181 mw153 mw193 mw201 mw202 [03:31:02] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 41% [03:31:11] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:31:20] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 4 backends are down. mw163 mw173 mw191 mw203 [03:31:21] PROBLEM - db161 PowerDNS Recursor on db161 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:31:23] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [03:31:26] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:31:39] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 12 backends are down. mw151 mw161 mw171 mw172 mw181 mw153 mw163 mw173 mw191 mw193 mw201 mw203 [03:31:44] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 10 backends are down. mw151 mw152 mw171 mw172 mw163 mw173 mw191 mw193 mw201 mw202 [03:32:59] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [03:33:02] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 90% [03:34:12] PROBLEM - db161 SSH on db161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:34:14] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.062 second response time [03:34:17] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 3.477 second response time [03:34:36] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [03:36:16] RECOVERY - db161 SSH on db161 is OK: SSH OK - OpenSSH_10.0p2 (protocol 2.0) [03:36:55] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 3.920 second response time [03:37:02] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 27% [03:38:02] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [03:38:36] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 13 backends are down. mw151 mw152 mw161 mw172 mw181 mw153 mw163 mw173 mw191 mw193 mw201 mw202 mw203 [03:39:09] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.930 second response time [03:40:05] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [03:40:22] @Infrastructure Specialists db161 at 98% RAM consumption which is probably why the whole farm is down [03:40:35] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [03:41:02] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [03:41:27] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:42:02] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.085 second response time [03:43:01] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [03:43:03] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [03:43:23] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [03:43:35] PROBLEM - db161 conntrack_table_size on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:43:39] No reponse from infra. I'll depool c2 to see if it improves [03:45:29] !log [petramagna@mwtask181] starting deploy of {'config': True, 'force': True} to all [03:45:41] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 9.561 second response time [03:45:42] RECOVERY - db161 conntrack_table_size on db161 is OK: OK: nf_conntrack is 0 % full [03:45:46] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 5.170 second response time [03:45:50] !log [petramagna@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 21s [03:45:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:46:02] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 4.018 second response time [03:46:05] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.483 second response time [03:46:32] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1518 bytes in 0.082 second response time [03:46:36] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [03:46:54] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.091 second response time [03:47:01] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.225 second response time [03:47:02] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 17% [03:47:20] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [03:47:29] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.249 second response time [03:47:30] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.070 second response time [03:47:31] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.230 second response time [03:47:38] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.084 second response time [03:47:39] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [03:47:44] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [03:48:10] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.082 second response time [03:55:08] PROBLEM - db161 SSH on db161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:57:06] RECOVERY - db161 SSH on db161 is OK: SSH OK - OpenSSH_10.0p2 (protocol 2.0) [04:03:08] RECOVERY - db161 PowerDNS Recursor on db161 is OK: DNS OK: 0.047 seconds response time. db161.fsslc.wtnet returns 10.0.16.128 [04:03:22] !log stop and start db161 from host [04:03:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:03:46] @posix_memalign ^ [04:04:51] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 1.08, 0.47, 0.18 [04:04:52] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 50 minutes ago with 0 failures [04:05:00] RECOVERY - db161 APT on db161 is OK: APT OK: 77 packages available for upgrade (0 critical updates). [04:08:40] !log [universalomega@mwtask181] starting deploy of {'config': True, 'force': True} to all [04:08:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:08:59] !log [universalomega@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 19s [04:09:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:09:54] Thanks! Looks like everything is good now. [04:26:44] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.64, 6.29, 4.20 [04:27:15] PROBLEM - bast161 NTP time on bast161 is WARNING: NTP WARNING: Offset -0.1094664931 secs [04:27:48] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.65, 7.02, 4.59 [04:29:15] RECOVERY - bast161 NTP time on bast161 is OK: NTP OK: Offset -0.01116475463 secs [04:29:47] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.25, 7.14, 4.93 [04:31:45] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.90, 6.68, 5.03 [04:32:44] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 6.50, 6.59, 5.06 [04:51:18] PROBLEM - bast161 NTP time on bast161 is WARNING: NTP WARNING: Offset -0.1108756661 secs [04:53:18] RECOVERY - bast161 NTP time on bast161 is OK: NTP OK: Offset -0.01095190644 secs [08:08:47] PROBLEM - bast161 NTP time on bast161 is WARNING: NTP WARNING: Offset -0.1105199456 secs [08:10:47] RECOVERY - bast161 NTP time on bast161 is OK: NTP OK: Offset -0.005504369736 secs [10:48:07] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.34, 16.22, 12.89 [10:50:07] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 18.32, 17.39, 13.76 [11:22:07] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 27.20, 23.92, 17.53 [11:26:08] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.75, 23.40, 18.72 [11:34:07] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 17.31, 19.75, 18.97 [11:45:50] PROBLEM - bast161 NTP time on bast161 is WARNING: NTP WARNING: Offset -0.1076876819 secs [11:47:50] RECOVERY - bast161 NTP time on bast161 is OK: NTP OK: Offset -0.008545309305 secs [11:57:50] PROBLEM - bast161 NTP time on bast161 is WARNING: NTP WARNING: Offset -0.107885778 secs [11:59:50] RECOVERY - bast161 NTP time on bast161 is OK: NTP OK: Offset -0.01875597239 secs [12:05:06] !log upgrade grafana to v13 [12:05:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:06:08] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.66, 19.91, 17.04 [12:10:07] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 16.21, 19.89, 17.86 [12:12:32] !log upgrade icinga2 on mon181 [12:12:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:29:43] !log apt-get dist-upgrade on mon181 and reboot [12:29:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:42] !log apt-get dist-upgrade on puppet181 and reboot [12:33:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:40:05] PROBLEM - changeprop202 Puppet on changeprop202 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:40:09] PROBLEM - swiftproxy161 Puppet on swiftproxy161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:40:10] PROBLEM - db172 Puppet on db172 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:40:36] PROBLEM - cp201 Puppet on cp201 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:41:19] PROBLEM - swiftproxy171 Puppet on swiftproxy171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:41:23] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): File[/home/agent],File[/home/paladox],File[/home/reception],File[/home/rhinos] [12:42:07] PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Puppet has 38 failures. Last run 2 minutes ago with 38 failures. Failed resources (up to 3 shown): File[/usr/local/bin/sugit],File[/usr/local/sbin/check-and-restart-php],File[/usr/local/bin/safe-service-restart],File[/etc/logrotate.d/puppet] [12:42:32] PROBLEM - mw202 Puppet on mw202 is CRITICAL: CRITICAL: Puppet has 38 failures. Last run 2 minutes ago with 38 failures. Failed resources (up to 3 shown): File[/usr/local/bin/sugit],File[/usr/local/sbin/check-and-restart-php],File[/usr/local/bin/safe-service-restart],File[/etc/logrotate.d/puppet] [12:42:50] PROBLEM - bast161 Puppet on bast161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:43:04] PROBLEM - rdb191 Puppet on rdb191 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:43:05] PROBLEM - mw203 Puppet on mw203 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:43:08] PROBLEM - llm191 Puppet on llm191 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:43:25] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Puppet has 60 failures. Last run 2 minutes ago with 60 failures. Failed resources (up to 3 shown): File[/etc/nagios/nrpe.cfg],File[/usr/lib/nagios/plugins/check_smart],File[/usr/lib/nagios/plugins/check_ipmi_sensors],File[/usr/local/bin/gen_fingerprints] [12:44:17] PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Puppet has 60 failures. Last run 3 minutes ago with 60 failures. Failed resources (up to 3 shown): File[/etc/nagios/nrpe.cfg],File[/usr/lib/nagios/plugins/check_smart],File[/usr/lib/nagios/plugins/check_ipmi_sensors],File[/usr/local/bin/gen_fingerprints] [13:00:18] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/b37e0bc49d36b8934f254a582fc84978d0fb76d9 [13:00:19] 02ssl/03main 07WikiTideBot 03b37e0bc Bot: Auto-update domain lists [13:07:10] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:08:50] RECOVERY - cp201 Puppet on cp201 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:08:58] RECOVERY - changeprop202 Puppet on changeprop202 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:09:02] RECOVERY - swiftproxy161 Puppet on swiftproxy161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:09:25] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [13:09:49] RECOVERY - db172 Puppet on db172 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:10:32] RECOVERY - llm191 Puppet on llm191 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [13:10:41] RECOVERY - mw202 Puppet on mw202 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:10:46] RECOVERY - bast161 Puppet on bast161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:11:01] RECOVERY - rdb191 Puppet on rdb191 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:11:23] RECOVERY - swiftproxy171 Puppet on swiftproxy171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:11:34] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [13:11:34] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [13:12:58] RECOVERY - mw203 Puppet on mw203 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [13:37:35] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 13.51, 13.74, 12.14 [13:39:35] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 11.75, 12.96, 12.05 [13:43:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.60, 14.79, 12.97 [13:45:35] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 15.28, 15.08, 13.32 [13:47:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 17.63, 16.02, 13.87 [13:49:35] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 15.94, 15.79, 14.03 [13:57:35] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 11.16, 13.26, 13.49 [14:04:33] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 15.30, 14.03, 13.70 [14:12:29] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 11.92, 12.92, 13.39 [14:13:19] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 21.40, 17.83, 14.04 [14:15:19] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 12.01, 15.45, 13.63 [14:18:26] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 15.11, 15.00, 14.19 [14:20:25] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 18.15, 16.28, 14.76 [14:26:22] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 13.41, 15.43, 14.97 [14:28:21] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 17.29, 15.75, 15.11 [14:32:49] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 126 packages available for upgrade (4 critical updates). [14:34:41] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 125 packages available for upgrade (4 critical updates). [14:36:33] PROBLEM - mon181 APT on mon181 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [14:38:04] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: APT CRITICAL: 82 packages available for upgrade (4 critical updates). [14:40:16] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 13.91, 15.22, 15.54 [14:43:11] PROBLEM - mwtask161 APT on mwtask161 is CRITICAL: APT CRITICAL: 123 packages available for upgrade (4 critical updates). [14:43:13] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 124 packages available for upgrade (4 critical updates). [14:44:32] PROBLEM - mwtask151 APT on mwtask151 is CRITICAL: APT CRITICAL: 123 packages available for upgrade (4 critical updates). [15:04:06] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 9.47, 11.45, 13.09 [15:22:44] [02puppet] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/75f343275e80 [15:22:44] 02puppet/03paladox-patch-2 07paladox 0375f3432 Update Matomo branch to 5.10.1 [15:22:48] [02puppet] 07paladox opened pull request #4861: Update Matomo branch to 5.10.1 (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4861 [15:22:54] [02puppet] 07paladox merged pull request #4861: Update Matomo branch to 5.10.1 (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4861 [15:22:54] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/adb279f7fa2df51f6f41ba7e8944894ba2e1e415 [15:22:54] 02puppet/03main 07paladox 03adb279f Update Matomo branch to 5.10.1 (#4861) [15:22:56] [02puppet] 07paladox 04deleted 03paladox-patch-2 at 0375f3432 13https://api.github.com/repos/miraheze/puppet/commit/75f3432 [15:58:07] RECOVERY - mwtask181 Disk Space on mwtask181 is OK: DISK OK - free space: / 94803MiB (42.7% inode=95%); [16:01:45] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/maintenance/importDump.php --wiki=sneakysasquatchwiki dump.xml --no-updates (START) [16:01:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:04:34] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/maintenance/importDump.php --wiki=sneakysasquatchwiki dump.xml --no-updates (END - exit=0) [16:04:35] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/maintenance/rebuildall.php --wiki=sneakysasquatchwiki (START) [16:04:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:04:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:46:05] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/maintenance/rebuildall.php --wiki=sneakysasquatchwiki (END - exit=0) [16:46:07] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/maintenance/initSiteStats.php --wiki=sneakysasquatchwiki --update (END - exit=0) [16:46:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:46:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:38:23] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 15.27, 13.74, 12.67 [17:42:21] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 11.82, 12.91, 12.58 [18:53:00] !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php nukeNS --wiki=fantendowiki --delete --ns=1 --ns=3002 --ns=3004 --ns=3006 --ns=3008 --ns=3010 --ns=3012 --ns=3016 --all (START) [18:53:01] !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php nukeNS --wiki=fantendowiki --delete --ns=1 --ns=3002 --ns=3004 --ns=3006 --ns=3008 --ns=3010 --ns=3012 --ns=3016 --all (END - exit=256) [18:53:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:53:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:54:01] !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php nukeNS --wiki=fantendowiki --delete --all --ns=1 (START) [18:54:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:59:28] not me scanning for any opportunities to react with a:OhGODNotAnother67EmojiPLEASESTOP: [19:11:23] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.70, 17.98, 13.64 [19:13:23] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 20.25, 18.60, 14.40 [19:15:10] !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php nukeNS --wiki=fantendowiki --delete --all --ns=1 (END - exit=0) [19:15:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:15:51] !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php nukeNS --wiki=fantendowiki --delete --all --ns=3002 (START) [19:15:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:20:14] !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php nukeNS --wiki=fantendowiki --delete --all --ns=3002 (END - exit=0) [19:20:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:21:09] !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php nukeNS --wiki=fantendowiki --delete --all --ns=3004 (START) [19:21:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:21:23] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.53, 20.11, 16.20 [20:23:23] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 17.33, 19.08, 16.32 [20:26:11] !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php nukeNS --wiki=fantendowiki --delete --all --ns=3004 (END - exit=0) [20:26:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:26:46] !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php nukeNS --wiki=fantendowiki --delete --all --ns=3006 (START) [20:26:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:31:38] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php namespaceDupes --wiki=fantendowiki --fix (END - exit=0) [20:31:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:32:27] [1/4] ``` [20:32:28] [2/4] 3118 links to fix, 3084 were resolvable, 34 were deleted. [20:32:28] [3/4] Looks good! [20:32:28] [4/4] ``` [20:34:48] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php nukeNS --wiki=fantendowiki --delete --all --ns=3010 (START) [20:34:50] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php nukeNS --wiki=fantendowiki --delete --all --ns=3010 (END - exit=0) [20:34:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:34:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:35:12] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php nukeNS --wiki=fantendowiki --delete --all --ns=3012 (START) [20:35:13] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php nukeNS --wiki=fantendowiki --delete --all --ns=3012 (END - exit=0) [20:35:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:35:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:35:29] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php nukeNS --wiki=fantendowiki --delete --all --ns=3016 (START) [20:35:30] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php nukeNS --wiki=fantendowiki --delete --all --ns=3016 (END - exit=0) [20:35:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:35:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:36:45] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php nukeNS --wiki=fantendowiki --delete --all --ns=3008 (START) [20:36:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:37:02] @thewwrnerdguy i got 3008+ fyi [20:37:09] inclusive [20:37:23] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php nukeNS --wiki=fantendowiki --delete --all --ns=3008 (END - exit=0) [20:37:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:40:47] damn you got all the easy ones [20:41:30] :3 [20:41:32] 3004 was running for an hour lmao [20:41:37] god damn [20:41:44] have you been using screen btw [20:41:54] even failing that you can have multiple ssh sessions [20:42:14] !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php nukeNS --wiki=fantendowiki --delete --all --ns=3006 (END - exit=0) [20:42:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:42:23] oh yeah I didn't think of doing that 🤦‍♀️ [20:49:27] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initSiteStats --wiki=fantendowiki (END - exit=0) [20:49:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:07:12] One day I will make mwscript auto open a screen/tmux if it's not inside one [21:08:19] One day might be in like 2050 though so don't hold your breath [21:08:29] do not press ctrl+a c at 2 am, worst mistake myself [21:08:32] Anyway good night as I need to be at train station in 9.5 hours [21:08:40] sleep well and happy travels [21:14:30] [02mw-config] 07pskyechology merged 07dependabot[bot]'s pull request #6432: Update phpunit/phpunit requirement from 13.1.13 to 13.2.0 (03main...03dependabot/composer/phpunit/phpunit-13.2.0) 13https://github.com/miraheze/mw-config/pull/6432 [21:14:31] [02mw-config] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/9bacf57a856d82c9cb61a62a739abeff2550d6f2 [21:14:31] 02mw-config/03main 07dependabot[bot] 039bacf57 Update phpunit/phpunit requirement from 13.1.13 to 13.2.0 (#6432)… [21:14:32] [02mw-config] 07pskyechology 04deleted 03dependabot/composer/phpunit/phpunit-13.2.0 at 0395a4607 13https://api.github.com/repos/miraheze/mw-config/commit/95a4607 [21:15:45] miraheze/mw-config - pskyechology the build passed. [21:16:12] [02mw-config] 07dependabot[bot] 04force-pushed 03dependabot/composer/justinrainbow/json-schema-6.9.0 from 0325da6df to 039e67584 13https://api.github.com/repos/miraheze/mw-config/commits/dependabot/composer/justinrainbow/json-schema-6.9.0 [21:16:12] 02mw-config/03dependabot/composer/justinrainbow/json-schema-6.9.0 07dependabot[bot] 039e67584 Update justinrainbow/json-schema requirement from 6.8.2 to 6.9.0… [21:17:20] miraheze/mw-config - dependabot[bot] the build passed. [21:27:00] [02mw-config] 07pskyechology merged 07dependabot[bot]'s pull request #6433: Update justinrainbow/json-schema requirement from 6.8.2 to 6.9.0 (03main...03dependabot/composer/justinrainbow/json-schema-6.9.0) 13https://github.com/miraheze/mw-config/pull/6433 [21:27:00] [02mw-config] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/b626cd0d0fb5b30a59f7b000620cf1feb5bb3955 [21:27:02] 02mw-config/03main 07dependabot[bot] 03b626cd0 Update justinrainbow/json-schema requirement from 6.8.2 to 6.9.0 (#6433)… [21:27:04] [02mw-config] 07pskyechology 04deleted 03dependabot/composer/justinrainbow/json-schema-6.9.0 at 039e67584 13https://api.github.com/repos/miraheze/mw-config/commit/9e67584 [21:28:12] miraheze/mw-config - pskyechology the build passed. [21:31:19] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 25.95, 20.17, 14.38 [21:33:19] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 23.14, 21.02, 15.40 [21:37:19] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 17.91, 19.86, 16.25 [21:41:19] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 22.61, 21.94, 17.96 [21:44:15] [02mw-config] 07pskyechology created 03configs (+1 new commit) 13https://github.com/miraheze/mw-config/commit/08f9274ba69c [21:44:15] 02mw-config/03configs 07Skye 0308f9274 Set configuration overrides… [21:45:04] [02mw-config] 07pskyechology opened pull request #6434: Set configuration overrides (03main...03configs) 13https://github.com/miraheze/mw-config/pull/6434 [21:46:09] !log [skye@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [21:46:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:46:21] [02mw-config] 07pskyechology merged pull request #6434: Set configuration overrides (03main...03configs) 13https://github.com/miraheze/mw-config/pull/6434 [21:46:23] [02mw-config] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/976d372616f4e455e557666e6dc551d341673a4a [21:46:24] miraheze/mw-config - pskyechology the build passed. [21:46:25] [02mw-config] 07pskyechology 04deleted 03configs at 0308f9274 13https://api.github.com/repos/miraheze/mw-config/commit/08f9274 [21:46:27] 02mw-config/03main 07Skye 03976d372 Set configuration overrides (#6434)… [21:46:27] !log [skye@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [21:46:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:46:34] !log [skye@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [21:46:35] !log [skye@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s [21:46:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:46:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:46:54] !log [skye@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 26s [21:46:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:34] miraheze/mw-config - pskyechology the build passed. [21:49:19] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 11.77, 18.14, 18.14 [21:54:18] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php rebuildall --wiki=fantendowiki (END - exit=-2) [21:54:19] !log [skye@mwtask171] Finished import for fantendowiki (XML: fantendo_pages_full.xml; Images: None) (END - exit=-2) [21:54:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:54:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:55:12] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php refreshLinks --wiki=fantendowiki (START) [21:55:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:43:23] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.48, 19.74, 15.85 [22:47:23] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.53, 22.18, 17.80 [22:49:23] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 16.67, 19.71, 17.40 [23:39:23] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.73, 20.58, 15.45 [23:41:23] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.95, 22.19, 16.65 [23:42:41] !log [void@puppet181] Upgraded packages on prometheus151: apache2, apache2-bin, apache2-data, and apache2-utils [23:42:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:43:04] !log [void@puppet181] Upgraded packages on mwtask151: apache2, apache2-bin, apache2-data, and apache2-utils [23:43:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:43:23] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.05, 22.36, 17.39 [23:43:29] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [23:44:23] !log [void@puppet181] Upgraded packages on mwtask161: apache2, apache2-bin, apache2-data, and apache2-utils [23:44:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:44:28] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 119 packages available for upgrade (0 critical updates). [23:44:38] !log [void@puppet181] Upgraded packages on mon181: apache2-bin and apache2-utils [23:44:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:45:07] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 119 packages available for upgrade (0 critical updates). [23:45:23] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.60, 23.13, 18.27 [23:45:41] !log [void@puppet181] Upgraded packages on mwtask171: apache2, apache2-bin, apache2-data, and apache2-utils [23:45:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:46:05] !log [void@puppet181] Upgraded packages on test151: apache2, apache2-bin, apache2-data, and apache2-utils [23:46:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:46:21] RECOVERY - mon181 APT on mon181 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:46:44] !log [void@puppet181] Upgraded packages on mwtask181: apache2, apache2-bin, apache2-data, and apache2-utils [23:46:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:46:48] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 121 packages available for upgrade (0 critical updates). [23:47:03] RECOVERY - test151 APT on test151 is OK: APT OK: 122 packages available for upgrade (0 critical updates). [23:47:23] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 19.85, 22.11, 18.52 [23:47:26] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 120 packages available for upgrade (0 critical updates). [23:51:23] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 16.78, 19.47, 18.28