[00:00:05] !log [www-data@mwtask181] Began automatic backing up [00:00:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:02:10] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/c18e2643518d58c3d6f3f39277bb04b83691a865 [00:02:10] 02statichelp/03main 07WikiTideBot 03c18e264 Bot: Auto-update Tech namespace pages 2026-07-01 00:02:07 [00:11:23] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 44247MiB (9.8% inode=100%); [00:13:29] !log [reception@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php MirahezeMagic:CheckWikiDatabases --wiki=loginwiki --delete (END - exit=0) [00:13:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:14:05] PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 11.58, 9.49, 6.76 [00:16:04] RECOVERY - db151 Current Load on db151 is OK: LOAD OK - total load average: 9.12, 9.72, 7.20 [00:21:23] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 84477MiB (18.6% inode=100%); [00:37:30] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 45283MiB (10.0% inode=100%); [00:40:51] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 44657MiB (9.8% inode=100%); [00:50:51] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 84337MiB (18.6% inode=100%); [01:03:35] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [01:11:30] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 85044MiB (18.8% inode=100%); [01:31:35] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [02:39:47] PROBLEM - db151 Current Load on db151 is CRITICAL: LOAD CRITICAL - total load average: 13.53, 11.09, 7.82 [02:41:47] PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 8.95, 10.32, 7.94 [02:43:47] RECOVERY - db151 Current Load on db151 is OK: LOAD OK - total load average: 9.39, 9.86, 8.06 [02:47:47] PROBLEM - db151 Current Load on db151 is CRITICAL: LOAD CRITICAL - total load average: 12.62, 10.99, 8.91 [02:49:47] PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 11.21, 11.30, 9.30 [02:53:47] PROBLEM - db151 Current Load on db151 is CRITICAL: LOAD CRITICAL - total load average: 14.37, 12.25, 10.07 [02:55:47] PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 10.97, 11.61, 10.11 [02:57:02] Any @Infrastructure Specialists around to see why db151 is now our problem child? [02:59:47] PROBLEM - db151 Current Load on db151 is CRITICAL: LOAD CRITICAL - total load average: 15.40, 12.48, 10.71 [03:07:47] PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 8.06, 10.81, 11.01 [03:09:47] RECOVERY - db151 Current Load on db151 is OK: LOAD OK - total load average: 1.41, 7.39, 9.73 [03:12:28] !log enable additional filtering for a domain that appears to be getting aggressively scraped [03:12:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:17:56] !log [void@puppet181] Upgraded packages on mattermost2: nginx and nginx-common [03:18:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:18:02] RECOVERY - mattermost2 APT on mattermost2 is OK: APT OK: 74 packages available for upgrade (0 critical updates). [03:18:14] !log [void@puppet181] Upgraded packages on matomo151: nginx and nginx-common [03:18:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:18:39] !log [void@puppet181] Upgraded packages on cp161: nginx and nginx-common [03:18:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:19:26] !log [void@puppet181] Upgraded packages on graylog161: nginx and nginx-common [03:19:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:19:34] RECOVERY - cp161 APT on cp161 is OK: APT OK: 76 packages available for upgrade (0 critical updates). [03:19:39] RECOVERY - graylog161 APT on graylog161 is OK: APT OK: 79 packages available for upgrade (0 critical updates). [03:19:45] !log [void@puppet181] Upgraded packages on llm191: nginx and nginx-common [03:19:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:20:05] RECOVERY - matomo151 APT on matomo151 is OK: APT OK: 77 packages available for upgrade (0 critical updates). [03:20:59] RECOVERY - llm191 APT on llm191 is OK: APT OK: 81 packages available for upgrade (0 critical updates). [03:21:33] RECOVERY - mw152 APT on mw152 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [03:22:23] !log [void@puppet181] Upgraded packages on mw152: nginx and nginx-common [03:22:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:24:00] !log [void@puppet181] Upgraded packages on mw171: nginx and nginx-common [03:24:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:24:17] RECOVERY - mw171 APT on mw171 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [03:24:26] RECOVERY - mw172 APT on mw172 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [03:26:42] !log [void@puppet181] Upgraded packages on mw172: nginx and nginx-common [03:26:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:27:42] !log [void@puppet181] Upgraded packages on mw173: nginx and nginx-common [03:27:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:28:06] !log [void@puppet181] Upgraded packages on mon181: nginx and nginx-common [03:28:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:28:30] RECOVERY - mw173 APT on mw173 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [03:28:36] RECOVERY - mon181 APT on mon181 is OK: APT OK: 7 packages available for upgrade (0 critical updates). [03:29:48] RECOVERY - mw151 APT on mw151 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [03:30:44] !log [void@puppet181] Upgraded packages on mw151: nginx and nginx-common [03:30:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:31:26] RECOVERY - mw162 APT on mw162 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [03:33:28] !log [void@puppet181] Upgraded packages on mw162: nginx and nginx-common [03:33:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:34:51] RECOVERY - mw181 APT on mw181 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [03:35:11] !log [void@puppet181] Upgraded packages on mw181: nginx and nginx-common [03:35:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:36:57] !log [void@puppet181] Upgraded packages on mw153: nginx and nginx-common [03:37:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:37:28] RECOVERY - mw153 APT on mw153 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [03:37:38] RECOVERY - mw161 APT on mw161 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [03:39:08] !log [void@puppet181] Upgraded packages on mw161: nginx and nginx-common [03:39:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:40:27] RECOVERY - mw191 APT on mw191 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [03:41:36] !log [void@puppet181] Upgraded packages on mw191: nginx and nginx-common [03:41:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:43:32] RECOVERY - mw163 APT on mw163 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [03:43:44] !log [void@puppet181] Upgraded packages on mw163: nginx and nginx-common [03:43:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:44:15] RECOVERY - mw193 APT on mw193 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [03:45:38] !log [void@puppet181] Upgraded packages on mw193: nginx and nginx-common [03:45:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:46:12] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 21.41, 18.13, 14.00 [03:47:40] RECOVERY - mw183 APT on mw183 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [03:47:55] !log [void@puppet181] Upgraded packages on mw183: nginx and nginx-common [03:47:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:48:01] PROBLEM - mw183 Current Load on mw183 is CRITICAL: LOAD CRITICAL - total load average: 26.79, 21.62, 16.66 [03:48:12] PROBLEM - mw193 Current Load on mw193 is CRITICAL: LOAD CRITICAL - total load average: 24.04, 20.18, 15.25 [03:49:28] !log [void@puppet181] Upgraded packages on mw192: nginx and nginx-common [03:49:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:50:06] RECOVERY - mw192 APT on mw192 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [03:50:12] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 20.54, 20.27, 15.89 [03:50:29] RECOVERY - mw182 APT on mw182 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [03:51:58] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 23.07, 22.53, 18.17 [03:52:09] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.96, 22.30, 16.77 [03:52:20] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 20.95, 18.98, 15.26 [03:52:40] !log [void@puppet181] Upgraded packages on mw182: nginx and nginx-common [03:52:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:53:59] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.14, 19.68, 15.99 [03:54:02] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 2 backends are down. mw191 mw193 [03:54:15] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 16.71, 18.71, 15.63 [03:54:19] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.04, 18.70, 14.43 [03:54:30] PROBLEM - mw201 Current Load on mw201 is CRITICAL: LOAD CRITICAL - total load average: 26.58, 21.50, 16.85 [03:54:34] RECOVERY - mw201 APT on mw201 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [03:54:47] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 23.47, 21.17, 16.89 [03:54:49] PROBLEM - mw153 Current Load on mw153 is WARNING: LOAD WARNING - total load average: 22.73, 18.26, 14.01 [03:55:24] !log [void@puppet181] Upgraded packages on mw201: nginx and nginx-common [03:55:28] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.14, 19.18, 14.88 [03:55:38] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 23.83, 20.82, 16.73 [03:55:48] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 23.26, 20.20, 15.88 [03:55:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:56:09] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.62, 23.03, 18.35 [03:56:30] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 21.53, 21.52, 17.45 [03:56:40] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.65, 19.63, 15.27 [03:56:43] PROBLEM - mw192 Current Load on mw192 is CRITICAL: LOAD CRITICAL - total load average: 24.28, 22.26, 17.79 [03:56:49] PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 23.00, 20.29, 15.74 [03:56:53] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.75, 19.04, 14.50 [03:57:18] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 21.23, 20.15, 16.57 [03:57:19] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.19, 19.55, 15.07 [03:57:38] PROBLEM - mw202 Current Load on mw202 is CRITICAL: LOAD CRITICAL - total load average: 27.30, 23.09, 18.06 [03:57:51] RECOVERY - mw202 APT on mw202 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [03:58:01] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.69, 19.09, 14.33 [03:58:02] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [03:58:35] !log [void@puppet181] Upgraded packages on mw202: nginx and nginx-common [03:58:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:58:40] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 15.53, 20.57, 17.78 [03:58:40] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 11.47, 17.04, 14.90 [03:58:49] RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 13.96, 18.80, 15.82 [03:58:49] RECOVERY - mw153 Current Load on mw153 is OK: LOAD OK - total load average: 13.96, 18.40, 15.22 [03:58:53] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 15.05, 18.11, 14.77 [03:59:18] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 14.39, 18.39, 16.40 [03:59:19] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 11.57, 16.83, 14.66 [03:59:28] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 11.34, 17.35, 15.34 [03:59:37] RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 10.98, 17.74, 16.04 [03:59:38] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 13.78, 20.38, 17.75 [03:59:44] !log [void@puppet181] Upgraded packages on mw203: nginx and nginx-common [03:59:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:59:49] RECOVERY - mw203 APT on mw203 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [03:59:59] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 10.84, 16.66, 16.17 [04:00:01] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 10.24, 15.46, 13.58 [04:00:09] !log [void@puppet181] Upgraded packages on mwtask151: nginx and nginx-common [04:00:09] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 11.35, 18.54, 17.77 [04:00:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:00:12] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 10.74, 17.70, 17.38 [04:00:19] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 9.46, 16.57, 15.31 [04:00:29] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [04:00:30] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 10.13, 17.41, 16.90 [04:00:31] !log [void@puppet181] Upgraded packages on mwtask161: nginx and nginx-common [04:00:33] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [04:00:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:00:36] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 9.25, 16.67, 16.68 [04:00:54] !log [void@puppet181] Upgraded packages on mwtask171: nginx and nginx-common [04:00:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:01:05] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 113 packages available for upgrade (0 critical updates). [04:01:14] !log [void@puppet181] Upgraded packages on os151: nginx and nginx-common [04:01:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:01:50] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 11.86, 17.37, 18.32 [04:01:53] !log [void@puppet181] Upgraded packages on os162: nginx and nginx-common [04:01:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:02:21] !log [void@puppet181] Upgraded packages on os202: nginx and nginx-common [04:02:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:02:34] RECOVERY - os151 APT on os151 is OK: APT OK: 76 packages available for upgrade (0 critical updates). [04:02:41] !log [void@puppet181] Upgraded packages on puppet181: nginx and nginx-common [04:02:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:02:52] RECOVERY - puppet181 APT on puppet181 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [04:03:00] !log [void@puppet181] Upgraded packages on phorge171: nginx and nginx-common [04:03:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:03:32] !log [void@puppet181] Upgraded packages on test151: nginx and nginx-common [04:03:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:03:50] RECOVERY - test151 APT on test151 is OK: APT OK: 114 packages available for upgrade (0 critical updates). [04:03:52] RECOVERY - os162 APT on os162 is OK: APT OK: 76 packages available for upgrade (0 critical updates). [04:03:53] RECOVERY - os202 APT on os202 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [04:03:53] RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [04:03:53] !log [void@puppet181] Upgraded packages on swiftproxy161: nginx and nginx-common [04:03:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:04:13] !log [void@puppet181] Upgraded packages on swiftproxy171: nginx and nginx-common [04:04:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:04:27] RECOVERY - swiftproxy161 APT on swiftproxy161 is OK: APT OK: 80 packages available for upgrade (0 critical updates). [04:04:37] !log [void@puppet181] Upgraded packages on mwtask181: nginx and nginx-common [04:04:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:05:23] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 114 packages available for upgrade (0 critical updates). [04:06:06] RECOVERY - swiftproxy171 APT on swiftproxy171 is OK: APT OK: 80 packages available for upgrade (0 critical updates). [04:11:38] !log [void@puppet181] Upgraded packages on reports171: nginx and nginx-common [04:11:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:13:29] RECOVERY - reports171 APT on reports171 is OK: APT OK: 79 packages available for upgrade (0 critical updates). [04:16:48] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 15.41, 23.05, 14.16 [04:18:48] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 6.37, 17.03, 13.03 [04:21:58] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 8.53, 10.61, 5.42 [04:23:58] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.29, 7.52, 4.92 [07:21:19] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 3 backends are down. mw161 mw163 mw201 [07:22:02] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 2 backends are down. mw161 mw193 [07:23:10] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 22.48, 17.04, 11.55 [07:23:28] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.04, 16.41, 10.76 [07:23:36] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [07:23:38] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 20.86, 17.12, 11.60 [07:23:38] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 21.35, 16.62, 11.02 [07:24:19] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.62, 17.12, 11.32 [07:25:18] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 20.87, 17.69, 12.11 [07:25:19] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.65, 17.17, 11.44 [07:25:28] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 19.85, 17.42, 11.82 [07:25:38] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 20.20, 18.09, 12.63 [07:25:41] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.215 second response time [07:26:02] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [07:26:11] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [07:27:11] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 20.24, 18.98, 13.63 [07:28:09] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.15, 19.08, 13.61 [07:28:13] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 2.636 second response time [07:29:19] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [07:29:19] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 16.67, 17.88, 13.12 [07:29:38] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 17.37, 18.70, 13.95 [07:30:09] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.02, 18.28, 14.04 [07:30:19] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 11.06, 16.61, 13.27 [07:31:18] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 8.94, 15.45, 13.28 [11:22:12] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 20.59, 16.09, 10.99 [11:23:05] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:23:11] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 20.59, 17.58, 12.60 [11:23:38] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 21.79, 17.75, 11.92 [11:24:12] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 19.71, 17.30, 12.06 [11:24:34] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 2 backends are down. mw181 mw203 [11:24:37] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 3 backends are down. mw163 mw193 mw201 [11:25:09] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 5.022 second response time [11:25:10] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 20.09, 18.48, 13.54 [11:25:38] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 19.79, 18.30, 12.80 [11:25:51] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 21.91, 18.56, 13.01 [11:26:24] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.16, 18.57, 12.98 [11:26:30] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [11:26:37] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [11:27:19] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.01, 18.71, 13.16 [11:28:01] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.10, 20.33, 13.92 [11:28:19] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.10, 19.68, 14.23 [11:29:11] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 21.56, 20.27, 15.41 [11:29:19] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 20.21, 19.30, 14.06 [11:29:39] PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 20.84, 19.63, 14.37 [11:29:47] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 20.87, 19.61, 14.78 [11:30:01] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 18.01, 19.71, 14.50 [11:30:09] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.97, 20.37, 15.53 [11:30:19] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 19.74, 19.99, 15.03 [11:30:20] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 15.59, 18.59, 14.35 [11:31:11] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 17.10, 19.57, 15.78 [11:31:34] RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 10.29, 16.28, 13.77 [11:31:41] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 13.19, 17.27, 14.50 [11:31:51] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 11.17, 17.00, 14.42 [11:32:09] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 11.51, 17.03, 14.90 [15:52:08] !log [macfan@test151] starting deploy of {'pull': 'world', 'world': True, 'versions': '1.46'} to test151 [15:52:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:52:18] !log [macfan@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [15:52:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:53:56] !log [macfan@test151] starting deploy of {'pull': 'world', 'world': True, 'versions': '1.46'} to test151 [15:53:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:54:20] !log [macfan@test151] finished deploy of {'pull': 'world', 'world': True, 'versions': '1.46'} to test151 - SUCCESS in 24s [15:54:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:05:05] !log [macfan@test151] starting deploy of {'pull': 'world', 'reset_world': True, 'world': True, 'versions': '1.46'} to test151 [16:05:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:07:07] !log [macfan@test151] starting deploy of {'pull': 'world', 'reset_world': True, 'world': True, 'versions': '1.46'} to test151 [16:07:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:08:30] !log [macfan@test151] starting deploy of {'pull': 'world', 'reset_world': True, 'world': True, 'versions': '1.46'} to test151 [16:08:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:09:37] !log [macfan@test151] starting deploy of {'pull': 'world', 'reset_world': True, 'world': True, 'versions': '1.46'} to test151 [16:09:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:08:51] [02python-functions] 07renovate[bot] pushed 1 new commit to 03main 13https://github.com/miraheze/python-functions/commit/0d971d1f14f39c9b884bc3046fc65db4947d1fca [20:08:51] 02python-functions/03main 07renovate[bot] 030d971d1 chore(deps): update actions/checkout action to v7 [20:08:52] [02python-functions] 07renovate[bot] 04deleted 03renovate/major-all-deps at 030d971d1 13https://api.github.com/repos/miraheze/python-functions/commit/0d971d1 [20:11:47] PROBLEM - db151 Current Load on db151 is CRITICAL: LOAD CRITICAL - total load average: 13.57, 10.24, 7.21 [20:12:13] miraheze/python-functions - renovate[bot] the build passed. [20:18:37] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 1 backends are down. mw163 [20:20:37] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [20:25:47] PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 1.22, 8.37, 11.08 [20:27:47] RECOVERY - db151 Current Load on db151 is OK: LOAD OK - total load average: 2.01, 6.25, 9.96 [20:44:31] !log [skye@mwtask171] Starting import for hoggieswiki (XML: None; Images: .) (START) [20:44:32] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importImages --wiki=hoggieswiki --sleep=1 '--comment=Importing images from https://hogwarts-school.net/wiki/index.php/Main_Page [[phorge:T15500|T15500]]' --search-recursively -- . (START) [20:44:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:44:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:49:07] !log [skye@mwtask171] Starting import for neavewiki (XML: None; Images: .) (START) [20:49:08] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importImages --wiki=neavewiki --sleep=0 '--comment=Importing images from neave-interactive.fandom.com [[phorge:T15534|T15534]]' -- . (START) [20:49:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:49:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:49:19] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importImages --wiki=neavewiki --sleep=0 '--comment=Importing images from neave-interactive.fandom.com [[phorge:T15534|T15534]]' -- . (END - exit=0) [20:49:20] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initSiteStats --wiki=neavewiki --update (START) [20:49:21] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initSiteStats --wiki=neavewiki --update (END - exit=0) [20:49:22] !log [skye@mwtask171] Finished import for neavewiki (XML: None; Images: .) (END - exit=0) [20:49:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:49:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:49:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:49:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:00:19] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/e83f4087e07ffdbe3789e00d2a4df987ea78149d [21:00:20] 02ssl/03main 07WikiTideBot 03e83f408 Bot: Auto-update domain lists [21:06:39] !log [skye@mwtask171] Starting import for supersciencewiki (XML: zhantimony_pages_full.xml; Images: None) (START) [21:06:40] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importDump --wiki=supersciencewiki --no-updates --username-prefix=fandom:cn.antimony -- zhantimony_pages_full.xml (START) [21:06:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:06:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:15:06] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importImages --wiki=hoggieswiki --sleep=1 '--comment=Importing images from https://hogwarts-school.net/wiki/index.php/Main_Page [[phorge:T15500|T15500]]' --search-recursively -- . (END - exit=0) [21:15:07] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initSiteStats --wiki=hoggieswiki --update (START) [21:15:08] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initSiteStats --wiki=hoggieswiki --update (END - exit=0) [21:15:09] !log [skye@mwtask171] Finished import for hoggieswiki (XML: None; Images: .) (END - exit=0) [21:15:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:15:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:15:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:15:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:51:58] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importDump --wiki=supersciencewiki --no-updates --username-prefix=fandom:cn.antimony -- zhantimony_pages_full.xml (END - exit=0) [21:51:59] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php rebuildall --wiki=supersciencewiki (START) [21:52:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:52:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:11:34] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php rebuildall --wiki=supersciencewiki (END - exit=0) [22:11:35] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initEditCount --wiki=supersciencewiki (START) [22:11:36] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initEditCount --wiki=supersciencewiki (END - exit=0) [22:11:37] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initSiteStats --wiki=supersciencewiki --update (START) [22:11:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:11:38] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initSiteStats --wiki=supersciencewiki --update (END - exit=0) [22:11:39] !log [skye@mwtask171] Finished import for supersciencewiki (XML: zhantimony_pages_full.xml; Images: None) (END - exit=0) [22:11:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:11:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:11:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:11:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:11:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:19:23] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 45310MiB (10.0% inode=100%);