[00:01:15] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/f9309f1acc62b0993c7b0f71ea9146a174756f2c [00:01:15] 02statichelp/03main 07WikiTideBot 03f9309f1 Bot: Auto-update Tech namespace pages 2026-07-09 00:01:13 [03:20:38] [02puppet] 07The-Voidwalker pushed 1 new commit to 03The-Voidwalker-patch-2 13https://github.com/miraheze/puppet/commit/d55021feee0f7ab3c0e1c998d19f729238d8bccf [03:20:38] 02puppet/03The-Voidwalker-patch-2 07The-Voidwalker 03d55021f Merge branch 'main' into The-Voidwalker-patch-2 [03:22:01] [02puppet] 07The-Voidwalker merged pull request #4860: varnish: add time header to cache object (03main...03The-Voidwalker-patch-2) 13https://github.com/miraheze/puppet/pull/4860 [03:22:01] [02puppet] 07The-Voidwalker pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/8ade71bd3348e287646fd3a51f7ebf66a949d21f [03:22:01] 02puppet/03main 07The-Voidwalker 038ade71b varnish: add time header to cache object (#4860)… [03:22:02] [02puppet] 07The-Voidwalker 04deleted 03The-Voidwalker-patch-2 at 03d55021f 13https://api.github.com/repos/miraheze/puppet/commit/d55021f [05:17:01] !log [reception@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/maintenance/importDump.php --wiki=moddedminecraftwiki /home/reception/ftb_gamepedia_pages_full.xml --username-prefix:wikia:ftb (START) [05:17:03] !log [reception@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/maintenance/importDump.php --wiki=moddedminecraftwiki /home/reception/ftb_gamepedia_pages_full.xml --username-prefix:wikia:ftb (END - exit=256) [05:17:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:17:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:17:40] !log [reception@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/maintenance/importDump.php --wiki=moddedminecraftwiki /home/reception/ftb_gamepedia_pages_full.xml --username-prefix=wikia:ftb (START) [05:17:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:05:24] [02mw-config] 07lihaohong6 merged pull request #6381: T15231: exclude localized special page titles in robots.txt (07miraheze:03main...07lihaohong6:03robots) 13https://github.com/miraheze/mw-config/pull/6381 [06:05:24] [02mw-config] 07lihaohong6 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/a4851e0f0d71b4ce3397fa2460e2c4ca5a8d1a74 [06:05:24] 02mw-config/03main 07Peter Li 03a4851e0 T15231: exclude localized special page titles in robots.txt (#6381) [06:08:54] [02mw-config] 07lihaohong6 opened pull request #6459: Set up LinkTarget for exttestwikibeta (07miraheze:03main...07lihaohong6:03linktarget) 13https://github.com/miraheze/mw-config/pull/6459 [06:10:37] PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 10.38, 10.11, 7.94 [06:11:32] miraheze/mw-config - lihaohong6 the build passed. [06:11:33] miraheze/mw-config - lihaohong6 the build passed. [06:12:37] PROBLEM - db151 Current Load on db151 is CRITICAL: LOAD CRITICAL - total load average: 14.57, 11.26, 8.59 [06:25:22] !log [petramagna@test151] starting deploy of {'config': True} to test151 [06:25:24] !log [petramagna@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [06:25:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:25:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:26:09] [02mw-config] 07lihaohong6 merged pull request #6459: Set up LinkTarget for exttestwikibeta (07miraheze:03main...07lihaohong6:03linktarget) 13https://github.com/miraheze/mw-config/pull/6459 [06:26:09] [02mw-config] 07lihaohong6 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/4b697e0892766a5be43391f04547bebefd23e6df [06:26:09] 02mw-config/03main 07Peter Li 034b697e0 Set up LinkTarget for exttestwikibeta (#6459) [06:26:30] !log [petramagna@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [06:26:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:27:00] !log [petramagna@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 29s [06:27:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:32:18] miraheze/mw-config - lihaohong6 the build passed. [06:34:37] PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 0.83, 5.83, 10.71 [06:36:37] RECOVERY - db151 Current Load on db151 is OK: LOAD OK - total load average: 0.29, 3.98, 9.44 [08:01:27] PROBLEM - mattermost2 SSH on mattermost2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:02:08] PROBLEM - mattermost2 HTTPS on mattermost2 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10005 milliseconds [08:03:12] PROBLEM - Host mattermost2 is DOWN: PING CRITICAL - Packet loss = 100% [08:11:59] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 14.72, 10.18, 7.47 [08:13:59] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 11.66, 10.45, 7.89 [08:15:59] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 14.37, 12.52, 9.01 [08:17:59] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 9.99, 11.48, 9.05 [08:25:59] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 13.18, 11.95, 10.05 [08:31:50] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 4 backends are down. mw182 mw183 mw193 mw201 [08:32:03] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 9 backends are down. mw161 mw172 mw163 mw173 mw183 mw191 mw192 mw193 mw203 [08:32:14] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 9 backends are down. mw151 mw181 mw182 mw153 mw173 mw191 mw192 mw193 mw203 [08:32:47] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 11 backends are down. mw151 mw152 mw161 mw172 mw181 mw153 mw163 mw183 mw191 mw201 mw202 [08:33:48] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:34:25] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.012 second response time [08:34:44] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:35:15] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [08:35:55] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.615 second response time [08:36:25] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.254 second response time [08:36:54] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [08:38:40] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.073 second response time [08:38:46] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [08:38:47] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.071 second response time [08:39:15] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 5% [08:39:51] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [08:40:03] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [08:40:14] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [08:46:03] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 1 backends are down. mw161 [08:46:14] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 3 backends are down. mw171 mw163 mw203 [08:47:50] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 5 backends are down. mw152 mw171 mw172 mw191 mw202 [08:48:13] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:48:41] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:49:06] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:49:41] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 7 backends are down. mw151 mw161 mw181 mw173 mw183 mw192 mw193 [08:50:18] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 5.836 second response time [08:50:48] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 8.122 second response time [08:51:01] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:51:02] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:14] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 7.635 second response time [08:51:29] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.013 second response time [08:52:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1783583510000&orgId=1&to=1783587140024 [08:53:06] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.436 second response time [08:53:09] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.021 second response time [08:53:31] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.956 second response time [08:55:12] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [08:56:46] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.315 second response time [08:57:10] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.805 second response time [08:57:16] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.749 second response time [08:57:35] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [08:59:50] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [09:00:03] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [09:00:14] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [09:01:06] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 15.98, 8.82, 4.50 [09:11:06] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 1.79, 10.26, 9.35 [09:12:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1783584470000&orgId=1&to=1783588070000 [09:12:28] RECOVERY - Host mattermost2 is UP: PING OK - Packet loss = 0%, RTA = 28.37 ms [09:12:40] RECOVERY - mattermost2 HTTPS on mattermost2 is OK: HTTP OK: HTTP/2 200 - 547 bytes in 0.127 second response time [09:13:06] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.15, 7.58, 8.47 [09:13:27] PROBLEM - mattermost2 Puppet on mattermost2 is WARNING: WARNING: Puppet last ran 1 hour ago [09:14:12] RECOVERY - mattermost2 SSH on mattermost2 is OK: SSH OK - OpenSSH_10.0p2 (protocol 2.0) [09:15:22] RECOVERY - mattermost2 Puppet on mattermost2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:21:59] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 2.32, 3.08, 11.36 [09:25:59] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 2.79, 3.05, 9.46 [11:34:53] !log [petramagna@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php CirrusSearch:UpdateSearchIndexConfig --wiki=casualtiesunknownwiki --startOver (END - exit=256) [11:34:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:35:49] !log [petramagna@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php ManageWiki:ResetWikiCaches --wiki=casualtiesunknownwiki (END - exit=0) [11:35:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:36:05] !log [petramagna@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php CirrusSearch:UpdateSearchIndexConfig --wiki=casualtiesunknownwiki --startOver (END - exit=0) [11:36:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:38:50] !log [petramagna@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php CirrusSearch:ForceSearchIndex --wiki=casualtiesunknownwiki (END - exit=0) [11:38:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:08:52] PROBLEM - test151 Check unit status of mediawiki_job_manage-inactive-wikis on test151 is CRITICAL: CRITICAL: Status of the systemd unit mediawiki_job_manage-inactive-wikis [12:36:46] PROBLEM - test151 Check unit status of mediawiki_job_mediamoderation-scan on test151 is CRITICAL: CRITICAL: Status of the systemd unit mediawiki_job_mediamoderation-scan [18:07:48] PROBLEM - test151 Check unit status of mediawiki_job_purge-abusefilter on test151 is CRITICAL: CRITICAL: Status of the systemd unit mediawiki_job_purge-abusefilter [19:15:11] !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.45', '1.46'], 'upgrade_skins': 'Refreshed'} to test151 [19:15:13] !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.45', '1.46'], 'upgrade_skins': 'Refreshed'} to test151 - SUCCESS in 1s [19:15:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:15:15] !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.45', 'upgrade_skins': 'Refreshed'} to all [19:15:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:15:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:15:40] !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.45', 'upgrade_skins': 'Refreshed'} to all - SUCCESS in 24s [19:15:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:03:01] PROBLEM - test151 Check unit status of mediawiki_job_purge-loginnotify on test151 is CRITICAL: CRITICAL: Status of the systemd unit mediawiki_job_purge-loginnotify [23:44:16] !log [void@test151] starting deploy of {'folders': '1.45/extensions/CreateWiki'} to test151 [23:44:17] !log [void@test151] finished deploy of {'folders': '1.45/extensions/CreateWiki'} to test151 - SUCCESS in 0s [23:44:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:44:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:44:53] folder was missing [23:45:44] !log [void@test151] starting deploy of {'folders': '1.45/extensions'} to test151 [23:45:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:45:48] !log [void@test151] finished deploy of {'folders': '1.45/extensions'} to test151 - SUCCESS in 4s [23:45:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:47:01] RECOVERY - test151 Check unit status of mediawiki_job_purge-loginnotify on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-loginnotify [23:47:40] !log [void@test151] restart failed mediawiki_job services [23:47:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:47:49] RECOVERY - test151 Check unit status of mediawiki_job_purge-abusefilter on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-abusefilter [23:48:46] RECOVERY - test151 Check unit status of mediawiki_job_mediamoderation-scan on test151 is OK: OK: Status of the systemd unit mediawiki_job_mediamoderation-scan [23:48:52] RECOVERY - test151 Check unit status of mediawiki_job_manage-inactive-wikis on test151 is OK: OK: Status of the systemd unit mediawiki_job_manage-inactive-wikis