[00:00:58] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/878db2f5d71cdc43e8603c23f2970d0324c69b2b [00:00:58] 02statichelp/03main 07WikiTideBot 03878db2f Bot: Auto-update Tech namespace pages 2026-11-12 00:00:55 [00:16:41] !log [waki285@test151] Starting deploy of {'versions': '1.47', 'upgrade_extensions': 'MirahezeRequests'} to test151 [00:16:42] !log [waki285@test151] finished deploy of {'versions': '1.47', 'upgrade_extensions': 'MirahezeRequests'} to test151 - SUCCESS in 0s [00:16:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112001641-bebc5c49 [00:16:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112001642-3b1a3d21 [00:16:52] !log [waki285@test151] Starting deploy of {'force_upgrade': True, 'versions': '1.47', 'upgrade_extensions': 'MirahezeRequests'} to test151 [00:16:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112001652-1984f9d3 [00:17:00] !log [waki285@test151] finished deploy of {'force_upgrade': True, 'versions': '1.47', 'upgrade_extensions': 'MirahezeRequests'} to test151 - SUCCESS in 8s [00:17:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112001700-70fcaf6d [00:19:56] !log [waki285@test151] Starting deploy of {'config': True} to test151 [00:19:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112001956-21582d85 [00:19:57] !log [waki285@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [00:19:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112001957-2702140e [00:20:47] !log [waki285@test151] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php sql --wiki=metawikibeta --file=/srv/mediawiki/1.47/extensions/MirahezeRequests/sql/mysql/tables-generated.sql (END - exit=1) [00:20:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112002047-06e7c0b5 [00:20:58] !log [waki285@test151] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php sql --wiki=metawikibeta /srv/mediawiki/1.47/extensions/MirahezeRequests/sql/mysql/tables-generated.sql (END - exit=1) [00:20:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112002058-ae8f25ac [00:22:11] !log [waki285@test151] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php sql --wiki=metawikibeta '--query=SHOW CREATE TABLE account_requests\G' (END - exit=1) [00:22:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112002211-b861ce66 [00:22:23] !log [waki285@test151] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php sql --wiki=metawikibeta '--query=SHOW CREATE TABLE account_requests;' (END - exit=0) [00:22:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112002223-951d0d95 [00:23:32] !log [waki285@test151] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php sql --wiki=metawikibeta '--query=SELECT request_id, request_status, request_username, request_timestamp FROM account_requests' (END - exit=0) [00:23:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112002332-d6d5d074 [00:24:48] !log [waki285@test151] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php sql --wiki=metawikibeta '--query=SELECT * FROM account_requests' (END - exit=0) [00:24:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112002448-7b98a3fb [00:25:05] !log [waki285@test151] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php sql --wiki=metawikibeta '--query=DROP TABLE account_requests' (END - exit=0) [00:25:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112002505-fe2a363e [00:25:12] !log [waki285@test151] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php sql --wiki=metawikibeta /srv/mediawiki/1.47/extensions/MirahezeRequests/sql/mysql/tables-generated.sql (END - exit=0) [00:25:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112002512-88bb33eb [00:31:02] !log [waki285@test151] Starting deploy of {'config': True, 'l10n': True, 'extension_list': True, 'versions': '1.47'} to test151 [00:31:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112003102-d31cd7cd [00:31:56] !log [waki285@test151] finished deploy of {'config': True, 'l10n': True, 'extension_list': True, 'versions': '1.47'} to test151 - SUCCESS in 53s [00:31:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112003156-670d1f6a [00:33:09] !log [waki285@test151] Starting deploy of {'force_upgrade': True, 'versions': '1.47', 'upgrade_extensions': 'MirahezeRequests'} to test151 [00:33:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112003309-1984f9d3 [00:33:17] !log [waki285@test151] finished deploy of {'force_upgrade': True, 'versions': '1.47', 'upgrade_extensions': 'MirahezeRequests'} to test151 - SUCCESS in 8s [00:33:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112003317-70fcaf6d [00:35:42] !log [waki285@test151] Starting deploy of {'config': True, 'l10n': True, 'versions': '1.47'} to test151 [00:35:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112003542-53c96682 [00:35:55] !log [waki285@test151] finished deploy of {'config': True, 'l10n': True, 'versions': '1.47'} to test151 - SUCCESS in 13s [00:35:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112003555-117ed1b0 [00:41:08] !log [waki285@test151] Starting deploy of {'config': True} to test151 [00:41:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112004108-21582d85 [00:41:09] !log [waki285@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [00:41:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112004109-2702140e [00:45:48] !log [waki285@test151] Starting deploy of {'config': True} to test151 [00:45:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112004548-21582d85 [00:45:49] !log [waki285@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [00:45:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112004549-2702140e [00:46:33] !log [waki285@test151] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php ManageWiki:ResetWikiCaches --wiki=metawikibeta (END - exit=0) [00:46:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112004633-5c50f0fc [01:19:36] !log [waki285@test151] Starting deploy of {'force_upgrade': True, 'versions': '1.47', 'upgrade_extensions': 'MirahezeRequests'} to test151 [01:19:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112011936-1984f9d3 [01:19:50] !log [waki285@test151] finished deploy of {'force_upgrade': True, 'versions': '1.47', 'upgrade_extensions': 'MirahezeRequests'} to test151 - SUCCESS in 13s [01:19:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112011950-e8765733 [01:20:49] !log [waki285@test151] Starting deploy of {'config': True, 'l10n': True, 'versions': '1.47'} to test151 [01:20:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112012049-53c96682 [01:21:13] !log [waki285@test151] finished deploy of {'config': True, 'l10n': True, 'versions': '1.47'} to test151 - SUCCESS in 23s [01:21:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112012113-158677ef [01:21:49] !log [waki285@test151] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php sql --wiki=metawikibeta '--query=ALTER TABLE account_requests MODIFY COLUMN request_status ENUM('''complete''','''declined''','''pending''','''starting''','''unconfirmed''','''mailfailed''','''sending''') NOT NULL;' (END - exit=0) [01:21:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112012149-d39ebd9c [01:22:34] !log [waki285@test151] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php sql --wiki=metawikibeta '--query=UPDATE account_requests SET request_status='''mailfailed''', request_completed_timestamp=NULL WHERE request_id=1 AND request_status='''complete''';' (END - exit=0) [01:22:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112012234-5bc750e7 [02:44:58] [02mw-config] 07suzu-test opened pull request #6640: Set wgReauthenticateForActions (07miraheze:03main...07suzu-test:03T16390) 13https://github.com/miraheze/mw-config/pull/6640 [02:46:02] miraheze/mw-config - suzu-test the build passed. [02:46:47] !log [waki285@test151] Starting deploy of {'config': True} to test151 [02:46:48] !log [waki285@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [02:46:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112024647-21582d85 [02:46:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112024648-2702140e [03:53:03] !log [waki285@test151] Starting deploy of {'config': True} to test151 [03:53:04] !log [waki285@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [03:53:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112035303-21582d85 [03:53:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112035304-2702140e [03:54:04] !log [waki285@test151] Starting deploy of {'force_upgrade': True, 'versions': '1.47', 'upgrade_extensions': 'MirahezeRequests'} to test151 [03:54:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112035404-1984f9d3 [03:54:11] !log [waki285@test151] finished deploy of {'force_upgrade': True, 'versions': '1.47', 'upgrade_extensions': 'MirahezeRequests'} to test151 - SUCCESS in 7s [03:54:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112035411-7f6c08e8 [03:55:48] !log [waki285@test151] Starting deploy of {'force_upgrade': True, 'versions': '1.47', 'upgrade_extensions': 'MirahezeRequests'} to test151 [03:55:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112035548-1984f9d3 [03:55:49] !log [waki285@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [03:55:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112035549-e84f20fe [03:55:51] !log [waki285@test151] Starting deploy of {'force_upgrade': True, 'versions': '1.47', 'upgrade_extensions': 'MirahezeRequests'} to test151 [03:55:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112035551-1984f9d3 [03:55:59] !log [waki285@test151] finished deploy of {'force_upgrade': True, 'versions': '1.47', 'upgrade_extensions': 'MirahezeRequests'} to test151 - SUCCESS in 7s [03:56:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112035559-7f6c08e8 [03:56:11] !log [waki285@test151] Starting deploy of {'config': True} to test151 [03:56:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112035611-21582d85 [03:56:12] !log [waki285@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [03:56:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112035612-2702140e [05:07:47] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php MirahezeMagic:UpgradeWiki --wiki=testwiki --json=/srv/mediawiki/1.47.json --change-version (START) [05:07:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112050747-b8f3ea51 [05:07:52] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php MirahezeMagic:UpgradeWiki --wiki=testwiki --json=/srv/mediawiki/1.47.json --change-version (END - exit=0; time=4s) [05:07:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112050752-284b8302 [05:08:23] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php MirahezeMagic:UpgradeWiki --wiki=randomwikistuffwiki --json=/srv/mediawiki/1.47.json --change-version (START) [05:08:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112050823-1792dcd4 [05:08:26] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php MirahezeMagic:UpgradeWiki --wiki=randomwikistuffwiki --json=/srv/mediawiki/1.47.json --change-version (END - exit=0; time=3s) [05:08:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112050826-79b676a8 [05:08:51] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php MirahezeMagic:UpgradeWiki --wiki=strinovawiki --json=/srv/mediawiki/1.47.json --change-version (START) [05:08:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112050851-c8d4c8cf [05:08:55] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php MirahezeMagic:UpgradeWiki --wiki=strinovawiki --json=/srv/mediawiki/1.47.json --change-version (END - exit=0; time=4s) [05:08:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112050855-4384d838 [05:09:18] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php MirahezeMagic:UpgradeWiki --wiki=ldapwikiwiki --json=/srv/mediawiki/1.47.json --change-version (START) [05:09:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112050918-90ad618d [05:09:21] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php MirahezeMagic:UpgradeWiki --wiki=ldapwikiwiki --json=/srv/mediawiki/1.47.json --change-version (END - exit=0; time=3s) [05:09:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112050921-dbf944c2 [05:09:42] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php OATHAuth:UpdateKeysToLatestVersion --wiki=ldapwikiwiki (END - exit=0) [05:09:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112050942-46c9fce4 [05:43:49] !log [petramagna@test151] Starting deploy of {'force_upgrade': True, 'versions': '1.47', 'upgrade_extensions': 'DiscussionTools', 'debug': True} to test151 (T16502) [05:43:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112054349-3fc23eb5 [05:43:50] !log [petramagna@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. (T16502) [05:43:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112054350-1a064367 [05:44:21] !log [petramagna@test151] Starting deploy of {'folders': '1.47/extensions/DiscussionTools', 'debug': True} to test151 (T16502) [05:44:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112054421-a57d8b85 [05:44:22] !log [petramagna@test151] finished deploy of {'folders': '1.47/extensions/DiscussionTools', 'debug': True} to test151 - SUCCESS in 0s (T16502) [05:44:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112054422-78e94ab3 [05:49:19] [02mediawiki-repos] 07lihaohong6 opened pull request #181: Remove DiscussionTools and ConfirmEdit pin and patches (07miraheze:03main...07lihaohong6:03main) 13https://github.com/miraheze/mediawiki-repos/pull/181 [06:02:12] [02mediawiki-repos] 07lihaohong6 pushed 1 new commit to 03main 13https://github.com/miraheze/mediawiki-repos/commit/1e63c42877906aef7e5680f8b97a616394d147c9 [06:02:12] 02mediawiki-repos/03main 07Peter Li 031e63c42 Remove DiscussionTools and ConfirmEdit pin and patches (#181)… [06:02:12] [02mediawiki-repos] 07lihaohong6 merged pull request #181: Remove DiscussionTools and ConfirmEdit pin and patches (07miraheze:03main...07lihaohong6:03main) 13https://github.com/miraheze/mediawiki-repos/pull/181 [06:04:23] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki-REL1_47 DiscussionTools] [06:34:54] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki-REL1_46 ConfirmEdit],Exec[git_checkout_MediaWiki-REL1_46 DiscussionTools],Exec[git_checkout_MediaWiki-REL1_47 ConfirmEdit],Exec[git_checkout_MediaWiki-REL1_47 DiscussionTools] [07:18:30] @posix_memalign ^ [07:29:29] Pretty sure whatever runs checkout has some issues because switching from pinned commits and branches always require manual intervention. [07:34:45] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [07:34:46] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:35:17] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 359.30, 153.07, 61.93 [07:35:52] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:35:55] PROBLEM - mw211 HTTPS on mw211 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [07:37:10] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [07:37:15] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [07:37:38] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [07:37:47] PROBLEM - mw211 MediaWiki Rendering on mw211 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.010 second response time [07:37:48] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [07:37:49] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.201 second response time [07:37:53] RECOVERY - mw211 HTTPS on mw211 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.082 second response time [07:37:54] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [07:38:06] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [07:38:12] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 1 backends are down. lb_mediawiki [07:38:37] PROBLEM - cp201 HAProxy Servers on cp201 is CRITICAL: CRITICAL: 2 servers DOWN: mw163 (mediawiki), mw203 (mediawiki) [07:38:51] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.455 second response time [07:39:08] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.309 second response time [07:39:13] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 1.726 second response time [07:39:47] RECOVERY - mw211 MediaWiki Rendering on mw211 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.143 second response time [07:39:48] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 3.917 second response time [07:40:08] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 5.767 second response time [07:40:13] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 12 backends are healthy [07:40:36] PROBLEM - db161 MariaDB Connections on db161 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Too many connections in /usr/lib/nagios/plugins/check_mysql_connections.php:66 Stack trace: #0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db161.fsslc.wtn...', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true) #1 {main} throw... [07:40:37] PROBLEM - db161 MariaDB on db161 is CRITICAL: Too many connections [07:42:37] RECOVERY - cp201 HAProxy Servers on cp201 is OK: OK: all 25 servers are up [07:42:38] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [07:42:47] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 3.301 second response time [07:45:54] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.087 second response time [07:45:59] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.078 second response time [07:46:37] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.074 second response time [07:48:30] RECOVERY - db161 MariaDB Connections on db161 is OK: OK connection usage: 48.9% Current connections: 489 [07:48:34] RECOVERY - db161 MariaDB on db161 is OK: Uptime: 131 Threads: 490 Questions: 10631 Slow queries: 19 Opens: 1859 Open tables: 1850 Queries per second avg: 81.152 [07:50:02] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [07:50:10] PROBLEM - cp161 HAProxy Servers on cp161 is CRITICAL: CRITICAL: 2 servers DOWN: mw153 (mediawiki), mw202 (mediawiki) [07:50:16] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [07:50:19] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [07:50:34] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [07:50:38] PROBLEM - cp201 HAProxy Servers on cp201 is CRITICAL: CRITICAL: 3 servers DOWN: mw192 (mediawiki), mw193 (mediawiki), mw213 (mediawiki) [07:50:56] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [07:51:03] PROBLEM - cp191 HAProxy Servers on cp191 is CRITICAL: CRITICAL: 4 servers DOWN: mw151 (mediawiki), mw171 (mediawiki), mw172 (mediawiki), mw203 (mediawiki) [07:51:08] PROBLEM - cp171 HAProxy Servers on cp171 is CRITICAL: CRITICAL: 5 servers DOWN: mw151 (mediawiki), mw172 (mediawiki), mw203 (mediawiki), mw211 (mediawiki), mw212 (mediawiki) [07:52:01] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 1.389 second response time [07:56:26] of course [07:58:15] RECOVERY - cp161 HAProxy Servers on cp161 is OK: OK: all 25 servers are up [07:58:53] RECOVERY - cp171 HAProxy Servers on cp171 is OK: OK: all 25 servers are up [07:59:37] !log [petramagna@mwtask181] Starting deploy of {'config': True, 'force': True, 'debug': True} to all [07:59:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112075937-93dcdb52 [08:00:01] !log [petramagna@mwtask181] finished deploy of {'config': True, 'force': True, 'debug': True} to all - SUCCESS in 24s [08:00:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112080001-6327b5d1 [08:00:24] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.076 second response time [08:00:37] RECOVERY - cp201 HAProxy Servers on cp201 is OK: OK: all 25 servers are up [08:00:39] RECOVERY - cp191 HAProxy Servers on cp191 is OK: OK: all 25 servers are up [08:00:42] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.312 second response time [08:00:51] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.087 second response time [08:01:08] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.090 second response time [08:03:38] !log [petramagna@mwtask181] Starting deploy of {'config': True, 'force': True, 'debug': True} to all [08:03:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112080338-93dcdb52 [08:03:58] !log [petramagna@mwtask181] finished deploy of {'config': True, 'force': True, 'debug': True} to all - SUCCESS in 20s [08:03:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112080358-de3e0880 [08:22:35] !log [petramagna@mwtask181] Starting deploy of {'versions': '1.47', 'upgrade_extensions': ['ConfirmEdit', 'DiscussionTools']} to all (T16502) [08:22:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112082235-4fa54803 [08:23:23] !log [petramagna@mwtask181] finished deploy of {'versions': '1.47', 'upgrade_extensions': ['ConfirmEdit', 'DiscussionTools']} to all - SUCCESS in 48s (T16502) [08:23:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112082323-f6d41e63 [08:32:07] !log [petramagna@mwtask181] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php MirahezeMagic:UpgradeWiki --wiki=newfurbywiki --json=/srv/mediawiki/1.47.json --change-version (START) [08:32:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112083207-75fedf32 [08:32:08] !log [petramagna@mwtask181] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php MirahezeMagic:UpgradeWiki --wiki=newfurbywiki --json=/srv/mediawiki/1.47.json --change-version (END - exit=1; time=1s) [08:32:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112083208-c80474fa [08:32:52] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [08:59:52] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 2.00, 1.94, 11.53 [09:03:52] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 1.15, 1.70, 9.25 [09:26:00] RECOVERY - mw172 Check unit status of php8.4-fpm_check_restart on mw172 is OK: OK: Status of the systemd unit php8.4-fpm_check_restart [10:00:34] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/ec44536e7945287b2fb283777541300f7fd54e39 [10:00:34] 02ssl/03main 07WikiTideBot 03ec44536 Bot: Auto-update domain lists [10:02:22] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [10:02:32] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (4 critical updates). [10:04:09] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (4 critical updates). [10:05:27] PROBLEM - mon181 APT on mon181 is CRITICAL: APT CRITICAL: 18 packages available for upgrade (2 critical updates). [10:06:53] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (4 critical updates). [10:12:23] PROBLEM - mwtask161 APT on mwtask161 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (4 critical updates). [10:12:59] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (4 critical updates). [10:14:54] PROBLEM - mwtask151 APT on mwtask151 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (4 critical updates). [10:34:08] RECOVERY - mw163 Check unit status of php8.4-fpm_check_restart on mw163 is OK: OK: Status of the systemd unit php8.4-fpm_check_restart [10:55:52] !log [petramagna@test151] Starting deploy of {'versions': 'all', 'upgrade_extensions': 'EditCountNeue', 'debug': True} to test151 (T16493) [10:55:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112105552-5b22d389 [10:55:53] !log [petramagna@test151] finished deploy of {'versions': 'all', 'upgrade_extensions': 'EditCountNeue', 'debug': True} to test151 - SUCCESS in 1s (T16493) [10:55:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112105553-2b842bdf [10:56:20] !log [petramagna@mwtask181] Starting deploy of {'versions': 'all', 'upgrade_extensions': 'EditCountNeue', 'debug': True} to all (T16493) [10:56:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112105620-7400d0ea [10:56:22] !log [petramagna@mwtask181] finished deploy of {'versions': 'all', 'upgrade_extensions': 'EditCountNeue', 'debug': True} to all - SUCCESS in 1s (T16493) [10:56:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112105622-00ecab2a [11:48:46] PROBLEM - mw181 Check unit status of php8.4-fpm_check_restart on mw181 is CRITICAL: CRITICAL: Status of the systemd unit php8.4-fpm_check_restart [15:51:16] [02mw-config] 07whostacking opened pull request #6641: T15986: share files from doorswiki with the DOORS language wikis (07miraheze:03main...07whostacking:03doors-shared-media) 13https://github.com/miraheze/mw-config/pull/6641 [15:52:22] miraheze/mw-config - whostacking the build passed. [17:05:10] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=fictionalmyempirewiki --new=altawiki --user=MacFan4000 (END - exit=0) (T16489) [17:05:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112170510-01b1ef0c [17:05:52] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php CreateWiki:SetContainersAccess --wiki=altawiki (END - exit=0) (T16489) [17:05:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112170552-db63f51f [17:10:20] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=repaircafeedesheimwiki --new=repaircafewiki --user=MacFan4000 (END - exit=0) (T16451) [17:10:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112171020-c2fa7aa9 [17:11:22] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php CreateWiki:SetContainersAccess --wiki=repaircafewiki (END - exit=0) (T16451) [17:11:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112171122-3d0b6c1a [19:04:12] !log [rhinos@puppet181] Upgraded packages on mon181: apache2-bin and apache2-utils [19:04:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112190411-185dcdbb [19:05:28] RECOVERY - mon181 APT on mon181 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [19:07:57] !log [rhinos@puppet181] Upgraded packages on mwtask151: apache2, apache2-bin, apache2-data, and apache2-utils [19:07:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112190757-07f94d39 [19:08:24] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [19:08:34] !log [rhinos@puppet181] Upgraded packages on mwtask161: apache2, apache2-bin, apache2-data, and apache2-utils [19:08:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112190834-d116a78f [19:08:52] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [19:08:59] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [19:09:16] !log [rhinos@puppet181] Upgraded packages on mwtask171: apache2, apache2-bin, apache2-data, and apache2-utils [19:09:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112190917-8602e56c [19:10:19] !log [rhinos@puppet181] Upgraded packages on mwtask181: apache2, apache2-bin, apache2-data, and apache2-utils [19:10:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112191019-c45dd332 [19:10:29] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [19:11:05] !log [rhinos@puppet181] Upgraded packages on prometheus151: apache2, apache2-bin, apache2-data, and apache2-utils [19:11:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112191105-52bf8f22 [19:11:35] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [19:13:07] !log [rhinos@puppet181] Upgraded packages on test151: apache2, apache2-bin, apache2-data, and apache2-utils [19:13:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112191307-c5570b42 [19:15:00] RECOVERY - test151 APT on test151 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [19:16:31] [02TSPortal] 07renovate[bot] 04force-pushed 03renovate/prod-deps from 03d3a5146 to 03db7f3d8 13https://api.github.com/repos/miraheze/TSPortal/commits/renovate/prod-deps [19:16:32] 02TSPortal/03renovate/prod-deps 07renovate[bot] 03db7f3d8 Update prod dependencies [19:16:53] !log bounced apache2 everywhere it's running [19:16:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112191653-05b1db96 [20:52:26] !log [rhinos@test151] Starting deploy of {'world': True, 'versions': '1.47'} to test151 [20:52:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112205226-30556cc0 [20:52:50] !log [rhinos@test151] finished deploy of {'world': True, 'versions': '1.47'} to test151 - SUCCESS in 23s [20:52:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112205250-79d7ea62 [20:55:11] !log [rhinos@test151] Starting deploy of {'world': True, 'versions': '1.47', 'apply_patches': '01-T444647.patch'} to test151 [20:55:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112205511-1ea85f4f [20:55:15] !log [rhinos@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [20:55:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261112205515-e3bd1a98 [21:12:45] [02python-functions] 07renovate[bot] 04force-pushed 03renovate/all-deps from 03bef2991 to 03cea7a41 13https://api.github.com/repos/miraheze/python-functions/commits/renovate/all-deps [21:12:45] 02python-functions/03renovate/all-deps 07renovate[bot] 03cea7a41 chore(deps): update all dependencies [21:14:14] miraheze/python-functions - renovate[bot] the build has errored. [22:55:00] [02mw-config] 07Universal-Omega left a file comment in pull request #6641 03093a4c5: I think it would be better to just add it to https://github.com/miraheze/mw-config/blob/c7a542e/LocalSettings.php#L3271 rather than this hack. 13https://github.com/miraheze/mw-config/pull/6641#discussion_r4557380245 [23:43:30] PROBLEM - llm191 Puppet on llm191 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[openwebui]