[00:00:57] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/0c59fee6db4afbb678d03b463da825cf179c013f [00:00:57] 02statichelp/03main 07WikiTideBot 030c59fee Bot: Auto-update Tech namespace pages 2026-02-26 00:00:55 [00:01:32] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 1 backends are down. mw183 [00:01:33] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.45', 'upgrade_extensions': 'Cargo'} to all [00:01:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:01:52] !log [paladox@test151] starting deploy of {'l10n': True, 'versions': '1.45', 'upgrade_extensions': 'Cargo'} to test151 [00:01:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:02:04] PROBLEM - mw183 Current Load on mw183 is CRITICAL: LOAD CRITICAL - total load average: 26.92, 23.21, 12.65 [00:03:32] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [00:04:02] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 9.13, 17.62, 11.83 [00:07:33] !log [paladox@test151] finished deploy of {'l10n': True, 'versions': '1.45', 'upgrade_extensions': 'Cargo'} to test151 - SUCCESS in 341s [00:07:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:10:12] !log [paladox@mwtask181] finished deploy of {'l10n': True, 'versions': '1.45', 'upgrade_extensions': 'Cargo'} to all - SUCCESS in 519s [00:10:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:15:59] !log [skye@test151] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php changePassword --wiki=metawikibeta --user= OATest --password= [00:16:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:13] not the actual password obviously [00:23:25] boutta log into OA's account now I know the password, ty skye [00:33:16] PROBLEM - mw172 SSH on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:33:32] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 2 backends are down. mw171 mw172 [00:33:47] PROBLEM - mw171 PowerDNS Recursor on mw171 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:33:52] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 2 backends are down. mw171 mw172 [00:33:53] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:34:11] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [00:34:19] PROBLEM - mw171 SSH on mw171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:34:25] PROBLEM - mw172 PowerDNS Recursor on mw172 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:34:39] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 2 backends are down. mw171 mw172 [00:34:40] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 61.26, 35.46, 17.41 [00:34:40] PROBLEM - mw172 php-fpm on mw172 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [00:34:45] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:35:01] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 2 backends are down. mw171 mw172 [00:35:13] RECOVERY - mw172 SSH on mw172 is OK: SSH OK - OpenSSH_10.0p2 (protocol 2.0) [00:35:38] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 35.44, 32.85, 17.63 [00:35:45] RECOVERY - mw171 PowerDNS Recursor on mw171 is OK: DNS OK: 0.050 seconds response time. mw171.fsslc.wtnet returns 10.0.17.122 [00:35:51] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.390 second response time [00:35:52] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [00:36:04] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4288 bytes in 0.055 second response time [00:36:14] RECOVERY - mw171 SSH on mw171 is OK: SSH OK - OpenSSH_10.0p2 (protocol 2.0) [00:36:21] RECOVERY - mw172 PowerDNS Recursor on mw172 is OK: DNS OK: 0.048 seconds response time. mw172.fsslc.wtnet returns 10.0.17.123 [00:36:36] RECOVERY - mw172 php-fpm on mw172 is OK: PROCS OK: 25 processes with command name 'php-fpm8.4' [00:36:39] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [00:36:44] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4288 bytes in 0.075 second response time [00:37:01] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [00:37:32] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [00:38:36] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 7.30, 19.05, 14.68 [00:39:27] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 7.45, 18.55, 15.10 [00:43:11] why is mw cooking [00:44:54] [1/2] wtf [00:44:54] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1476379465188900957/image.png?ex=69a0e906&is=699f9786&hm=2e01f3bd9507c7e060ad40c9c6cf1268ed7ccf40b410a311d720d23dc5a02ac9& [01:08:18] !log [blankeclair@mwtask171] Starting import for kaiyaripediawiki (XML: None; Images: all_images/) (START) [01:08:19] !log [blankeclair@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importImages --wiki=kaiyaripediawiki --sleep=0 '--comment=Importing images from https://kaiyari.fandom.com ([[phorge:T15018|T15018]])' -- all_images/ (START) [01:08:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:08:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:12:08] !log [blankeclair@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importImages --wiki=kaiyaripediawiki --sleep=0 '--comment=Importing images from https://kaiyari.fandom.com ([[phorge:T15018|T15018]])' -- all_images/ (END - exit=0) [01:12:09] !log [blankeclair@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initSiteStats --wiki=kaiyaripediawiki --update (START) [01:12:10] !log [blankeclair@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initSiteStats --wiki=kaiyaripediawiki --update (END - exit=0) [01:12:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:12:11] !log [blankeclair@mwtask171] Finished import for kaiyaripediawiki (XML: None; Images: all_images/) (END - exit=0) [01:12:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:12:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:12:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:26:34] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/c45258ffaf71b16ca8c59ec313463e33cb8965b8 [01:26:34] 02puppet/03main 07CosmicAlpha 03c45258f beta: use public network again [03:14:30] [02puppet] 07The-Voidwalker created 03The-Voidwalker-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/e8ecb715976f [03:14:31] 02puppet/03The-Voidwalker-patch-1 07The-Voidwalker 03e8ecb71 gen_fingerprints use POSIX character class for space matching… [03:14:42] [02puppet] 07The-Voidwalker opened pull request #4796: gen_fingerprints use POSIX character class for space matching (03main...03The-Voidwalker-patch-1) 13https://github.com/miraheze/puppet/pull/4796 [03:16:38] [02puppet] 07The-Voidwalker pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/89f1ef19359d4de2967eec737b5bba0ccad5dcc6 [03:16:39] 02puppet/03main 07The-Voidwalker 0389f1ef1 gen_fingerprints use POSIX character class for space matching (#4796)… [03:16:41] [02puppet] 07The-Voidwalker merged pull request #4796: gen_fingerprints use POSIX character class for space matching (03main...03The-Voidwalker-patch-1) 13https://github.com/miraheze/puppet/pull/4796 [03:16:43] [02puppet] 07The-Voidwalker 04deleted 03The-Voidwalker-patch-1 at 03e8ecb71 13https://api.github.com/repos/miraheze/puppet/commit/e8ecb71 [03:42:19] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/8f4ce63d258918ce925635d7d3c9a70247d06005 [03:42:19] 02mw-config/03main 07CosmicAlpha 038f4ce63 Only allow loginwiki in $wgCrossSiteAjaxDomains [03:42:25] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [03:42:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:42:31] !log [universalomega@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [03:42:32] !log [universalomega@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s [03:42:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:42:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:42:50] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s [03:42:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:43:32] miraheze/mw-config - Universal-Omega the build passed. [03:53:33] miraheze/mw-config - pskyechology the build passed. [03:54:45] [02mw-config] 07pskyechology merged pull request #6223: Set wgWmincExistingWikis to an empty array (07miraheze:03main...07pskyechology:03T14782) 13https://github.com/miraheze/mw-config/pull/6223 [03:54:45] [02mw-config] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/c7701dd297fec4b703273f997511aef7e92f2cba [03:54:47] 02mw-config/03main 07Skye 03c7701dd Set wgWmincExistingWikis to an empty array (#6223)… [03:54:55] !log [skye@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [03:54:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:55:19] !log [skye@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 24s [03:55:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:55:58] miraheze/mw-config - pskyechology the build passed. [05:25:26] !log [blankeclair@mwtask161] Starting import for gunvoltwiki (XML: azurestrikergunvolt_pages_full.xml.7z; Images: None) (START) [05:25:27] !log [blankeclair@mwtask161] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importDump --wiki=gunvoltwiki --no-updates --username-prefix=fandom:azurestrikergunvolt -- azurestrikergunvolt_pages_full.xml.7z (START) [05:25:28] !log [blankeclair@mwtask161] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importDump --wiki=gunvoltwiki --no-updates --username-prefix=fandom:azurestrikergunvolt -- azurestrikergunvolt_pages_full.xml.7z (END - exit=1) [05:25:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:25:29] !log [blankeclair@mwtask161] Finished import for gunvoltwiki (XML: azurestrikergunvolt_pages_full.xml.7z; Images: None) (END - exit=1) [05:25:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:25:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:25:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:25:46] !log [blankeclair@mwtask161] Starting import for gunvoltwiki (XML: azurestrikergunvolt_pages_full.xml; Images: None) (START) [05:25:47] !log [blankeclair@mwtask161] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importDump --wiki=gunvoltwiki --no-updates --username-prefix=fandom:azurestrikergunvolt -- azurestrikergunvolt_pages_full.xml (START) [05:25:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:25:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:00:41] !log [blankeclair@mwtask161] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importDump --wiki=gunvoltwiki --no-updates --username-prefix=fandom:azurestrikergunvolt -- azurestrikergunvolt_pages_full.xml (END - exit=0) [06:00:42] !log [blankeclair@mwtask161] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php rebuildall --wiki=gunvoltwiki (START) [06:00:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:00:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:12:09] !log [blankeclair@mwtask161] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php rebuildall --wiki=gunvoltwiki (END - exit=0) [06:12:10] !log [blankeclair@mwtask161] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initEditCount --wiki=gunvoltwiki (START) [06:12:11] !log [blankeclair@mwtask161] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initEditCount --wiki=gunvoltwiki (END - exit=0) [06:12:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:12:12] !log [blankeclair@mwtask161] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initSiteStats --wiki=gunvoltwiki --update (START) [06:12:13] !log [blankeclair@mwtask161] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initSiteStats --wiki=gunvoltwiki --update (END - exit=0) [06:12:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:12:14] !log [blankeclair@mwtask161] Finished import for gunvoltwiki (XML: azurestrikergunvolt_pages_full.xml; Images: None) (END - exit=0) [06:12:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:12:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:12:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:12:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:10:41] [02WikiDiscover] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/WikiDiscover/commit/220ef96708debe25300828d58d6e4973a90ab27f [12:10:41] 02WikiDiscover/03main 07translatewiki.net 03220ef96 Localisation updates from https://translatewiki.net. [12:10:43] [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/ba550970be8d0d9169eda76eb2d62116d875b7ec [12:10:43] 02MirahezeMagic/03main 07translatewiki.net 03ba55097 Localisation updates from https://translatewiki.net. [12:15:44] miraheze/WikiDiscover - translatewiki the build passed. [12:18:43] miraheze/MirahezeMagic - translatewiki the build passed. [15:33:55] PROBLEM - swift-lb.wikitide.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikitide.net' expires in 14 day(s) (Fri 13 Mar 2026 03:30:27 PM GMT +0000). [16:58:47] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/e4c4f784a6fe0f6c1ed1dcaba569482a257dc6c0 [16:58:47] 02puppet/03main 07CosmicAlpha 03e4c4f78 puppetdb: remove bookworm [17:03:00] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772121740000&orgId=1&to=1772125380215 [17:08:00] [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772121740000&orgId=1&to=1772125430000 [17:17:15] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/05ef9dde7754dbc81c83f81b82c5c426618b68ff [17:17:15] 02puppet/03main 07CosmicAlpha 0305ef9dd postgresql: remove bookworm [17:30:27] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/14d127f7ebc8c3e44d1a8e2a02d9aaefee3a0193 [17:30:28] 02ssl/03main 07WikiTideBot 0314d127f Bot: Auto-update domain lists [19:48:43] PROBLEM - os151 APT on os151 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [19:49:30] PROBLEM - cloud15 APT on cloud15 is CRITICAL: APT CRITICAL: 202 packages available for upgrade (1 critical updates). [19:49:36] PROBLEM - mw193 APT on mw193 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:49:53] PROBLEM - cloud21 APT on cloud21 is CRITICAL: APT CRITICAL: 16 packages available for upgrade (1 critical updates). [19:50:05] !log change hostnames for cloud* to fsslc.wtnet [19:50:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:28] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/1c068dc7d5a9bec5dde6cc74a903e973dc0031a6 [19:50:28] 02puppet/03main 07CosmicAlpha 031c068dc Remove wikitide.net node for cloud [19:51:21] PROBLEM - mw171 APT on mw171 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:52:54] PROBLEM - mw191 APT on mw191 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:53:23] PROBLEM - mw153 APT on mw153 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:54:11] PROBLEM - kafka182 APT on kafka182 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [19:55:49] PROBLEM - cloud19 APT on cloud19 is CRITICAL: APT CRITICAL: 62 packages available for upgrade (1 critical updates). [19:57:32] PROBLEM - os162 APT on os162 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [19:57:54] PROBLEM - mw192 APT on mw192 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:59:09] PROBLEM - graylog161 APT on graylog161 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (1 critical updates). [19:59:12] PROBLEM - mw151 APT on mw151 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:59:23] PROBLEM - mw163 APT on mw163 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:00:41] PROBLEM - os202 APT on os202 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [20:02:04] PROBLEM - cloud18 APT on cloud18 is CRITICAL: APT CRITICAL: 199 packages available for upgrade (1 critical updates). [20:02:47] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [20:02:48] PROBLEM - os161 APT on os161 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [20:02:55] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:03:03] PROBLEM - mw201 APT on mw201 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:03:37] PROBLEM - cloud16 APT on cloud16 is CRITICAL: APT CRITICAL: 199 packages available for upgrade (1 critical updates). [20:03:42] PROBLEM - mw162 APT on mw162 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:03:47] PROBLEM - mw181 APT on mw181 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:03:49] PROBLEM - mw172 APT on mw172 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:03:49] PROBLEM - os191 APT on os191 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [20:03:50] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:03:50] PROBLEM - puppet181 APT on puppet181 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (1 critical updates). [20:03:54] PROBLEM - mw203 APT on mw203 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:03:59] PROBLEM - mw173 APT on mw173 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:04:11] PROBLEM - cloud20 APT on cloud20 is CRITICAL: APT CRITICAL: 203 packages available for upgrade (1 critical updates). [20:04:12] PROBLEM - cloud17 APT on cloud17 is CRITICAL: APT CRITICAL: 199 packages available for upgrade (1 critical updates). [20:04:17] PROBLEM - mwtask151 APT on mwtask151 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:04:23] PROBLEM - mw152 APT on mw152 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:04:34] PROBLEM - mw202 APT on mw202 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:04:41] PROBLEM - mwtask161 APT on mwtask161 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:04:44] PROBLEM - mw183 APT on mw183 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:04:48] PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:04:48] PROBLEM - mw161 APT on mw161 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:05:03] PROBLEM - os201 APT on os201 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [20:09:13] !log [universalomega@puppet181] Upgraded packages on test151: libnss3 [20:09:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:10:28] !log [universalomega@puppet181] Upgraded packages on cloud21: libnss3 [20:10:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:10:47] RECOVERY - test151 APT on test151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [20:11:30] !log [universalomega@puppet181] Upgraded packages on cloud15: libnss3 [20:11:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:11:48] !log [universalomega@puppet181] Upgraded packages on graylog161: libnss3 [20:11:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:11:52] RECOVERY - cloud21 APT on cloud21 is OK: APT OK: 15 packages available for upgrade (0 critical updates). [20:11:56] RECOVERY - cloud15 APT on cloud15 is OK: APT OK: 201 packages available for upgrade (0 critical updates). [20:12:02] !log [universalomega@puppet181] Upgraded packages on cloud16: libnss3 [20:12:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:12:17] !log [universalomega@puppet181] Upgraded packages on cloud20: libnss3 [20:12:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:12:46] !log [universalomega@puppet181] Upgraded packages on cloud17: libnss3 [20:12:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:13:02] !log [universalomega@puppet181] Upgraded packages on cloud18: libnss3 [20:13:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:13:09] RECOVERY - graylog161 APT on graylog161 is OK: APT OK: 3 packages available for upgrade (0 critical updates). [20:13:15] !log [universalomega@puppet181] Upgraded packages on cloud19: libnss3 [20:13:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:13:24] RECOVERY - cloud19 APT on cloud19 is OK: APT OK: 61 packages available for upgrade (0 critical updates). [20:13:35] RECOVERY - cloud16 APT on cloud16 is OK: APT OK: 198 packages available for upgrade (0 critical updates). [20:14:09] RECOVERY - cloud20 APT on cloud20 is OK: APT OK: 202 packages available for upgrade (0 critical updates). [20:14:11] RECOVERY - cloud18 APT on cloud18 is OK: APT OK: 198 packages available for upgrade (0 critical updates). [20:14:23] !log [universalomega@puppet181] Upgraded packages on kafka182: libnss3 [20:14:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:14:30] RECOVERY - cloud17 APT on cloud17 is OK: APT OK: 198 packages available for upgrade (0 critical updates). [20:15:02] !log [universalomega@puppet181] Upgraded packages on mw151: libnss3 [20:15:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:15:19] RECOVERY - mw151 APT on mw151 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:15:31] !log [universalomega@puppet181] Upgraded packages on mw153: libnss3 [20:15:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:15:44] !log [universalomega@puppet181] Upgraded packages on mw162: libnss3 [20:15:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:15:58] !log [universalomega@puppet181] Upgraded packages on mw163: libnss3 [20:16:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:16:11] !log [universalomega@puppet181] Upgraded packages on mw152: libnss3 [20:16:11] RECOVERY - kafka182 APT on kafka182 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [20:16:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:16:22] RECOVERY - mw152 APT on mw152 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:16:24] !log [universalomega@puppet181] Upgraded packages on mw172: libnss3 [20:16:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:16:38] !log [universalomega@puppet181] Upgraded packages on mw161: libnss3 [20:16:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:16:51] !log [universalomega@puppet181] Upgraded packages on mw171: libnss3 [20:16:52] RECOVERY - mw161 APT on mw161 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:16:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:17:05] !log [universalomega@puppet181] Upgraded packages on mw182: libnss3 [20:17:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:17:19] !log [universalomega@puppet181] Upgraded packages on mw191: libnss3 [20:17:20] RECOVERY - mw171 APT on mw171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:17:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:17:22] RECOVERY - mw153 APT on mw153 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:17:22] RECOVERY - mw163 APT on mw163 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:17:33] !log [universalomega@puppet181] Upgraded packages on mw173: libnss3 [20:17:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:17:39] RECOVERY - mw162 APT on mw162 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:17:45] RECOVERY - mw172 APT on mw172 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:17:47] !log [universalomega@puppet181] Upgraded packages on mw181: libnss3 [20:17:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:17:57] RECOVERY - mw173 APT on mw173 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:18:01] !log [universalomega@puppet181] Upgraded packages on mw192: libnss3 [20:18:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:18:15] !log [universalomega@puppet181] Upgraded packages on mw193: libnss3 [20:18:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:18:30] !log [universalomega@puppet181] Upgraded packages on mw183: libnss3 [20:18:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:18:44] !log [universalomega@puppet181] Upgraded packages on mw201: libnss3 [20:18:44] RECOVERY - mw183 APT on mw183 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:18:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:18:54] RECOVERY - mw182 APT on mw182 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:18:55] RECOVERY - mw191 APT on mw191 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:18:58] !log [universalomega@puppet181] Upgraded packages on mw202: libnss3 [20:19:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:19:02] RECOVERY - mw201 APT on mw201 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:19:11] !log [universalomega@puppet181] Upgraded packages on mwtask151: libnss3 [20:19:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:19:33] !log [universalomega@puppet181] Upgraded packages on mw203: libnss3 [20:19:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:19:35] RECOVERY - mw193 APT on mw193 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:19:46] RECOVERY - mw181 APT on mw181 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:19:49] !log [universalomega@puppet181] Upgraded packages on mwtask171: libnss3 [20:19:49] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:19:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:19:51] RECOVERY - mw203 APT on mw203 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:19:51] RECOVERY - mw192 APT on mw192 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:20:03] !log [universalomega@puppet181] Upgraded packages on mwtask161: libnss3 [20:20:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:20:17] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:20:22] !log [universalomega@puppet181] Upgraded packages on os151: libnss3 [20:20:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:20:32] RECOVERY - mw202 APT on mw202 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:20:40] RECOVERY - os151 APT on os151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [20:20:40] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:20:43] !log [universalomega@puppet181] Upgraded packages on os162: libnss3 [20:20:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:20:57] !log [universalomega@puppet181] Upgraded packages on os161: libnss3 [20:20:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:21:14] !log [universalomega@puppet181] Upgraded packages on mwtask181: libnss3 [20:21:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:21:27] !log [universalomega@puppet181] Upgraded packages on puppet181: libnss3 [20:21:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:21:32] RECOVERY - os162 APT on os162 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [20:21:40] !log [universalomega@puppet181] Upgraded packages on os201: libnss3 [20:21:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:21:54] !log [universalomega@puppet181] Upgraded packages on os191: libnss3 [20:21:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:21:57] RECOVERY - puppet181 APT on puppet181 is OK: APT OK: 3 packages available for upgrade (0 critical updates). [20:22:30] !log [universalomega@puppet181] Upgraded packages on os202: libnss3 [20:22:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:22:41] RECOVERY - os202 APT on os202 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [20:22:48] RECOVERY - os161 APT on os161 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [20:22:52] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:23:27] RECOVERY - os201 APT on os201 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [20:23:46] RECOVERY - os191 APT on os191 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [20:36:09] [02mw-config] 07Universal-Omega created 03replace-trigger_error (+1 new commit) 13https://github.com/miraheze/mw-config/commit/0b706da66c3c [20:36:09] 02mw-config/03replace-trigger_error 07CosmicAlpha 030b706da Replace deprecated trigger_error [20:36:16] [02mw-config] 07Universal-Omega opened pull request #6330: Replace deprecated trigger_error (03main...03replace-trigger_error) 13https://github.com/miraheze/mw-config/pull/6330 [20:37:26] miraheze/mw-config - Universal-Omega the build passed. [20:45:05] [02mw-config] 07Universal-Omega pushed 1 new commit to 03replace-trigger_error 13https://github.com/miraheze/mw-config/commit/e01962e9b93535ae6c6ecd12e0466fd02b43ae62 [20:45:06] 02mw-config/03replace-trigger_error 07CosmicAlpha 03e01962e Print [20:46:21] miraheze/mw-config - Universal-Omega the build passed. [20:46:24] [02mw-config] 07Universal-Omega merged pull request #6330: Replace deprecated trigger_error (03main...03replace-trigger_error) 13https://github.com/miraheze/mw-config/pull/6330 [20:46:25] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/4448ff46dfcfd051bd6473e893d151f373466197 [20:46:25] 02mw-config/03main 07CosmicAlpha 034448ff4 Replace deprecated trigger_error [20:46:26] [02mw-config] 07Universal-Omega 04deleted 03replace-trigger_error at 03e01962e 13https://api.github.com/repos/miraheze/mw-config/commit/e01962e [20:47:31] miraheze/mw-config - Universal-Omega the build passed. [20:48:26] !log [universalomega@cloud21] Clear SEL [20:48:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:49:28] RECOVERY - cloud21 IPMI Sensors on cloud21 is OK: IPMI Status: OK [20:50:35] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [20:50:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:50:58] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 23s [20:51:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:02:44] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772136110000&orgId=1&to=1772139764503 [21:07:44] [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1772136110000&orgId=1&to=1772139830000 [22:30:37] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/3656c79bde7365670701a0376dec993434c574a2 [22:30:37] 02ssl/03main 07WikiTideBot 033656c79 Bot: Auto-update domain lists [22:32:10] [02CreateWiki] 07Universal-Omega 04force-pushed 03strict_types from 03adfe8c1 to 036a4c3d0 13https://api.github.com/repos/miraheze/CreateWiki/commits/strict_types [22:32:10] 02CreateWiki/03strict_types 07CosmicAlpha 03a74c094 Use strict_types [22:32:11] 02CreateWiki/03strict_types 07CosmicAlpha 03f8976e8 - [22:32:12] 02CreateWiki/03strict_types 07CosmicAlpha 0331cf3f6 Fix [22:32:13] 02CreateWiki/03strict_types 07CosmicAlpha 036a4c3d0 Fix [22:40:47] miraheze/CreateWiki - Universal-Omega the build passed. [22:42:05] [02CreateWiki] 07Universal-Omega created 03log-linkrenderer (+1 new commit) 13https://github.com/miraheze/CreateWiki/commit/5b9e08be3c6b [22:42:05] 02CreateWiki/03log-linkrenderer 07CosmicAlpha 035b9e08b FarmerLogFormatter: don't inject LinkRenderer, use getLinkRenderer() [22:42:12] [02CreateWiki] 07Universal-Omega opened pull request #808: FarmerLogFormatter: don't inject LinkRenderer, use getLinkRenderer() (03main...03log-linkrenderer) 13https://github.com/miraheze/CreateWiki/pull/808 [22:43:57] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03log-linkrenderer 13https://github.com/miraheze/CreateWiki/commit/078e2bf8e2bb9be76e32b90e68e5f3c19098b1ef [22:43:58] 02CreateWiki/03log-linkrenderer 07CosmicAlpha 03078e2bf - [22:52:50] miraheze/CreateWiki - Universal-Omega the build passed. [23:00:48] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/155951bcbffbe3112ac34d57b5cdcd51fff03b50 [23:00:48] 02ssl/03main 07WikiTideBot 03155951b Bot: Auto-update domain lists [23:04:54] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/2e4c2128908f11c1c5f1890ff2516b20f5ebb289 [23:04:54] 02puppet/03main 07CosmicAlpha 032e4c212 mattermost: upgrade to 11.4.2 [23:07:58] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/707d3f8bd9c6f9ed4a6c850476153867cfebf028 [23:07:58] 02puppet/03main 07CosmicAlpha 03707d3f8 mattermost: remove cp from firewall rules [23:13:24] [02puppet] 07Universal-Omega created 03mattermost-cleanup (+1 new commit) 13https://github.com/miraheze/puppet/commit/d26b412c1eef [23:13:24] 02puppet/03mattermost-cleanup 07CosmicAlpha 03d26b412 mattermost: some cleanup and param validation [23:13:30] [02puppet] 07Universal-Omega opened pull request #4797: mattermost: some cleanup and param validation (03main...03mattermost-cleanup) 13https://github.com/miraheze/puppet/pull/4797 [23:16:29] [02puppet] 07Universal-Omega pushed 1 new commit to 03mattermost-cleanup 13https://github.com/miraheze/puppet/commit/38ed5da52a3eea460416c9825d623635d4bd3bfe [23:16:30] 02puppet/03mattermost-cleanup 07CosmicAlpha 0338ed5da - [23:18:22] [02puppet] 07Universal-Omega pushed 1 new commit to 03mattermost-cleanup 13https://github.com/miraheze/puppet/commit/bff48787b06c7b86dbd19a231b6edca702240a29 [23:18:23] 02puppet/03mattermost-cleanup 07CosmicAlpha 03bff4878 Update [23:22:28] [02puppet] 07Universal-Omega retitled PR #4797: "mattermost: some cleanup and param validation" ➜ "mattermost: add param validation" 13https://github.com/miraheze/puppet/pull/4797 [23:24:07] [02puppet] 07Universal-Omega merged pull request #4797: mattermost: add param validation (03main...03mattermost-cleanup) 13https://github.com/miraheze/puppet/pull/4797 [23:24:07] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/75c971540a4b3302ff240175deeafef2f95f8532 [23:24:07] 02puppet/03main 07CosmicAlpha 0375c9715 mattermost: add param validation… [23:24:09] [02puppet] 07Universal-Omega 04deleted 03mattermost-cleanup at 03bff4878 13https://api.github.com/repos/miraheze/puppet/commit/bff4878 [23:30:31] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/bd3fe535238414929217e52245b9361714c1dce6 [23:30:31] 02ssl/03main 07WikiTideBot 03bd3fe53 Bot: Auto-update domain lists