[00:01:03] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/763a94d8954dae78fc1c83ea9e40d70cf9368839 [00:01:03] 02statichelp/03main 07WikiTideBot 03763a94d Bot: Auto-update Tech namespace pages 2026-10-27 00:01:00 [00:50:03] [02dns] 07Universal-Omega created 03cloud21-mw (+1 new commit) 13https://github.com/miraheze/dns/commit/52647c893a84 [00:50:03] 02dns/03cloud21-mw 07CosmicAlpha 0352647c8 Add mw21* to dns [00:50:09] [02dns] 07Universal-Omega opened pull request #719: Add mw21* to dns (03main...03cloud21-mw) 13https://github.com/miraheze/dns/pull/719 [00:51:01] [02dns] 07Universal-Omega pushed 1 new commit to 03cloud21-mw 13https://github.com/miraheze/dns/commit/2b2389ce61512ff092543487b3b0c3e290b3e809 [00:51:01] 02dns/03cloud21-mw 07CosmicAlpha 032b2389c Update [00:51:03] miraheze/dns - Universal-Omega the build passed. [00:51:37] [02dns] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/4895c0c16a6989b08c53b0a31856a6ab5ba9a1ae [00:51:38] 02dns/03main 07CosmicAlpha 034895c0c Add mw21* to dns [00:51:40] [02dns] 07Universal-Omega merged pull request #719: Add mw21* to dns (03main...03cloud21-mw) 13https://github.com/miraheze/dns/pull/719 [00:51:42] [02dns] 07Universal-Omega 04deleted 03cloud21-mw at 032b2389c 13https://api.github.com/repos/miraheze/dns/commit/2b2389c [00:51:51] miraheze/dns - Universal-Omega the build passed. [00:52:22] miraheze/dns - Universal-Omega the build passed. [02:05:08] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/504b84c3aee9f139ff4617565886910622169a14 [02:05:08] 02puppet/03main 07CosmicAlpha 03504b84c squid: allow proxying puppet [02:11:33] !log create VMs for mw211, mw212, and mw213 [02:11:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027021133-4185db89 [02:18:34] [02puppet] 07Universal-Omega created 03cloud21-mw (+1 new commit) 13https://github.com/miraheze/puppet/commit/363d2da7dd35 [02:18:34] 02puppet/03cloud21-mw 07CosmicAlpha 03363d2da Add new hiera for mw21*… [02:18:39] [02puppet] 07Universal-Omega opened pull request #5122: Add new hiera for mw21* (03main...03cloud21-mw) 13https://github.com/miraheze/puppet/pull/5122 [02:19:07] [02puppet] 07Universal-Omega pushed 1 new commit to 03cloud21-mw 13https://github.com/miraheze/puppet/commit/a359f18391871b876f0ff923312741e7ab811619 [02:19:08] 02puppet/03cloud21-mw 07CosmicAlpha 03a359f18 + [02:19:26] [02puppet] 07Universal-Omega pushed 1 new commit to 03cloud21-mw 13https://github.com/miraheze/puppet/commit/02141d400dba0a423a66c425c30bc0289c077235 [02:19:27] 02puppet/03cloud21-mw 07CosmicAlpha 0302141d4 + [02:20:12] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/b33e843a4c3132b924e097e083b964b85ad403da [02:20:12] 02puppet/03main 07CosmicAlpha 03b33e843 Add new hiera for mw21*… [02:20:13] [02puppet] 07Universal-Omega merged pull request #5122: Add new hiera for mw21* (03main...03cloud21-mw) 13https://github.com/miraheze/puppet/pull/5122 [02:20:14] [02puppet] 07Universal-Omega 04deleted 03cloud21-mw at 0302141d4 13https://api.github.com/repos/miraheze/puppet/commit/02141d4 [02:20:43] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/09976d625d2bdf684dbb0e84306aa967190bede2 [02:20:43] 02puppet/03main 07CosmicAlpha 0309976d6 mediawiki: remove support for xhprof [02:20:44] [02puppet] 07Universal-Omega merged pull request #5109: mediawiki: remove support for xhprof (03main...03remove-xhprof) 13https://github.com/miraheze/puppet/pull/5109 [02:20:45] [02puppet] 07Universal-Omega 04deleted 03remove-xhprof at 03b3b17d5 13https://api.github.com/repos/miraheze/puppet/commit/b3b17d5 [02:31:43] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/0133f22793597c51d7c3954778efd3b26ff58942 [02:31:43] 02puppet/03main 07CosmicAlpha 030133f22 Use libboost 1.83 [02:34:14] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/36eb4325b5c69fbf0294dca0dd3e2f27294b2107 [02:34:14] 02puppet/03main 07CosmicAlpha 0336eb432 Use libfmt10 [02:34:15] PROBLEM - mw211 PowerDNS Recursor on mw211 is CRITICAL: Domain 'mw211.fsslc.wtnet' was not found by the server [02:34:24] PROBLEM - mw211.fsslc.wtnet SSL Check on sslhost is CRITICAL: connect to address mw211.fsslc.wtnet and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:34:27] PROBLEM - mw211 chrony on mw211 is WARNING: chrony stratum too high: 10 > 5 [02:34:29] PROBLEM - mw211 HTTPS on mw211 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw211.fsslc.wtnet port 443 after 0 ms: Could not connect to server [02:34:42] PROBLEM - mw211 Puppet on mw211 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:34:47] PROBLEM - mw211 MediaWiki Rendering on mw211 is CRITICAL: connect to address 10.0.21.175 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:35:13] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/f912075876b4be3ee08fecff3770754ee31d5ef6 [02:35:13] 02puppet/03main 07CosmicAlpha 03f912075 Fix [02:38:05] PROBLEM - mw211 APT on mw211 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [02:46:05] RECOVERY - mw211 APT on mw211 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [03:32:52] [02puppet] 07Universal-Omega created 03mcrouter-trixie (+1 new commit) 13https://github.com/miraheze/puppet/commit/da3513d0c57d [03:32:52] 02puppet/03mcrouter-trixie 07CosmicAlpha 03da3513d mcrouter: use version built for trixie [03:32:58] [02puppet] 07Universal-Omega opened pull request #5123: mcrouter: use version built for trixie (03main...03mcrouter-trixie) 13https://github.com/miraheze/puppet/pull/5123 [03:35:45] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/203cb66c001d98e3a4a144c109dc71fcb0c025c2 [03:35:45] 02puppet/03main 07CosmicAlpha 03203cb66 mediawiki: only use internal IPs [03:36:14] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/29c1943ba44a5e85d69857cd74d015c6eb5dcf77 [03:36:15] 02puppet/03main 07CosmicAlpha 0329c1943 mcrouter: use version built for trixie [03:36:17] [02puppet] 07Universal-Omega merged pull request #5123: mcrouter: use version built for trixie (03main...03mcrouter-trixie) 13https://github.com/miraheze/puppet/pull/5123 [03:36:19] [02puppet] 07Universal-Omega 04deleted 03mcrouter-trixie at 03da3513d 13https://api.github.com/repos/miraheze/puppet/commit/da3513d [03:38:15] RECOVERY - mw211 PowerDNS Recursor on mw211 is OK: DNS OK: 0.030 seconds response time. mw211.fsslc.wtnet returns 10.0.21.175 [03:39:20] PROBLEM - mw211 MediaWiki Rendering on mw211 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 305 bytes in 0.013 second response time [03:39:51] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/0b581dc2c75c32ce8105ecc1d8498b840c7a7de9 [03:39:51] 02puppet/03main 07CosmicAlpha 030b581dc mcrouter: install libevent-2.1-7t64 [03:40:11] PROBLEM - mw211 APT on mw211 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [03:42:11] RECOVERY - mw211 APT on mw211 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:01:27] RECOVERY - mw211.fsslc.wtnet SSL Check on sslhost is OK: OK - Certificate 'CloudFlare Origin Certificate' will expire on Tue 03 Apr 2040 12:14:00 PM GMT +0000. [04:04:44] PROBLEM - mw212 Current Load on mw212 is CRITICAL: connect to address 10.0.21.176 port 5666: Connection refusedconnect to host 10.0.21.176 port 5666: Connection refused [04:04:46] PROBLEM - mw212 Disk Space on mw212 is CRITICAL: connect to address 10.0.21.176 port 5666: Connection refusedconnect to host 10.0.21.176 port 5666: Connection refused [04:04:55] PROBLEM - mw212 php-fpm on mw212 is CRITICAL: connect to address 10.0.21.176 port 5666: Connection refusedconnect to host 10.0.21.176 port 5666: Connection refused [04:04:56] PROBLEM - mw212 APT on mw212 is CRITICAL: connect to address 10.0.21.176 port 5666: Connection refusedconnect to host 10.0.21.176 port 5666: Connection refused [04:04:57] PROBLEM - mw212.fsslc.wtnet SSL Check on sslhost is CRITICAL: connect to address mw212.fsslc.wtnet and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [04:05:00] PROBLEM - mw212 LB agent on mw212 is CRITICAL: connect to address 10.0.21.176 and port 9590: Connection refused [04:05:03] PROBLEM - mw212 HTTPS on mw212 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw212.fsslc.wtnet port 443 after 0 ms: Could not connect to server [04:05:05] PROBLEM - mw212 PowerDNS Recursor on mw212 is CRITICAL: connect to address 10.0.21.176 port 5666: Connection refusedconnect to host 10.0.21.176 port 5666: Connection refused [04:05:18] PROBLEM - mw212 conntrack_table_size on mw212 is CRITICAL: connect to address 10.0.21.176 port 5666: Connection refusedconnect to host 10.0.21.176 port 5666: Connection refused [04:05:20] PROBLEM - mw212 Check unit status of php8.4-fpm_check_restart on mw212 is CRITICAL: connect to address 10.0.21.176 port 5666: Connection refusedconnect to host 10.0.21.176 port 5666: Connection refused [04:05:22] PROBLEM - mw212 MediaWiki Rendering on mw212 is CRITICAL: connect to address 10.0.21.176 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [04:05:24] PROBLEM - mw212 Puppet on mw212 is CRITICAL: connect to address 10.0.21.176 port 5666: Connection refusedconnect to host 10.0.21.176 port 5666: Connection refused [04:05:26] PROBLEM - mw212 chrony on mw212 is CRITICAL: connect to address 10.0.21.176 port 5666: Connection refusedconnect to host 10.0.21.176 port 5666: Connection refused [04:05:35] PROBLEM - mw212 nftables_active on mw212 is CRITICAL: connect to address 10.0.21.176 port 5666: Connection refusedconnect to host 10.0.21.176 port 5666: Connection refused [04:06:20] RECOVERY - mw211 chrony on mw211 is OK: chrony OK: System Time 0.000000175 seconds fast of NTP time, Stratum 3 [04:07:10] PROBLEM - mw211 MediaWiki Rendering on mw211 is CRITICAL: connect to address 10.0.21.175 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [04:08:07] !log rebuild mcrouter for trixie [04:08:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027040808-4ee9de5c [04:09:22] PROBLEM - mw211 MediaWiki Rendering on mw211 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 305 bytes in 0.012 second response time [04:10:38] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/e9d50b4c30f9d92aafd5a02b823113a8db80c56a [04:10:38] 02puppet/03main 07CosmicAlpha 03e9d50b4 mwdeploy: add mw211, mw212, and mw213 [04:11:24] PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libevent-2.1-7t64] [04:11:37] PROBLEM - mw202 Puppet on mw202 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libevent-2.1-7t64] [04:12:21] miraheze/puppet - Universal-Omega the build passed. [04:12:32] PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libevent-2.1-7t64] [04:12:54] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libevent-2.1-7t64] [04:13:23] !log [universalomega@mwtask181] Starting new install of {'versions': '1.46', 'new_install': True} to mw211 [04:13:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027041323-bf6a02c3 [04:13:54] !log [universalomega@mwtask181] finished new install of {'versions': '1.46', 'new_install': True} to mw211 - SUCCESS in 31s [04:13:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027041354-d8b02d07 [04:14:26] RECOVERY - mw211 MediaWiki Rendering on mw211 is OK: HTTP OK: HTTP/1.1 200 OK - 1188 bytes in 0.296 second response time [04:14:28] PROBLEM - mw203 Puppet on mw203 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libevent-2.1-7t64] [04:15:07] RECOVERY - mw211 Puppet on mw211 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [04:15:18] PROBLEM - mw212 MediaWiki Rendering on mw212 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 305 bytes in 0.013 second response time [04:15:21] PROBLEM - mw212 nftables_active on mw212 is UNKNOWN: NRPE: Unable to read output [04:16:13] RECOVERY - mw212 Current Load on mw212 is OK: LOAD OK - total load average: 0.72, 0.87, 0.55 [04:16:23] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/76e96fc94136f2b11c85038598b935b5d396e07e [04:16:23] 02puppet/03main 07CosmicAlpha 0376e96fc mcrouter: use the libgoogle-glog0v6t64 package [04:16:24] RECOVERY - mw212 Disk Space on mw212 is OK: DISK OK - free space: / 43700MiB (88.2% inode=95%); [04:16:28] RECOVERY - mw212 APT on mw212 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:16:44] RECOVERY - mw212 php-fpm on mw212 is OK: PROCS OK: 37 processes with command name 'php-fpm8.4' [04:16:44] RECOVERY - mw212 Check unit status of php8.4-fpm_check_restart on mw212 is OK: OK: Status of the systemd unit php8.4-fpm_check_restart [04:16:48] RECOVERY - mw212 LB agent on mw212 is OK: TCP OK - 0.000 second response time on 10.0.21.176 port 9590 [04:17:00] RECOVERY - mw212 PowerDNS Recursor on mw212 is OK: DNS OK: 0.045 seconds response time. mw212.fsslc.wtnet returns 10.0.21.176 [04:17:07] RECOVERY - mw212 conntrack_table_size on mw212 is OK: OK: nf_conntrack is 0 % full [04:17:12] RECOVERY - mw212 chrony on mw212 is OK: chrony OK: System Time 0.000001334 seconds slow of NTP time, Stratum 3 [04:17:18] RECOVERY - mw212 nftables_active on mw212 is OK: OK nftables input default policy is set [04:19:18] PROBLEM - mw212 MediaWiki Rendering on mw212 is CRITICAL: connect to address 10.0.21.176 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [04:21:18] PROBLEM - mw212 MediaWiki Rendering on mw212 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 305 bytes in 0.013 second response time [04:22:25] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libgoogle-glog0v6t64] [04:22:57] PROBLEM - mw191 Puppet on mw191 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libgoogle-glog0v6t64] [04:23:28] PROBLEM - mw153 Puppet on mw153 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libgoogle-glog0v6t64] [04:24:42] RECOVERY - mw212.fsslc.wtnet SSL Check on sslhost is OK: OK - Certificate 'CloudFlare Origin Certificate' will expire on Tue 03 Apr 2040 12:14:00 PM GMT +0000. [04:29:14] PROBLEM - mw163 Puppet on mw163 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libgoogle-glog0v6t64] [04:29:38] PROBLEM - mw192 Puppet on mw192 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libgoogle-glog0v6t64] [04:30:27] PROBLEM - mw151 Puppet on mw151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libgoogle-glog0v6t64] [04:31:07] [02puppet] 07Universal-Omega created 03aptproxy (+1 new commit) 13https://github.com/miraheze/puppet/commit/3947a697079e [04:31:07] 02puppet/03aptproxy 07CosmicAlpha 033947a69 Unify apt proxy and set if http proxy is set [04:31:12] [02puppet] 07Universal-Omega opened pull request #5124: Unify apt proxy and set if http proxy is set (03main...03aptproxy) 13https://github.com/miraheze/puppet/pull/5124 [04:32:01] [02puppet] 07Universal-Omega pushed 1 new commit to 03aptproxy 13https://github.com/miraheze/puppet/commit/56ec9a12841b5569057dd2f23f9e9cdacda06a97 [04:32:01] 02puppet/03aptproxy 07CosmicAlpha 0356ec9a1 Update [04:32:37] [02puppet] 07Universal-Omega pushed 1 new commit to 03aptproxy 13https://github.com/miraheze/puppet/commit/30adace9f3eaac69c99707a81236c8dfc5dd1570 [04:32:38] 02puppet/03aptproxy 07CosmicAlpha 0330adace - [04:32:42] [02TSPortal] 07renovate[bot] created 03renovate/dev-deps (+1 new commit) 13https://github.com/miraheze/TSPortal/commit/794a3e6f1541 [04:32:42] 02TSPortal/03renovate/dev-deps 07renovate[bot] 03794a3e6 Update dependency phpunit/phpunit to v13.4.5 [04:33:06] [02puppet] 07Universal-Omega pushed 1 new commit to 03aptproxy 13https://github.com/miraheze/puppet/commit/a104bbfca08315bb321cdc5e3269003a39f1d9be [04:33:06] 02puppet/03aptproxy 07CosmicAlpha 03a104bbf - [04:33:11] PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libgoogle-glog0v6t64] [04:33:56] PROBLEM - mw213 MediaWiki Rendering on mw213 is CRITICAL: connect to address 10.0.21.177 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [04:33:58] PROBLEM - mw213 nftables_active on mw213 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.21.177: Connection reset by peer [04:33:58] PROBLEM - mw213 Check unit status of php8.4-fpm_check_restart on mw213 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.21.177: Connection reset by peer [04:34:00] PROBLEM - mw213 chrony on mw213 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.21.177: Connection reset by peer [04:34:07] PROBLEM - mw213 APT on mw213 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.21.177: Connection reset by peer [04:34:08] PROBLEM - mw213 Current Load on mw213 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.21.177: Connection reset by peer [04:34:12] PROBLEM - mw213 PowerDNS Recursor on mw213 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.21.177: Connection reset by peer [04:34:14] [02puppet] 07Universal-Omega pushed 1 new commit to 03aptproxy 13https://github.com/miraheze/puppet/commit/36a11f8d1f55468b006180e1e698a806a448a4b9 [04:34:14] PROBLEM - mw213 php-fpm on mw213 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.21.177: Connection reset by peer [04:34:14] 02puppet/03aptproxy 07CosmicAlpha 0336a11f8 - [04:34:33] PROBLEM - mw213 LB agent on mw213 is CRITICAL: connect to address 10.0.21.177 and port 9590: Connection refused [04:34:35] PROBLEM - mw213 Disk Space on mw213 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.21.177: Connection reset by peer [04:34:35] PROBLEM - mw213 Puppet on mw213 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.21.177: Connection reset by peer [04:34:38] PROBLEM - mw213 conntrack_table_size on mw213 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.21.177: Connection reset by peer [04:34:39] PROBLEM - mw213 HTTPS on mw213 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw213.fsslc.wtnet port 443 after 0 ms: Could not connect to server [04:34:52] PROBLEM - mw213.fsslc.wtnet SSL Check on sslhost is CRITICAL: connect to address mw213.fsslc.wtnet and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [04:34:56] [02puppet] 07Universal-Omega pushed 1 new commit to 03aptproxy 13https://github.com/miraheze/puppet/commit/076cf0dc1009e5115a9bd62567698e5bf10cd5d4 [04:34:56] 02puppet/03aptproxy 07CosmicAlpha 03076cf0d - [04:35:31] PROBLEM - mw201 Puppet on mw201 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libgoogle-glog0v6t64] [04:35:42] [02puppet] 07Universal-Omega pushed 1 new commit to 03aptproxy 13https://github.com/miraheze/puppet/commit/6a33de151d82c1d496d9d06baf3a55ac355034be [04:35:42] 02puppet/03aptproxy 07CosmicAlpha 036a33de1 - [04:36:07] PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libgoogle-glog0v6t64] [04:36:25] [02puppet] 07Universal-Omega pushed 1 new commit to 03aptproxy 13https://github.com/miraheze/puppet/commit/cac54a4ad2a5eeb8406362e0c4af003f44a3f2ca [04:36:25] 02puppet/03aptproxy 07CosmicAlpha 03cac54a4 - [04:37:00] [02puppet] 07Universal-Omega pushed 1 new commit to 03aptproxy 13https://github.com/miraheze/puppet/commit/0e335aa3e993a4209732fd0326fb4eb2c214b585 [04:37:00] 02puppet/03aptproxy 07CosmicAlpha 030e335aa - [04:37:11] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libgoogle-glog0v6t64] [04:37:29] PROBLEM - mw183 Puppet on mw183 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libgoogle-glog0v6t64] [04:37:31] [02puppet] 07Universal-Omega pushed 1 new commit to 03aptproxy 13https://github.com/miraheze/puppet/commit/54c97b55ec3808a7573bbf2fd6be7dbf36c7ef3b [04:37:31] 02puppet/03aptproxy 07CosmicAlpha 0354c97b5 - [04:37:43] [02puppet] 07Universal-Omega pushed 1 new commit to 03aptproxy 13https://github.com/miraheze/puppet/commit/22566574542ef850616b99bb766646f2d7a098f7 [04:37:43] 02puppet/03aptproxy 07CosmicAlpha 032256657 - [04:37:43] PROBLEM - mw213 MediaWiki Rendering on mw213 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 305 bytes in 0.017 second response time [04:37:53] RECOVERY - mw213 Check unit status of php8.4-fpm_check_restart on mw213 is OK: OK: Status of the systemd unit php8.4-fpm_check_restart [04:37:55] RECOVERY - mw213 chrony on mw213 is OK: chrony OK: System Time 0.000002180 seconds fast of NTP time, Stratum 3 [04:37:58] PROBLEM - mw213 nftables_active on mw213 is UNKNOWN: NRPE: Unable to read output [04:38:00] RECOVERY - mw213 APT on mw213 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:38:00] RECOVERY - mw213 PowerDNS Recursor on mw213 is OK: DNS OK: 0.041 seconds response time. mw213.fsslc.wtnet returns 10.0.21.177 [04:38:03] RECOVERY - mw213 Current Load on mw213 is OK: LOAD OK - total load average: 0.77, 0.91, 0.56 [04:38:08] RECOVERY - mw213 php-fpm on mw213 is OK: PROCS OK: 37 processes with command name 'php-fpm8.4' [04:38:30] PROBLEM - mw213 Puppet on mw213 is UNKNOWN: NRPE: Unable to read output [04:38:32] RECOVERY - mw213 Disk Space on mw213 is OK: DISK OK - free space: / 43703MiB (88.2% inode=95%); [04:38:32] RECOVERY - mw213 LB agent on mw213 is OK: TCP OK - 0.000 second response time on 10.0.21.177 port 9590 [04:38:33] [02puppet] 07Universal-Omega pushed 1 new commit to 03aptproxy 13https://github.com/miraheze/puppet/commit/4fd11ff908986337c889b9c975be54e76165bc0b [04:38:34] 02puppet/03aptproxy 07CosmicAlpha 034fd11ff - [04:38:35] PROBLEM - mw213 conntrack_table_size on mw213 is UNKNOWN: NRPE: Unable to read output [04:39:00] [02puppet] 07Universal-Omega pushed 1 new commit to 03aptproxy 13https://github.com/miraheze/puppet/commit/ad0dc1abf6aa2d8b5ed653510c77de94e9a05b3a [04:39:00] 02puppet/03aptproxy 07CosmicAlpha 03ad0dc1a - [04:39:21] [02puppet] 07Universal-Omega pushed 1 new commit to 03aptproxy 13https://github.com/miraheze/puppet/commit/8799a886dc2f574bcb18eb6662f796fc4208739b [04:39:21] 02puppet/03aptproxy 07CosmicAlpha 038799a88 - [04:39:40] [02puppet] 07Universal-Omega pushed 1 new commit to 03aptproxy 13https://github.com/miraheze/puppet/commit/31cf29d830cdecabe6f1a46102140f982f512cf7 [04:39:40] 02puppet/03aptproxy 07CosmicAlpha 0331cf29d - [04:39:58] RECOVERY - mw213 nftables_active on mw213 is OK: OK nftables input default policy is set [04:40:16] [02puppet] 07Universal-Omega pushed 1 new commit to 03aptproxy 13https://github.com/miraheze/puppet/commit/4f364b370af89a0f40cf818f75895401f2f95b40 [04:40:16] 02puppet/03aptproxy 07CosmicAlpha 034f364b3 - [04:40:29] PROBLEM - mw213 Puppet on mw213 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.46/skins/Femiwiki/node_modules] [04:40:33] RECOVERY - mw213 conntrack_table_size on mw213 is OK: OK: nf_conntrack is 0 % full [04:40:51] [02puppet] 07Universal-Omega pushed 1 new commit to 03aptproxy 13https://github.com/miraheze/puppet/commit/c3187ca5245fdae960b6c2d075947ed7310b6d41 [04:40:51] 02puppet/03aptproxy 07CosmicAlpha 03c3187ca - [04:41:55] [02puppet] 07Universal-Omega merged pull request #5124: Unify apt proxy and set if http proxy is set (03main...03aptproxy) 13https://github.com/miraheze/puppet/pull/5124 [04:41:56] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/4f51086e4027ab2d46128474e5118f8b95ab8e8c [04:41:57] 02puppet/03main 07CosmicAlpha 034f51086 Unify apt proxy and set if http proxy is set [04:41:59] [02puppet] 07Universal-Omega 04deleted 03aptproxy at 03c3187ca 13https://api.github.com/repos/miraheze/puppet/commit/c3187ca [04:43:19] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 44.06, 22.71, 9.46 [04:44:54] PROBLEM - mw173 Puppet on mw173 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libgoogle-glog0v6t64] [04:46:35] !log [universalomega@mwtask181] Starting new install of {'versions': '1.46', 'new_install': True} to ['mw212', 'mw213'] [04:46:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027044635-8132717e [04:46:54] !log [universalomega@mwtask181] finished new install of {'versions': '1.46', 'new_install': True} to ['mw212', 'mw213'] - SUCCESS in 18s [04:46:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027044654-6cde7312 [04:47:10] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 2.25, 11.31, 7.71 [04:47:12] PROBLEM - mw213 MediaWiki Rendering on mw213 is CRITICAL: connect to address 10.0.21.177 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [04:47:19] RECOVERY - mw212 MediaWiki Rendering on mw212 is OK: HTTP OK: HTTP/1.1 200 OK - 1188 bytes in 0.292 second response time [04:48:19] PROBLEM - mw193 Puppet on mw193 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libgoogle-glog0v6t64] [04:49:06] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.70, 8.25, 7.01 [04:49:25] RECOVERY - mw212 Puppet on mw212 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [04:49:29] RECOVERY - mw193 Puppet on mw193 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [04:50:20] RECOVERY - mw213 Puppet on mw213 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:51:06] RECOVERY - mw213 MediaWiki Rendering on mw213 is OK: HTTP OK: HTTP/1.1 200 OK - 1188 bytes in 0.025 second response time [04:52:37] RECOVERY - mw173 Puppet on mw173 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [04:52:38] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [04:52:38] RECOVERY - mw153 Puppet on mw153 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [04:52:43] RECOVERY - mw183 Puppet on mw183 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [04:52:48] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [04:52:53] RECOVERY - mw192 Puppet on mw192 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [04:52:53] RECOVERY - mw213.fsslc.wtnet SSL Check on sslhost is OK: OK - Certificate 'CloudFlare Origin Certificate' will expire on Tue 03 Apr 2040 12:14:00 PM GMT +0000. [04:52:54] RECOVERY - mw191 Puppet on mw191 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [04:52:59] RECOVERY - mw163 Puppet on mw163 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [04:53:00] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [04:53:12] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [04:53:14] RECOVERY - mw201 Puppet on mw201 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [04:53:18] RECOVERY - mw202 Puppet on mw202 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:53:22] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:53:30] RECOVERY - mw203 Puppet on mw203 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:53:31] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:53:34] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:53:41] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:03:02] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/2bd7e0522e3ff83e813ef21d87eb9644548c075f [05:03:02] 02puppet/03main 07CosmicAlpha 032bd7e05 Remove legacy apt proxy cleanup [05:03:07] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/c416936da4b849033db6a1cd0fef7b4b873748cd [05:03:07] 02puppet/03main 07CosmicAlpha 03c416936 cache: pool in mw211, mw212, and mw213 [05:04:46] PROBLEM - cp171 Haproxy balancer mediawiki on cp171 is CRITICAL: connect to address localhost and port 8150: Connection refused [05:05:45] !log [universalomega@mwtask181] Starting deploy of {'pull': 'config', 'config': True} to ['mw211', 'mw212', 'mw213'] [05:05:46] !log [universalomega@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw211 [05:05:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027050545-27f562aa [05:05:55] An exception was raised while trying to log your message, MirahezeLSBot [05:05:58] !log [universalomega@mwtask181] Starting deploy of {'pull': 'config', 'config': True, 'force': True} to ['mw211', 'mw212', 'mw213'] [05:06:01] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True} to ['mw211', 'mw212', 'mw213'] - SUCCESS in 2s [05:06:05] !log [universalomega@mwtask181] Starting deploy of {'pull': 'config', 'config': True} to ['mw211', 'mw212', 'mw213'] [05:06:06] !log [universalomega@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw211 [05:06:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027050559-de12e539 [05:06:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027050608-0a690dce [05:06:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027050610-27f562aa [05:06:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027050612-6ecb0234 [05:06:33] !log [universalomega@mwtask181] Starting deploy of {'world': True, 'versions': 'all'} to ['mw211', 'mw212', 'mw213'] [05:06:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027050633-e4bf9f59 [05:06:36] !log [universalomega@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw211 [05:06:37] PROBLEM - cp171 Haproxy TLS backend for mon181 on cp171 is CRITICAL: connect to address localhost and port 8201: Connection refused [05:06:44] PROBLEM - cp171 Haproxy TLS backend for reports171 on cp171 is CRITICAL: connect to address localhost and port 8205: Connection refused [05:06:45] An exception was raised while trying to log your message, MirahezeLSBot [05:06:46] PROBLEM - cp171 Haproxy TLS backend for mw163 on cp171 is CRITICAL: connect to address localhost and port 8123: Connection refused [05:06:46] PROBLEM - cp171 Haproxy TLS backend for swiftproxy171 on cp171 is CRITICAL: connect to address localhost and port 8207: Connection refused [05:06:49] PROBLEM - cp171 Haproxy balancer matomo on cp171 is CRITICAL: connect to address localhost and port 8153: Connection refused [05:06:49] RECOVERY - mw213 HTTPS on mw213 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.084 second response time [05:06:51] PROBLEM - cp171 Haproxy TLS backend for mwtask161 on cp171 is CRITICAL: connect to address localhost and port 8163: Connection refused [05:06:52] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to cp171.wikitide.net port 443 after 0 ms: Could not connect to server [05:06:55] PROBLEM - cp171 Haproxy TLS backend for mw192 on cp171 is CRITICAL: connect to address localhost and port 8130: Connection refused [05:06:58] PROBLEM - cp171 Haproxy TLS backend for test151 on cp171 is CRITICAL: connect to address localhost and port 8181: Connection refused [05:07:01] PROBLEM - cp171 Haproxy TLS backend for mw152 on cp171 is CRITICAL: connect to address localhost and port 8114: Connection refused [05:07:02] PROBLEM - cp171 Haproxy TLS backend for mw201 on cp171 is CRITICAL: connect to address localhost and port 8132: Connection refused [05:07:03] PROBLEM - cp201 Puppet on cp201 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[load-new-vcl-file] [05:07:03] PROBLEM - cp171 Haproxy TLS backend for mw172 on cp171 is CRITICAL: connect to address localhost and port 8118: Connection refused [05:07:03] PROBLEM - cp171 Haproxy TLS backend for mw182 on cp171 is CRITICAL: connect to address localhost and port 8120: Connection refused [05:07:07] PROBLEM - cp171 Haproxy TLS backend for matomo211 on cp171 is CRITICAL: connect to address localhost and port 8209: Connection refused [05:07:09] PROBLEM - cp171 Haproxy TLS backend for mw193 on cp171 is CRITICAL: connect to address localhost and port 8131: Connection refused [05:07:12] PROBLEM - cp171 Haproxy balancer thumb on cp171 is CRITICAL: connect to address localhost and port 8151: Connection refused [05:07:13] PROBLEM - cp171 Haproxy TLS backend for matomo151 on cp171 is CRITICAL: connect to address localhost and port 8203: Connection refused [05:07:16] RECOVERY - mw211 HTTPS on mw211 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.077 second response time [05:07:19] PROBLEM - cp171 Puppet on cp171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[load-new-vcl-file] [05:07:23] PROBLEM - cp171 Haproxy TLS backend for mw161 on cp171 is CRITICAL: connect to address localhost and port 8115: Connection refused [05:07:23] PROBLEM - cp171 Haproxy TLS backend for swiftproxy161 on cp171 is CRITICAL: connect to address localhost and port 8206: Connection refused [05:07:24] PROBLEM - cp171 Haproxy TLS backend for mw181 on cp171 is CRITICAL: connect to address localhost and port 8119: Connection refused [05:07:26] PROBLEM - cp171 Haproxy TLS backend for mw173 on cp171 is CRITICAL: connect to address localhost and port 8125: Connection refused [05:07:28] PROBLEM - cp171 Haproxy TLS backend for mw153 on cp171 is CRITICAL: connect to address localhost and port 8121: Connection refused [05:07:29] PROBLEM - cp171 Haproxy TLS backend for mw203 on cp171 is CRITICAL: connect to address localhost and port 8134: Connection refused [05:07:30] PROBLEM - cp171 Haproxy TLS backend for puppet181 on cp171 is CRITICAL: connect to address localhost and port 8204: Connection refused [05:07:30] PROBLEM - cp171 Haproxy TLS backend for mw162 on cp171 is CRITICAL: connect to address localhost and port 8116: Connection refused [05:07:30] PROBLEM - cp171 Haproxy TLS backend for mw183 on cp171 is CRITICAL: connect to address localhost and port 8127: Connection refused [05:07:32] PROBLEM - cp171 Haproxy TLS backend for mw171 on cp171 is CRITICAL: connect to address localhost and port 8117: Connection refused [05:07:33] !log [universalomega@mwtask181] Starting deploy of {'world': True, 'force': True, 'versions': 'all'} to ['mw211', 'mw212', 'mw213'] [05:07:34] PROBLEM - cp171 Haproxy TLS backend for mwtask151 on cp171 is CRITICAL: connect to address localhost and port 8162: Connection refused [05:07:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027050733-ca82a4e3 [05:07:43] !log [universalomega@mwtask181] finished deploy of {'world': True, 'force': True, 'versions': 'all'} to ['mw211', 'mw212', 'mw213'] - SUCCESS in 10s [05:07:45] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 4 backends are down. lb_mediawiki lb_thumb lb_swift lb_matomo [05:07:45] PROBLEM - cp171 Haproxy TLS backend for mw191 on cp171 is CRITICAL: connect to address localhost and port 8129: Connection refused [05:07:47] PROBLEM - cp171 Haproxy balancer swift on cp171 is CRITICAL: connect to address localhost and port 8152: Connection refused [05:07:52] RECOVERY - mw212 HTTPS on mw212 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.077 second response time [05:07:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027050743-2c6f5263 [05:08:02] PROBLEM - cp171 Haproxy TLS backend for phorge171 on cp171 is CRITICAL: connect to address localhost and port 8202: Connection refused [05:08:02] !log [universalomega@mwtask181] Starting deploy of {'pull': 'config', 'config': True} to ['mw211', 'mw212', 'mw213'] [05:08:04] PROBLEM - cp171 Haproxy TLS backend for mwtask171 on cp171 is CRITICAL: connect to address localhost and port 8161: Connection refused [05:08:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027050803-27f562aa [05:08:06] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to ['mw211', 'mw212', 'mw213'] - SUCCESS in 3s [05:08:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027050806-e5418fe5 [05:08:11] PROBLEM - cp171 Haproxy TLS backend for mw202 on cp171 is CRITICAL: connect to address localhost and port 8133: Connection refused [05:08:16] PROBLEM - cp171 Haproxy TLS backend for mw151 on cp171 is CRITICAL: connect to address localhost and port 8113: Connection refused [05:08:20] PROBLEM - mw212 MediaWiki Rendering on mw212 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 866 bytes in 0.096 second response time [05:08:20] PROBLEM - cp171 Haproxy TLS backend for mwtask181 on cp171 is CRITICAL: connect to address localhost and port 8160: Connection refused [05:08:22] PROBLEM - cp171 Haproxy TLS backend for speedscope211 on cp171 is CRITICAL: connect to address localhost and port 8208: Connection refused [05:08:27] PROBLEM - cp171 health.wikitide.net HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to health.wikitide.net port 443 after 0 ms: Could not connect to server [05:09:02] RECOVERY - cp201 Puppet on cp201 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:09:21] PROBLEM - mw213 MediaWiki Rendering on mw213 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 866 bytes in 0.094 second response time [05:09:44] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 9 backends are healthy [05:09:49] RECOVERY - cp171 Puppet on cp171 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [05:10:42] RECOVERY - cp171 Haproxy TLS backend for reports171 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8205 [05:10:43] RECOVERY - cp171 Haproxy balancer swift on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8152 [05:10:46] RECOVERY - cp171 Haproxy TLS backend for mw163 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8123 [05:10:46] RECOVERY - cp171 Haproxy TLS backend for mwtask161 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8163 [05:10:46] RECOVERY - cp171 Haproxy TLS backend for swiftproxy171 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8207 [05:10:51] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.073 second response time [05:10:52] RECOVERY - cp171 Haproxy balancer mediawiki on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8150 [05:10:55] RECOVERY - cp171 Haproxy TLS backend for mw192 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8130 [05:10:58] RECOVERY - cp171 Haproxy TLS backend for test151 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8181 [05:11:01] RECOVERY - cp171 Haproxy TLS backend for mw152 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8114 [05:11:02] RECOVERY - cp171 Haproxy TLS backend for mw201 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8132 [05:11:03] RECOVERY - cp171 Haproxy balancer thumb on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8151 [05:11:03] RECOVERY - cp171 Haproxy TLS backend for mw172 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8118 [05:11:03] RECOVERY - cp171 Haproxy TLS backend for mw182 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8120 [05:11:07] RECOVERY - cp171 Haproxy TLS backend for matomo211 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8209 [05:11:07] RECOVERY - cp171 Haproxy TLS backend for mwtask171 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8161 [05:11:08] RECOVERY - cp171 Haproxy TLS backend for matomo151 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8203 [05:11:09] RECOVERY - cp171 Haproxy TLS backend for mw193 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8131 [05:11:16] RECOVERY - cp171 Haproxy TLS backend for mw161 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8115 [05:11:19] PROBLEM - mw211 MediaWiki Rendering on mw211 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 866 bytes in 0.121 second response time [05:11:23] RECOVERY - cp171 Haproxy TLS backend for mw173 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8125 [05:11:23] RECOVERY - cp171 Haproxy TLS backend for swiftproxy161 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8206 [05:11:24] RECOVERY - cp171 Haproxy TLS backend for mw171 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8117 [05:11:24] RECOVERY - cp171 Haproxy TLS backend for mw181 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8119 [05:11:26] RECOVERY - cp171 Haproxy TLS backend for mw153 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8121 [05:11:28] RECOVERY - cp171 Haproxy TLS backend for puppet181 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8204 [05:11:29] RECOVERY - cp171 Haproxy TLS backend for mw203 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8134 [05:11:30] RECOVERY - cp171 Haproxy TLS backend for mw162 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8116 [05:11:30] RECOVERY - cp171 Haproxy TLS backend for mw183 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8127 [05:11:32] RECOVERY - cp171 Haproxy TLS backend for mwtask151 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8162 [05:11:40] RECOVERY - cp171 Haproxy TLS backend for mw191 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8129 [05:11:44] RECOVERY - cp171 health.wikitide.net HTTPS on cp171 is OK: HTTP OK: HTTP/2 200 - 112 bytes in 0.011 second response time [05:11:50] RECOVERY - cp171 Haproxy TLS backend for speedscope211 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8208 [05:11:56] RECOVERY - cp171 Haproxy TLS backend for mwtask181 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8160 [05:12:01] RECOVERY - cp171 Haproxy TLS backend for phorge171 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8202 [05:12:03] RECOVERY - cp171 Haproxy TLS backend for mw151 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8113 [05:12:06] RECOVERY - cp171 Haproxy TLS backend for mw202 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8133 [05:12:11] RECOVERY - cp171 Haproxy balancer matomo on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8153 [05:12:14] RECOVERY - cp171 Haproxy TLS backend for mon181 on cp171 is OK: TCP OK - 0.000 second response time on localhost port 8201 [05:13:20] !log [universalomega@mwtask181] Starting deploy of {'l10n': True, 'versions': 'all'} to ['mw211', 'mw212', 'mw213'] [05:13:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027051320-0d6fe6c4 [05:13:31] !log [universalomega@mwtask181] finished deploy of {'l10n': True, 'versions': 'all'} to ['mw211', 'mw212', 'mw213'] - SUCCESS in 10s [05:13:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027051331-7b96e8bf [05:15:39] !log [universalomega@mwtask181] Starting deploy of {'l10n': True, 'versions': 'all', 'ignore_time': True} to ['mw211', 'mw212', 'mw213'] [05:15:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027051539-eb2aa0ea [05:15:45] RECOVERY - mw211 MediaWiki Rendering on mw211 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.207 second response time [05:15:46] !log [universalomega@mwtask181] finished deploy of {'l10n': True, 'versions': 'all', 'ignore_time': True} to ['mw211', 'mw212', 'mw213'] - SUCCESS in 6s [05:15:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027051546-76b134c5 [05:15:54] RECOVERY - mw212 MediaWiki Rendering on mw212 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.232 second response time [05:15:58] RECOVERY - mw213 MediaWiki Rendering on mw213 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.367 second response time [06:03:42] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 25.75, 15.08, 7.98 [06:06:21] [02mw-config] 07Aeywoo opened pull request #6611: Updating footer icons for DOORS Wikis (07miraheze:03main...07Aeywoo:03patch-2) 13https://github.com/miraheze/mw-config/pull/6611 [06:07:36] miraheze/mw-config - Aeywoo the build passed. [06:07:42] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 4.22, 9.43, 7.32 [06:12:38] [02WikiTideDebug] 07Universal-Omega created 03cloud21-mw (+1 new commit) 13https://github.com/miraheze/WikiTideDebug/commit/369da52a85d0 [06:12:38] 02WikiTideDebug/03cloud21-mw 07CosmicAlpha 03369da52 Add mw211, mw212, and mw213 [06:12:44] [02WikiTideDebug] 07Universal-Omega opened pull request #39: Add mw211, mw212, and mw213 (03main...03cloud21-mw) 13https://github.com/miraheze/WikiTideDebug/pull/39 [06:15:52] [02mw-config] 07Universal-Omega merged 07Aeywoo's pull request #6611: Updating footer icons for DOORS Wikis (07miraheze:03main...07Aeywoo:03patch-2) 13https://github.com/miraheze/mw-config/pull/6611 [06:15:52] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/afcdfed80afeec7f37e1c24e1725f1ccc09e290e [06:15:52] 02mw-config/03main 07Aeywoo 03afcdfed Updating footer icons for DOORS Wikis (#6611)… [06:16:05] !log [universalomega@mwtask181] Starting deploy of {'pull': 'config', 'config': True} to all [06:16:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027061605-4d1b4212 [06:16:33] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 27s [06:16:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027061633-6464652e [06:16:36] miraheze/mw-config - Universal-Omega the build has errored. [06:44:59] [02mw-config] 07Aeywoo opened pull request #6612: Bumping phpunit to 13.4.5 (07miraheze:03main...07Aeywoo:03patch-2) 13https://github.com/miraheze/mw-config/pull/6612 [06:45:50] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/10efc4ceaa4f74a39ab7f0acbd6ff30e0a13d61a [06:45:50] 02mw-config/03main 07Aeywoo 0310efc4c Bumping phpunit to 13.4.5 (#6612)… [06:45:51] [02mw-config] 07Universal-Omega merged 07Aeywoo's pull request #6612: Bumping phpunit to 13.4.5 (07miraheze:03main...07Aeywoo:03patch-2) 13https://github.com/miraheze/mw-config/pull/6612 [06:46:09] miraheze/mw-config - Aeywoo the build passed. [06:46:50] miraheze/mw-config - Universal-Omega the build passed. [11:02:33] [02TSPortal] 07renovate[bot] merged 03renovate/dev-deps into 03main 13https://github.com/miraheze/TSPortal/compare/d168720f9a79...794a3e6f1541 [11:02:35] [02TSPortal] 07renovate[bot] 04deleted 03renovate/dev-deps at 03794a3e6 13https://api.github.com/repos/miraheze/TSPortal/commit/794a3e6 [11:03:01] [02TSPortal] 07renovate[bot] 04force-pushed 03renovate/prod-deps from 03af79ada to 0389257f7 13https://api.github.com/repos/miraheze/TSPortal/commits/renovate/prod-deps [11:03:01] 02TSPortal/03renovate/prod-deps 07renovate[bot] 0389257f7 Update prod dependencies [11:28:34] PROBLEM - matomo211 Current Load on matomo211 is CRITICAL: LOAD CRITICAL - total load average: 18.25, 14.65, 11.69 [11:29:23] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.66, 14.98, 11.95 [11:30:32] PROBLEM - matomo211 Current Load on matomo211 is WARNING: LOAD WARNING - total load average: 15.77, 14.67, 12.05 [11:31:23] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 13.90, 14.60, 12.17 [11:32:31] RECOVERY - matomo211 Current Load on matomo211 is OK: LOAD OK - total load average: 9.31, 12.96, 11.76 [11:33:23] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 8.11, 12.26, 11.61 [11:47:47] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 164.48, 81.37, 34.60 [11:48:17] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [11:48:23] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [11:48:28] I will depool db 161 [11:48:32] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [11:48:58] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [11:49:19] !log [petramagna@mwtask181] Starting deploy of {'config': True, 'force': True} to all [11:49:41] !log [petramagna@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 22s [11:49:57] An exception was raised while trying to log your message, MirahezeLSBot [11:50:15] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1485 bytes in 0.230 second response time [11:50:25] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1485 bytes in 8.556 second response time [11:50:30] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 1.490 second response time [11:50:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027114956-01ef2076 [11:50:54] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.095 second response time [11:51:02] @Infrastructure Specialists db161 unresponsive for the past 5 minutes [11:52:07] It's back but still under high load [11:52:28] Will redeploy if load goes down [11:53:13] Repooling [11:53:21] !log [petramagna@mwtask181] Starting deploy of {'config': True, 'force': True} to all [11:53:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027115321-c291307c [11:53:42] !log [petramagna@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 21s [11:53:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027115342-0df86a34 [12:15:42] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 3.08, 3.85, 11.22 [12:19:42] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 3.23, 3.49, 9.38 [12:53:23] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 14.06, 13.03, 10.79 [12:55:23] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.77, 13.95, 11.36 [12:57:40] PROBLEM - matomo211 Current Load on matomo211 is CRITICAL: LOAD CRITICAL - total load average: 18.61, 15.80, 12.63 [13:17:23] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 11.20, 15.22, 15.55 [13:17:40] PROBLEM - matomo211 Current Load on matomo211 is WARNING: LOAD WARNING - total load average: 10.87, 15.35, 15.71 [13:27:23] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 8.13, 10.32, 12.89 [13:27:40] RECOVERY - matomo211 Current Load on matomo211 is OK: LOAD OK - total load average: 6.96, 10.39, 13.16 [13:41:23] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.61, 13.08, 11.97 [13:41:40] PROBLEM - matomo211 Current Load on matomo211 is CRITICAL: LOAD CRITICAL - total load average: 17.16, 13.85, 12.54 [13:43:23] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 14.38, 13.18, 12.14 [13:43:40] PROBLEM - matomo211 Current Load on matomo211 is WARNING: LOAD WARNING - total load average: 15.19, 13.87, 12.69 [13:45:23] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 8.07, 11.39, 11.62 [13:45:40] RECOVERY - matomo211 Current Load on matomo211 is OK: LOAD OK - total load average: 8.91, 11.79, 12.06 [13:49:23] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 18.03, 14.45, 12.76 [13:49:41] PROBLEM - matomo211 Current Load on matomo211 is CRITICAL: LOAD CRITICAL - total load average: 17.45, 14.67, 13.14 [14:12:28] [02mw-config] 07dependabot[bot] created 03dependabot/composer/justinrainbow/json-schema-6.14.0 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/dab4f4b4a6b2 [14:12:28] 02mw-config/03dependabot/composer/justinrainbow/json-schema-6.14.0 07dependabot[bot] 03dab4f4b Update justinrainbow/json-schema requirement from 6.13.1 to 6.14.0… [14:12:31] [02mw-config] 07dependabot[bot] opened pull request #6613: Update justinrainbow/json-schema requirement from 6.13.1 to 6.14.0 (03main...03dependabot/composer/justinrainbow/json-schema-6.14.0) 13https://github.com/miraheze/mw-config/pull/6613 [14:12:32] [02mw-config] 07dependabot[bot] added the label 'dependencies' to pull request #6613 (Update justinrainbow/json-schema requirement from 6.13.1 to 6.14.0) 13https://github.com/miraheze/mw-config/pull/6613 [14:12:34] [02mw-config] 07dependabot[bot] added the label 'php' to pull request #6613 (Update justinrainbow/json-schema requirement from 6.13.1 to 6.14.0) 13https://github.com/miraheze/mw-config/pull/6613 [14:13:31] miraheze/mw-config - dependabot[bot] the build passed. [15:12:44] [02TSPortal] 07renovate[bot] 04force-pushed 03renovate/prod-deps from 0389257f7 to 03124802e 13https://api.github.com/repos/miraheze/TSPortal/commits/renovate/prod-deps [15:12:44] 02TSPortal/03renovate/prod-deps 07renovate[bot] 03124802e Update prod dependencies [15:29:40] PROBLEM - matomo211 Current Load on matomo211 is WARNING: LOAD WARNING - total load average: 6.55, 12.21, 15.93 [15:31:23] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.06, 10.66, 14.95 [15:37:40] RECOVERY - matomo211 Current Load on matomo211 is OK: LOAD OK - total load average: 9.92, 10.88, 13.49 [15:39:23] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 7.69, 10.14, 13.02 [15:49:23] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.98, 12.76, 12.42 [15:49:40] PROBLEM - matomo211 Current Load on matomo211 is CRITICAL: LOAD CRITICAL - total load average: 17.12, 13.06, 12.38 [16:05:51] [02puppet] 07Universal-Omega created 03lb-ctl-jobrunner (+1 new commit) 13https://github.com/miraheze/puppet/commit/f94bf1ef2781 [16:05:51] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 03f94bf1e haproxy: expand lb-ctl to jobrunner also [16:05:59] [02puppet] 07Universal-Omega opened pull request #5125: haproxy: expand lb-ctl to jobrunner also (03main...03lb-ctl-jobrunner) 13https://github.com/miraheze/puppet/pull/5125 [16:07:44] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/a6a6a765f5d5b8017b172775583e4ceffc1ea995 [16:07:44] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 03a6a6a76 Update [16:08:28] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/0de2e953483c572279371a1c7dc0098c3bcfbd3d [16:08:28] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 030de2e95 + [16:09:04] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/9c81b8fce2e541e63e558b035bcca56e6d271143 [16:09:04] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 039c81b8f + [16:09:23] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 11.18, 15.14, 15.64 [16:09:40] PROBLEM - matomo211 Current Load on matomo211 is WARNING: LOAD WARNING - total load average: 12.02, 14.58, 15.22 [16:09:58] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/c67edbbb8c1d81eeaa9d6d384b81b9875383da8b [16:09:58] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 03c67edbb + [16:10:46] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/662204b1bf92a9da7abbe2b4b40628b281894afe [16:10:46] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 03662204b + [16:11:23] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.46, 15.72, 15.78 [16:11:34] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/1b1b3b122383b6d2facc3506b0718ecaeb10923e [16:11:34] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 031b1b3b1 + [16:11:40] PROBLEM - matomo211 Current Load on matomo211 is CRITICAL: LOAD CRITICAL - total load average: 16.63, 15.34, 15.41 [16:11:45] miraheze/puppet - Universal-Omega the build passed. [16:12:26] miraheze/puppet - Universal-Omega the build passed. [16:12:33] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/fa8c38b0410c666405b4dafc5117a2d3eda2443f [16:12:33] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 03fa8c38b + [16:13:16] miraheze/puppet - Universal-Omega the build passed. [16:13:22] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/ac4933c5fc99d1152ea03fd97fe7d6ac24e6b9c1 [16:13:22] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 03ac4933c + [16:13:23] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 11.58, 14.72, 15.45 [16:13:40] PROBLEM - matomo211 Current Load on matomo211 is WARNING: LOAD WARNING - total load average: 11.86, 14.23, 15.02 [16:14:10] miraheze/puppet - Universal-Omega the build passed. [16:14:21] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/ac412e17a05799bbe4b7934f6ef174eef2682c0e [16:14:21] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 03ac412e1 + [16:15:13] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/a9c30d3b6c2b0e42f2f774e839e16d2473bbea67 [16:15:13] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 03a9c30d3 + [16:15:37] miraheze/puppet - Universal-Omega the build passed. [16:16:08] miraheze/puppet - Universal-Omega the build passed. [16:17:24] PROBLEM - ping on mw211 is CRITICAL: CRITICAL - Host Unreachable (10.0.21.175) [16:18:05] PROBLEM - Host mw211 is DOWN: CRITICAL - Host Unreachable (10.0.21.175) [16:18:18] PROBLEM - cp161 HAProxy Backends on cp161 is CRITICAL: CRITICAL: 1 backend down: mw211 (mediawiki) [16:18:32] miraheze/puppet - Universal-Omega the build passed. [16:19:01] PROBLEM - cp171 HAProxy Backends on cp171 is CRITICAL: CRITICAL: 1 backend down: mw211 (mediawiki) [16:19:04] PROBLEM - cp191 HAProxy Backends on cp191 is CRITICAL: CRITICAL: 1 backend down: mw211 (mediawiki) [16:19:23] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 6.56, 9.86, 13.14 [16:19:26] PROBLEM - cp201 HAProxy Backends on cp201 is CRITICAL: CRITICAL: 1 backend down: mw211 (mediawiki) [16:19:40] RECOVERY - matomo211 Current Load on matomo211 is OK: LOAD OK - total load average: 6.17, 9.60, 12.79 [16:41:05] PROBLEM - mw211.fsslc.wtnet SSL Check on sslhost is CRITICAL: connect to address mw211.fsslc.wtnet and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [17:17:42] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 32.55, 26.43, 13.74 [17:23:42] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 4.81, 11.11, 10.76 [17:25:42] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 3.04, 8.42, 9.82 [19:26:00] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/74e5adf34aae0e930046cd933434e2ab05d3b86f [19:26:00] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 0374e5adf Update [19:26:32] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/9d488e4c7a0612612414c7683fa938a0c2b921ae [19:26:32] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 039d488e4 Update [19:27:20] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/34fa1f10aa7bcbd7db7b00b2a831751521336471 [19:27:20] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 0334fa1f1 - [19:28:22] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/2181d96f67a9d36570b0e6040e94487f8f8a109b [19:28:23] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 032181d96 - [19:28:53] miraheze/puppet - Universal-Omega the build passed. [19:29:01] miraheze/puppet - Universal-Omega the build passed. [19:29:02] miraheze/puppet - Universal-Omega the build passed. [19:30:06] miraheze/puppet - Universal-Omega the build passed. [19:30:28] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/4a9b189fe5cce65e508deb5c4690c43588775008 [19:30:28] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 034a9b189 Update [19:32:24] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/0cd31aa2f45001b13c7e81aadca502ef61c9c21d [19:32:24] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 030cd31aa Update [19:32:28] miraheze/puppet - Universal-Omega the build passed. [19:33:05] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/41a4d2dc587a6fae7506a26127a81a596e0768a8 [19:33:05] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 0341a4d2d Update [19:33:53] miraheze/puppet - Universal-Omega the build passed. [19:34:09] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/35a24294410f55815c20cc3b754b2f8f9523a93a [19:34:09] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 0335a2429 Update [19:35:40] miraheze/puppet - Universal-Omega the build passed. [19:36:05] miraheze/puppet - Universal-Omega the build passed. [19:38:04] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/f5b96bd6a30829eb2ee6b0681d7aef805b8948da [19:38:04] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 03f5b96bd Update [19:39:07] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/a664a8b09dc9b2521794d9a1918a3f022aef7ffd [19:39:07] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 03a664a8b Update comment [19:39:59] miraheze/puppet - Universal-Omega the build passed. [19:40:36] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/5e384384742424b2edb424a5036d8ef09b5ec265 [19:40:36] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 035e38438 Update [19:40:50] miraheze/puppet - Universal-Omega the build passed. [19:42:20] miraheze/puppet - Universal-Omega the build passed. [19:42:30] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/39368185c2ca38054a72fb999dd195aa2622ac24 [19:42:30] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 033936818 Update [19:43:35] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/1108c83f3013b41f64fab84f8f30bff60fb293af [19:43:35] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 031108c83 Update [19:44:10] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/b04d2f5de049fdf82372a75187f8de2d55ae20d6 [19:44:10] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 03b04d2f5 Update [19:44:55] miraheze/puppet - Universal-Omega the build passed. [19:45:33] miraheze/puppet - Universal-Omega the build passed. [19:46:40] miraheze/puppet - Universal-Omega the build passed. [19:52:14] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/7f420286a93fb83c963d400e99059629c56017c2 [19:52:14] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 037f42028 Fix [19:53:55] miraheze/puppet - Universal-Omega the build passed. [19:54:39] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/08b7ab75febe8b6ceaed861daeb5cb31aa3efd58 [19:54:39] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 0308b7ab7 Update [19:55:54] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/9f518b68c2aad2c47496d9bca23322027ac5686a [19:55:54] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 039f518b6 Update [19:58:30] miraheze/puppet - Universal-Omega the build passed. [19:58:36] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/f840e7523f222598cb614c40c1667578b0e87830 [19:58:36] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 03f840e75 Update [19:58:40] PROBLEM - graylog161 APT on graylog161 is CRITICAL: APT CRITICAL: 9 packages available for upgrade (4 critical updates). [19:59:00] PROBLEM - os162 APT on os162 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (4 critical updates). [19:59:07] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/6ecc2ae0f6f327a174aa207da32bed995f125f17 [19:59:07] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 036ecc2ae Update [19:59:41] PROBLEM - os202 APT on os202 is CRITICAL: APT CRITICAL: 7 packages available for upgrade (4 critical updates). [20:00:41] miraheze/puppet - Universal-Omega the build passed. [20:03:03] PROBLEM - os161 APT on os161 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (4 critical updates). [20:03:16] PROBLEM - os191 APT on os191 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (4 critical updates). [20:05:22] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/dd318cdc723df9b412ac845f97c9626a051029c6 [20:05:22] 02puppet/03main 07CosmicAlpha 03dd318cd varnish: delete unused nginx template [20:05:39] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/5c7ad5f4151b3454274e6c5c4742a301261cbff2 [20:05:39] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 035c7ad5f Merge branch 'main' into lb-ctl-jobrunner [20:10:12] PROBLEM - mwtask171 Puppet on mwtask171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 30 minutes ago with 0 failures [20:11:20] PROBLEM - changeprop202 Puppet on changeprop202 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 4 minutes ago with 0 failures [20:11:35] PROBLEM - mwtask161 Puppet on mwtask161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 30 minutes ago with 0 failures [20:12:02] PROBLEM - mwtask151 Puppet on mwtask151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 30 minutes ago with 0 failures [20:12:50] PROBLEM - cp171 Puppet on cp171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:13:26] PROBLEM - puppet181 APT on puppet181 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (4 critical updates). [20:13:35] RECOVERY - mwtask161 Puppet on mwtask161 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [20:14:02] RECOVERY - mwtask151 Puppet on mwtask151 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [20:14:10] RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [20:14:39] PROBLEM - cp191 Puppet on cp191 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 20 minutes ago with 0 failures [20:15:12] PROBLEM - cp161 Puppet on cp161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 20 minutes ago with 0 failures [20:15:30] PROBLEM - cp201 Puppet on cp201 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 7 minutes ago with 0 failures [20:16:37] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/d0ddec1da405508c2e1787e175a95cbd98374991 [20:16:37] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 03d0ddec1 Fix [20:17:26] PROBLEM - os201 APT on os201 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (4 critical updates). [20:17:42] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 45.91, 20.26, 11.83 [20:18:15] PROBLEM - os151 APT on os151 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (4 critical updates). [20:18:23] miraheze/puppet - Universal-Omega the build passed. [20:19:00] PROBLEM - cp171 HAProxy Backends on cp171 is UNKNOWN: NRPE: Command 'check_HAProxy Backends' not defined [20:19:20] RECOVERY - cp171 Puppet on cp171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:19:30] RECOVERY - cp201 Puppet on cp201 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:20:18] PROBLEM - cp161 HAProxy Backends on cp161 is UNKNOWN: NRPE: Command 'check_HAProxy Backends' not defined [20:20:38] RECOVERY - cp191 Puppet on cp191 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [20:21:04] PROBLEM - cp191 HAProxy Backends on cp191 is UNKNOWN: NRPE: Command 'check_HAProxy Backends' not defined [20:21:13] RECOVERY - cp161 Puppet on cp161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:21:26] PROBLEM - cp201 HAProxy Backends on cp201 is UNKNOWN: NRPE: Command 'check_HAProxy Backends' not defined [20:23:07] PROBLEM - kafka182 APT on kafka182 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (2 critical updates). [20:25:42] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 5.33, 11.08, 11.70 [20:31:42] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 4.67, 6.99, 9.68 [20:33:28] [02puppet] 07Universal-Omega pushed 1 new commit to 03lb-ctl-jobrunner 13https://github.com/miraheze/puppet/commit/e301711edb4f08a604163bb57eb982e15abfa9fc [20:33:28] 02puppet/03lb-ctl-jobrunner 07CosmicAlpha 03e301711 - [20:34:03] RECOVERY - Host mw211 is UP: PING OK - Packet loss = 0%, RTA = 0.42 ms [20:34:23] PROBLEM - mw211 HTTPS on mw211 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw211.fsslc.wtnet port 443 after 0 ms: Could not connect to server [20:34:33] PROBLEM - mw211 Puppet on mw211 is WARNING: WARNING: Puppet last ran 4 hours ago [20:34:42] RECOVERY - ping on mw211 is OK: PING OK - Packet loss = 0%, RTA = 0.34 ms [20:35:00] miraheze/puppet - Universal-Omega the build passed. [20:35:03] PROBLEM - cp191 HAProxy Servers on cp191 is CRITICAL: CRITICAL: 1 server DOWN: mw211 (mediawiki) [20:35:20] RECOVERY - changeprop202 Puppet on changeprop202 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:35:22] PROBLEM - cp161 HAProxy Servers on cp161 is CRITICAL: CRITICAL: 2 servers DOWN: mw212 (mediawiki), mw213 (mediawiki) [20:35:25] PROBLEM - cp171 HAProxy Servers on cp171 is CRITICAL: CRITICAL: 2 servers DOWN: mw212 (mediawiki), mw213 (mediawiki) [20:35:25] PROBLEM - cp201 HAProxy Servers on cp201 is CRITICAL: CRITICAL: 2 servers DOWN: mw212 (mediawiki), mw213 (mediawiki) [20:36:21] RECOVERY - mw211 HTTPS on mw211 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.117 second response time [20:36:31] RECOVERY - mw211 Puppet on mw211 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:37:09] PROBLEM - mw213 MediaWiki Rendering on mw213 is CRITICAL: connect to address 10.0.21.177 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:38:13] PROBLEM - mw212 HTTPS on mw212 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw212.fsslc.wtnet port 443 after 0 ms: Could not connect to server [20:38:27] PROBLEM - mw212 MediaWiki Rendering on mw212 is CRITICAL: connect to address 10.0.21.176 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:38:31] PROBLEM - mw213 HTTPS on mw213 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw213.fsslc.wtnet port 443 after 0 ms: Could not connect to server [20:39:01] RECOVERY - cp191 HAProxy Servers on cp191 is OK: OK: all 29 servers are up [20:39:10] RECOVERY - mw213 MediaWiki Rendering on mw213 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.817 second response time [20:39:14] RECOVERY - cp201 HAProxy Servers on cp201 is OK: OK: all 29 servers are up [20:39:14] RECOVERY - cp161 HAProxy Servers on cp161 is OK: OK: all 29 servers are up [20:39:14] RECOVERY - cp171 HAProxy Servers on cp171 is OK: OK: all 29 servers are up [20:39:58] RECOVERY - mw213 HTTPS on mw213 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.064 second response time [20:40:03] RECOVERY - mw212 MediaWiki Rendering on mw212 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.254 second response time [20:40:08] RECOVERY - mw211.fsslc.wtnet SSL Check on sslhost is OK: OK - Certificate 'CloudFlare Origin Certificate' will expire on Tue 03 Apr 2040 12:14:00 PM GMT +0000. [20:40:08] RECOVERY - mw212 HTTPS on mw212 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 1.404 second response time [20:41:27] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/a4cecbc8c320cb72a53bc24465ee79b53459e818 [20:41:27] 02puppet/03main 07CosmicAlpha 03a4cecbc haproxy: expand lb-ctl to jobrunner also [20:41:29] [02puppet] 07Universal-Omega merged pull request #5125: haproxy: expand lb-ctl to jobrunner also (03main...03lb-ctl-jobrunner) 13https://github.com/miraheze/puppet/pull/5125 [20:41:31] [02puppet] 07Universal-Omega 04deleted 03lb-ctl-jobrunner at 03e301711 13https://api.github.com/repos/miraheze/puppet/commit/e301711 [20:43:12] miraheze/puppet - Universal-Omega the build passed. [20:51:24] [02mw-config] 07Universal-Omega created 03remove-xhprof (+1 new commit) 13https://github.com/miraheze/mw-config/commit/3eb4f1c06c6d [20:51:24] 02mw-config/03remove-xhprof 07CosmicAlpha 033eb4f1c Remove support for xhprof… [20:51:38] [02mw-config] 07Universal-Omega opened pull request #6614: Remove support for xhprof (03main...03remove-xhprof) 13https://github.com/miraheze/mw-config/pull/6614 [20:52:52] miraheze/mw-config - Universal-Omega the build passed. [20:53:04] [02mw-config] 07Universal-Omega merged pull request #6614: Remove support for xhprof (03main...03remove-xhprof) 13https://github.com/miraheze/mw-config/pull/6614 [20:53:06] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/5eccd6b7182bb2f74287d8ac0df971ef9a628b38 [20:53:07] 02mw-config/03main 07CosmicAlpha 035eccd6b Remove support for xhprof… [20:53:09] !log [universalomega@test151] Starting deploy of {'pull': 'config', 'config': True} to test151 [20:53:09] [02mw-config] 07Universal-Omega 04deleted 03remove-xhprof at 033eb4f1c 13https://api.github.com/repos/miraheze/mw-config/commit/3eb4f1c [20:53:10] !log [universalomega@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 1s [20:53:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027205309-ab9b245d [20:53:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027205311-b90d12cf [20:53:17] !log [universalomega@mwtask181] Starting deploy of {'pull': 'config', 'config': True} to all [20:53:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027205317-4d1b4212 [20:53:47] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 30s [20:53:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027205347-9c4b905e [20:54:14] miraheze/mw-config - Universal-Omega the build passed. [20:55:51] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/0c1d809a82fa87e4b9295d2579d96a13fcbb6ba7 [20:55:51] 02mw-config/03main 07dependabot[bot] 030c1d809 Update justinrainbow/json-schema requirement from 6.13.1 to 6.14.0 (#6613)… [20:55:51] [02mw-config] 07Universal-Omega merged 07dependabot[bot]'s pull request #6613: Update justinrainbow/json-schema requirement from 6.13.1 to 6.14.0 (03main...03dependabot/composer/justinrainbow/json-schema-6.14.0) 13https://github.com/miraheze/mw-config/pull/6613 [20:55:52] [02mw-config] 07Universal-Omega 04deleted 03dependabot/composer/justinrainbow/json-schema-6.14.0 at 03dab4f4b 13https://api.github.com/repos/miraheze/mw-config/commit/dab4f4b [21:04:52] !log [universalomega@mwtask181] Starting deploy of {'l10n': True, 'versions': 'all', 'ignore_time': True} to all [21:04:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027210452-104889a9 [21:05:58] !log [universalomega@mwtask181] finished deploy of {'l10n': True, 'versions': 'all', 'ignore_time': True} to all - SUCCESS in 65s [21:06:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027210558-a382bfa1 [21:13:22] !log [universalomega@mwtask181] Starting deploy of {'l10n': True, 'versions': 'all', 'ignore_time': True} to all [21:13:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027211322-104889a9 [21:15:38] !log [universalomega@mwtask181] finished deploy of {'l10n': True, 'versions': 'all', 'ignore_time': True} to all - SUCCESS in 135s [21:15:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027211538-c2920851 [21:55:28] [02mw-config] 07Universal-Omega created 03db-conn (+1 new commit) 13https://github.com/miraheze/mw-config/commit/909a314ed435 [21:55:28] 02mw-config/03db-conn 07CosmicAlpha 03909a314 Test some improvements to database connection [21:55:34] [02mw-config] 07Universal-Omega opened pull request #6615: Test some improvements to database connection (03main...03db-conn) 13https://github.com/miraheze/mw-config/pull/6615 [21:56:42] miraheze/mw-config - Universal-Omega the build passed. [21:57:59] [02mw-config] 07Universal-Omega pushed 1 new commit to 03db-conn 13https://github.com/miraheze/mw-config/commit/a37f06038b7ccf606773908f33fd99afbf17e618 [21:58:00] 02mw-config/03db-conn 07CosmicAlpha 03a37f060 + [21:59:10] miraheze/mw-config - Universal-Omega the build passed. [22:00:11] [02python-functions] 07renovate[bot] created 03renovate/all-deps (+1 new commit) 13https://github.com/miraheze/python-functions/commit/2911c1c8a2f4 [22:00:11] 02python-functions/03renovate/all-deps 07renovate[bot] 032911c1c chore(deps): update dependency pytest to v9.2.0 [22:00:35] [02mw-config] 07lihaohong6 left a file comment in pull request #6461 03ad484f8: Maybe be more specific about weird bugs. They mainly stem from global variables leaking between invocations. 13https://github.com/miraheze/mw-config/pull/6461#discussion_r4394683017 [22:25:38] PROBLEM - db161 APT on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:26:45] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 30.60, 15.57, 8.74 [22:29:01] PROBLEM - db161 Puppet on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:43:02] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 29 minutes ago with 0 failures [22:49:18] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [22:49:40] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [22:50:02] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [22:50:02] PROBLEM - db161 Puppet on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:51:14] PROBLEM - db161 PowerDNS Recursor on db161 is CRITICAL: CRITICAL - Plugin timed out while executing system call [22:52:31] !log [universalomega@mwtask181] Starting deploy of {'config': True, 'force': True} to all [22:52:50] !log [universalomega@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 19s [22:53:08] An exception was raised while trying to log your message, MirahezeLSBot [22:53:46] An exception was raised while trying to log your message, MirahezeLSBot [22:54:36] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 4.320 second response time [22:54:43] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 2.737 second response time [22:54:49] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.398 second response time [22:55:04] RECOVERY - db161 PowerDNS Recursor on db161 is OK: DNS OK: 7.692 seconds response time. db161.fsslc.wtnet returns 10.0.16.128 [23:00:59] !log [universalomega@mwtask181] Starting deploy of {'config': True, 'force': True} to all [23:01:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027230100-d97a3a2b [23:01:22] !log [universalomega@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 22s [23:01:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027230122-eec26513 [23:02:26] !log [universalomega@mwtask181] Starting deploy of {'config': True, 'force': True} to all [23:02:46] !log [universalomega@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 20s [23:03:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027230226-d97a3a2b [23:03:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027230333-a2ffa6bd [23:05:39] !log stop and start db161 from host [23:05:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027230539-2c165129 [23:05:53] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 0.39, 0.10, 0.03 [23:06:15] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 52 minutes ago with 0 failures [23:06:18] RECOVERY - db161 APT on db161 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [23:06:44] !log [universalomega@mwtask181] Starting deploy of {'config': True, 'force': True} to all [23:06:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027230644-d97a3a2b [23:07:07] !log [universalomega@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 23s [23:07:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261027230707-5dc1a8bb [23:54:04] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 39.09, 17.86, 9.72 [23:59:18] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [23:59:24] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [23:59:51] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received