[00:18:00] PROBLEM - Host swiftobject191 is DOWN: check_ping: Invalid hostname/address - Usage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4|-6] [00:18:34] PROBLEM - cloud20 IPMI Sensors on cloud20 is CRITICAL: IPMI Status: Critical [5 system event log (SEL) entries present] [00:20:14] [02puppet] 07paladox created 03paladox-patch-5 (+1 new commit) 13https://github.com/miraheze/puppet/commit/8d837fd504fc [00:20:14] 02puppet/03paladox-patch-5 07paladox 038d837fd ircecho: Fix print statement… [00:20:17] [02puppet] 07paladox opened pull request #4209: ircecho: Fix print statement (03master...03paladox-patch-5) 13https://github.com/miraheze/puppet/pull/4209 [00:20:24] [02puppet] 07coderabbitai[bot] commented on pull request #4209: --- […] 13https://github.com/miraheze/puppet/pull/4209#issuecomment-2686499452 [00:21:27] miraheze/puppet - paladox the build passed. [00:23:43] [02puppet] 07paladox created 03paladox-patch-6 (+1 new commit) 13https://github.com/miraheze/puppet/commit/af25c5c3feba [00:23:43] 02puppet/03paladox-patch-6 07paladox 03af25c5c Install swiftobject191 [00:23:46] [02puppet] 07paladox opened pull request #4210: Install swiftobject191 (03master...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4210 [00:23:53] [02puppet] 07coderabbitai[bot] commented on pull request #4210: --- […] 13https://github.com/miraheze/puppet/pull/4210#issuecomment-2686503727 [00:24:13] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-6 13https://github.com/miraheze/puppet/commit/f119032c49fe67b36c5eedd5d0c78dadd8dfc1d8 [00:24:13] 02puppet/03paladox-patch-6 07paladox 03f119032 Create swiftobject191.yaml [00:24:24] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/1cff5de1653fd87046415c08f4b71377d324d0cf [00:24:24] 02puppet/03master 07paladox 031cff5de Install swiftobject191 (#4210) [00:24:26] [02puppet] 07paladox merged pull request #4210: Install swiftobject191 (03master...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4210 [00:24:28] [02puppet] 07paladox 04deleted 03paladox-patch-6 at 03f119032 13https://github.com/miraheze/puppet/commit/f119032 [00:27:11] RECOVERY - cloud20 IPMI Sensors on cloud20 is OK: IPMI Status: OK [00:27:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:27:32] !log [void@cloud20.wikitide.net] clear ipmi sel (contained some resolved inlet temperature alerts) [00:27:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:48:07] RECOVERY - Host swiftobject191 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [00:48:36] RECOVERY - swiftobject191 SSH on swiftobject191 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u5 (protocol 2.0) [00:49:11] RECOVERY - swiftobject191 ferm_active on swiftobject191 is OK: OK ferm input default policy is set [00:56:18] [02puppet] 07paladox created 03paladox-patch-6 (+1 new commit) 13https://github.com/miraheze/puppet/commit/dbe5eb4f1eba [00:56:18] 02puppet/03paladox-patch-6 07paladox 03dbe5eb4 Install swiftobject121 [00:56:21] [02puppet] 07paladox opened pull request #4211: Install swiftobject121 (03master...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4211 [00:56:27] [02puppet] 07coderabbitai[bot] commented on pull request #4211: --- […] 13https://github.com/miraheze/puppet/pull/4211#issuecomment-2686541314 [00:56:29] [02puppet] 07paladox retitled PR #4211: "Install swiftobject121" ➜ "Install swiftobject201" 13https://github.com/miraheze/puppet/pull/4211 [00:58:06] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-6 13https://github.com/miraheze/puppet/commit/1edb4f4b2370a8f28037060d5bfbb7665892fb53 [00:58:06] 02puppet/03paladox-patch-6 07paladox 031edb4f4 Update site.pp [00:58:34] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-6 13https://github.com/miraheze/puppet/commit/382709ff62fdc737507078298318202e39cf75af [00:58:34] 02puppet/03paladox-patch-6 07paladox 03382709f Create swiftobject201.yaml [00:59:44] PROBLEM - swiftobject201 SSH on swiftobject201 is UNKNOWN: Usage:check_ssh [-4|-6] [-t ] [-r ] [-p ] [00:59:50] PROBLEM - swiftobject201 Current Load on swiftobject201 is WARNING: Could not resolve hostname : Name or service not known [00:59:53] PROBLEM - swiftobject201 APT on swiftobject201 is WARNING: Could not resolve hostname : Name or service not known [01:00:04] PROBLEM - Host swiftobject201 is DOWN: check_ping: Invalid hostname/address - Usage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4|-6] [01:02:09] [02puppet] 07paladox merged pull request #4211: Install swiftobject201 (03master...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4211 [01:02:09] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/fb136a0020cf3b77b0abb5349124d577b3e2afec [01:02:09] 02puppet/03master 07paladox 03fb136a0 Install swiftobject201 (#4211) [01:02:11] [02puppet] 07paladox 04deleted 03paladox-patch-6 at 03382709f 13https://github.com/miraheze/puppet/commit/382709f [01:04:04] !log [void@cloud15] clear ipmi sel (resolved inlet temperature alerts) [01:04:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:05:06] RECOVERY - cloud15 IPMI Sensors on cloud15 is OK: IPMI Status: OK [01:06:05] RECOVERY - Host swiftobject201 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [01:06:21] PROBLEM - swiftobject201 Swift Object Service on swiftobject201 is CRITICAL: connect to address 10.0.20.145 and port 6000: Connection refused [01:07:01] RECOVERY - swiftobject201 Current Load on swiftobject201 is OK: LOAD OK - total load average: 0.08, 0.20, 0.09 [01:07:16] RECOVERY - swiftobject201 SSH on swiftobject201 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u5 (protocol 2.0) [01:07:16] RECOVERY - swiftobject201 APT on swiftobject201 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [01:17:57] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/66fab5897aa56617cb9405b1be487a87a46c1a19 [01:17:57] 02puppet/03master 07paladox 0366fab58 Add swiftobject191 to swift::storage::devices [01:23:13] PROBLEM - swiftobject181 Disk Space on swiftobject181 is WARNING: DISK WARNING - free space: / 81518MiB (6% inode=84%); [01:23:13] PROBLEM - swiftobject191 Current Load on swiftobject191 is CRITICAL: LOAD CRITICAL - total load average: 14.56, 8.16, 3.52 [01:23:33] PROBLEM - swiftobject151 Disk Space on swiftobject151 is WARNING: DISK WARNING - free space: / 93129MiB (6% inode=84%); [01:23:34] PROBLEM - swiftobject171 Disk Space on swiftobject171 is WARNING: DISK WARNING - free space: / 87389MiB (6% inode=84%); [01:24:03] PROBLEM - swiftobject161 Disk Space on swiftobject161 is WARNING: DISK WARNING - free space: / 95468MiB (7% inode=84%); [01:29:29] RECOVERY - swiftobject171 Disk Space on swiftobject171 is OK: DISK OK - free space: / 150608MiB (11% inode=85%); [01:29:33] RECOVERY - swiftobject151 Disk Space on swiftobject151 is OK: DISK OK - free space: / 170073MiB (12% inode=85%); [01:29:49] RECOVERY - swiftobject161 Disk Space on swiftobject161 is OK: DISK OK - free space: / 149091MiB (11% inode=85%); [01:31:06] RECOVERY - swiftobject181 Disk Space on swiftobject181 is OK: DISK OK - free space: / 171259MiB (12% inode=85%); [01:32:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:32:23] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/71ea04bcd2f108ac599f87c7e18f5d7f1e0afbc8 [01:32:23] 02puppet/03master 07paladox 0371ea04b Add swiftobject201 to swift::storage::devices [01:35:12] RECOVERY - swiftobject201 Swift Object Service on swiftobject201 is OK: TCP OK - 0.000 second response time on 10.0.20.145 port 6000 [01:36:53] PROBLEM - swiftobject191 Current Load on swiftobject191 is WARNING: LOAD WARNING - total load average: 4.83, 10.22, 8.65 [01:38:08] PROBLEM - swiftobject201 Current Load on swiftobject201 is CRITICAL: LOAD CRITICAL - total load average: 12.63, 7.00, 2.85 [01:38:51] RECOVERY - swiftobject191 Current Load on swiftobject191 is OK: LOAD OK - total load average: 7.04, 8.94, 8.34 [01:40:05] RECOVERY - swiftobject201 Current Load on swiftobject201 is OK: LOAD OK - total load average: 9.44, 7.64, 3.58 [01:43:57] PROBLEM - swiftobject201 Current Load on swiftobject201 is CRITICAL: LOAD CRITICAL - total load average: 15.40, 11.76, 6.17 [01:49:48] PROBLEM - swiftobject201 Current Load on swiftobject201 is WARNING: LOAD WARNING - total load average: 10.55, 11.37, 7.83 [01:51:45] PROBLEM - swiftobject201 Current Load on swiftobject201 is CRITICAL: LOAD CRITICAL - total load average: 12.40, 11.65, 8.33 [01:53:42] PROBLEM - swiftobject201 Current Load on swiftobject201 is WARNING: LOAD WARNING - total load average: 9.00, 10.81, 8.45 [01:55:39] RECOVERY - swiftobject201 Current Load on swiftobject201 is OK: LOAD OK - total load average: 3.29, 8.32, 7.83 [03:42:11] [02puppet] 07MacFan4000 opened pull request #4212: fix backups log locations for grafana and db (07miraheze:03master...07MacFan4000:03patch-61) 13https://github.com/miraheze/puppet/pull/4212 [03:42:15] [02puppet] 07MacFan4000 un-readied pull request #4212: fix backups log locations for grafana and db (07miraheze:03master...07MacFan4000:03patch-61) 13https://github.com/miraheze/puppet/pull/4212 [03:42:17] [02puppet] 07coderabbitai[bot] commented on pull request #4212: --- […] 13https://github.com/miraheze/puppet/pull/4212#issuecomment-2686776154 [03:46:24] [02puppet] 07MacFan4000 retitled PR #4212: "fix backups log locations for grafana and db" ➜ "fix backups log location for grafana" 13https://github.com/miraheze/puppet/pull/4212 [03:46:45] [02puppet] 07MacFan4000 readied pull request #4212: fix backups log location for grafana (07miraheze:03master...07MacFan4000:03patch-61) 13https://github.com/miraheze/puppet/pull/4212 [04:12:17] [02ssl] 07MacFan4000 created 03MacFan4000-patch-1 (+1 new commit) 13https://github.com/miraheze/ssl/commit/5256dd0e77ef [04:12:18] 02ssl/03MacFan4000-patch-1 07MacFan4000 035256dd0 remove non-pointed domains [04:12:23] [02ssl] 07MacFan4000 opened pull request #825: remove non-pointed domains (03master...03MacFan4000-patch-1) 13https://github.com/miraheze/ssl/pull/825 [04:13:12] [02ssl] 07MacFan4000 pushed 1 new commit to 03MacFan4000-patch-1 13https://github.com/miraheze/ssl/commit/0301f61c6540861ada9d80db0eb4ca89ed4cc529 [04:13:13] 02ssl/03MacFan4000-patch-1 07MacFan4000 030301f61 Update certs.yaml [04:13:38] [02ssl] 07coderabbitai[bot] commented on pull request #825: --- […] 13https://github.com/miraheze/ssl/pull/825#issuecomment-2686819307 [04:15:23] [02ssl] 07MacFan4000 merged pull request #825: remove non-pointed domains (03master...03MacFan4000-patch-1) 13https://github.com/miraheze/ssl/pull/825 [04:15:25] [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/93b3689842c4ea26ecff762044bfa4c0ed80ddc8 [04:15:26] 02ssl/03master 07MacFan4000 0393b3689 remove non-pointed domains (#825) [04:15:28] [02ssl] 07MacFan4000 04deleted 03MacFan4000-patch-1 at 030301f61 13https://github.com/miraheze/ssl/commit/0301f61 [05:12:50] [02puppet] 07Universal-Omega requested changes on pull request #4212 13https://github.com/miraheze/puppet/pull/4212#pullrequestreview-2646632611 [05:12:50] [02puppet] 07Universal-Omega left a file comment in pull request #4212 03df1f80b: This is wrong to many grafana-backup 13https://github.com/miraheze/puppet/pull/4212#discussion_r1972882908 [05:16:29] [02puppet] 07Universal-Omega left a file comment in pull request #4212 03df1f80b: ```suggestion […] 13https://github.com/miraheze/puppet/pull/4212#discussion_r1972885300 [05:23:07] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 22.96, 14.16, 6.97 [05:25:58] [02puppet] 07Universal-Omega merged 07MacFan4000's pull request #4212: fix backups log location for grafana (07miraheze:03master...07MacFan4000:03patch-61) 13https://github.com/miraheze/puppet/pull/4212 [05:26:00] [02puppet] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/532dcbc0e73db534d2e278cf8237cfa6ff940daa [05:26:02] 02puppet/03master 07MacFan4000 03532dcbc fix backups log location for grafana (#4212) [05:46:32] RECOVERY - mon181 Backups Grafana on mon181 is OK: FILE_AGE OK: /var/log/grafana-backup/grafana-backup/grafana-backup.log is 182771 seconds old and 375 bytes [05:46:43] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 12.78, 19.94, 18.36 [05:50:37] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 23.05, 13.93, 6.66 [05:58:33] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 3.61, 13.68, 10.48 [08:36:43] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 25.74, 17.20, 8.27 [08:44:43] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 18.38, 21.86, 14.69 [08:48:43] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 2.79, 14.06, 13.41 [08:56:45] PROBLEM - kingdomdeath.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kingdomdeath.wiki' expires in 14 day(s) (Fri 14 Mar 2025 08:40:52 AM GMT +0000). [08:57:04] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/19fb6f622fed2779a36240c21391faef12620119 [08:57:04] 02ssl/03master 07WikiTideSSLBot 0319fb6f6 Bot: Update SSL cert for kingdomdeath.wiki [09:14:57] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 27207MiB (5% inode=99%); [09:26:14] RECOVERY - kingdomdeath.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kingdomdeath.wiki' will expire on Wed 28 May 2025 07:58:21 AM GMT +0000. [11:22:30] PROBLEM - cloud20 IPMI Sensors on cloud20 is WARNING: IPMI Status: Warning [1 system event log (SEL) entry present] [11:38:44] PROBLEM - Host mattermost1 is DOWN: PING CRITICAL - Packet loss = 100% [11:41:11] RECOVERY - Host mattermost1 is UP: PING WARNING - Packet loss = 93%, RTA = 31.08 ms [12:09:18] !log double cache times for new cf caches [12:09:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:10:25] [02CreateWiki] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/CreateWiki/commit/c6a595a91d315cfdb42583a451715dcb4692a544 [12:10:25] 02CreateWiki/03master 07translatewiki.net 03c6a595a Localisation updates from https://translatewiki.net. [12:10:27] [02RequestSSL] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/RequestSSL/commit/5cfa96f118cab708a35ab54bec57b1b44c7de117 [12:10:29] [02landing] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/landing/commit/afab6c47fa1f64fe493b9af6ea9ec10a7c83ee31 [12:10:31] 02RequestSSL/03master 07translatewiki.net 035cfa96f Localisation updates from https://translatewiki.net. [12:10:32] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 13.21, 8.36, 3.96 [12:10:32] 02landing/03master 07translatewiki.net 03afab6c4 Localisation updates from https://translatewiki.net. [12:10:51] oh i guess it's 12:00 then icinga-miraheze [12:11:27] miraheze/landing - translatewiki the build passed. [12:12:32] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 11.25, 8.66, 4.57 [12:14:32] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 15.78, 10.51, 5.69 [12:16:32] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 7.85, 8.77, 5.60 [12:22:07] miraheze/CreateWiki - translatewiki the build passed. [12:22:47] miraheze/RequestSSL - translatewiki the build passed. [12:28:32] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 18.51, 12.79, 8.13 [12:31:38] PROBLEM - mw154 MediaWiki Rendering on mw154 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4714 bytes in 0.065 second response time [12:33:36] RECOVERY - mw154 MediaWiki Rendering on mw154 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.166 second response time [12:34:32] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.24, 8.59, 8.31 [12:52:30] [02dns] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/909b1f964543ba5076b62f95066e3ebb7e41dc76 [12:52:30] 02dns/03master 07paladox 03909b1f9 Add cp38 to dns [12:53:28] miraheze/dns - paladox the build passed. [13:14:49] [02puppet] 07paladox created 03paladox-patch-6 (+1 new commit) 13https://github.com/miraheze/puppet/commit/459c5dee720a [13:14:49] 02puppet/03paladox-patch-6 07paladox 03459c5de Install cp38 [13:14:53] [02puppet] 07paladox opened pull request #4213: Install cp38 (03master...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4213 [13:15:01] [02puppet] 07coderabbitai[bot] commented on pull request #4213: --- […] 13https://github.com/miraheze/puppet/pull/4213#issuecomment-2687932168 [13:15:10] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-6 13https://github.com/miraheze/puppet/commit/ed05b46b0eb66986cba3e354e7448b965b12ed39 [13:15:10] 02puppet/03paladox-patch-6 07paladox 03ed05b46 Update site.pp [13:15:18] [02puppet] 07paladox merged pull request #4213: Install cp38 (03master...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4213 [13:15:18] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/b3fc3d6783b62dae9319d8868ec3f37fe3ee7ab7 [13:15:18] 02puppet/03master 07paladox 03b3fc3d6 Install cp38 (#4213) [13:15:19] [02puppet] 07paladox 04deleted 03paladox-patch-6 at 03ed05b46 13https://github.com/miraheze/puppet/commit/ed05b46 [13:19:05] PROBLEM - cp38 HTTP 4xx/5xx ERROR Rate on cp38 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [13:29:03] PROBLEM - cp38 NTP time on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:20] PROBLEM - cp38 Nginx Backend for mw164 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:23] PROBLEM - cp38 Nginx Backend for mw161 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:23] PROBLEM - cp38 APT on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:26] PROBLEM - cp38 Nginx Backend for mw172 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:26] PROBLEM - cp38 Nginx Backend for matomo151 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:26] PROBLEM - cp38 Puppet on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:26] PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:29] PROBLEM - cp38 Nginx Backend for mw154 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:32] PROBLEM - cp38 Nginx Backend for mw153 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:33] oh [13:29:36] PROBLEM - cp38 Nginx Backend for mw184 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:36] PROBLEM - cp38 Nginx Backend for mw171 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:36] PROBLEM - cp38 Nginx Backend for mw181 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:36] PROBLEM - cp38 Nginx Backend for swiftproxy171 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:36] that's fine [13:29:39] PROBLEM - cp38 Nginx Backend for mw182 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:42] PROBLEM - cp38 Nginx Backend for reports171 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:46] PROBLEM - cp38 Nginx Backend for mon181 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:49] PROBLEM - cp38 Nginx Backend for mw163 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:52] PROBLEM - cp38 SSH on cp38 is CRITICAL: connect to address 10.0.19.146 and port 22: No route to host [13:29:52] PROBLEM - cp38 Nginx Backend for mwtask161 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:52] PROBLEM - cp38 Nginx Backend for mw174 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:55] PROBLEM - cp38 Nginx Backend for mw183 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:55] PROBLEM - cp38 HTTPS on cp38 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to cp38.wikitide.net port 443 after 1010 ms: Couldn't connect to server [13:29:56] PROBLEM - cp38 Nginx Backend for puppet181 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:56] PROBLEM - cp38 Nginx Backend for swiftproxy161 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:56] PROBLEM - cp38 Nginx Backend for mwtask171 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:56] PROBLEM - ping on cp38 is CRITICAL: CRITICAL - Host Unreachable (10.0.19.146) [13:29:58] PROBLEM - cp38 conntrack_table_size on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:58] PROBLEM - cp38 Disk Space on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:29:59] PROBLEM - cp38 Current Load on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:30:01] PROBLEM - cp38 Nginx Backend for mwtask151 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:30:02] PROBLEM - cp38 Nginx Backend for test151 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:30:05] PROBLEM - cp38 Nginx Backend for mw151 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:30:08] PROBLEM - cp38 PowerDNS Recursor on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:30:11] PROBLEM - cp38 ferm_active on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:30:12] PROBLEM - cp38 Nginx Backend for mw162 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:30:21] PROBLEM - cp38 Nginx Backend for mwtask181 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:30:24] PROBLEM - cp38 Nginx Backend for phorge171 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:30:28] PROBLEM - cp38 Nginx Backend for mw173 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:30:28] PROBLEM - cp38 Nginx Backend for mw152 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host [13:30:38] PROBLEM - Host cp38 is DOWN: CRITICAL - Host Unreachable (10.0.19.146) [13:31:03] !log delete cp36 from cf [13:31:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:33:22] !log add cp38 to cf and disable it [13:33:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:56:15] PROBLEM - cp38 Nginx Backend for mwtask151 on cp38 is WARNING: Could not resolve hostname : Name or service not known [13:56:15] PROBLEM - cp38 Nginx Backend for mw183 on cp38 is WARNING: Could not resolve hostname : Name or service not known [13:56:18] PROBLEM - cp38 HTTPS on cp38 is CRITICAL: HTTP CRITICAL - Unable to lookup IP address for '': getaddrinfo returned -2 - Name or service not known [13:56:21] PROBLEM - cp38 Nginx Backend for reports171 on cp38 is WARNING: Could not resolve hostname : Name or service not known [13:56:23] PROBLEM - cp38 Nginx Backend for mw163 on cp38 is WARNING: Could not resolve hostname : Name or service not known [13:56:24] PROBLEM - cp38 Nginx Backend for test151 on cp38 is WARNING: Could not resolve hostname : Name or service not known [13:56:24] PROBLEM - cp38 Nginx Backend for mw181 on cp38 is WARNING: Could not resolve hostname : Name or service not known [13:56:25] PROBLEM - cp38 Nginx Backend for matomo151 on cp38 is WARNING: Could not resolve hostname : Name or service not known [13:56:27] PROBLEM - cp38 PowerDNS Recursor on cp38 is WARNING: Could not resolve hostname : Name or service not known [13:56:28] PROBLEM - cp38 Nginx Backend for swiftproxy161 on cp38 is WARNING: Could not resolve hostname : Name or service not known [13:56:28] PROBLEM - cp38 Nginx Backend for mw171 on cp38 is WARNING: Could not resolve hostname : Name or service not known [13:56:29] PROBLEM - cp38 Puppet on cp38 is WARNING: Could not resolve hostname : Name or service not known [13:56:30] PROBLEM - cp38 Current Load on cp38 is WARNING: Could not resolve hostname : Name or service not known [13:56:32] PROBLEM - cp38 Nginx Backend for mw164 on cp38 is WARNING: Could not resolve hostname : Name or service not known [13:56:33] PROBLEM - cp38 Nginx Backend for mw184 on cp38 is WARNING: Could not resolve hostname : Name or service not known [13:56:34] PROBLEM - cp38 Nginx Backend for mw153 on cp38 is WARNING: Could not resolve hostname : Name or service not known [13:56:37] PROBLEM - cp38 Nginx Backend for mwtask171 on cp38 is WARNING: Could not resolve hostname : Name or service not known [13:56:39] PROBLEM - cp38 Nginx Backend for mw172 on cp38 is WARNING: Could not resolve hostname : Name or service not known [13:56:41] PROBLEM - Host cp38 is DOWN: check_ping: Invalid hostname/address - Usage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4|-6] [13:57:45] hmm [14:00:39] RECOVERY - Host cp38 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [14:01:38] RECOVERY - cp38 Nginx Backend for mw163 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8123 [14:01:38] RECOVERY - cp38 Nginx Backend for reports171 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8205 [14:01:43] RECOVERY - cp38 HTTPS on cp38 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4073 bytes in 0.061 second response time [14:02:03] RECOVERY - cp38 Nginx Backend for mwtask151 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8162 [14:02:08] RECOVERY - cp38 Nginx Backend for test151 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8181 [14:02:13] RECOVERY - cp38 Nginx Backend for matomo151 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8203 [14:02:18] RECOVERY - cp38 Nginx Backend for mw183 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8127 [14:02:18] RECOVERY - cp38 Nginx Backend for mw164 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8124 [14:02:23] RECOVERY - cp38 Puppet on cp38 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [14:02:23] RECOVERY - cp38 Nginx Backend for swiftproxy161 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8206 [14:02:23] RECOVERY - cp38 Current Load on cp38 is OK: LOAD OK - total load average: 0.47, 0.23, 0.10 [14:02:23] RECOVERY - cp38 Nginx Backend for mw171 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8117 [14:02:23] RECOVERY - cp38 Nginx Backend for mw181 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8119 [14:02:23] RECOVERY - cp38 Nginx Backend for mw184 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8128 [14:02:28] RECOVERY - cp38 Nginx Backend for mw153 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8121 [14:02:28] RECOVERY - cp38 PowerDNS Recursor on cp38 is OK: DNS OK: 0.035 seconds response time. wikitide.net returns 2602:294:0:b13::110,38.46.223.205 [14:02:30] !log pool cp38 to 10% of traffic [14:02:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:02:33] RECOVERY - cp38 Nginx Backend for mw172 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8118 [14:02:38] RECOVERY - cp38 Nginx Backend for mwtask171 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8161 [14:06:20] [02mw-config] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/fa625795e62d [14:06:20] 02mw-config/03paladox-patch-2 07paladox 03fa62579 Add cp38 to wgCdnServers [14:06:24] [02mw-config] 07paladox opened pull request #5836: Add cp38 to wgCdnServers (03master...03paladox-patch-2) 13https://github.com/miraheze/mw-config/pull/5836 [14:06:32] [02mw-config] 07paladox merged pull request #5836: Add cp38 to wgCdnServers (03master...03paladox-patch-2) 13https://github.com/miraheze/mw-config/pull/5836 [14:06:34] [02mw-config] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/7cdba55b1f38993ee1016573e4d44b8af422fec6 [14:06:36] [02mw-config] 07paladox 04deleted 03paladox-patch-2 at 03fa62579 13https://github.com/miraheze/mw-config/commit/fa62579 [14:06:37] 02mw-config/03master 07paladox 037cdba55 Add cp38 to wgCdnServers (#5836) [14:06:56] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [14:06:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:07:15] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 18s [14:07:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:07:26] miraheze/mw-config - paladox the build passed. [14:07:32] miraheze/mw-config - paladox the build passed. [14:07:49] [02mw-config] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/5b37882ee4ec [14:07:49] 02mw-config/03paladox-patch-2 07paladox 035b37882 Uninstall MultiPurge [14:07:56] [02mw-config] 07paladox opened pull request #5837: Uninstall MultiPurge (03master...03paladox-patch-2) 13https://github.com/miraheze/mw-config/pull/5837 [14:08:33] [02mw-config] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/mw-config/commit/b67b44f452f0e4e3d943e558efb335ee5884663e [14:08:34] 02mw-config/03paladox-patch-2 07paladox 03b67b44f Update LocalSettings.php [14:08:54] miraheze/mw-config - paladox the build passed. [14:09:15] PROBLEM - fashiondreamer.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fashiondreamer.wiki' expires in 14 day(s) (Fri 14 Mar 2025 01:50:18 PM GMT +0000). [14:09:31] miraheze/mw-config - paladox the build passed. [14:13:04] [02mw-config] 07paladox merged pull request #5837: Uninstall MultiPurge (03master...03paladox-patch-2) 13https://github.com/miraheze/mw-config/pull/5837 [14:13:05] [02mw-config] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/78f2ed3cb764b7e7e85a4e54146d801d690a4b0a [14:13:05] 02mw-config/03master 07paladox 0378f2ed3 Uninstall MultiPurge (#5837) [14:13:06] [02mw-config] 07paladox 04deleted 03paladox-patch-2 at 03b67b44f 13https://github.com/miraheze/mw-config/commit/b67b44f [14:13:09] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [14:13:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:13:25] !log [paladox@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [14:13:26] !log [paladox@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 1s [14:13:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:13:28] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 18s [14:13:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:13:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:13:58] miraheze/mw-config - paladox the build passed. [14:14:05] !log ramp cp38 up to 25% of traffic [14:14:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:14:37] [02mediawiki-repos] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/mediawiki-repos/commit/38ce9d9fccce [14:14:37] 02mediawiki-repos/03paladox-patch-1 07paladox 0338ce9d9 Remove MultiPurge… [14:14:40] [02mediawiki-repos] 07paladox opened pull request #50: Remove MultiPurge (03master...03paladox-patch-1) 13https://github.com/miraheze/mediawiki-repos/pull/50 [14:14:47] [02mediawiki-repos] 07coderabbitai[bot] commented on pull request #50: --- […] 13https://github.com/miraheze/mediawiki-repos/pull/50#issuecomment-2688087712 [14:14:50] [02mediawiki-repos] 07paladox merged pull request #50: Remove MultiPurge (03master...03paladox-patch-1) 13https://github.com/miraheze/mediawiki-repos/pull/50 [14:14:50] [02mediawiki-repos] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/mediawiki-repos/commit/657e40b86074be68316b67a5466d65732552d602 [14:14:51] 02mediawiki-repos/03master 07paladox 03657e40b Remove MultiPurge (#50)… [14:14:53] [02mediawiki-repos] 07paladox 04deleted 03paladox-patch-1 at 0338ce9d9 13https://github.com/miraheze/mediawiki-repos/commit/38ce9d9 [14:16:28] !log [paladox@test151] starting deploy of {'l10n': True, 'versions': ['1.43', '1.44'], 'upgrade_extensions': 'CreateWiki'} to test151 [14:16:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:18:43] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.43', 'upgrade_extensions': 'CreateWiki'} to all [14:18:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:23:28] !log [paladox@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw151.wikitide.net [14:23:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:25:39] [02mw-config] 07dependabot[bot] created 03dependabot/composer/justinrainbow/json-schema-6.2.0 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/7fee35221da2 [14:25:39] 02mw-config/03dependabot/composer/justinrainbow/json-schema-6.2.0 07dependabot[bot] 037fee352 Update justinrainbow/json-schema requirement from 6.0.0 to 6.2.0… [14:25:41] [02mw-config] 07dependabot[bot] added the label 'dependencies' to pull request #5838 (Update justinrainbow/json-schema requirement from 6.0.0 to 6.2.0) 13https://github.com/miraheze/mw-config/pull/5838 [14:25:41] [02mw-config] 07dependabot[bot] opened pull request #5838: Update justinrainbow/json-schema requirement from 6.0.0 to 6.2.0 (03master...03dependabot/composer/justinrainbow/json-schema-6.2.0) 13https://github.com/miraheze/mw-config/pull/5838 [14:25:43] [02mw-config] 07dependabot[bot] added the label 'php' to pull request #5838 (Update justinrainbow/json-schema requirement from 6.0.0 to 6.2.0) 13https://github.com/miraheze/mw-config/pull/5838 [14:25:45] [02mw-config] 07dependabot[bot] commented on pull request #5805: Superseded by #5838. 13https://github.com/miraheze/mw-config/pull/5805#issuecomment-2688118124 [14:25:47] [02mw-config] 07dependabot[bot] closed pull request #5805: Update justinrainbow/json-schema requirement from 6.0.0 to 6.1.0 (03master...03dependabot/composer/justinrainbow/json-schema-6.1.0) 13https://github.com/miraheze/mw-config/pull/5805 [14:25:49] [02mw-config] 07dependabot[bot] 04deleted 03dependabot/composer/justinrainbow/json-schema-6.1.0 at 031a045c6 13https://github.com/miraheze/mw-config/commit/1a045c6 [14:25:53] !log [paladox@test151] finished deploy of {'l10n': True, 'versions': ['1.43', '1.44'], 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 565s [14:25:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:26:36] miraheze/mw-config - dependabot[bot] the build has errored. [14:27:34] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.43', 'upgrade_extensions': 'CreateWiki'} to all [14:27:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:30:54] !log [paladox@mwtask181] finished deploy of {'l10n': True, 'versions': '1.43', 'upgrade_extensions': 'CreateWiki'} to all - SUCCESS in 200s [14:30:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:47:18] PROBLEM - wikimoma.art - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikimoma.art' expires in 14 day(s) (Fri 14 Mar 2025 02:42:04 PM GMT +0000). [14:47:30] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/f188ed0d64a9a5f3b8ce8139be53ffda2cb616af [14:47:30] 02ssl/03master 07WikiTideSSLBot 03f188ed0 Bot: Update SSL cert for wikimoma.art [14:47:46] PROBLEM - thegreatwar.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'thegreatwar.uk' expires in 14 day(s) (Fri 14 Mar 2025 02:25:51 PM GMT +0000). [14:51:52] PROBLEM - keitaiwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'keitaiwiki.com' expires in 14 day(s) (Fri 14 Mar 2025 02:33:57 PM GMT +0000). [14:52:02] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/527eadf80833dbfc1898c6fa7ffa7ff4e810c15f [14:52:02] 02ssl/03master 07WikiTideSSLBot 03527eadf Bot: Update SSL cert for keitaiwiki.com [14:53:18] PROBLEM - worldsanskrit.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'worldsanskrit.net' expires in 14 day(s) (Fri 14 Mar 2025 02:35:17 PM GMT +0000). [14:53:22] PROBLEM - dragonquestwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dragonquestwiki.com' expires in 14 day(s) (Fri 14 Mar 2025 02:26:07 PM GMT +0000). [14:53:30] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/612f59df51732e95aa72e9135766b8570d2bdfda [14:53:30] 02ssl/03master 07WikiTideSSLBot 03612f59d Bot: Update SSL cert for worldsanskrit.net [14:53:40] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/33e65c92960f399537f4f036d4e2be0998316794 [14:53:40] 02ssl/03master 07WikiTideSSLBot 0333e65c9 Bot: Update SSL cert for dragonquestwiki.com [14:55:04] !log increase cp38 traffic from 25% to 50% (fully pooled) [14:55:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:56:21] PROBLEM - yokaiwatchwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'yokaiwatchwiki.com' expires in 14 day(s) (Fri 14 Mar 2025 02:31:09 PM GMT +0000). [14:56:31] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/b43fcd7da5693219d9a8c26ca8a850b004b780f8 [14:56:32] 02ssl/03master 07WikiTideSSLBot 03b43fcd7 Bot: Update SSL cert for yokaiwatchwiki.com [15:21:04] RECOVERY - keitaiwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'keitaiwiki.com' will expire on Wed 28 May 2025 01:53:26 PM GMT +0000. [15:21:58] RECOVERY - dragonquestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dragonquestwiki.com' will expire on Wed 28 May 2025 01:55:04 PM GMT +0000. [15:22:57] RECOVERY - worldsanskrit.net - LetsEncrypt on sslhost is OK: OK - Certificate 'worldsanskrit.net' will expire on Wed 28 May 2025 01:54:54 PM GMT +0000. [15:26:10] RECOVERY - yokaiwatchwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'yokaiwatchwiki.com' will expire on Wed 28 May 2025 01:57:56 PM GMT +0000. [15:27:39] [02dns] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/dns/commit/e078b71da604 [15:27:39] 02dns/03paladox-patch-1 07paladox 03e078b71 Add mw191/192/193 and mw201/202/203 to dns [15:27:43] [02dns] 07paladox opened pull request #647: Add mw191/192/193 and mw201/202/203 to dns (03master...03paladox-patch-1) 13https://github.com/miraheze/dns/pull/647 [15:28:41] [02dns] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/dns/commit/53c25ec844e71c1f6e6f7cb2086dc4b3f50b1374 [15:28:42] 02dns/03paladox-patch-1 07paladox 0353c25ec Update wikitide.net [15:28:44] miraheze/dns - paladox the build passed. [15:29:50] [02dns] 07paladox merged pull request #647: Add mw191/192/193 and mw201/202/203 to dns (03master...03paladox-patch-1) 13https://github.com/miraheze/dns/pull/647 [15:29:50] [02dns] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/4d909692108ffe98551199e3571bfdde2d02f0f3 [15:29:50] 02dns/03master 07paladox 034d90969 Add mw191/192/193 and mw201/202/203 to dns (#647) [15:29:51] [02dns] 07paladox 04deleted 03paladox-patch-1 at 0353c25ec 13https://github.com/miraheze/dns/commit/53c25ec [15:30:19] miraheze/dns - paladox the build passed. [15:30:48] miraheze/dns - paladox the build passed. [15:45:22] RECOVERY - wikimoma.art - LetsEncrypt on sslhost is OK: OK - Certificate 'wikimoma.art' will expire on Wed 28 May 2025 01:48:54 PM GMT +0000. [16:04:48] [02puppet] 07paladox created 03paladox-patch-6 (+1 new commit) 13https://github.com/miraheze/puppet/commit/241dae6b0717 [16:04:48] 02puppet/03paladox-patch-6 07paladox 03241dae6 Install mw191/192/193 [16:04:52] [02puppet] 07paladox opened pull request #4214: Install mw191/192/193 (03master...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4214 [16:04:59] [02puppet] 07coderabbitai[bot] commented on pull request #4214: --- […] 13https://github.com/miraheze/puppet/pull/4214#issuecomment-2688412404 [16:05:45] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-6 13https://github.com/miraheze/puppet/commit/1706cea356f9086df5da78ca325aff02e0eeac82 [16:05:45] 02puppet/03paladox-patch-6 07paladox 031706cea Update common.yaml [16:06:38] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-6 13https://github.com/miraheze/puppet/commit/8ce7767cbd7f96babf3cf81f2bd575acbbae20f0 [16:06:38] 02puppet/03paladox-patch-6 07paladox 038ce7767 Update mwdeploy.py [16:07:59] miraheze/puppet - paladox the build passed. [16:12:53] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/f49c91f2dd3db1fd14d4981d9599e2f94620b727 [16:12:53] 02puppet/03master 07paladox 03f49c91f Install mw191/192/193 (#4214) [16:12:53] [02puppet] 07paladox merged pull request #4214: Install mw191/192/193 (03master...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4214 [16:12:54] [02puppet] 07paladox 04deleted 03paladox-patch-6 at 038ce7767 13https://github.com/miraheze/puppet/commit/8ce7767 [16:14:02] miraheze/puppet - paladox the build passed. [16:19:02] PROBLEM - mw191 php-fpm on mw191 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2' [16:19:03] PROBLEM - mw193 Puppet on mw193 is WARNING: Could not resolve hostname : Name or service not known [16:19:04] PROBLEM - Host mw193 is DOWN: check_ping: Invalid hostname/address - Usage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4|-6] [16:19:20] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [16:19:22] PROBLEM - mw191 APT on mw191 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [16:19:30] PROBLEM - mw192 APT on mw192 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [16:19:40] PROBLEM - mw192 php-fpm on mw192 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2' [16:19:45] PROBLEM - mw191.wikitide.net SSL Check on sslhost is CRITICAL: connect to address mw191.wikitide.net and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [16:19:46] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:19:51] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: connect to address 10.0.19.160 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [16:19:53] PROBLEM - mw192.wikitide.net SSL Check on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:19:54] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw191.wikitide.net port 443 after 0 ms: Couldn't connect to server [16:20:01] PROBLEM - mw191 Puppet on mw191 is CRITICAL: CRITICAL: Puppet has 95 failures. Last run 2 minutes ago with 95 failures. Failed resources (up to 3 shown): Package[mcrouter],Package[cgroup-tools],Package[imagemagick],Package[webp] [16:20:12] PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: 3 backends are down. mw191 mw192 mw193 [16:20:18] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 3 backends are down. mw191 mw192 mw193 [16:20:27] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 3 backends are down. mw191 mw192 mw193 [16:24:52] PROBLEM - mw193.wikitide.net SSL Check on sslhost is CRITICAL: connect to address mw193.wikitide.net and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [16:24:59] RECOVERY - Host mw193 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [16:25:03] PROBLEM - mw193 Puppet on mw193 is CRITICAL: CRITICAL: Puppet has 93 failures. Last run 24 seconds ago with 93 failures. Failed resources (up to 3 shown): Package[mcrouter],Package[cgroup-tools],Package[imagemagick],Package[webp] [16:25:35] PROBLEM - mw193 APT on mw193 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [16:25:42] PROBLEM - mw192 Puppet on mw192 is CRITICAL: CRITICAL: Puppet has 93 failures. Last run 1 minute ago with 93 failures. Failed resources (up to 3 shown): Package[mcrouter],Package[cgroup-tools],Package[imagemagick],Package[webp] [16:26:17] [02puppet] 07paladox created 03paladox-patch-6 (+1 new commit) 13https://github.com/miraheze/puppet/commit/358f3795eb98 [16:26:17] 02puppet/03paladox-patch-6 07paladox 03358f379 mcrouter: install some packages required by mcrouter [16:26:21] [02puppet] 07paladox opened pull request #4215: mcrouter: install some packages required by mcrouter (03master...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4215 [16:26:29] [02puppet] 07coderabbitai[bot] commented on pull request #4215: --- […] 13https://github.com/miraheze/puppet/pull/4215#issuecomment-2688477372 [16:26:30] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw193.wikitide.net port 443 after 0 ms: Couldn't connect to server [16:26:45] PROBLEM - mw193 php-fpm on mw193 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2' [16:26:53] [02puppet] 07paladox merged pull request #4215: mcrouter: install some packages required by mcrouter (03master...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4215 [16:26:54] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/42af6526065ab170bbf0f96ea6f2b3ed68ce1250 [16:26:54] 02puppet/03master 07paladox 0342af652 mcrouter: install some packages required by mcrouter (#4215) [16:26:54]