[00:08:01] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 53960MiB (11% inode=99%); [01:27:33] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 3.57, 16.67, 23.01 [01:31:33] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 2.25, 8.66, 18.29 [01:52:01] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49824MiB (10% inode=99%); [02:01:33] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.29, 19.75, 13.79 [02:03:33] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.35, 21.05, 15.00 [02:09:33] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 8.59, 17.67, 15.83 [02:45:16] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 21.87, 18.92, 11.36 [02:49:16] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 25.20, 21.25, 13.98 [02:51:16] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 15.08, 20.10, 14.53 [03:10:56] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 25.57, 16.31, 7.77 [03:18:56] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 3.86, 15.46, 11.96 [03:19:55] PROBLEM - os161 Current Load on os161 is WARNING: LOAD WARNING - total load average: 3.57, 3.40, 2.56 [03:21:55] RECOVERY - os161 Current Load on os161 is OK: LOAD OK - total load average: 2.43, 3.15, 2.58 [03:43:30] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 22.24, 14.13, 6.45 [03:45:30] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 8.44, 12.68, 6.93 [03:54:56] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 28.08, 14.34, 7.25 [03:58:56] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 23.14, 19.54, 11.05 [04:04:56] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 16.10, 20.24, 14.31 [04:13:39] [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/9707d71f9efe013813a60377470b021d719181e6 [04:13:39] 02ssl/03master 07MacFan4000 039707d71 add cpzero.wiki and wiki.ethereal.mov [04:28:16] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 27.60, 22.36, 18.24 [05:12:55] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 7.77, 19.36, 23.34 [05:16:55] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 7.88, 12.24, 19.44 [06:02:39] PROBLEM - os161 Disk Space on os161 is WARNING: DISK WARNING - free space: / 23603MiB (10% inode=99%); [06:07:00] PROBLEM - os161 Current Load on os161 is WARNING: LOAD WARNING - total load average: 3.62, 3.12, 2.54 [06:09:00] RECOVERY - os161 Current Load on os161 is OK: LOAD OK - total load average: 3.14, 3.13, 2.62 [06:12:39] RECOVERY - os161 Disk Space on os161 is OK: DISK OK - free space: / 24690MiB (11% inode=99%); [06:16:38] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 23.09, 16.59, 9.85 [06:20:35] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 25.04, 20.79, 13.09 [06:30:27] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 16.88, 21.10, 17.46 [06:32:26] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 16.05, 19.31, 17.24 [07:24:06] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 21949MiB (9% inode=99%); [07:26:29] RECOVERY - wiki.ooer.ooo - LetsEncrypt on sslhost is OK: OK - Certificate 'ooer.ooo' will expire on Wed 21 May 2025 06:18:05 AM GMT +0000. [07:36:06] RECOVERY - os151 Disk Space on os151 is OK: DISK OK - free space: / 25372MiB (11% inode=99%); [07:41:17] PROBLEM - os161 Disk Space on os161 is WARNING: DISK WARNING - free space: / 22997MiB (10% inode=99%); [07:42:06] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 23066MiB (10% inode=99%); [07:43:13] RECOVERY - os161 Disk Space on os161 is OK: DISK OK - free space: / 25045MiB (11% inode=99%); [07:46:06] RECOVERY - os151 Disk Space on os151 is OK: DISK OK - free space: / 25029MiB (11% inode=99%); [07:47:04] PROBLEM - os161 Disk Space on os161 is WARNING: DISK WARNING - free space: / 23578MiB (10% inode=99%); [07:50:06] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 23229MiB (10% inode=99%); [07:56:06] RECOVERY - os151 Disk Space on os151 is OK: DISK OK - free space: / 24409MiB (11% inode=99%); [07:57:30] PROBLEM - wiki.ooer.ooo - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [08:00:06] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 23540MiB (10% inode=99%); [08:22:49] PROBLEM - matomo151 HTTPS on matomo151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to matomo151.wikitide.net port 443 after 0 ms: Couldn't connect to server [08:24:49] RECOVERY - matomo151 HTTPS on matomo151 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 0.284 second response time [10:27:10] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.77, 6.26, 5.48 [10:29:07] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 5.60, 5.86, 5.43 [10:40:39] PROBLEM - os161 Disk Space on os161 is CRITICAL: DISK CRITICAL - free space: / 13257MiB (5% inode=99%); [10:44:06] PROBLEM - os151 Disk Space on os151 is CRITICAL: DISK CRITICAL - free space: / 12276MiB (5% inode=99%); [10:44:39] PROBLEM - os161 Disk Space on os161 is WARNING: DISK WARNING - free space: / 15194MiB (6% inode=99%); [10:56:06] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 16204MiB (7% inode=99%); [11:42:35] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.66, 16.59, 8.40 [11:46:35] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 17.76, 18.95, 11.36 [12:08:36] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.32, 19.59, 16.10 [12:09:00] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw181 [12:09:13] PROBLEM - rct.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address rct.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:09:13] PROBLEM - wikitide.net - LetsEncrypt on sslhost is CRITICAL: connect to address wikitide.net and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:09:18] PROBLEM - cp36 Nginx Backend for mw174 on cp36 is CRITICAL: connect to address localhost and port 8126: Connection refused [12:09:21] PROBLEM - yokaiwatchwiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address yokaiwatchwiki.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:09:23] PROBLEM - rosettacode.org - LetsEncrypt on sslhost is CRITICAL: connect to address rosettacode.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:09:26] PROBLEM - cp36 Nginx Backend for mw163 on cp36 is CRITICAL: connect to address localhost and port 8123: Connection refused [12:09:28] PROBLEM - cp36 Nginx Backend for mw171 on cp36 is CRITICAL: connect to address localhost and port 8117: Connection refused [12:09:29] PROBLEM - balloonfightwiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address balloonfightwiki.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:09:33] PROBLEM - archivesofhavnor.org - LetsEncrypt on sslhost is CRITICAL: connect to address archivesofhavnor.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:09:33] PROBLEM - creativecommons.id - LetsEncrypt on sslhost is CRITICAL: connect to address creativecommons.id and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:09:36] PROBLEM - cp36 Nginx Backend for mw172 on cp36 is CRITICAL: connect to address localhost and port 8118: Connection refused [12:09:37] PROBLEM - cp36 Nginx Backend for mwtask171 on cp36 is CRITICAL: connect to address localhost and port 8161: Connection refused [12:09:37] PROBLEM - cp36 Nginx Backend for mw182 on cp36 is CRITICAL: connect to address localhost and port 8120: Connection refused [12:09:38] PROBLEM - cp36 Nginx Backend for puppet181 on cp36 is CRITICAL: connect to address localhost and port 8204: Connection refused [12:09:38] PROBLEM - metroidpedia.com - LetsEncrypt on sslhost is CRITICAL: connect to address metroidpedia.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:09:40] PROBLEM - fairytail.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address fairytail.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:09:40] PROBLEM - bebaskanpengetahuan.id - LetsEncrypt on sslhost is CRITICAL: connect to address bebaskanpengetahuan.id and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:09:41] PROBLEM - cp36 Nginx Backend for mwtask151 on cp36 is CRITICAL: connect to address localhost and port 8162: Connection refused [12:09:45] PROBLEM - cp36 Nginx Backend for mw181 on cp36 is CRITICAL: connect to address localhost and port 8119: Connection refused [12:09:48] PROBLEM - thestarsareright.org - LetsEncrypt on sslhost is CRITICAL: connect to address thestarsareright.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:09:53] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 17 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw154 mw163 mw164 mw173 mw174 mw183 mw184 mediawiki [12:09:53] PROBLEM - apeirology.com - LetsEncrypt on sslhost is CRITICAL: connect to address apeirology.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:09:57] PROBLEM - electowiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address electowiki.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:09:59] PROBLEM - cp36 Nginx Backend for phorge171 on cp36 is CRITICAL: connect to address localhost and port 8202: Connection refused [12:09:59] PROBLEM - cp36 Nginx Backend for mw173 on cp36 is CRITICAL: connect to address localhost and port 8125: Connection refused [12:10:01] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 38.46.223.205/cpweb, 2602:294:0:b13::110/cpweb [12:10:02] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 38.46.223.205/cpweb, 2602:294:0:b13::110/cpweb [12:10:05] PROBLEM - cp36 Nginx Backend for mon181 on cp36 is CRITICAL: connect to address localhost and port 8201: Connection refused [12:10:06] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to cp36.wikitide.net port 443 after 0 ms: Couldn't connect to server [12:10:10] PROBLEM - puzzles.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address puzzles.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:10:22] PROBLEM - cp36 Nginx Backend for swiftproxy161 on cp36 is CRITICAL: connect to address localhost and port 8206: Connection refused [12:10:22] PROBLEM - cp36 Nginx Backend for mwtask181 on cp36 is CRITICAL: connect to address localhost and port 8160: Connection refused [12:10:22] PROBLEM - cp36 Nginx Backend for mw152 on cp36 is CRITICAL: connect to address localhost and port 8114: Connection refused [12:10:32] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 85% [12:10:35] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.89, 19.42, 16.47 [12:10:36] PROBLEM - cp36 Nginx Backend for reports171 on cp36 is CRITICAL: connect to address localhost and port 8205: Connection refused [12:10:37] PROBLEM - theenglishlakes.uk - LetsEncrypt on sslhost is CRITICAL: connect to address theenglishlakes.uk and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:10:37] PROBLEM - cp36 Nginx Backend for mw184 on cp36 is CRITICAL: connect to address localhost and port 8128: Connection refused [12:10:38] PROBLEM - cp36 Nginx Backend for swiftproxy171 on cp36 is CRITICAL: connect to address localhost and port 8207: Connection refused [12:10:46] PROBLEM - cp36 Nginx Backend for mwtask161 on cp36 is CRITICAL: connect to address localhost and port 8163: Connection refused [12:10:48] [02CreateWiki] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/CreateWiki/commit/7c9b2b32141bd2b268d994df3a7a10c4bf818475 [12:10:48] 02CreateWiki/03master 07translatewiki.net 037c9b2b3 Localisation updates from https://translatewiki.net. [12:10:49] [02ErrorPages] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/ErrorPages/commit/4ec735f9a9c8f66ff485b4a1c92e58557c4a22e0 [12:10:49] 02ErrorPages/03master 07translatewiki.net 034ec735f Localisation updates from https://translatewiki.net. [12:10:49] PROBLEM - cp36 Nginx Backend for matomo151 on cp36 is CRITICAL: connect to address localhost and port 8203: Connection refused [12:10:49] PROBLEM - cp36 Nginx Backend for mw151 on cp36 is CRITICAL: connect to address localhost and port 8113: Connection refused [12:10:50] [02ImportDump] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/ImportDump/commit/b838e22e0b153d584c7ee4c760c72097e0b03dc2 [12:10:52] 02ImportDump/03master 07translatewiki.net 03b838e22 Localisation updates from https://translatewiki.net. [12:10:54] [02landing] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/landing/commit/34a6ec439d9c5973b526df9ad0531d2e5e4b374a [12:10:56] [02DataDump] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/DataDump/commit/d51fd561a2ccae1a90198ed2ffe7fe47bf036b8f [12:10:58] [02ManageWiki] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/ManageWiki/commit/da81829cd228f43e560c93c2f84f4a0f4e731f52 [12:11:00] [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/MirahezeMagic/commit/fbcea743bc6423006f81604ba8a01a06ae2f2b4d [12:11:00] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [12:11:02] [02RemovePII] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/RemovePII/commit/db52bb65005a54b34605cec02936fe2c53a48c9d [12:11:03] PROBLEM - cp36 Nginx Backend for mw164 on cp36 is CRITICAL: connect to address localhost and port 8124: Connection refused [12:11:03] 02RemovePII/03master 07translatewiki.net 03db52bb6 Localisation updates from https://translatewiki.net. [12:11:05] [02WikiDiscover] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/WikiDiscover/commit/35238198505b469d667ea0ab8e6b972ba2b0acde [12:11:07] 02DataDump/03master 07translatewiki.net 03d51fd56 Localisation updates from https://translatewiki.net. [12:11:07] PROBLEM - cp36 Nginx Backend for mw154 on cp36 is CRITICAL: connect to address localhost and port 8122: Connection refused [12:11:08] PROBLEM - cp36 Nginx Backend for mw183 on cp36 is CRITICAL: connect to address localhost and port 8127: Connection refused [12:11:08] PROBLEM - cp36 Nginx Backend for mw162 on cp36 is CRITICAL: connect to address localhost and port 8116: Connection refused [12:11:08] 02ManageWiki/03master 07translatewiki.net 03da81829 Localisation updates from https://translatewiki.net. [12:11:09] PROBLEM - voecwiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address voecwiki.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:11:10] PROBLEM - cp36 Nginx Backend for mw161 on cp36 is CRITICAL: connect to address localhost and port 8115: Connection refused [12:11:10] 02MirahezeMagic/03master 07translatewiki.net 03fbcea74 Localisation updates from https://translatewiki.net. [12:11:11] PROBLEM - cp36 Nginx Backend for mw153 on cp36 is CRITICAL: connect to address localhost and port 8121: Connection refused [12:11:12] 02landing/03master 07translatewiki.net 0334a6ec4 Localisation updates from https://translatewiki.net. [12:11:13] 02WikiDiscover/03master 07translatewiki.net 033523819 Localisation updates from https://translatewiki.net. [12:11:14] PROBLEM - yepedia.xyz - LetsEncrypt on sslhost is CRITICAL: connect to address yepedia.xyz and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:11:14] PROBLEM - cp36 Nginx Backend for test151 on cp36 is CRITICAL: connect to address localhost and port 8181: Connection refused [12:11:40] miraheze/ErrorPages - translatewiki the build passed. [12:11:48] miraheze/landing - translatewiki the build passed. [12:12:03] PROBLEM - looneypyramids.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address looneypyramids.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:12:11] PROBLEM - holostars.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address holostars.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:12:14] PROBLEM - fnafmw.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address fnafmw.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:12:16] PROBLEM - evilgeniuswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address evilgeniuswiki.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:12:17] PROBLEM - opendatascot.org - LetsEncrypt on sslhost is CRITICAL: connect to address opendatascot.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:12:39] PROBLEM - wonderfuleveryday.org - LetsEncrypt on sslhost is CRITICAL: connect to address wonderfuleveryday.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:12:43] PROBLEM - clubpenguinfanon.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address clubpenguinfanon.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:12:44] PROBLEM - kingdomdeath.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address kingdomdeath.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:12:51] PROBLEM - *.wikitide.org - LetsEncrypt on sslhost is CRITICAL: connect to address wikitide.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:13:00] PROBLEM - wikimoma.art - LetsEncrypt on sslhost is CRITICAL: connect to address wikimoma.art and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:13:00] PROBLEM - lostmediawiki.ru - LetsEncrypt on sslhost is CRITICAL: connect to address lostmediawiki.ru and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:13:24] PROBLEM - worldsanskrit.net - LetsEncrypt on sslhost is CRITICAL: connect to address worldsanskrit.net and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:13:40] PROBLEM - sptwiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address sptwiki.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:13:40] PROBLEM - nonciclopedia.org - LetsEncrypt on sslhost is CRITICAL: connect to address nonciclopedia.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:13:53] PROBLEM - transgender.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address transgender.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:14:25] PROBLEM - kodiak.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address kodiak.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:14:35] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.25, 21.43, 17.97 [12:14:41] PROBLEM - rainverse.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address rainverse.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:14:46] PROBLEM - urbanshade.org - LetsEncrypt on sslhost is CRITICAL: connect to address urbanshade.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:15:10] PROBLEM - yoshipedia.com - LetsEncrypt on sslhost is CRITICAL: connect to address yoshipedia.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:15:14] PROBLEM - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is CRITICAL: connect to address marinebiodiversitymatrix.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:15:20] miraheze/DataDump - translatewiki the build passed. [12:15:28] miraheze/RemovePII - translatewiki the build passed. [12:15:31] PROBLEM - maxcapacity.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address maxcapacity.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:15:35] PROBLEM - podpedia.org - LetsEncrypt on sslhost is CRITICAL: connect to address podpedia.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:15:39] miraheze/WikiDiscover - translatewiki the build passed. [12:15:40] PROBLEM - issue-tracker.wikitide.org - LetsEncrypt on sslhost is CRITICAL: connect to address issue-tracker.wikitide.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:15:52] PROBLEM - journeytheword.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address journeytheword.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:15:55] miraheze/ManageWiki - translatewiki the build passed. [12:16:11] PROBLEM - terraforming.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address terraforming.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:16:20] PROBLEM - wikappedia.cc - LetsEncrypt on sslhost is CRITICAL: connect to address wikappedia.cc and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:16:39] PROBLEM - worldless.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address worldless.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:17:02] PROBLEM - fanojo.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address fanojo.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:17:16] PROBLEM - project-patterns.com - LetsEncrypt on sslhost is CRITICAL: connect to address project-patterns.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:17:17] PROBLEM - weavefarers.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address weavefarers.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:17:53] PROBLEM - oxboxtra.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address oxboxtra.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:18:20] PROBLEM - poupedia.com - LetsEncrypt on sslhost is CRITICAL: connect to address poupedia.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:18:26] PROBLEM - roll-of-arms.com - LetsEncrypt on sslhost is CRITICAL: connect to address roll-of-arms.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:18:29] PROBLEM - speleo.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address speleo.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:18:32] PROBLEM - bristolstudenthousingcoop.org - LetsEncrypt on sslhost is CRITICAL: connect to address bristolstudenthousingcoop.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:18:35] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 18.52, 20.10, 18.23 [12:18:48] PROBLEM - keitaiwiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address keitaiwiki.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:18:58] PROBLEM - cnidaria.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address cnidaria.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:19:03] PROBLEM - fashiondreamer.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address fashiondreamer.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [12:19:07] RECOVERY - cp36 Nginx Backend for mw154 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8122 [12:19:08] RECOVERY - cp36 Nginx Backend for mw183 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8127 [12:19:08] RECOVERY - cp36 Nginx Backend for mw162 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8116 [12:19:10] RECOVERY - cp36 Nginx Backend for mw161 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8115 [12:19:11] RECOVERY - cp36 Nginx Backend for mw153 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8121 [12:19:14] RECOVERY - cp36 Nginx Backend for test151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8181 [12:19:18] RECOVERY - cp36 Nginx Backend for mw174 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8126 [12:19:26] RECOVERY - cp36 Nginx Backend for mw163 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8123 [12:19:28] RECOVERY - cp36 Nginx Backend for mw171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8117 [12:19:36] RECOVERY - cp36 Nginx Backend for mw172 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8118 [12:19:37] RECOVERY - cp36 Nginx Backend for mwtask171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8161 [12:19:37] RECOVERY - cp36 Nginx Backend for mw182 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8120 [12:19:38] RECOVERY - cp36 Nginx Backend for puppet181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8204 [12:19:41] RECOVERY - cp36 Nginx Backend for mwtask151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8162 [12:19:45] RECOVERY - cp36 Nginx Backend for mw181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8119 [12:19:53] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [12:19:59] RECOVERY - cp36 Nginx Backend for phorge171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8202 [12:19:59] RECOVERY - cp36 Nginx Backend for mw173 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8125 [12:20:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:20:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:20:05] RECOVERY - cp36 Nginx Backend for mon181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8201 [12:20:11] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4111 bytes in 4.457 second response time [12:20:22] RECOVERY - cp36 Nginx Backend for mwtask181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8160 [12:20:22] RECOVERY - cp36 Nginx Backend for swiftproxy161 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8206 [12:20:22] RECOVERY - cp36 Nginx Backend for mw152 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8114 [12:20:32] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 16% [12:20:36] RECOVERY - cp36 Nginx Backend for reports171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8205 [12:20:37] RECOVERY - cp36 Nginx Backend for mw184 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8128 [12:20:38] RECOVERY - cp36 Nginx Backend for swiftproxy171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8207 [12:20:46] RECOVERY - cp36 Nginx Backend for mwtask161 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8163 [12:20:49] RECOVERY - cp36 Nginx Backend for matomo151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8203 [12:20:49] RECOVERY - cp36 Nginx Backend for mw151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8113 [12:21:03] RECOVERY - cp36 Nginx Backend for mw164 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8124 [12:22:40] miraheze/ImportDump - translatewiki the build passed. [12:22:49] miraheze/CreateWiki - translatewiki the build passed. [12:23:00] miraheze/MirahezeMagic - translatewiki the build passed. [12:23:53] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw181 [12:25:00] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 3 backends are down. mw163 mw174 mw184 [12:25:02] PROBLEM - mw154 Current Load on mw154 is WARNING: LOAD WARNING - total load average: 21.02, 16.58, 10.78 [12:25:35] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.60, 17.32, 11.08 [12:25:48] PROBLEM - mw153 Current Load on mw153 is WARNING: LOAD WARNING - total load average: 22.87, 17.63, 11.66 [12:25:52] PROBLEM - mw164 Current Load on mw164 is WARNING: LOAD WARNING - total load average: 22.74, 16.96, 10.92 [12:25:58] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.96, 17.88, 11.68 [12:26:56] PROBLEM - psycho.engineering - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'psycho.engineering' expires in 14 day(s) (Tue 11 Mar 2025 12:21:00 PM GMT +0000). [12:27:02] RECOVERY - mw154 Current Load on mw154 is OK: LOAD OK - total load average: 15.41, 16.40, 11.46 [12:27:10] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/26f61ca88f4122dc96078950ba87293dec8ff4ee [12:27:10] 02ssl/03master 07WikiTideSSLBot 0326f61ca Bot: Update SSL cert for psycho.engineering [12:27:48] RECOVERY - mw153 Current Load on mw153 is OK: LOAD OK - total load average: 17.40, 16.71, 12.01 [12:27:52] RECOVERY - mw164 Current Load on mw164 is OK: LOAD OK - total load average: 16.45, 17.08, 11.74 [12:27:58] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 13.62, 17.22, 12.26 [12:28:52] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [12:29:35] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 19.61, 17.12, 12.32 [12:30:54] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4114 bytes in 4.611 second response time [12:35:06] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [12:37:32] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 13.99, 11.18, 8.21 [12:37:45] RECOVERY - wikitide.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Fri 28 Mar 2025 03:25:53 AM GMT +0000. [12:38:06] RECOVERY - balloonfightwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'balloonfightwiki.org' will expire on Sun 06 Apr 2025 03:42:27 PM GMT +0000. [12:38:06] RECOVERY - creativecommons.id - LetsEncrypt on sslhost is OK: OK - Certificate 'creativecommons.id' will expire on Fri 23 May 2025 05:22:27 PM GMT +0000. [12:38:06] RECOVERY - archivesofhavnor.org - LetsEncrypt on sslhost is OK: OK - Certificate 'archivesofhavnor.org' will expire on Thu 24 Apr 2025 01:55:22 PM GMT +0000. [12:38:19] RECOVERY - fairytail.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fairytail.wiki' will expire on Thu 20 Mar 2025 09:03:07 PM GMT +0000. [12:38:19] RECOVERY - bebaskanpengetahuan.id - LetsEncrypt on sslhost is OK: OK - Certificate 'bebaskanpengetahuan.id' will expire on Wed 21 May 2025 10:11:11 AM GMT +0000. [12:38:36] RECOVERY - thestarsareright.org - LetsEncrypt on sslhost is OK: OK - Certificate 'thestarsareright.org' will expire on Thu 22 May 2025 10:38:46 AM GMT +0000. [12:38:38] RECOVERY - rct.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'rct.wiki' will expire on Thu 22 May 2025 09:01:22 AM GMT +0000. [12:38:39] RECOVERY - yokaiwatchwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'yokaiwatchwiki.com' will expire on Fri 14 Mar 2025 02:31:09 PM GMT +0000. [12:39:10] RECOVERY - metroidpedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'metroidpedia.com' will expire on Fri 23 May 2025 03:01:56 PM GMT +0000. [12:39:16] RECOVERY - rosettacode.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rosettacode.org' will expire on Sun 06 Apr 2025 01:20:04 AM GMT +0000. [12:39:20] RECOVERY - puzzles.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'puzzles.wiki' will expire on Tue 08 Apr 2025 10:21:57 PM GMT +0000. [12:39:29] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 11.73, 11.21, 8.57 [12:39:33] PROBLEM - electowiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'electowiki.org' expires in 14 day(s) (Tue 11 Mar 2025 12:09:09 PM GMT +0000). [12:39:44] RECOVERY - voecwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'voecwiki.com' will expire on Tue 25 Mar 2025 12:53:32 AM GMT +0000. [12:39:45] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/e8dcd2b9aa5add251f92cf98b631016917b04425 [12:39:45] 02ssl/03master 07WikiTideSSLBot 03e8dcd2b Bot: Update SSL cert for electowiki.org [12:39:46] RECOVERY - apeirology.com - LetsEncrypt on sslhost is OK: OK - Certificate 'apeirology.com' will expire on Fri 23 May 2025 03:52:38 PM GMT +0000. [12:39:48] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [12:39:58] RECOVERY - yepedia.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'yepedia.xyz' will expire on Thu 24 Apr 2025 02:10:34 PM GMT +0000. [12:40:14] RECOVERY - theenglishlakes.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'theenglishlakes.uk' will expire on Thu 17 Apr 2025 05:44:53 PM GMT +0000. [12:41:01] RECOVERY - looneypyramids.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'looneypyramids.wiki' will expire on Fri 23 May 2025 07:45:51 AM GMT +0000. [12:41:17] PROBLEM - evilgeniuswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'evilgeniuswiki.com' expires in 14 day(s) (Tue 11 Mar 2025 12:24:51 PM GMT +0000). [12:41:23] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4114 bytes in 5.262 second response time [12:41:23] RECOVERY - wonderfuleveryday.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wonderfuleveryday.org' will expire on Fri 02 May 2025 01:12:45 AM GMT +0000. [12:41:26] RECOVERY - clubpenguinfanon.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'clubpenguinfanon.wiki' will expire on Tue 25 Mar 2025 11:01:22 PM GMT +0000. [12:41:27] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 6.30, 9.82, 8.42 [12:41:28] RECOVERY - holostars.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'holostars.wiki' will expire on Mon 17 Mar 2025 04:06:52 PM GMT +0000. [12:41:29] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/14cf019dd8cd370e0353aeb887b67cb680373643 [12:41:29] 02ssl/03master 07WikiTideSSLBot 0314cf019 Bot: Update SSL cert for evilgeniuswiki.com [12:41:49] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4113 bytes in 6.644 second response time [12:42:00] RECOVERY - wikimoma.art - LetsEncrypt on sslhost is OK: OK - Certificate 'wikimoma.art' will expire on Fri 14 Mar 2025 02:42:04 PM GMT +0000. [12:42:00] RECOVERY - lostmediawiki.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'lostmediawiki.ru' will expire on Mon 07 Apr 2025 01:25:09 PM GMT +0000. [12:42:05] RECOVERY - opendatascot.org - LetsEncrypt on sslhost is OK: OK - Certificate 'opendatascot.org' will expire on Wed 12 Mar 2025 08:48:09 AM GMT +0000. [12:42:06] RECOVERY - fnafmw.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fnafmw.wiki' will expire on Thu 15 May 2025 11:15:17 PM GMT +0000. [12:42:25] RECOVERY - kingdomdeath.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kingdomdeath.wiki' will expire on Fri 14 Mar 2025 08:40:52 AM GMT +0000. [12:42:42] RECOVERY - *.wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Sat 12 Apr 2025 07:36:30 PM GMT +0000. [12:42:48] RECOVERY - worldsanskrit.net - LetsEncrypt on sslhost is OK: OK - Certificate 'worldsanskrit.net' will expire on Fri 14 Mar 2025 02:35:17 PM GMT +0000. [12:43:19] RECOVERY - kodiak.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kodiak.wiki' will expire on Wed 09 Apr 2025 02:48:10 AM GMT +0000. [12:43:20] RECOVERY - sptwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'sptwiki.org' will expire on Fri 11 Apr 2025 05:33:12 PM GMT +0000. [12:43:20] RECOVERY - nonciclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nonciclopedia.org' will expire on Wed 21 May 2025 06:04:48 AM GMT +0000. [12:43:44] RECOVERY - transgender.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'transgender.wiki' will expire on Sun 30 Mar 2025 04:29:43 PM GMT +0000. [12:44:01] RECOVERY - maxcapacity.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'maxcapacity.wiki' will expire on Sun 04 May 2025 02:42:43 PM GMT +0000. [12:44:19] RECOVERY - issue-tracker.wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Sat 12 Apr 2025 07:36:30 PM GMT +0000. [12:44:26] RECOVERY - rainverse.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'rainverse.wiki' will expire on Mon 21 Apr 2025 12:52:28 AM GMT +0000. [12:44:31] RECOVERY - urbanshade.org - LetsEncrypt on sslhost is OK: OK - Certificate 'urbanshade.org' will expire on Mon 24 Mar 2025 02:48:59 PM GMT +0000. [12:44:43] RECOVERY - journeytheword.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'journeytheword.wiki' will expire on Tue 08 Apr 2025 10:26:03 PM GMT +0000. [12:44:45] RECOVERY - podpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'podpedia.org' will expire on Wed 21 May 2025 06:59:07 PM GMT +0000. [12:44:50] RECOVERY - yoshipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'yoshipedia.com' will expire on Thu 17 Apr 2025 06:15:41 PM GMT +0000. [12:44:59] RECOVERY - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Tue 20 May 2025 09:01:38 PM GMT +0000. [12:45:21] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 27147MiB (5% inode=99%); [12:45:22] RECOVERY - terraforming.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'terraforming.wiki' will expire on Wed 07 May 2025 06:27:23 PM GMT +0000. [12:45:33] RECOVERY - fanojo.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fanojo.wiki' will expire on Fri 28 Mar 2025 03:19:12 AM GMT +0000. [12:45:39] RECOVERY - wikappedia.cc - LetsEncrypt on sslhost is OK: OK - Certificate 'wikappedia.cc' will expire on Sun 23 Mar 2025 01:58:38 PM GMT +0000. [12:46:01] RECOVERY - project-patterns.com - LetsEncrypt on sslhost is OK: OK - Certificate 'project-patterns.com' will expire on Tue 22 Apr 2025 01:04:41 AM GMT +0000. [12:46:07] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [12:46:18] RECOVERY - worldless.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'worldless.wiki' will expire on Tue 06 May 2025 06:11:44 AM GMT +0000. [12:46:43] RECOVERY - weavefarers.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'weavefarers.wiki' will expire on Mon 05 May 2025 12:07:17 AM GMT +0000. [12:47:15] RECOVERY - oxboxtra.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'oxboxtra.wiki' will expire on Mon 14 Apr 2025 06:14:39 AM GMT +0000. [12:47:36] RECOVERY - keitaiwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'keitaiwiki.com' will expire on Fri 14 Mar 2025 02:33:57 PM GMT +0000. [12:47:48] RECOVERY - roll-of-arms.com - LetsEncrypt on sslhost is OK: OK - Certificate 'roll-of-arms.com' will expire on Thu 22 May 2025 11:08:14 PM GMT +0000. [12:48:05] RECOVERY - cnidaria.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'cnidaria.wiki' will expire on Sun 04 May 2025 05:19:11 PM GMT +0000. [12:48:05] RECOVERY - fashiondreamer.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fashiondreamer.wiki' will expire on Fri 14 Mar 2025 01:50:18 PM GMT +0000. [12:48:07] RECOVERY - speleo.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'speleo.wiki' will expire on Tue 20 May 2025 08:21:38 PM GMT +0000. [12:48:15] RECOVERY - poupedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'poupedia.com' will expire on Mon 28 Apr 2025 01:12:42 PM GMT +0000. [12:48:31] RECOVERY - bristolstudenthousingcoop.org - LetsEncrypt on sslhost is OK: OK - Certificate 'bristolstudenthousingcoop.org' will expire on Wed 21 May 2025 09:47:11 AM GMT +0000. [12:49:45] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [12:50:11] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4113 bytes in 0.238 second response time [12:51:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:51:42] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4114 bytes in 0.089 second response time [12:52:18] what now [12:53:00] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [12:53:53] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [12:55:58] RECOVERY - psycho.engineering - LetsEncrypt on sslhost is OK: OK - Certificate 'psycho.engineering' will expire on Sun 25 May 2025 11:28:33 AM GMT +0000. [12:56:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:09:08] RECOVERY - electowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'electowiki.org' will expire on Sun 25 May 2025 11:41:08 AM GMT +0000. [13:10:19] RECOVERY - evilgeniuswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'evilgeniuswiki.com' will expire on Sun 25 May 2025 11:42:53 AM GMT +0000. [13:41:35] PROBLEM - index.pdcommunity.ir - Cloudflare on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [14:10:26] RECOVERY - index.pdcommunity.ir - Cloudflare on sslhost is OK: OK - Certificate 'index.pdcommunity.ir' will expire on Sat 24 May 2025 03:29:01 PM GMT +0000. [14:20:42] PROBLEM - vesc.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vesc.wiki' expires in 14 day(s) (Tue 11 Mar 2025 01:54:22 PM GMT +0000). [14:20:55] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/86c9ee7cc845c09f0146fb29de7b827cb8f17d4d [14:20:55] 02ssl/03master 07WikiTideSSLBot 0386c9ee7 Bot: Update SSL cert for vesc.wiki [14:48:55] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 22.76, 13.43, 5.97 [14:49:54] RECOVERY - vesc.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'vesc.wiki' will expire on Sun 25 May 2025 01:22:17 PM GMT +0000. [14:52:55] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 19.92, 17.58, 9.45 [15:25:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 27.60, 13.82, 5.97 [15:27:05] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.42, 16.46, 7.96 [16:23:51] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.16, 6.80, 5.75 [16:27:40] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 6.62, 6.68, 5.96 [16:34:22] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.69, 7.00, 6.38 [16:36:17] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 5.39, 6.51, 6.28 [17:29:07] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.26, 6.88, 6.57 [17:31:07] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 5.77, 6.61, 6.52 [17:33:43] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 24.94, 22.29, 15.26 [17:34:39] PROBLEM - wiki.case-clicker.com - Cloudflare on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [17:39:34] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 18.09, 22.23, 17.65 [17:41:31] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 3.51, 15.38, 15.68 [19:13:21] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 53406MiB (11% inode=99%); [20:08:35] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.41, 15.95, 8.90 [20:12:35] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 15.68, 18.48, 11.71 [20:41:09] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 26.92, 17.85, 10.48 [20:43:09] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 12.39, 16.79, 11.08 [20:57:21] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49792MiB (10% inode=99%); [21:48:39] RECOVERY - os161 Disk Space on os161 is OK: DISK OK - free space: / 69947MiB (31% inode=99%); [21:50:06] RECOVERY - os151 Disk Space on os151 is OK: DISK OK - free space: / 210038MiB (94% inode=99%); [21:56:35] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 29.78, 20.08, 10.81 [22:01:01] [02CreateWiki] 07Universal-Omega created 03schema-vars (+1 new commit) 13https://github.com/miraheze/CreateWiki/commit/78d9ecf8f585 [22:01:01] 02CreateWiki/03schema-vars 07CosmicAlpha 0378d9ecf Apply SchemaVars when creating databases [22:01:07] [02CreateWiki] 07Universal-Omega opened pull request #673: Apply SchemaVars when creating databases (03master...03schema-vars) 13https://github.com/miraheze/CreateWiki/pull/673 [22:01:16] [02CreateWiki] 07coderabbitai[bot] commented on pull request #673: --- […] 13https://github.com/miraheze/CreateWiki/pull/673#issuecomment-2679753672 [22:05:50] [02CreateWiki] 07Universal-Omega commented on pull request #673: Seems like this might actually already do this. So maybe this isn't needed actually... https://github.com/wikimedia/mediawiki/blob/3abc64a/includes/libs/rdbms/database/Database.php#L2796 13https://github.com/miraheze/CreateWiki/pull/673#issuecomment-2679762615 [22:13:30] miraheze/CreateWiki - Universal-Omega the build passed. [22:22:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [22:43:24] [02CreateWiki] 07dependabot[bot] created 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0 (+1 new commit) 13https://github.com/miraheze/CreateWiki/commit/05b93e697af5 [22:43:24] 02CreateWiki/03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0 07dependabot[bot] 0305b93e6 Bump eslint-config-wikimedia from 0.28.2 to 0.29.0… [22:43:25] [02CreateWiki] 07dependabot[bot] added the label 'dependencies' to pull request #674 (Bump eslint-config-wikimedia from 0.28.2 to 0.29.0) 13https://github.com/miraheze/CreateWiki/pull/674 [22:43:27] [02CreateWiki] 07dependabot[bot] added the label 'javascript' to pull request #674 (Bump eslint-config-wikimedia from 0.28.2 to 0.29.0) 13https://github.com/miraheze/CreateWiki/pull/674 [22:43:29] [02CreateWiki] 07dependabot[bot] opened pull request #674: Bump eslint-config-wikimedia from 0.28.2 to 0.29.0 (03master...03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0) 13https://github.com/miraheze/CreateWiki/pull/674 [22:43:32] [02CreateWiki] 07coderabbitai[bot] commented on pull request #674: --- […] 13https://github.com/miraheze/CreateWiki/pull/674#issuecomment-2679837610 [22:55:12] miraheze/CreateWiki - dependabot[bot] the build passed. [22:58:35] [02CreateWiki] 07BlankEclair approved pull request #663 13https://github.com/miraheze/CreateWiki/pull/663#pullrequestreview-2638641297 [23:10:20] [02CreateWiki] 07Universal-Omega merged 07dependabot[bot]'s pull request #674: Bump eslint-config-wikimedia from 0.28.2 to 0.29.0 (03master...03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0) 13https://github.com/miraheze/CreateWiki/pull/674 [23:10:22] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/CreateWiki/commit/f0963f2fed0467767477e8a92bed1b1706d5d6fe [23:10:24] [02CreateWiki] 07Universal-Omega 04deleted 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0 at 0305b93e6 13https://github.com/miraheze/CreateWiki/commit/05b93e6 [23:10:25] 02CreateWiki/03master 07dependabot[bot] 03f0963f2 Bump eslint-config-wikimedia from 0.28.2 to 0.29.0 (#674)… [23:17:54] [02IncidentReporting] 07dependabot[bot] created 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0 (+1 new commit) 13https://github.com/miraheze/IncidentReporting/commit/22521b5d53e4 [23:17:54] 02IncidentReporting/03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0 07dependabot[bot] 0322521b5 Bump eslint-config-wikimedia from 0.28.2 to 0.29.0… [23:17:55] [02IncidentReporting] 07dependabot[bot] opened pull request #99: Bump eslint-config-wikimedia from 0.28.2 to 0.29.0 (03master...03dependabot/npm_and_yarn/eslint-config-wikimedia-0.29.0) 13https://github.com/miraheze/IncidentReporting/pull/99 [23:17:56] [02IncidentReporting] 07dependabot[bot] added the label 'dependencies' to pull request #99 (Bump eslint-config-wikimedia from 0.28.2 to 0.29.0) 13https://github.com/miraheze/IncidentReporting/pull/99 [23:17:58] [02IncidentReporting] 07dependabot[bot] added the label 'javascript' to pull request #99 (Bump eslint-config-wikimedia from 0.28.2 to 0.29.0) 13https://github.com/miraheze/IncidentReporting/pull/99 [23:18:02] [02IncidentReporting] 07coderabbitai[bot] commented on pull request #99: --- […] 13https://github.com/miraheze/IncidentReporting/pull/99#issuecomment-2679907628 [23:22:05] miraheze/CreateWiki - Universal-Omega the build passed. [23:22:35] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 15.42, 18.91, 23.95 [23:22:53] miraheze/IncidentReporting - dependabot[bot] the build passed. [23:26:35] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 2.10, 10.88, 19.65 [23:42:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1