[00:01:20] PROBLEM - swiftobject151 Disk Space on swiftobject151 is WARNING: DISK WARNING - free space: / 80341MiB (6% inode=85%); [01:01:36] PROBLEM - worldtriggerwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'worldtriggerwiki.com' expires in 14 day(s) (Tue 31 Dec 2024 12:40:15 AM GMT +0000). [01:01:54] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/d4f8e38d17b551d70c425145d50e9adfe3ab5d87 [01:01:54] 02ssl/03master 07WikiTideSSLBot 03d4f8e38 Bot: Update SSL cert for worldtriggerwiki.com [01:05:37] PROBLEM - cp37 Current Load on cp37 is CRITICAL: LOAD CRITICAL - total load average: 15.22, 10.15, 4.40 [01:05:55] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is WARNING: WARNING - NGINX Error Rate is 59% [01:06:05] PROBLEM - cp36 Current Load on cp36 is CRITICAL: LOAD CRITICAL - total load average: 11.68, 7.99, 3.64 [01:07:55] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is CRITICAL: CRITICAL - NGINX Error Rate is 96% [01:09:55] RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 32% [01:12:05] RECOVERY - cp36 Current Load on cp36 is OK: LOAD OK - total load average: 2.51, 6.69, 4.73 [01:13:37] RECOVERY - cp37 Current Load on cp37 is OK: LOAD OK - total load average: 1.53, 5.99, 5.09 [01:31:12] RECOVERY - worldtriggerwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'worldtriggerwiki.com' will expire on Sun 16 Mar 2025 12:03:12 AM GMT +0000. [03:47:20] PROBLEM - swiftobject151 Disk Space on swiftobject151 is CRITICAL: DISK CRITICAL - free space: / 80122MiB (5% inode=85%); [04:09:20] PROBLEM - swiftobject151 Disk Space on swiftobject151 is WARNING: DISK WARNING - free space: / 80137MiB (6% inode=85%); [04:23:20] PROBLEM - swiftobject151 Disk Space on swiftobject151 is CRITICAL: DISK CRITICAL - free space: / 80123MiB (5% inode=85%); [04:28:47] sir [04:28:55] you went down [04:29:11] by 14 MiB [04:29:18] and went from warning [04:29:20] PROBLEM - swiftobject151 Disk Space on swiftobject151 is WARNING: DISK WARNING - free space: / 80131MiB (6% inode=85%); [04:29:29] to critical- [04:29:31] okay [04:30:29] MacFan4000 can you tell icinga critical is more critical then warning [04:32:44] Mmm, might need to keep a closer eye on it [04:33:47] If I see it get low enough, then I’ll push the phorge task to UBN [04:35:20] MacFan4000 you mean high enough [04:37:02] The number in the alert is the space remaining [04:40:55] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildall.php --wiki=deathbattlewiki (END - exit=2) [04:40:59] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/initSiteStats.php --wiki=deathbattlewiki --update (END - exit=0) [04:41:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:41:02] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importDump.php --wiki=deathbattlewiki deathbattle_pages_full.xml --no-updates (START) [04:41:04] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importDump.php --wiki=deathbattlewiki deathbattle_pages_full.xml --no-updates (END - exit=2) [04:41:05] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildall.php --wiki=deathbattlewiki (START) [04:41:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:41:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:41:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:41:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:43:20] PROBLEM - swiftobject151 Disk Space on swiftobject151 is CRITICAL: DISK CRITICAL - free space: / 80125MiB (5% inode=85%); [05:05:37] PROBLEM - cp37 Current Load on cp37 is CRITICAL: LOAD CRITICAL - total load average: 14.47, 7.63, 3.19 [05:05:45] phorge forgetting CSS exists for anyone else [05:05:55] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 38.46.223.205/cpweb, 38.46.223.206/cpweb, 2602:294:0:b13::110/cpweb, 2602:294:0:b23::112/cpweb [05:06:05] PROBLEM - cp36 Current Load on cp36 is CRITICAL: LOAD CRITICAL - total load average: 19.20, 8.31, 3.42 [05:06:42] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 38.46.223.205/cpweb, 38.46.223.206/cpweb, 2602:294:0:b13::110/cpweb [05:07:03] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 50% [05:07:04] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [05:07:33] hm [05:07:50] does phorge happen to use varnish? [05:09:00] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to cp36.wikitide.net port 443 after 0 ms: Couldn't connect to server [05:09:03] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [05:09:13] PROBLEM - cp36 Nginx Backend for puppet181 on cp36 is CRITICAL: connect to address localhost and port 8204: Connection refused [05:09:21] PROBLEM - cp36 Nginx Backend for mw171 on cp36 is CRITICAL: connect to address localhost and port 8117: Connection refused [05:09:23] PROBLEM - cp36 Nginx Backend for mw153 on cp36 is CRITICAL: connect to address localhost and port 8121: Connection refused [05:09:29] PROBLEM - cp36 Nginx Backend for mw184 on cp36 is CRITICAL: connect to address localhost and port 8128: Connection refused [05:09:30] PROBLEM - cp36 Nginx Backend for mw152 on cp36 is CRITICAL: connect to address localhost and port 8114: Connection refused [05:09:31] PROBLEM - cp36 Nginx Backend for mw162 on cp36 is CRITICAL: connect to address localhost and port 8116: Connection refused [05:09:35] PROBLEM - cp36 Nginx Backend for swiftproxy171 on cp36 is CRITICAL: connect to address localhost and port 8207: Connection refused [05:09:43] PROBLEM - cp36 Nginx Backend for swiftproxy161 on cp36 is CRITICAL: connect to address localhost and port 8206: Connection refused [05:09:49] PROBLEM - cp36 Nginx Backend for mw181 on cp36 is CRITICAL: connect to address localhost and port 8119: Connection refused [05:09:49] It is being served by CF, if you look at the ssl, its a google cert [05:09:53] PROBLEM - cp36 Nginx Backend for mon181 on cp36 is CRITICAL: connect to address localhost and port 8201: Connection refused [05:09:53] PROBLEM - speleo.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:10:02] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 17 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw154 mw163 mw164 mw173 mw174 mw183 mw184 mediawiki [05:10:05] RECOVERY - cp36 Current Load on cp36 is OK: LOAD OK - total load average: 1.74, 6.66, 4.11 [05:10:06] PROBLEM - cp36 Nginx Backend for mw164 on cp36 is CRITICAL: connect to address localhost and port 8124: Connection refused [05:10:11] PROBLEM - cp36 Nginx Backend for mw172 on cp36 is CRITICAL: connect to address localhost and port 8118: Connection refused [05:10:11] PROBLEM - cp36 Nginx Backend for reports171 on cp36 is CRITICAL: connect to address localhost and port 8205: Connection refused [05:10:13] PROBLEM - cp36 Nginx Backend for test151 on cp36 is CRITICAL: connect to address localhost and port 8181: Connection refused [05:10:16] PROBLEM - cp36 Nginx Backend for phorge171 on cp36 is CRITICAL: connect to address localhost and port 8202: Connection refused [05:10:18] PROBLEM - cp36 Nginx Backend for mwtask181 on cp36 is CRITICAL: connect to address localhost and port 8160: Connection refused [05:10:21] PROBLEM - cp36 Nginx Backend for mw182 on cp36 is CRITICAL: connect to address localhost and port 8120: Connection refused [05:10:22] PROBLEM - cp36 Nginx Backend for mw173 on cp36 is CRITICAL: connect to address localhost and port 8125: Connection refused [05:10:32] PROBLEM - cp36 Nginx Backend for mw154 on cp36 is CRITICAL: connect to address localhost and port 8122: Connection refused [05:10:33] PROBLEM - cp36 Nginx Backend for mwtask151 on cp36 is CRITICAL: connect to address localhost and port 8162: Connection refused [05:10:35] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 17 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw154 mw163 mw164 mw173 mw174 mw183 mw184 mediawiki [05:10:35] PROBLEM - cp36 Nginx Backend for mw161 on cp36 is CRITICAL: connect to address localhost and port 8115: Connection refused [05:10:38] PROBLEM - cp36 Nginx Backend for mw151 on cp36 is CRITICAL: connect to address localhost and port 8113: Connection refused [05:10:44] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is CRITICAL: CRITICAL - NGINX Error Rate is 99% [05:10:47] PROBLEM - cp36 Nginx Backend for mw174 on cp36 is CRITICAL: connect to address localhost and port 8126: Connection refused [05:10:47] PROBLEM - cp36 Nginx Backend for mwtask161 on cp36 is CRITICAL: connect to address localhost and port 8163: Connection refused [05:10:49] PROBLEM - cp36 Nginx Backend for mwtask171 on cp36 is CRITICAL: connect to address localhost and port 8161: Connection refused [05:10:51] NGINX is collasping. again [05:10:53] PROBLEM - icclopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:10:58] PROBLEM - cp36 Nginx Backend for mw183 on cp36 is CRITICAL: connect to address localhost and port 8127: Connection refused [05:11:02] PROBLEM - cp36 Nginx Backend for mw163 on cp36 is CRITICAL: connect to address localhost and port 8123: Connection refused [05:11:02] PROBLEM - cp36 Nginx Backend for matomo151 on cp36 is CRITICAL: connect to address localhost and port 8203: Connection refused [05:11:19] I just pinged the entire tech team [05:12:04] thank you mac [05:12:14] well back to watching the world burn [05:12:26] i think its worse this time [05:12:30] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [05:12:33] was Phorge affected before [05:12:38] wow [05:12:41] and grafana [05:12:59] snowstacked: are you wowing the volume of complaints [05:13:15] no idea why cp* going down would affect services that don't use it [05:13:20] no im wowing the problems [05:13:22] 😍 [05:13:29] must be so fun right [05:13:33] damn it stacked [05:13:37] RECOVERY - cp37 Current Load on cp37 is OK: LOAD OK - total load average: 2.55, 6.27, 4.72 [05:13:48] see my blessing has recovered it [05:13:52] you lost me a perfectly good joke poking fun at US electoral systems [05:13:59] oh [05:14:01] im sorry [05:14:18] i didnt mean to [05:14:23] >:( [05:14:31] PROBLEM - corru.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address corru.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:14:34] PROBLEM - dkwiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address dkwiki.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:14:36] oh no [05:14:43] so about that volume [05:14:47] PROBLEM - cp37 Nginx Backend for mw163 on cp37 is CRITICAL: connect to address localhost and port 8123: Connection refused [05:14:48] PROBLEM - gimkit.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address gimkit.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:14:49] PROBLEM - cp37 Nginx Backend for mw182 on cp37 is CRITICAL: connect to address localhost and port 8120: Connection refused [05:14:49] a ha ha ha [05:14:50] PROBLEM - sytbay.site - LetsEncrypt on sslhost is CRITICAL: connect to address sytbay.site and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:14:55] PROBLEM - cp37 Nginx Backend for mw174 on cp37 is CRITICAL: connect to address localhost and port 8126: Connection refused [05:14:57] ice spice blessings [05:14:59] MacFan4000: lets run [05:14:59] PROBLEM - ncuwiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address ncuwiki.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:15:05] PROBLEM - cp37 Nginx Backend for mw162 on cp37 is CRITICAL: connect to address localhost and port 8116: Connection refused [05:15:05] PROBLEM - cp37 Nginx Backend for mw152 on cp37 is CRITICAL: connect to address localhost and port 8114: Connection refused [05:15:07] PROBLEM - rothwell-leeds.co.uk - Cloudflare on sslhost is CRITICAL: connect to address rothwell-leeds.co.uk and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:15:10] PROBLEM - cp37 Nginx Backend for mw164 on cp37 is CRITICAL: connect to address localhost and port 8124: Connection refused [05:15:10] PROBLEM - cp37 Nginx Backend for mw153 on cp37 is CRITICAL: connect to address localhost and port 8121: Connection refused [05:15:14] PROBLEM - clubpenguinfanon.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address clubpenguinfanon.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:15:19] PROBLEM - 48plus.org - LetsEncrypt on sslhost is CRITICAL: connect to address 48plus.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:15:20] PROBLEM - cp37 Nginx Backend for mw173 on cp37 is CRITICAL: connect to address localhost and port 8125: Connection refused [05:15:20] PROBLEM - cp37 Nginx Backend for mwtask161 on cp37 is CRITICAL: connect to address localhost and port 8163: Connection refused [05:15:22] PROBLEM - bobobay.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address bobobay.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:15:23] PROBLEM - cp37 Nginx Backend for mon181 on cp37 is CRITICAL: connect to address localhost and port 8201: Connection refused [05:15:30] PROBLEM - pizzatower.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address pizzatower.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:15:31] PROBLEM - cp37 Nginx Backend for test151 on cp37 is CRITICAL: connect to address localhost and port 8181: Connection refused [05:15:33] PROBLEM - cp37 Nginx Backend for mw184 on cp37 is CRITICAL: connect to address localhost and port 8128: Connection refused [05:15:34] PROBLEM - cp37 Nginx Backend for mw183 on cp37 is CRITICAL: connect to address localhost and port 8127: Connection refused [05:15:36] PROBLEM - cp37 Nginx Backend for mw171 on cp37 is CRITICAL: connect to address localhost and port 8117: Connection refused [05:15:37] PROBLEM - wikimas.kr - LetsEncrypt on sslhost is CRITICAL: connect to address wikimas.kr and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:15:41] PROBLEM - thestarsareright.org - LetsEncrypt on sslhost is CRITICAL: connect to address thestarsareright.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:15:41] PROBLEM - pyramidgames.wiki - Cloudflare on sslhost is CRITICAL: connect to address pyramidgames.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:15:42] PROBLEM - cp37 Nginx Backend for mw151 on cp37 is CRITICAL: connect to address localhost and port 8113: Connection refused [05:15:46] PROBLEM - cp37 Nginx Backend for swiftproxy161 on cp37 is CRITICAL: connect to address localhost and port 8206: Connection refused [05:15:46] PROBLEM - cp37 Nginx Backend for mw154 on cp37 is CRITICAL: connect to address localhost and port 8122: Connection refused [05:15:47] PROBLEM - cp37 Nginx Backend for phorge171 on cp37 is CRITICAL: connect to address localhost and port 8202: Connection refused [05:15:49] PROBLEM - solaswiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address solaswiki.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:15:51] PROBLEM - cp37 Nginx Backend for swiftproxy171 on cp37 is CRITICAL: connect to address localhost and port 8207: Connection refused [05:15:53] PROBLEM - cp37 Nginx Backend for mw181 on cp37 is CRITICAL: connect to address localhost and port 8119: Connection refused [05:15:53] PROBLEM - cp37 Nginx Backend for mwtask171 on cp37 is CRITICAL: connect to address localhost and port 8161: Connection refused [05:15:54] PROBLEM - kingdomway.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address kingdomway.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:15:55] PROBLEM - cp37 Nginx Backend for reports171 on cp37 is CRITICAL: connect to address localhost and port 8205: Connection refused [05:15:57] PROBLEM - rippaversewiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address rippaversewiki.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:16:00] PROBLEM - theenglishlakes.uk - LetsEncrypt on sslhost is CRITICAL: connect to address theenglishlakes.uk and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:16:02] PROBLEM - cp37 Nginx Backend for matomo151 on cp37 is CRITICAL: connect to address localhost and port 8203: Connection refused [05:16:02] PROBLEM - cp37 Nginx Backend for mw161 on cp37 is CRITICAL: connect to address localhost and port 8115: Connection refused [05:16:05] PROBLEM - comprehensibleinputwiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address comprehensibleinputwiki.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:16:16] PROBLEM - cp37 Nginx Backend for mw172 on cp37 is CRITICAL: connect to address localhost and port 8118: Connection refused [05:16:19] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 17 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw154 mw163 mw164 mw173 mw174 mw183 mw184 mediawiki [05:16:22] PROBLEM - cp37 Nginx Backend for mwtask151 on cp37 is CRITICAL: connect to address localhost and port 8162: Connection refused [05:16:29] PROBLEM - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is CRITICAL: connect to address encyclopediaofastrobiology.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:16:30] PROBLEM - cp37 Nginx Backend for puppet181 on cp37 is CRITICAL: connect to address localhost and port 8204: Connection refused [05:16:31] PROBLEM - cp37 Nginx Backend for mwtask181 on cp37 is CRITICAL: connect to address localhost and port 8160: Connection refused [05:16:47] PROBLEM - kagaga.jp - LetsEncrypt on sslhost is CRITICAL: connect to address kagaga.jp and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:16:52] PROBLEM - n64brew.dev - LetsEncrypt on sslhost is CRITICAL: connect to address n64brew.dev and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:16:54] PROBLEM - persist.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address persist.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:17:03] PROBLEM - beidipedia.com - LetsEncrypt on sslhost is CRITICAL: connect to address beidipedia.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:17:15] PROBLEM - dccomicswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address dccomicswiki.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:17:16] PROBLEM - blackmagic.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address blackmagic.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:17:34] PROBLEM - rosettacode.org - LetsEncrypt on sslhost is CRITICAL: connect to address rosettacode.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:17:43] jesus fuck [05:17:52] can we just get like [05:18:03] PROBLEM - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is CRITICAL: connect to address marinebiodiversitymatrix.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:18:11] PROBLEM - maxcapacity.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address maxcapacity.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:18:14] PROBLEM - acuralegendwiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address acuralegendwiki.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:18:16] > icinga-miraheze: PROBLEM - yes - LetsEncrypt on sslhost is CRITICAL: connect to address all of them and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:18:22] PROBLEM - holocron.net - LetsEncrypt on sslhost is CRITICAL: connect to address holocron.net and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:18:30] PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: connect to address antiguabarbudacalypso.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [05:18:46] RECOVERY - cp36 Nginx Backend for mw174 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8126 [05:18:47] RECOVERY - cp36 Nginx Backend for mwtask161 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8163 [05:18:49] RECOVERY - cp36 Nginx Backend for mwtask171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8161 [05:18:58] RECOVERY - cp36 Nginx Backend for mw183 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8127 [05:19:02] RECOVERY - cp36 Nginx Backend for mw163 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8123 [05:19:03] RECOVERY - cp36 Nginx Backend for matomo151 on cp36 is OK: TCP OK - 0.004 second response time on localhost port 8203 [05:19:13] RECOVERY - cp36 Nginx Backend for puppet181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8204 [05:19:20] RECOVERY - cp36 Nginx Backend for mw171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8117 [05:19:23] RECOVERY - cp36 Nginx Backend for mw153 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8121 [05:19:29] RECOVERY - cp36 Nginx Backend for mw184 on cp36 is OK: TCP OK - 0.001 second response time on localhost port 8128 [05:19:30] RECOVERY - cp36 Nginx Backend for mw152 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8114 [05:19:31] RECOVERY - cp36 Nginx Backend for mw162 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8116 [05:19:35] RECOVERY - cp36 Nginx Backend for swiftproxy171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8207 [05:19:42] PROBLEM - buildabearwiki.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:19:43] RECOVERY - cp36 Nginx Backend for swiftproxy161 on cp36 is OK: TCP OK - 0.001 second response time on localhost port 8206 [05:19:49] RECOVERY - cp36 Nginx Backend for mw181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8119 [05:19:53] RECOVERY - cp36 Nginx Backend for mon181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8201 [05:20:06] RECOVERY - cp36 Nginx Backend for mw164 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8124 [05:20:11] RECOVERY - cp36 Nginx Backend for mw172 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8118 [05:20:11] RECOVERY - cp36 Nginx Backend for reports171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8205 [05:20:14] RECOVERY - cp36 Nginx Backend for test151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8181 [05:20:16] RECOVERY - cp36 Nginx Backend for phorge171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8202 [05:20:18] RECOVERY - cp36 Nginx Backend for mwtask181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8160 [05:20:21] RECOVERY - cp36 Nginx Backend for mw182 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8120 [05:20:22] RECOVERY - cp36 Nginx Backend for mw173 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8125 [05:20:33] RECOVERY - cp36 Nginx Backend for mw154 on cp36 is OK: TCP OK - 0.003 second response time on localhost port 8122 [05:20:33] RECOVERY - cp36 Nginx Backend for mwtask151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8162 [05:20:35] RECOVERY - cp36 Nginx Backend for mw161 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8115 [05:20:38] RECOVERY - cp36 Nginx Backend for mw151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8113 [05:22:05] PROBLEM - cp36 Current Load on cp36 is WARNING: LOAD WARNING - total load average: 6.99, 4.56, 3.51 [05:22:12] PROBLEM - lotrmc.ru - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:22:14] PROBLEM - edapedia.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:22:16] PROBLEM - osdev.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:23:50] PROBLEM - exmormon.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:24:05] PROBLEM - cp36 Current Load on cp36 is CRITICAL: LOAD CRITICAL - total load average: 8.85, 5.86, 4.10 [05:24:53] AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA [05:26:05] PROBLEM - cp36 Current Load on cp36 is WARNING: LOAD WARNING - total load average: 7.04, 6.23, 4.46 [05:27:29] PROBLEM - agesofconflict.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:28:47] RECOVERY - cp37 Nginx Backend for mw163 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8123 [05:28:49] RECOVERY - cp37 Nginx Backend for mw182 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8120 [05:28:55] RECOVERY - cp37 Nginx Backend for mw174 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8126 [05:28:58] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4140 bytes in 0.057 second response time [05:29:03] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 3% [05:29:05] RECOVERY - cp37 Nginx Backend for mw162 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8116 [05:29:05] RECOVERY - cp37 Nginx Backend for mw152 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8114 [05:29:12] RECOVERY - cp37 Nginx Backend for mw153 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8121 [05:29:13] RECOVERY - cp37 Nginx Backend for mw164 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8124 [05:29:20] RECOVERY - cp37 Nginx Backend for mw173 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8125 [05:29:20] RECOVERY - cp37 Nginx Backend for mwtask161 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8163 [05:29:23] RECOVERY - cp37 Nginx Backend for mon181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8201 [05:29:31] PixDeVI: looks like we are being DDOS'd [05:29:31] RECOVERY - cp37 Nginx Backend for test151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8181 [05:29:33] RECOVERY - cp37 Nginx Backend for mw184 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8128 [05:29:34] RECOVERY - cp37 Nginx Backend for mw183 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8127 [05:29:36] RECOVERY - cp37 Nginx Backend for mw171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8117 [05:29:42] RECOVERY - cp37 Nginx Backend for mw151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8113 [05:29:48] RECOVERY - cp37 Nginx Backend for swiftproxy161 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8206 [05:29:48] RECOVERY - cp37 Nginx Backend for mw154 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8122 [05:29:48] RECOVERY - cp37 Nginx Backend for phorge171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8202 [05:29:49] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [05:29:51] RECOVERY - cp37 Nginx Backend for swiftproxy171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8207 [05:29:53] RECOVERY - cp37 Nginx Backend for mwtask171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8161 [05:29:53] RECOVERY - cp37 Nginx Backend for mw181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8119 [05:29:54] PisDeVl: [05:29:54] RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 4% [05:29:55] RECOVERY - cp37 Nginx Backend for reports171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8205 [05:29:55] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:30:02] RECOVERY - cp37 Nginx Backend for matomo151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8203 [05:30:02] RECOVERY - cp37 Nginx Backend for mw161 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8115 [05:30:02] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [05:30:05] RECOVERY - cp36 Current Load on cp36 is OK: LOAD OK - total load average: 1.70, 4.70, 4.32 [05:30:06] PixDeVl: [05:30:16] RECOVERY - cp37 Nginx Backend for mw172 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8118 [05:30:22] RECOVERY - cp37 Nginx Backend for mwtask151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8162 [05:30:33] ho ho ho! [05:30:35] RECOVERY - cp37 Nginx Backend for puppet181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8204 [05:30:35] RECOVERY - cp37 Nginx Backend for mwtask181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8160 [05:30:39] holiday traditions [05:30:42] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [05:31:01] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4139 bytes in 0.060 second response time [05:31:54] santa felt jolly today [05:35:39] PROBLEM - cp37 Current Load on cp37 is CRITICAL: LOAD CRITICAL - total load average: 12.19, 6.85, 3.88 [05:35:55] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 38.46.223.205/cpweb, 38.46.223.206/cpweb, 2602:294:0:b13::110/cpweb [05:36:05] PROBLEM - cp36 Current Load on cp36 is CRITICAL: LOAD CRITICAL - total load average: 16.78, 11.34, 7.06 [05:36:42] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 38.46.223.206/cpweb, 2602:294:0:b13::110/cpweb, 2602:294:0:b23::112/cpweb [05:37:16] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 141.57 ms [05:38:02] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [05:39:19] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 141.57 ms [05:39:51] RECOVERY - speleo.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'speleo.wiki' will expire on Thu 06 Mar 2025 09:11:56 PM GMT +0000. [05:40:19] RECOVERY - icclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'icclopedia.org' will expire on Sun 09 Mar 2025 02:26:09 AM GMT +0000. [05:40:59] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 8 backends are down. mw151 mw181 mw153 mw163 mw164 mw174 mw183 mw184 [05:41:31] https://github.com/miraheze/dns/blob/master/zones/wikitide.net#L159 [05:41:31] [GitHub] [miraheze/dns] zones/wikitide.net @ master | L159: phorge-static DYNA geoip!cp [05:41:40] that explains why phorge is affected [05:41:51] the static part of it still goes through cp* [05:41:54] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [05:41:55] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 2% [05:42:55] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [05:43:37] PROBLEM - cp37 Current Load on cp37 is WARNING: LOAD WARNING - total load average: 3.88, 7.07, 5.50 [05:43:54] RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 0% [05:43:55] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:43:58] RECOVERY - rothwell-leeds.co.uk - Cloudflare on sslhost is OK: OK - Certificate 'CloudFlare Origin Certificate' will expire on Sat 27 Aug 2039 04:37:00 PM GMT +0000. [05:44:09] RECOVERY - dkwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dkwiki.org' will expire on Sat 08 Mar 2025 03:26:18 PM GMT +0000. [05:44:22] RECOVERY - gimkit.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'gimkit.wiki' will expire on Wed 05 Mar 2025 02:35:49 PM GMT +0000. [05:44:23] RECOVERY - pyramidgames.wiki - Cloudflare on sslhost is OK: OK - Certificate 'CloudFlare Origin Certificate' will expire on Sat 27 Aug 2039 04:37:00 PM GMT +0000. [05:44:23] RECOVERY - thestarsareright.org - LetsEncrypt on sslhost is OK: OK - Certificate 'thestarsareright.org' will expire on Sat 08 Mar 2025 11:10:21 AM GMT +0000. [05:44:28] RECOVERY - ncuwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'ncuwiki.com' will expire on Sun 09 Mar 2025 05:24:06 PM GMT +0000. [05:44:29] RECOVERY - corru.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'corru.wiki' will expire on Wed 22 Jan 2025 02:58:47 AM GMT +0000. [05:44:37] RECOVERY - 48plus.org - LetsEncrypt on sslhost is OK: OK - Certificate '48plus.org' will expire on Wed 08 Jan 2025 04:02:38 PM GMT +0000. [05:44:42] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [05:44:43] RECOVERY - sytbay.site - LetsEncrypt on sslhost is OK: OK - Certificate 'sytbay.site' will expire on Mon 03 Feb 2025 10:49:41 PM GMT +0000. [05:44:47] RECOVERY - kingdomway.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kingdomway.wiki' will expire on Wed 29 Jan 2025 06:48:21 AM GMT +0000. [05:45:00] RECOVERY - theenglishlakes.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'theenglishlakes.uk' will expire on Sat 01 Feb 2025 06:17:35 PM GMT +0000. [05:45:04] RECOVERY - wikimas.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'wikimas.kr' will expire on Sat 08 Mar 2025 09:56:22 AM GMT +0000. [05:45:09] RECOVERY - pizzatower.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'pizzatower.wiki' will expire on Sat 15 Mar 2025 06:20:47 PM GMT +0000. [05:45:13] RECOVERY - clubpenguinfanon.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'clubpenguinfanon.wiki' will expire on Thu 09 Jan 2025 11:54:12 PM GMT +0000. [05:45:17] RECOVERY - bobobay.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'bobobay.wiki' will expire on Wed 12 Feb 2025 02:26:39 PM GMT +0000. [05:45:24] RECOVERY - rippaversewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'rippaversewiki.com' will expire on Tue 14 Jan 2025 01:24:58 PM GMT +0000. [05:45:28] RECOVERY - solaswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'solaswiki.org' will expire on Sun 09 Mar 2025 11:41:38 AM GMT +0000. [05:45:37] RECOVERY - cp37 Current Load on cp37 is OK: LOAD OK - total load average: 5.12, 6.03, 5.28 [05:45:58] RECOVERY - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediaofastrobiology.org' will expire on Fri 07 Mar 2025 08:41:41 AM GMT +0000. [05:46:07] RECOVERY - comprehensibleinputwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'comprehensibleinputwiki.org' will expire on Tue 11 Feb 2025 02:39:50 PM GMT +0000. [05:46:07] RECOVERY - blackmagic.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'blackmagic.wiki' will expire on Sun 23 Feb 2025 07:21:15 PM GMT +0000. [05:46:17] RECOVERY - kagaga.jp - LetsEncrypt on sslhost is OK: OK - Certificate 'kagaga.jp' will expire on Sat 04 Jan 2025 05:19:43 PM GMT +0000. [05:46:30] RECOVERY - dccomicswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dccomicswiki.com' will expire on Fri 28 Feb 2025 11:41:34 PM GMT +0000. [05:46:37] RECOVERY - rosettacode.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rosettacode.org' will expire on Tue 21 Jan 2025 01:55:49 AM GMT +0000. [05:46:46] RECOVERY - n64brew.dev - LetsEncrypt on sslhost is OK: OK - Certificate 'n64brew.dev' will expire on Fri 07 Mar 2025 06:15:58 PM GMT +0000. [05:46:46] RECOVERY - beidipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'beidipedia.com' will expire on Wed 22 Jan 2025 02:50:31 PM GMT +0000. [05:46:47] RECOVERY - persist.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'persist.wiki' will expire on Sat 08 Mar 2025 10:58:16 PM GMT +0000. [05:47:31] RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Sat 08 Mar 2025 06:18:35 PM GMT +0000. [05:47:41] RECOVERY - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Thu 06 Mar 2025 09:33:11 PM GMT +0000. [05:48:10] RECOVERY - maxcapacity.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'maxcapacity.wiki' will expire on Tue 18 Feb 2025 03:36:47 PM GMT +0000. [05:48:11] RECOVERY - acuralegendwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'acuralegendwiki.org' will expire on Sun 09 Mar 2025 02:34:56 AM GMT +0000. [05:48:14] RECOVERY - holocron.net - LetsEncrypt on sslhost is OK: OK - Certificate 'holocron.net' will expire on Mon 24 Feb 2025 03:12:55 AM GMT +0000. [05:48:42] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 38.46.223.206/cpweb, 2602:294:0:b23::112/cpweb [05:49:27] RECOVERY - buildabearwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'buildabearwiki.info' will expire on Sat 08 Mar 2025 11:49:05 PM GMT +0000. [05:49:31] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [05:49:37] PROBLEM - cp37 Current Load on cp37 is CRITICAL: LOAD CRITICAL - total load average: 19.80, 11.99, 7.74 [05:49:55] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2602:294:0:b23::112/cpweb [05:50:42] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [05:50:58] RECOVERY - edapedia.net - LetsEncrypt on sslhost is OK: OK - Certificate 'edapedia.net' will expire on Wed 22 Jan 2025 02:47:33 AM GMT +0000. [05:50:58] RECOVERY - lotrmc.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'lotrmc.ru' will expire on Sun 02 Feb 2025 04:49:27 PM GMT +0000. [05:51:33] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4162 bytes in 3.706 second response time [05:51:35] RECOVERY - osdev.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'osdev.wiki' will expire on Fri 07 Feb 2025 09:54:48 PM GMT +0000. [05:51:55] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:54:20] PROBLEM - familiacorsi.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:54:34] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [05:54:38] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 99% [05:54:42] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 38.46.223.205/cpweb, 38.46.223.206/cpweb, 2602:294:0:b13::110/cpweb, 2602:294:0:b23::112/cpweb [05:55:43] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to cp37.wikitide.net port 443 after 0 ms: Couldn't connect to server [05:55:55] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 38.46.223.205/cpweb, 38.46.223.206/cpweb, 2602:294:0:b13::110/cpweb, 2602:294:0:b23::112/cpweb [05:56:28] RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 38% [05:57:05] PROBLEM - cp37 Nginx Backend for mw163 on cp37 is CRITICAL: connect to address localhost and port 8123: Connection refused [05:57:05] PROBLEM - cp37 Nginx Backend for mw182 on cp37 is CRITICAL: connect to address localhost and port 8120: Connection refused [05:57:05] PROBLEM - cp37 Nginx Backend for mw174 on cp37 is CRITICAL: connect to address localhost and port 8126: Connection refused [05:57:05] PROBLEM - cp37 Nginx Backend for mw162 on cp37 is CRITICAL: connect to address localhost and port 8116: Connection refused [05:57:05] PROBLEM - cp37 Nginx Backend for mw152 on cp37 is CRITICAL: connect to address localhost and port 8114: Connection refused [05:57:10] PROBLEM - cp37 Nginx Backend for mw164 on cp37 is CRITICAL: connect to address localhost and port 8124: Connection refused [05:57:10] PROBLEM - cp37 Nginx Backend for mw153 on cp37 is CRITICAL: connect to address localhost and port 8121: Connection refused [05:57:21] PROBLEM - cp37 Nginx Backend for mw173 on cp37 is CRITICAL: connect to address localhost and port 8125: Connection refused [05:57:21] PROBLEM - cp37 Nginx Backend for mwtask161 on cp37 is CRITICAL: connect to address localhost and port 8163: Connection refused [05:57:23] PROBLEM - cp37 Nginx Backend for mon181 on cp37 is CRITICAL: connect to address localhost and port 8201: Connection refused [05:57:29] oh its downing [05:57:31] PROBLEM - cp37 Nginx Backend for test151 on cp37 is CRITICAL: connect to address localhost and port 8181: Connection refused [05:57:33] PROBLEM - cp37 Nginx Backend for mw184 on cp37 is CRITICAL: connect to address localhost and port 8128: Connection refused [05:57:34] PROBLEM - cp37 Nginx Backend for mw183 on cp37 is CRITICAL: connect to address localhost and port 8127: Connection refused [05:57:36] PROBLEM - cp37 Nginx Backend for mw171 on cp37 is CRITICAL: connect to address localhost and port 8117: Connection refused [05:57:42] PROBLEM - cp37 Nginx Backend for mw151 on cp37 is CRITICAL: connect to address localhost and port 8113: Connection refused [05:57:46] PROBLEM - cp37 Nginx Backend for swiftproxy161 on cp37 is CRITICAL: connect to address localhost and port 8206: Connection refused [05:57:46] PROBLEM - cp37 Nginx Backend for mw154 on cp37 is CRITICAL: connect to address localhost and port 8122: Connection refused [05:57:47] PROBLEM - cp37 Nginx Backend for phorge171 on cp37 is CRITICAL: connect to address localhost and port 8202: Connection refused [05:57:51] PROBLEM - cp37 Nginx Backend for swiftproxy171 on cp37 is CRITICAL: connect to address localhost and port 8207: Connection refused [05:57:53] PROBLEM - cp37 Nginx Backend for mw181 on cp37 is CRITICAL: connect to address localhost and port 8119: Connection refused [05:57:53] PROBLEM - cp37 Nginx Backend for mwtask171 on cp37 is CRITICAL: connect to address localhost and port 8161: Connection refused [05:57:55] PROBLEM - cp37 Nginx Backend for reports171 on cp37 is CRITICAL: connect to address localhost and port 8205: Connection refused [05:58:02] PROBLEM - cp37 Nginx Backend for mw161 on cp37 is CRITICAL: connect to address localhost and port 8115: Connection refused [05:58:02] PROBLEM - cp37 Nginx Backend for matomo151 on cp37 is CRITICAL: connect to address localhost and port 8203: Connection refused [05:58:17] PROBLEM - cp37 Nginx Backend for mw172 on cp37 is CRITICAL: connect to address localhost and port 8118: Connection refused [05:58:22] PROBLEM - cp37 Nginx Backend for mwtask151 on cp37 is CRITICAL: connect to address localhost and port 8162: Connection refused [05:58:30] PROBLEM - cp37 Nginx Backend for puppet181 on cp37 is CRITICAL: connect to address localhost and port 8204: Connection refused [05:58:31] PROBLEM - cp37 Nginx Backend for mwtask181 on cp37 is CRITICAL: connect to address localhost and port 8160: Connection refused [05:58:39] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 17 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw154 mw163 mw164 mw173 mw174 mw183 mw184 mediawiki [05:58:41] PROBLEM - cp36 Varnish Backends on cp36 is WARNING: No backends detected. If this is an error, see readme.txt [05:59:04] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 56 - Recv failure: Connection reset by peer [05:59:05] RECOVERY - cp37 Nginx Backend for mw162 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8116 [05:59:06] RECOVERY - cp37 Nginx Backend for mw152 on cp37 is OK: TCP OK - 0.001 second response time on localhost port 8114 [05:59:10] RECOVERY - cp37 Nginx Backend for mw164 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8124 [05:59:10] RECOVERY - cp37 Nginx Backend for mw153 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8121 [05:59:20] RECOVERY - cp37 Nginx Backend for mw173 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8125 [05:59:21] RECOVERY - cp37 Nginx Backend for mwtask161 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8163 [05:59:23] RECOVERY - cp37 Nginx Backend for mon181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8201 [05:59:31] RECOVERY - cp37 Nginx Backend for test151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8181 [05:59:33] RECOVERY - cp37 Nginx Backend for mw184 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8128 [05:59:39] RECOVERY - cp37 Nginx Backend for mw183 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8127 [05:59:39] RECOVERY - cp37 Nginx Backend for mw171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8117 [05:59:39] PROBLEM - cp37 Current Load on cp37 is WARNING: LOAD WARNING - total load average: 4.75, 6.73, 7.28 [05:59:42] RECOVERY - cp37 Nginx Backend for mw151 on cp37 is OK: TCP OK - 0.004 second response time on localhost port 8113 [05:59:46] RECOVERY - cp37 Nginx Backend for swiftproxy161 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8206 [05:59:46] RECOVERY - cp37 Nginx Backend for mw154 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8122 [05:59:47] RECOVERY - cp37 Nginx Backend for phorge171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8202 [05:59:51] RECOVERY - cp37 Nginx Backend for swiftproxy171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8207 [05:59:53] RECOVERY - cp37 Nginx Backend for mw181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8119 [05:59:53] RECOVERY - cp37 Nginx Backend for mwtask171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8161 [05:59:55] RECOVERY - cp37 Nginx Backend for reports171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8205 [06:00:02] RECOVERY - cp37 Nginx Backend for matomo151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8203 [06:00:02] RECOVERY - cp37 Nginx Backend for mw161 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8115 [06:00:16] RECOVERY - cp37 Nginx Backend for mw172 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8118 [06:00:22] RECOVERY - cp37 Nginx Backend for mwtask151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8162 [06:00:30] RECOVERY - cp37 Nginx Backend for puppet181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8204 [06:00:31] RECOVERY - cp37 Nginx Backend for mwtask181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8160 [06:00:38] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 17 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw154 mw163 mw164 mw173 mw174 mw183 mw184 mediawiki [06:00:47] RECOVERY - cp37 Nginx Backend for mw163 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8123 [06:00:49] RECOVERY - cp37 Nginx Backend for mw182 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8120 [06:00:55] RECOVERY - cp37 Nginx Backend for mw174 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8126 [06:01:13] PROBLEM - cp36 Nginx Backend for puppet181 on cp36 is CRITICAL: connect to address localhost and port 8204: Connection refused [06:01:20]