[00:00:50] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [00:02:23] RECOVERY - cp1 Disk Space on cp1 is OK: DISK OK - free space: / 6133MiB (11% inode=97%); [00:13:11] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [00:19:32] PROBLEM - graylog1 Current Load on graylog1 is CRITICAL: LOAD CRITICAL - total load average: 9.05, 7.36, 5.69 [00:23:32] PROBLEM - graylog1 Current Load on graylog1 is WARNING: LOAD WARNING - total load average: 4.83, 6.98, 5.98 [00:29:53] PROBLEM - graylog1 Current Load on graylog1 is CRITICAL: LOAD CRITICAL - total load average: 41.35, 16.60, 9.62 [00:36:25] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com All nameservers failed to answer the query. [00:38:16] PROBLEM - graylog1 APT on graylog1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:40:13] RECOVERY - graylog1 APT on graylog1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [00:42:18] PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org All nameservers failed to answer the query. [00:58:53] PROBLEM - graylog1 APT on graylog1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:03:24] PROBLEM - graylog1 HTTPS on graylog1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [01:03:36] PROBLEM - graylog1 SSH on graylog1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:05:27] PROBLEM - graylog1 conntrack_table_size on graylog1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:05:37] PROBLEM - graylog1 NTP time on graylog1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:06:37] PROBLEM - graylog1 PowerDNS Recursor on graylog1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:06:42] PROBLEM - graylog1 ferm_active on graylog1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:07:55] RECOVERY - graylog1 NTP time on graylog1 is OK: NTP OK: Offset -0.008310079575 secs [01:08:41] RECOVERY - graylog1 ferm_active on graylog1 is OK: OK ferm input default policy is set [01:08:41] RECOVERY - graylog1 PowerDNS Recursor on graylog1 is OK: DNS OK: 0.737 seconds response time. wikitide.org returns 2604:4300:a:24::113,63.141.240.6 [01:09:30] RECOVERY - graylog1 HTTPS on graylog1 is OK: HTTP OK: HTTP/2 200 - 646 bytes in 2.130 second response time [01:09:47] RECOVERY - graylog1 SSH on graylog1 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [01:12:19] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [01:12:23] PROBLEM - cp1 Disk Space on cp1 is WARNING: DISK WARNING - free space: / 6073MiB (10% inode=97%); [01:13:51] PROBLEM - graylog1 HTTPS on graylog1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:13:55] PROBLEM - graylog1 PowerDNS Recursor on graylog1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:14:11] PROBLEM - graylog1 SSH on graylog1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:17:21] RECOVERY - graylog1 conntrack_table_size on graylog1 is OK: OK: nf_conntrack is 0 % full [01:17:40] RECOVERY - graylog1 APT on graylog1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [01:17:50] RECOVERY - graylog1 HTTPS on graylog1 is OK: HTTP OK: HTTP/2 200 - 646 bytes in 0.008 second response time [01:18:05] RECOVERY - graylog1 PowerDNS Recursor on graylog1 is OK: DNS OK: 0.883 seconds response time. wikitide.org returns 2604:4300:a:24::113,63.141.240.6 [01:18:13] RECOVERY - graylog1 SSH on graylog1 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [01:36:27] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [01:38:43] [02WikiTideOrg/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/763214ec41f4...ba2d3f7bd315 [01:38:44] [02WikiTideOrg/dns] 07AgentIsai 03ba2d3f7 - Set public IPs [01:53:42] [02WikiTideOrg/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±8] 13https://github.com/WikiTideOrg/puppet/compare/44eb751e140e...449a3af94bde [01:53:43] [02WikiTideOrg/puppet] 07AgentIsai 03449a3af - Update server hostnames [01:54:57] WikiTideOrg/puppet - AgentIsai the build passed. [01:57:07] PROBLEM - graylog1 APT on graylog1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:59:06] RECOVERY - graylog1 APT on graylog1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [02:05:42] PROBLEM - graylog1 SSH on graylog1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:22] RECOVERY - mlrpgspeedruns.com - reverse DNS on sslhost is OK: SSL OK - mlrpgspeedruns.com reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [02:06:48] PROBLEM - graylog1 APT on graylog1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:07:42] RECOVERY - graylog1 SSH on graylog1 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [02:09:14] RECOVERY - graylog1 APT on graylog1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [02:12:14] RECOVERY - rosettacode.org - reverse DNS on sslhost is OK: SSL OK - rosettacode.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [02:23:33] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 864 bytes in 0.399 second response time [02:24:30] PROBLEM - graylog1 APT on graylog1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:25:32] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.158 second response time [02:38:03] PROBLEM - graylog1 HTTPS on graylog1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [02:38:10] PROBLEM - graylog1 SSH on graylog1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:38:29] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mlrpgspeedruns.com. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [02:38:50] PROBLEM - graylog1 PowerDNS Recursor on graylog1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:39:02] RECOVERY - graylog1 APT on graylog1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [02:39:58] RECOVERY - graylog1 HTTPS on graylog1 is OK: HTTP OK: HTTP/2 200 - 646 bytes in 0.043 second response time [02:40:06] RECOVERY - graylog1 SSH on graylog1 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [02:40:32] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [02:40:45] RECOVERY - graylog1 PowerDNS Recursor on graylog1 is OK: DNS OK: 0.129 seconds response time. wikitide.org returns 2604:4300:a:24::113,63.141.240.6 [02:42:11] RECOVERY - www.rosettacode.org - reverse DNS on sslhost is OK: SSL OK - www.rosettacode.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [02:49:16] PROBLEM - graylog1 PowerDNS Recursor on graylog1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:49:28] PROBLEM - graylog1 NTP time on graylog1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:50:02] !log [agent@jobrunner21] starting deploy of {'force': True, 'folders': '1.40', 'versions': ['']} to [jobrunner21, mw21] [02:50:05] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [02:50:26] !log [agent@jobrunner21] finished deploy of {'force': True, 'folders': '1.40', 'versions': ['']} to [jobrunner21, mw21] - SUCCESS in 24s [02:50:29] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [02:50:39] !log [agent@jobrunner21] starting deploy of {'force': True, 'folders': '1.41', 'versions': ['']} to [jobrunner21, mw21] [02:50:41] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [02:51:01] !log [agent@jobrunner21] finished deploy of {'force': True, 'folders': '1.41', 'versions': ['']} to [jobrunner21, mw21] - SUCCESS in 21s [02:51:03] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [02:51:12] RECOVERY - graylog1 PowerDNS Recursor on graylog1 is OK: DNS OK: 1.072 second response time. wikitide.org returns 2604:4300:a:24::113,63.141.240.6 [02:51:31] RECOVERY - graylog1 NTP time on graylog1 is OK: NTP OK: Offset -0.02759519219 secs [02:51:54] PROBLEM - graylog1 APT on graylog1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:58:36] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [02:59:42] PROBLEM - graylog1 PowerDNS Recursor on graylog1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:01:42] RECOVERY - graylog1 PowerDNS Recursor on graylog1 is OK: DNS OK: 3.398 seconds response time. wikitide.org returns 2604:4300:a:24::113,63.141.240.6 [03:02:40] RECOVERY - graylog1 APT on graylog1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [03:07:21] RECOVERY - mlrpgspeedruns.com - reverse DNS on sslhost is OK: SSL OK - mlrpgspeedruns.com reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [03:10:03] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [03:10:26] PROBLEM - graylog1 PowerDNS Recursor on graylog1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:10:36] PROBLEM - graylog1 SSH on graylog1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:11:42] PROBLEM - graylog1 APT on graylog1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:11:47] PROBLEM - graylog1 HTTPS on graylog1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [03:12:39] RECOVERY - graylog1 SSH on graylog1 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [03:14:02] !log [agent@jobrunner21] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['']} to [mw21, mw22, jobrunner21] [03:14:04] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:14:05] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query rosettacode.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [03:14:12] !log [agent@jobrunner21] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['']} to [mw21, mw22, jobrunner21] - SUCCESS in 10s [03:14:14] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:14:22] PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org All nameservers failed to answer the query. [03:14:35] !log [agent@jobrunner21] starting deploy of {'force': True, 'folders': '1.40,1.41', 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [03:14:37] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:14:49] PROBLEM - graylog1 conntrack_table_size on graylog1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:15:07] !log [agent@jobrunner21] finished deploy of {'force': True, 'folders': '1.40,1.41', 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 32s [03:15:09] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:16:26] !log [agent@jobrunner21] starting deploy of {'upgrade_vendor': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [03:16:27] !log [agent@jobrunner21] finished deploy of {'upgrade_vendor': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 0s [03:16:28] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:16:28] !log [agent@jobrunner21] starting deploy of {'upgrade_vendor': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [03:16:30] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:16:32] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:16:38] !log [agent@jobrunner21] finished deploy of {'upgrade_vendor': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 12s [03:16:40] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:16:43] RECOVERY - graylog1 conntrack_table_size on graylog1 is OK: OK: nf_conntrack is 0 % full [03:16:46] RECOVERY - graylog1 APT on graylog1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [03:16:55] !log [agent@jobrunner21] starting deploy of {'upgrade_vendor': True, 'force': True, 'versions': ['1.41']} to [mw21, mw22, jobrunner21] [03:16:56] !log [agent@jobrunner21] finished deploy of {'upgrade_vendor': True, 'force': True, 'versions': ['1.41']} to [mw21, mw22, jobrunner21] - SUCCESS in 0s [03:16:57] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:16:57] !log [agent@jobrunner21] starting deploy of {'upgrade_vendor': True, 'force': True, 'versions': ['1.41']} to [mw21, mw22, jobrunner21] [03:16:59] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:17:01] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:17:07] !log [agent@jobrunner21] finished deploy of {'upgrade_vendor': True, 'force': True, 'versions': ['1.41']} to [mw21, mw22, jobrunner21] - SUCCESS in 12s [03:17:10] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:17:28] RECOVERY - graylog1 PowerDNS Recursor on graylog1 is OK: DNS OK: 0.271 seconds response time. wikitide.org returns 2604:4300:a:24::113,63.141.240.6 [03:17:55] !log [agent@jobrunner21] starting deploy of {'l10n': True, 'force': True, 'versions': ['1.40', '1.41']} to [mw21, mw22, jobrunner21] [03:17:56] !log [agent@jobrunner21] finished deploy of {'l10n': True, 'force': True, 'versions': ['1.40', '1.41']} to [mw21, mw22, jobrunner21] - SUCCESS in 0s [03:17:57] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:17:57] !log [agent@jobrunner21] starting deploy of {'l10n': True, 'force': True, 'versions': ['1.40', '1.41']} to [mw21, mw22, jobrunner21] [03:17:58] !log [agent@jobrunner21] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [03:17:59] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:18:01] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:18:02] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:18:20] !log [agent@jobrunner21] starting deploy of {'pull': 'errorpages', 'errorpages': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [03:18:22] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:18:25] !log [agent@jobrunner21] finished deploy of {'pull': 'errorpages', 'errorpages': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 4s [03:18:27] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:18:36] !log [agent@jobrunner21] starting deploy of {'pull': 'landing', 'landing': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [03:18:38] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:18:40] !log [agent@jobrunner21] finished deploy of {'pull': 'landing', 'landing': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 4s [03:18:42] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:30:48] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [03:38:27] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com All nameservers failed to answer the query. [03:39:53] PROBLEM - graylog1 Current Load on graylog1 is WARNING: LOAD WARNING - total load average: 0.03, 0.44, 7.19 [03:41:48] RECOVERY - graylog1 Current Load on graylog1 is OK: LOAD OK - total load average: 0.02, 0.31, 6.35 [03:43:11] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [03:43:51] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.rosettacode.org All nameservers failed to answer the query. [03:48:07] [02WikiTideOrg/puppet] 07AgentIsai pushed 031 commit to 03master [+1/-1/±1] 13https://github.com/WikiTideOrg/puppet/compare/449a3af94bde...1dc060de8c05 [03:48:09] [02WikiTideOrg/puppet] 07AgentIsai 031dc060d - Move swiftproxy to cloud2 [03:51:03] [02WikiTideOrg/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/ba2d3f7bd315...6443675b1770 [03:51:04] [02WikiTideOrg/dns] 07AgentIsai 036443675 - Add swiftproxy21 [04:07:24] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [04:13:33] RECOVERY - www.rosettacode.org - reverse DNS on sslhost is OK: SSL OK - www.rosettacode.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [04:36:22] RECOVERY - mlrpgspeedruns.com - reverse DNS on sslhost is OK: SSL OK - mlrpgspeedruns.com reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [04:38:22] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - valorantjp.wiki All nameservers failed to answer the query. [04:42:14] RECOVERY - rosettacode.org - reverse DNS on sslhost is OK: SSL OK - rosettacode.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [04:45:30] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query rosettacode.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [05:07:07] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query valorantjp.wiki. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [05:10:22] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [05:14:23] PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org All nameservers failed to answer the query. [05:15:12] RECOVERY - www.rosettacode.org - reverse DNS on sslhost is OK: SSL OK - www.rosettacode.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [05:38:29] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mlrpgspeedruns.com. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [05:39:59] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query nexttide.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [05:58:36] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [06:09:32] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [06:12:18] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query rosettacode.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [06:13:16] [02WikiTideOrg/mw-config] 07AgentIsai pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/ad50fad500f6...88f074a9a12b [06:13:18] [02WikiTideOrg/mw-config] 07AgentIsai 0388f074a - Start database maintenance [06:13:49] !log [agent@jobrunner1] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] [06:13:51] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [06:13:51] !log [agent@jobrunner1] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] - SUCCESS in 2s [06:14:17] WikiTideOrg/mw-config - AgentIsai the build passed. [06:17:01] !log [@jobrunner21] starting deploy of {'config': True, 'versions': ['']} to [mw21, mw22, jobrunner21] [06:17:04] !log [@jobrunner21] DEPLOY ABORTED: Canary check failed for meta.wikitide.org@mw21.wikitide.net [06:27:32] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 864 bytes in 0.093 second response time [06:28:10] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 864 bytes in 0.070 second response time [06:28:19] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 864 bytes in 0.070 second response time [06:28:50] [02WikiTideOrg/mw-config] 07AgentIsai pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/88f074a9a12b...53af3954177e [06:28:52] [02WikiTideOrg/mw-config] 07AgentIsai 0353af395 - Update hostnames [06:29:20] [02WikiTideOrg/mw-config] 07AgentIsai pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/53af3954177e...bb22eea4dd77 [06:29:22] [02WikiTideOrg/mw-config] 07AgentIsai 03bb22eea - Update database hostname [06:29:40] PROBLEM - db1 MariaDB on db1 is CRITICAL: Can't connect to server on 'db1.wikitide.net' (115) [06:29:50] WikiTideOrg/mw-config - AgentIsai the build passed. [06:30:02] PROBLEM - matomo1 HTTPS on matomo1 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 426 bytes in 0.051 second response time [06:30:18] PROBLEM - db1 MariaDB Connections on db1 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:47Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(47): mysqli_real_connect(Object(mysqli), 'db1.wikitide.ne...', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, false)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_con [06:30:18] ine 47Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:47Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(47): mysqli_real_connect(Object(mysqli), 'db1.wikitide.ne...', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, false)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 47 [06:30:26] WikiTideOrg/mw-config - AgentIsai the build passed. [06:30:45] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [06:30:46] PROBLEM - phorge1 phorge-static.wikitide.org HTTPS on phorge1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [06:31:18] PROBLEM - phorge1 issue-tracker.wikitide.org HTTPS on phorge1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4248 bytes in 0.024 second response time [06:31:42] !log [salt-user@jobrunner21] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [06:31:45] !log [salt-user@jobrunner21] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 3s [06:32:32] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is CRITICAL: CRITICAL - NGINX Error Rate is 70% [06:33:15] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 54% [06:34:32] RECOVERY - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is OK: OK - NGINX Error Rate is 15% [06:35:15] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 28% [06:36:14] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [06:38:23] PROBLEM - cp1 Disk Space on cp1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.113: Connection reset by peer [06:39:16] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [06:39:55] PROBLEM - cp1 Nginx Backend for mw1 on cp1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.113: Connection reset by peer [06:40:02] PROBLEM - cp1 Nginx Backend for mw2 on cp1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.113: Connection reset by peer [06:40:05] PROBLEM - cp1 Nginx Backend for phorge1 on cp1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.113: Connection reset by peer [06:40:14] PROBLEM - cp1 ferm_active on cp1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.113: Connection reset by peer [06:40:17] PROBLEM - cp1 Current Load on cp1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.113: Connection reset by peer [06:40:22] PROBLEM - cp1 SSH on cp1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:40:23] PROBLEM - cp1 NTP time on cp1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.113: Connection reset by peer [06:40:35] PROBLEM - cp1 conntrack_table_size on cp1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.113: Connection reset by peer [06:40:38] PROBLEM - cp1 PowerDNS Recursor on cp1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.113: Connection reset by peer [06:40:48] PROBLEM - cp1 Nginx Backend for mail1 on cp1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.113: Connection reset by peer [06:40:49] PROBLEM - cp1 Nginx Backend for matomo1 on cp1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.113: Connection reset by peer [06:40:50] PROBLEM - cp1 Nginx Backend for test1 on cp1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.113: Connection reset by peer [06:41:04] PROBLEM - cp1 Varnish Backends on cp1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.113: Connection reset by peer [06:41:23] PROBLEM - cp1 HTTPS on cp1 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 2571 bytes in 0.007 second response time [06:41:24] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.113: Connection reset by peer [06:41:31] PROBLEM - cp1 APT on cp1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.113: Connection reset by peer [06:41:33] PROBLEM - cp1 Nginx Backend for puppet1 on cp1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.113: Connection reset by peer [06:42:22] RECOVERY - rosettacode.org - reverse DNS on sslhost is OK: SSL OK - rosettacode.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [06:43:23] PROBLEM - cp2 ferm_active on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [06:43:27] PROBLEM - cp2 Nginx Backend for test1 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [06:43:29] PROBLEM - cp2 Current Load on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [06:43:30] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [06:43:33] PROBLEM - cp2 Nginx Backend for mail1 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [06:43:33] PROBLEM - cp2 PowerDNS Recursor on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [06:43:33] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 46% [06:43:34] PROBLEM - cp2 Nginx Backend for mw2 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [06:43:49] PROBLEM - cp2 Nginx Backend for mw1 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [06:44:08] PROBLEM - cp2 conntrack_table_size on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [06:44:09] PROBLEM - cp2 NTP time on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [06:44:19] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [06:44:25] PROBLEM - cp2 APT on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [06:44:27] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 2592 bytes in 0.153 second response time [06:44:38] PROBLEM - cp2 Nginx Backend for matomo1 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [06:44:41] PROBLEM - cp2 Disk Space on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [06:44:41] PROBLEM - cp2 Nginx Backend for phorge1 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [06:44:46] PROBLEM - cp2 Nginx Backend for puppet1 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [06:44:58] PROBLEM - cp2 SSH on cp2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:45:33] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 24% [06:46:27] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/2 200 - 2917 bytes in 0.165 second response time [06:46:37] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 63.141.240.6/cpweb, 2604:4300:a:24::113/cpweb [06:47:32] PROBLEM - cp3 GDNSD Datacenters on cp3 is CRITICAL: CRITICAL - 2 datacenters are down: 63.141.240.6/cpweb, 2604:4300:a:24::113/cpweb [06:47:47] PROBLEM - hsck.lophocmatngu.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - hsck.lophocmatngu.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [06:49:39] PROBLEM - meta.sagan4.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - meta.sagan4.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [06:49:39] PROBLEM - beta.sagan4.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - beta.sagan4.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [06:49:46] PROBLEM - www.farthestfrontier.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.farthestfrontier.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [06:49:49] PROBLEM - alpha.sagan4.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - alpha.sagan4.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [06:51:17] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 58% [06:51:38] PROBLEM - realitylongterms.studio - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - realitylongterms.studio reverse DNS resolves to vps-d267e842.vps.ovh.ca [06:51:46] PROBLEM - thegreatwar.uk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - thegreatwar.uk reverse DNS resolves to vps-d267e842.vps.ovh.ca [06:51:49] PROBLEM - wikitide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikitide.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [06:53:17] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [06:53:19] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 2593 bytes in 0.158 second response time [06:54:01] PROBLEM - wikitide.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikitide.net reverse DNS resolves to vps-d267e842.vps.ovh.ca [06:54:01] PROBLEM - farthestfrontier.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - farthestfrontier.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [06:54:01] PROBLEM - greatamerica.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - greatamerica.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [06:57:13] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/2 200 - 2916 bytes in 0.190 second response time [06:58:34] PROBLEM - www.lgbtqia.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.lgbtqia.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [06:58:34] PROBLEM - wiki.colleimadcat.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.colleimadcat.com reverse DNS resolves to vps-d267e842.vps.ovh.ca [06:59:23] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net reverse DNS resolves to vps-d267e842.vps.ovh.ca [06:59:33] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 52% [07:01:18] PROBLEM - sdiy.info - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - sdiy.info reverse DNS resolves to vps-d267e842.vps.ovh.ca [07:02:49] PROBLEM - projects.dmvpetridish.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - projects.dmvpetridish.com reverse DNS resolves to vps-d267e842.vps.ovh.ca [07:05:33] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 33% [07:06:21] PROBLEM - www.greatamerica.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.greatamerica.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [07:06:23] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com reverse DNS resolves to vps-d267e842.vps.ovh.ca [07:06:57] PROBLEM - wiki.burning.art - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.burning.art reverse DNS resolves to vps-d267e842.vps.ovh.ca [07:06:57] PROBLEM - www.wikitide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.wikitide.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [07:07:33] PROBLEM - www.polandballwiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.polandballwiki.com reverse DNS resolves to vps-d267e842.vps.ovh.ca [07:08:10] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - valorantjp.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [07:08:20] PROBLEM - wiki.chevrine.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.chevrine.com reverse DNS resolves to vps-d267e842.vps.ovh.ca [07:09:19] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 55% [07:10:12] PROBLEM - data.lophocmatngu.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - data.lophocmatngu.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [07:10:20] PROBLEM - distrowiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - distrowiki.net reverse DNS resolves to vps-d267e842.vps.ovh.ca [07:10:20] PROBLEM - polandballwiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - polandballwiki.com reverse DNS resolves to vps-d267e842.vps.ovh.ca [07:10:34] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [07:12:14] PROBLEM - mason.sagan4.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mason.sagan4.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [07:13:19] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [07:14:01] PROBLEM - hkrail.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - hkrail.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [07:14:01] PROBLEM - lophocmatngu.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - lophocmatngu.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [07:15:33] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 41% [07:41:33] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 11% [07:44:14] PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [07:49:22] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 49% [07:50:48] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 2592 bytes in 0.155 second response time [07:55:23] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [07:57:23] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 5% [07:57:33] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 54% [07:57:41] RECOVERY - db1 MariaDB on db1 is OK: Uptime: 7 Threads: 32 Questions: 901 Slow queries: 40 Opens: 134 Open tables: 128 Queries per second avg: 128.714 [07:58:03] RECOVERY - matomo1 HTTPS on matomo1 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 0.980 second response time [07:58:10] PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.406 second response time [07:58:18] RECOVERY - db1 MariaDB Connections on db1 is OK: OK connection usage: 1.6%Current connections: 8 [07:58:19] PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.154 second response time [07:58:36] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/2 200 - 2917 bytes in 0.154 second response time [07:58:46] RECOVERY - phorge1 phorge-static.wikitide.org HTTPS on phorge1 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 58195 bytes in 0.285 second response time [07:59:18] RECOVERY - phorge1 issue-tracker.wikitide.org HTTPS on phorge1 is OK: HTTP OK: HTTP/1.1 200 OK - 114452 bytes in 0.158 second response time [07:59:32] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.234 second response time [07:59:33] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 25% [08:08:27] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 2591 bytes in 0.160 second response time [08:10:27] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/2 200 - 2915 bytes in 0.158 second response time [08:17:46] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.rosettacode.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [08:28:39] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [08:55:20] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 2592 bytes in 0.150 second response time [08:57:17] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/2 200 - 2917 bytes in 0.154 second response time [08:58:38] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net reverse DNS resolves to vps-d267e842.vps.ovh.ca [09:06:00] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 2593 bytes in 0.150 second response time [09:07:57] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/2 200 - 2939 bytes in 0.166 second response time [09:14:43] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 2593 bytes in 0.161 second response time [09:17:17] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [09:20:34] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/2 200 - 2916 bytes in 0.165 second response time [09:23:57] PROBLEM - swiftproxy1 Current Load on swiftproxy1 is CRITICAL: LOAD CRITICAL - total load average: 5.59, 9.81, 4.71 [09:25:51] RECOVERY - swiftproxy1 Current Load on swiftproxy1 is OK: LOAD OK - total load average: 0.96, 6.75, 4.19 [09:42:19] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [09:46:58] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.rosettacode.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [09:54:40] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 034 commits to 03suffix [+0/-0/±5] 13https://github.com/WikiTideOrg/mw-config/compare/ea7063b83157...3aa978a20c09 [09:54:43] [02WikiTideOrg/mw-config] 07Universal-Omega 033aa978a - Merge branch 'main' into suffix [09:54:45] [02mw-config] 07Universal-Omega synchronize pull request 03#25: Use 'wiki' suffix - 13https://github.com/WikiTideOrg/mw-config/pull/25 [09:55:35] WikiTideOrg/mw-config - Universal-Omega the build passed. [09:57:27] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 2593 bytes in 0.161 second response time [09:58:39] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [10:05:16] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/2 200 - 2939 bytes in 0.158 second response time [10:06:14] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [10:06:27] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [10:28:38] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net reverse DNS resolves to vps-d267e842.vps.ovh.ca [10:36:12] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - valorantjp.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [10:36:21] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com reverse DNS resolves to vps-d267e842.vps.ovh.ca [10:37:17] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 2592 bytes in 0.161 second response time [10:39:14] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/2 200 - 2938 bytes in 0.171 second response time [10:42:18] PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [11:06:14] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [11:06:58] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [11:28:45] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 2593 bytes in 0.152 second response time [11:30:42] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/2 200 - 2894 bytes in 0.164 second response time [11:36:12] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - valorantjp.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [11:36:32] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [12:06:08] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [12:22:11] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 2592 bytes in 0.169 second response time [12:23:48] !log [agent@jobrunner21] starting deploy of {'l10n': True, 'force': True, 'versions': ['1.40', '1.41']} to [mw21, mw22, jobrunner21] [12:23:49] !log [agent@jobrunner21] finished deploy of {'l10n': True, 'force': True, 'versions': ['1.40', '1.41']} to [mw21, mw22, jobrunner21] - SUCCESS in 0s [12:23:50] !log [agent@jobrunner21] starting deploy of {'l10n': True, 'force': True, 'versions': ['1.40', '1.41']} to [mw21, mw22, jobrunner21] [12:23:53] !log [agent@jobrunner21] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [12:23:55] !log [agent@jobrunner21] starting deploy of {'l10n': True, 'force': True, 'versions': ['1.40', '1.41']} to [mw21, mw22, jobrunner21] [12:23:56] !log [agent@jobrunner21] finished deploy of {'l10n': True, 'force': True, 'versions': ['1.40', '1.41']} to [mw21, mw22, jobrunner21] - SUCCESS in 1s [12:23:57] !log [agent@jobrunner21] starting deploy of {'l10n': True, 'force': True, 'versions': ['1.40', '1.41']} to [mw21, mw22, jobrunner21] [12:28:44] !log [agent@jobrunner21] finished deploy of {'l10n': True, 'force': True, 'versions': ['1.40', '1.41']} to [mw21, mw22, jobrunner21] - SUCCESS in 288s [12:28:45] !log [agent@jobrunner21] starting deploy of {'l10n': True, 'force': True, 'versions': ['1.40', '1.41']} to [mw21, mw22, jobrunner21] [12:30:00] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/2 200 - 2895 bytes in 0.165 second response time [12:30:43] PROBLEM - cp3 APT on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [12:30:43] PROBLEM - cp3 Nginx Backend for mail1 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [12:30:53] PROBLEM - cp3 HTTPS on cp3 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 2571 bytes in 0.299 second response time [12:31:01] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [12:31:06] PROBLEM - cp3 Nginx Backend for puppet1 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [12:31:12] PROBLEM - cp3 NTP time on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [12:31:22] PROBLEM - cp3 Nginx Backend for matomo1 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [12:31:32] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [12:31:38] PROBLEM - cp3 Nginx Backend for mw2 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [12:31:39] PROBLEM - cp3 Nginx Backend for mw1 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [12:31:58] PROBLEM - cp3 conntrack_table_size on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [12:32:03] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [12:32:19] PROBLEM - cp3 Nginx Backend for test1 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [12:32:22] PROBLEM - cp3 ferm_active on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [12:32:34] PROBLEM - cp3 Nginx Backend for phorge1 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [12:32:41] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [12:32:46] !log [agent@jobrunner21] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [12:32:49] !log [agent@jobrunner21] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 2s [12:32:56] PROBLEM - cp3 SSH on cp3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:33:44] PROBLEM - cp4 Nginx Backend for puppet1 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [12:33:45] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [12:33:51] !log [agent@jobrunner21] finished deploy of {'l10n': True, 'force': True, 'versions': ['1.40', '1.41']} to [mw21, mw22, jobrunner21] - SUCCESS in 596s [12:33:54] PROBLEM - cp4 Nginx Backend for test1 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [12:33:55] PROBLEM - cp4 APT on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [12:33:57] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [12:34:14] !log [agent@jobrunner21] starting deploy of {'pull': 'errorpages', 'errorpages': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [12:34:16] !log [agent@jobrunner21] finished deploy of {'pull': 'errorpages', 'errorpages': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 2s [12:34:32] PROBLEM - cp4 PowerDNS Recursor on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [12:34:41] PROBLEM - cp4 NTP time on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [12:34:41] PROBLEM - cp4 Nginx Backend for mw2 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [12:34:46] PROBLEM - cp4 conntrack_table_size on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [12:34:51] PROBLEM - cp4 ferm_active on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [12:34:54] PROBLEM - cp4 Nginx Backend for mail1 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [12:34:55] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [12:34:59] PROBLEM - cp4 Nginx Backend for phorge1 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [12:35:06] PROBLEM - cp4 SSH on cp4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:35:13] PROBLEM - cp4 Disk Space on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [12:35:33] PROBLEM - cp4 HTTPS on cp4 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 2571 bytes in 0.414 second response time [12:35:36] PROBLEM - cp4 Nginx Backend for matomo1 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [12:35:37] PROBLEM - cp4 Nginx Backend for mw1 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [12:35:41] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [12:36:06] !log [agent@jobrunner21] starting deploy of {'force': True, 'folders': '1.40', 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [12:36:29] !log [agent@jobrunner21] finished deploy of {'force': True, 'folders': '1.40', 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 22s [12:36:51] !log [agent@jobrunner21] starting deploy of {'force': True, 'folders': '1.41', 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [12:37:14] !log [agent@jobrunner21] finished deploy of {'force': True, 'folders': '1.41', 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 23s [12:42:00] PROBLEM - cp5 Nginx Backend for mw1 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [12:42:01] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [12:42:03] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [12:42:30] PROBLEM - cp5 Current Load on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [12:42:44] PROBLEM - cp5 Nginx Backend for matomo1 on cp5 is CRITICAL: CHECK_NRPE: (ssl_err != 5) Error - Could not complete SSL handshake with 15.235.167.159: 1 [12:42:47] PROBLEM - cp5 Nginx Backend for mw2 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [12:42:50] PROBLEM - cp5 ferm_active on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [12:42:56] PROBLEM - cp5 NTP time on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [12:42:59] PROBLEM - cp5 Nginx Backend for test1 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [12:43:01] PROBLEM - cp5 HTTPS on cp5 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 2571 bytes in 0.693 second response time [12:43:09] PROBLEM - cp5 Nginx Backend for puppet1 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [12:43:13] PROBLEM - cp5 Nginx Backend for phorge1 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [12:43:13] PROBLEM - cp5 PowerDNS Recursor on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [12:43:19] PROBLEM - cp5 Nginx Backend for mail1 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [12:43:37] PROBLEM - cp5 APT on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [12:43:38] PROBLEM - cp5 conntrack_table_size on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [12:43:45] PROBLEM - cp5 Disk Space on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [12:43:52] PROBLEM - cp5 SSH on cp5 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:45:21] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [12:46:45] PROBLEM - cp6 conntrack_table_size on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 139.99.236.151: Connection reset by peer [12:46:49] PROBLEM - cp6 APT on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 139.99.236.151: Connection reset by peer [12:46:51] PROBLEM - cp6 PowerDNS Recursor on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 139.99.236.151: Connection reset by peer [12:46:58]