[00:00:34] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [00:01:15] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:06:15] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 38% [00:10:12] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 53% [00:11:36] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to cp2.wikitide.net port 443 after 31 ms: Couldn't connect to server [00:11:39] PROBLEM - cp2 Disk Space on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [00:11:40] PROBLEM - cp2 conntrack_table_size on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [00:11:45] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [00:11:45] PROBLEM - cp2 PowerDNS Recursor on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [00:11:48] PROBLEM - cp2 Nginx Backend for matomo1 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [00:11:49] PROBLEM - cp2 ferm_active on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [00:11:50] PROBLEM - cp2 Nginx Backend for mw2 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [00:11:52] PROBLEM - cp2 Nginx Backend for puppet1 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [00:12:01] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [00:12:03] PROBLEM - cp2 Nginx Backend for mw1 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [00:12:04] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [00:12:08] PROBLEM - cp2 Nginx Backend for phorge1 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [00:12:09] PROBLEM - cp2 Current Load on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [00:12:13] PROBLEM - cp2 APT on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [00:12:19] PROBLEM - cp2 Nginx Backend for mail1 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [00:12:19] PROBLEM - cp2 Nginx Backend for test1 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [00:12:20] PROBLEM - cp2 NTP time on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [00:15:58] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 52% [00:16:12] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [00:17:54] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 38% [00:18:12] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 58% [00:20:12] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 79% [00:20:21] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/4ed62e68c939...b47ae05dde81 [00:20:24] [02WikiTideOrg/puppet] 07Universal-Omega 03b47ae05 - Update ports [00:20:33] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:32] PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.133 second response time [00:23:28] PROBLEM - cp2 Nginx Backend for mail21 on cp2 is CRITICAL: connect to address localhost and port 8202: Connection refused [00:23:39] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/b47ae05dde81...a8723ef1fd8c [00:23:41] [02WikiTideOrg/puppet] 07Universal-Omega 03a8723ef - Revert [00:24:56] PROBLEM - cp2 APT on cp2 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [00:24:56] PROBLEM - cp2 Nginx Backend for matomo21 on cp2 is CRITICAL: connect to address localhost and port 8204: Connection refused [00:25:24] RECOVERY - cp2 Nginx Backend for mail21 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8202 [00:26:12] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 41% [00:26:48] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 40% [00:26:51] RECOVERY - cp2 Nginx Backend for matomo21 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8204 [00:28:58] PROBLEM - cp2 APT on cp2 is CRITICAL: APT CRITICAL: 32 packages available for upgrade (3 critical updates). [00:30:57] PROBLEM - cp2 APT on cp2 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [00:32:12] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 69% [00:32:58] PROBLEM - cp2 APT on cp2 is CRITICAL: APT CRITICAL: 32 packages available for upgrade (3 critical updates). [00:34:12] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 47% [00:35:00] RECOVERY - cp2 APT on cp2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:36:12] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 74% [00:36:44] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 38% [00:41:27] PROBLEM - cp2 Varnish Backends on cp2 is WARNING: No backends detected. If this is an error, see readme.txt [00:42:44] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 88% [00:43:27] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mediawiki [00:44:12] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 36% [00:44:15] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [00:45:08] PROBLEM - cp2 Nginx Backend for matomo21 on cp2 is CRITICAL: connect to address localhost and port 8204: Connection refused [00:45:15] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 0% [00:46:10] PROBLEM - cp2 Nginx Backend for mail21 on cp2 is CRITICAL: connect to address localhost and port 8202: Connection refused [00:46:44] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 52% [00:48:12] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 53% [00:48:44] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 70% [00:50:45] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 41% [00:52:12] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [00:52:35] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 40% [00:54:12] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 50% [00:54:35] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 38% [00:56:12] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [00:56:46] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [00:58:12] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 51% [00:58:44] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 54% [01:00:12] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 29% [01:00:33] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [01:00:44] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 38% [01:01:27] I am getting database timeouts on https://rosettacode.org/ [01:14:49] PROBLEM - meta.sagan4.org - reverse DNS on sslhost is UNKNOWN: PROBLEM - www.greatamerica.wiki - reverse DNS on sslhost is UNKNOWN: [01:15:50] PROBLEM - beta.sagan4.org - reverse DNS on sslhost is UNKNOWN: PROBLEM - www.polandballwiki.com - reverse DNS on sslhost is UNKNOWN: [01:15:50] PROBLEM - hsck.lophocmatngu.wiki - reverse DNS on sslhost is UNKNOWN: [01:15:52] PROBLEM - matomo21 HTTPS on matomo21 is UNKNOWN: [01:16:06] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 49% [01:17:52] PROBLEM - mon21 Current Load on mon21 is CRITICAL: LOAD CRITICAL - total load average: 20.29, 12.58, 5.55 [01:18:45] PROBLEM - matomo21 HTTPS on matomo21 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10038 milliseconds [01:19:04] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 36% PROBLEM - phorge21 HTTPS on phorge21 is UNKNOWN: [01:20:08] PROBLEM - www.farthestfrontier.wiki - reverse DNS on sslhost is UNKNOWN: PROBLEM - puppet21 HTTPS on puppet21 is UNKNOWN: PROBLEM - mw21 HTTPS on mw21 is UNKNOWN: PROBLEM - data.lophocmatngu.wiki - reverse DNS on sslhost is UNKNOWN: 15 (Terminated).> PROBLEM - cp2 HTTPS on cp2 is UNKNOWN: [01:20:21] PROBLEM - wiki.burning.art - reverse DNS on sslhost is UNKNOWN: [01:20:22] PROBLEM - mon21 grafana.wikitide.net HTTPS on mon21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:21:19] PROBLEM - mason.sagan4.org - reverse DNS on sslhost is UNKNOWN: [01:23:18] PROBLEM - matomo21 HTTPS on matomo21 is UNKNOWN: [01:44:35] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:47:20] PROBLEM - mon21 PowerDNS Recursor on mon21 is CRITICAL: CRITICAL - Plugin timed out while executing system call PROBLEM - mem21 Current Load on mem21 is WARNING: LOAD WARNING - total load average: 3.62, 3.57, 2.89 [01:47:21] PROBLEM - mon21 HTTPS on mon21 is UNKNOWN: PROBLEM - mon21 HTTPS on mon21 is UNKNOWN: [01:47:21] PROBLEM - os21 Current Load on os21 is UNKNOWN: [01:47:34] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/a8723ef1fd8c...6d8a06934bd0 [01:47:36] [02WikiTideOrg/puppet] 07Universal-Omega 036d8a069 - Update servers [01:48:35] PROBLEM - projects.dmvpetridish.com - reverse DNS on sslhost is UNKNOWN: PROBLEM - mon21 monitoring.wikitide.net HTTPS on mon21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:48:37] PROBLEM - mon21 monitoring.wikitide.net HTTPS on mon21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:48:38] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:48:58] WikiTideOrg/puppet - Universal-Omega the build has errored. [01:49:36] PROBLEM - mem21 Current Load on mem21 is CRITICAL: LOAD CRITICAL - total load average: 5.10, 3.84, 3.48 [01:50:05] PROBLEM - farthestfrontier.wiki - reverse DNS on sslhost is UNKNOWN: PROBLEM - jobrunner21 HTTPS on jobrunner21 is UNKNOWN: PROBLEM - cp6 HTTPS on cp6 is UNKNOWN: PROBLEM - mail21 HTTPS on mail21 is UNKNOWN: PROBLEM - mlrpgspeedruns.com [01:50:05] sslhost is UNKNOWN: [01:50:49] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [01:50:57] PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 7.028 second response time [01:51:14] PROBLEM - mw1 MediaWiki Rendering on mw1 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [01:51:23] PROBLEM - mon21 Check correctness of the icinga configuration on mon21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:51:23] PROBLEM - wiki.chevrine.com - reverse DNS on sslhost is UNKNOWN: [01:51:23] PROBLEM - newcascadia.net - reverse DNS on sslhost is UNKNOWN: [01:51:23] PROBLEM - mon21 Check correctness of the icinga configuration on mon21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:51:23] PROBLEM - cp4 HTTPS on cp4 is UNKNOWN: [01:51:24] PROBLEM - wiki.myehs.eu - reverse DNS on sslhost is UNKNOWN: [01:51:24] PROBLEM - www.lgbtqia.wiki - reverse DNS on sslhost is UNKNOWN: [01:51:25] PROBLEM - nexttide.org - reverse DNS on sslhost is UNKNOWN: [01:51:25] PROBLEM - wikitide.com - reverse DNS on sslhost is UNKNOWN: [01:51:26] PROBLEM - wiki.colleimadcat.com - reverse DNS on sslhost is UNKNOWN: [01:51:26] PROBLEM - alpha.sagan4.org - reverse DNS on sslhost is UNKNOWN: [01:51:27] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to cp2.wikitide.net port 443 after 10 ms: Couldn't connect to server [01:51:27] RECOVERY - mon21 HTTPS on mon21 is OK: HTTP OK: HTTP/2 302 - 374 bytes in 0.159 second response time [01:51:28] PROBLEM - os21 Current Load on os21 is CRITICAL: LOAD CRITICAL - total load average: 4.78, 4.16, 3.65 [01:51:28] RECOVERY - mon21 monitoring.wikitide.net HTTPS on mon21 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.259 second response time [01:51:29] RECOVERY - mon21 grafana.wikitide.net HTTPS on mon21 is OK: HTTP OK: HTTP/1.1 302 Found - 429 bytes in 0.362 second response time [01:51:29] PROBLEM - cp3 HTTPS on cp3 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 2628 bytes in 0.328 second response time [01:51:30] PROBLEM - cp5 HTTPS on cp5 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 2628 bytes in 0.754 second response time [01:51:30] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 85% [01:52:14] PROBLEM - mw21 HTTPS on mw21 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10083 milliseconds PROBLEM - puppet21 HTTPS on puppet21 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10072 milliseconds [01:52:26] RECOVERY - mon21 PowerDNS Recursor on mon21 is OK: DNS OK: 0.094 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [01:52:26] RECOVERY - jobrunner21 HTTPS on jobrunner21 is OK: HTTP OK: HTTP/2 200 - 370 bytes in 0.029 second response time [01:52:30] RECOVERY - cp6 HTTPS on cp6 is OK: HTTP OK: HTTP/2 200 - 2995 bytes in 1.616 second response time PROBLEM - phorge21 HTTPS on phorge21 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 666 bytes in 2.156 second response time [01:52:56] PROBLEM - mw2 MediaWiki Rendering on mw2 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [01:53:35] PROBLEM - matomo21 HTTPS on matomo21 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10043 milliseconds [01:53:35] PROBLEM - mail21 HTTPS on mail21 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10014 milliseconds [01:53:37] RECOVERY - cp4 HTTPS on cp4 is OK: HTTP OK: HTTP/2 200 - 2996 bytes in 0.702 second response time [01:53:38] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 16% [01:53:41] RECOVERY - cp5 HTTPS on cp5 is OK: HTTP OK: HTTP/2 200 - 2974 bytes in 5.089 second response time [01:54:20] PROBLEM - mem21 Current Load on mem21 is WARNING: LOAD WARNING - total load average: 3.43, 3.95, 3.66 [01:54:23] PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Puppet last ran 12 hours ago [01:55:34] PROBLEM - rosettacode.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:34] PROBLEM - rosettacode.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:35] PROBLEM - lophocmatngu.wiki - reverse DNS on sslhost is UNKNOWN: [01:55:35] PROBLEM - wikitide.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:35] PROBLEM - polandballwiki.com - reverse DNS on sslhost is UNKNOWN: [01:55:35] PROBLEM - wikitide.net - reverse DNS on sslhost is UNKNOWN: [01:55:35] PROBLEM - hkrail.wiki - reverse DNS on sslhost is UNKNOWN: [01:55:36] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is UNKNOWN: PROBLEM - www.wikitide.org - reverse DNS on sslhost is UNKNOWN: [01:55:36] RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/2 200 - 2974 bytes in 0.565 second response time [01:55:43] RECOVERY - mon21 Check correctness of the icinga configuration on mon21 is OK: Icinga configuration is correct [01:55:49] PROBLEM - wikitide.org - reverse DNS on sslhost is UNKNOWN: [01:55:49] PROBLEM - greatamerica.wiki - reverse DNS on sslhost is UNKNOWN: PROBLEM - sdiy.info - reverse DNS on sslhost is UNKNOWN: [01:55:49] PROBLEM - rosettacode.org - reverse DNS on sslhost is UNKNOWN: [01:55:49] PROBLEM - distrowiki.net - reverse DNS on sslhost is UNKNOWN: [01:57:43] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 42% [01:59:40] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 20% [02:00:03] RECOVERY - mem21 Current Load on mem21 is OK: LOAD OK - total load average: 2.38, 2.98, 3.35 [02:00:12] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 76% [02:07:31] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 43% [02:09:27] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 32% [02:10:16] PROBLEM - phorge1 issue-tracker.wikitide.org HTTPS on phorge1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4248 bytes in 0.032 second response time [02:10:27] PROBLEM - db1 MariaDB on db1 is CRITICAL: Can't connect to server on 'db1.wikitide.net' (115) [02:11:33] PROBLEM - db1 MariaDB Connections on db1 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:47Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(47): mysqli_real_connect(Object(mysqli), 'db1.wikitide.ne...', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, false)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_conne [02:11:33] ine 47Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:47Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(47): mysqli_real_connect(Object(mysqli), 'db1.wikitide.ne...', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, false)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 47 [02:11:33] PROBLEM - matomo1 HTTPS on matomo1 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 426 bytes in 0.034 second response time [02:11:48] PROBLEM - phorge1 phorge-static.wikitide.org HTTPS on phorge1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [02:18:14] PROBLEM - projects.dmvpetridish.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns0.wixdns.net.', 'ns1.wixdns.net.'], 'CNAME': 'mw-lb.wikitide.org.'} [02:18:54] RECOVERY - farthestfrontier.wiki - reverse DNS on sslhost is OK: SSL OK - farthestfrontier.wiki reverse DNS resolves to cp5.wikitide.net - CNAME FLAT [02:19:04] PROBLEM - wiki.chevrine.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns1.digitalocean.com.', 'ns2.digitalocean.com.', 'ns3.digitalocean.com.'], 'CNAME': 'mw-lb.wikitide.org.'} [02:19:08] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [02:19:34] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com All nameservers failed to answer the query. [02:19:56] PROBLEM - www.lgbtqia.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['grannbo.ns.cloudflare.com.', 'rodrigo.ns.cloudflare.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [02:19:57] PROBLEM - wiki.colleimadcat.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['dns29.hichina.com.', 'dns30.hichina.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [02:20:03] PROBLEM - meta.sagan4.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns59.domaincontrol.com.', 'ns60.domaincontrol.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [02:20:03] PROBLEM - www.greatamerica.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['john.ns.cloudflare.com.', 'vida.ns.cloudflare.com.'], 'CNAME': 'mw-lb.wikitide.org.'} [02:20:25] PROBLEM - wikitide.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikitide.com could not be found [02:20:26] PROBLEM - beta.sagan4.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns59.domaincontrol.com.', 'ns60.domaincontrol.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [02:20:27] PROBLEM - www.polandballwiki.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['john.ns.cloudflare.com.', 'vida.ns.cloudflare.com.'], 'CNAME': 'mw-lb.wikitide.org.'} [02:20:29] PROBLEM - alpha.sagan4.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns59.domaincontrol.com.', 'ns60.domaincontrol.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [02:20:38] PROBLEM - hsck.lophocmatngu.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['lia.ns.cloudflare.com.', 'gerald.ns.cloudflare.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [02:20:47] PROBLEM - wiki.myehs.eu - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['fred.ns.cloudflare.com.', 'ivy.ns.cloudflare.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [02:21:02] PROBLEM - data.lophocmatngu.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['lia.ns.cloudflare.com.', 'gerald.ns.cloudflare.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [02:21:11] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [02:21:23] PROBLEM - www.farthestfrontier.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['cleo.ns.cloudflare.com.', 'may.ns.cloudflare.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [02:21:39] RECOVERY - wiki.burning.art - reverse DNS on sslhost is OK: SSL OK - wiki.burning.art reverse DNS resolves to cp5.wikitide.net - CNAME OK [02:23:00] PROBLEM - mon21 Current Load on mon21 is WARNING: LOAD WARNING - total load average: 0.06, 0.31, 5.99 [02:23:35] PROBLEM - mason.sagan4.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns59.domaincontrol.com.', 'ns60.domaincontrol.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [02:24:13] RECOVERY - polandballwiki.com - reverse DNS on sslhost is OK: SSL OK - polandballwiki.com reverse DNS resolves to cp5.wikitide.net - CNAME FLAT [02:24:18] RECOVERY - lophocmatngu.wiki - reverse DNS on sslhost is OK: SSL OK - lophocmatngu.wiki reverse DNS resolves to cp5.wikitide.net - CNAME FLAT [02:24:19] RECOVERY - hkrail.wiki - reverse DNS on sslhost is OK: SSL OK - hkrail.wiki reverse DNS resolves to cp5.wikitide.net - CNAME FLAT [02:24:22] RECOVERY - rosettacode.org - reverse DNS on sslhost is OK: SSL OK - rosettacode.org reverse DNS resolves to cp5.wikitide.net - NS RECORDS OK [02:24:27] RECOVERY - www.rosettacode.org - reverse DNS on sslhost is OK: SSL OK - www.rosettacode.org reverse DNS resolves to cp5.wikitide.net - NS RECORDS OK [02:24:34] RECOVERY - www.wikitide.org - reverse DNS on sslhost is OK: SSL OK - www.wikitide.org reverse DNS resolves to cp3.wikitide.net - NS RECORDS OK [02:24:35] RECOVERY - rosettacode.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rosettacode.org' will expire on Sat 20 Jan 2024 06:52:33 AM GMT +0000. [02:24:40] RECOVERY - wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Tue 20 Feb 2024 12:20:10 PM GMT +0000. [02:24:55] RECOVERY - wikitide.org - reverse DNS on sslhost is OK: SSL OK - wikitide.org reverse DNS resolves to cp3.wikitide.net - NS RECORDS OK [02:25:03] RECOVERY - wikitide.net - reverse DNS on sslhost is OK: SSL OK - wikitide.net reverse DNS resolves to cp3.wikitide.net - NS RECORDS OK [02:25:36] RECOVERY - greatamerica.wiki - reverse DNS on sslhost is OK: SSL OK - greatamerica.wiki reverse DNS resolves to cp5.wikitide.net - CNAME FLAT [02:25:37] RECOVERY - distrowiki.net - reverse DNS on sslhost is OK: SSL OK - distrowiki.net reverse DNS resolves to cp3.wikitide.net - CNAME FLAT [02:25:37] RECOVERY - sdiy.info - reverse DNS on sslhost is OK: SSL OK - sdiy.info reverse DNS resolves to cp5.wikitide.net - NS RECORDS OK [02:27:00] RECOVERY - mon21 Current Load on mon21 is OK: LOAD OK - total load average: 0.06, 0.18, 4.64 [02:28:10] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/6d8a06934bd0...1db17177b6bf [02:28:12] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 43% [02:28:12] [02WikiTideOrg/puppet] 07Universal-Omega 031db1717 - Spacing [02:32:04] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [02:34:03] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 49% [02:36:01] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 73% [02:37:57] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 42% [02:43:46] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 38% [02:49:00] PROBLEM - cp2 HTTPS on cp2 is UNKNOWN: [02:59:33] RECOVERY - db1 MariaDB Connections on db1 is OK: OK connection usage: 0.2%Current connections: 1 [03:00:27] RECOVERY - db1 MariaDB on db1 is OK: Uptime: 68 Threads: 2 Questions: 177 Slow queries: 36 Opens: 34 Open tables: 12 Queries per second avg: 2.602 [03:03:33] RECOVERY - matomo1 HTTPS on matomo1 is OK: HTTP OK: HTTP/2 200 - 224 bytes in 0.208 second response time [03:15:38] PROBLEM - mem21 memcached on mem21 is UNKNOWN: [03:15:38] PROBLEM - bast21 Puppet on bast21 is UNKNOWN: [03:15:41] PROBLEM - puppet21 HTTPS on puppet21 is UNKNOWN: [03:17:01] PROBLEM - wiki.colleimadcat.com - reverse DNS on sslhost is UNKNOWN: PROBLEM - newcascadia.net - reverse DNS on sslhost is UNKNOWN: [03:17:01] PROBLEM - data.lophocmatngu.wiki - reverse DNS on sslhost is UNKNOWN: PROBLEM - wiki.chevrine.com - reverse DNS on sslhost is UNKNOWN: PROBLEM - www.lgbtqia.wiki - reverse DNS on sslhost is UNKNOWN: PROBLEM - alpha.sagan4.org - reverse DNS on sslhost is UN [03:17:01] xceeded.> PROBLEM - projects.dmvpetridish.com - reverse DNS on sslhost is UNKNOWN: [03:17:01] PROBLEM - hsck.lophocmatngu.wiki - reverse DNS on sslhost is UNKNOWN: [03:17:01] PROBLEM - meta.sagan4.org - reverse DNS on sslhost is UNKNOWN: [03:17:02] PROBLEM - www.greatamerica.wiki - reverse DNS on sslhost is UNKNOWN: [03:17:02] PROBLEM - www.polandballwiki.com - reverse DNS on sslhost is UNKNOWN: [03:17:03] PROBLEM - www.farthestfrontier.wiki - reverse DNS on sslhost is UNKNOWN: [03:17:03] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is UNKNOWN: [03:17:04] PROBLEM - wiki.myehs.eu - reverse DNS on sslhost is UNKNOWN: PROBLEM - wikitide.com - reverse DNS on sslhost is UNKNOWN: [03:17:04] PROBLEM - beta.sagan4.org - reverse DNS on sslhost is UNKNOWN: [03:17:05] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to cp2.wikitide.net port 443 after 31 ms: Couldn't connect to server [03:17:25] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 40% [03:18:04] PROBLEM - nexttide.org - reverse DNS on sslhost is UNKNOWN: [03:18:04] PROBLEM - mason.sagan4.org - reverse DNS on sslhost is UNKNOWN: [03:18:05] PROBLEM - bast21 Puppet on bast21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:18:13] PROBLEM - mem21 memcached on mem21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:18:15] PROBLEM - puppet21 HTTPS on puppet21 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [03:18:57] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 22% [03:19:01] PROBLEM - mon21 Current Load on mon21 is CRITICAL: LOAD CRITICAL - total load average: 20.15, 34.36, 30.60 [03:19:07] PROBLEM - mem21 Current Load on mem21 is WARNING: LOAD WARNING - total load average: 3.15, 3.53, 3.37 [03:21:02] RECOVERY - mem21 Current Load on mem21 is OK: LOAD OK - total load average: 2.61, 3.24, 3.28 [03:25:31] RECOVERY - phorge1 issue-tracker.wikitide.org HTTPS on phorge1 is OK: HTTP OK: HTTP/1.1 200 OK - 115674 bytes in 0.321 second response time [03:25:48] RECOVERY - phorge1 phorge-static.wikitide.org HTTPS on phorge1 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 58195 bytes in 0.376 second response time [03:26:49] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.188 second response time [03:27:42] PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.155 second response time [03:35:47] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:45] PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.116 second response time [03:44:26] [02WikiTideOrg/ssl] 07WikiTideSSL pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/WikiTideOrg/ssl/compare/1491bd65da1c...d869af88bf75 [03:44:26] PROBLEM - jobrunner1 Disk Space on jobrunner1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.104: Connection reset by peer [03:44:28] [02WikiTideOrg/ssl] 07WikiTideSSL 03d869af8 - Bot: Adding SSL cert for wiki.myehs.eu Certificate committed by universalomega [03:44:30] PROBLEM - jobrunner1 php-fpm on jobrunner1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.104: Connection reset by peer [03:44:33] PROBLEM - jobrunner1 PowerDNS Recursor on jobrunner1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.104: Connection reset by peer [03:44:42] PROBLEM - jobrunner1 NTP time on jobrunner1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.104: Connection reset by peer [03:44:47] PROBLEM - jobrunner1 ferm_active on jobrunner1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.104: Connection reset by peer [03:44:56] PROBLEM - jobrunner1 JobRunner Service on jobrunner1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.104: Connection reset by peer [03:45:00] PROBLEM - mon21 Current Load on mon21 is WARNING: LOAD WARNING - total load average: 0.32, 0.31, 5.75 [03:45:06] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.104: Connection reset by peer [03:45:13] [02WikiTideOrg/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/ssl/compare/d869af88bf75...92c03967bf16 [03:45:15] [02WikiTideOrg/ssl] 07Universal-Omega 0392c0396 - Remove duplicate [03:45:16] PROBLEM - jobrunner1 conntrack_table_size on jobrunner1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.104: Connection reset by peer [03:45:25] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [03:45:35] PROBLEM - www.lgbtqia.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['grannbo.ns.cloudflare.com.', 'rodrigo.ns.cloudflare.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [03:45:36] PROBLEM - wiki.colleimadcat.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['dns29.hichina.com.', 'dns30.hichina.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [03:45:39] PROBLEM - data.lophocmatngu.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['lia.ns.cloudflare.com.', 'gerald.ns.cloudflare.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [03:45:41] PROBLEM - www.greatamerica.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['john.ns.cloudflare.com.', 'vida.ns.cloudflare.com.'], 'CNAME': 'mw-lb.wikitide.org.'} [03:45:41] PROBLEM - meta.sagan4.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns59.domaincontrol.com.', 'ns60.domaincontrol.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [03:45:45] PROBLEM - jobrunner1 nutcracker process on jobrunner1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.104: Connection reset by peer [03:45:46] PROBLEM - jobrunner1 nutcracker port on jobrunner1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.104: Connection reset by peer [03:46:01] PROBLEM - www.farthestfrontier.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['cleo.ns.cloudflare.com.', 'may.ns.cloudflare.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [03:46:01] PROBLEM - wiki.chevrine.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns1.digitalocean.com.', 'ns2.digitalocean.com.', 'ns3.digitalocean.com.'], 'CNAME': 'mw-lb.wikitide.org.'} [03:46:04] PROBLEM - wikitide.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikitide.com could not be found [03:46:05] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [03:46:06] PROBLEM - www.polandballwiki.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['john.ns.cloudflare.com.', 'vida.ns.cloudflare.com.'], 'CNAME': 'mw-lb.wikitide.org.'} [03:46:06] PROBLEM - beta.sagan4.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns59.domaincontrol.com.', 'ns60.domaincontrol.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [03:46:08] PROBLEM - alpha.sagan4.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns59.domaincontrol.com.', 'ns60.domaincontrol.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [03:46:09] PROBLEM - cp3 ferm_active on cp3 is CRITICAL: ERROR ferm input drop default policy not set, ferm might not have been started correctly [03:46:16] PROBLEM - hsck.lophocmatngu.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['lia.ns.cloudflare.com.', 'gerald.ns.cloudflare.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [03:46:26] PROBLEM - wiki.myehs.eu - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['fred.ns.cloudflare.com.', 'ivy.ns.cloudflare.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [03:46:40] PROBLEM - projects.dmvpetridish.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns1.wixdns.net.', 'ns0.wixdns.net.'], 'CNAME': 'mw-lb.wikitide.org.'} [03:46:40] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com All nameservers failed to answer the query. [03:47:00] RECOVERY - mon21 Current Load on mon21 is OK: LOAD OK - total load average: 0.15, 0.25, 5.07 [03:47:14]