[01:00:26] RECOVERY - phorge21 Backups Phorge Static on phorge21 is OK: FILE_AGE OK: /var/log/phorge-backup.log is 25 seconds old and 0 bytes [01:07:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.97, 7.41, 7.99 [01:09:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 7.20, 7.63, 8.02 [01:11:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.66, 7.39, 7.86 [01:15:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.45, 7.81, 7.91 [01:17:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.36, 7.78, 7.90 [01:19:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.66, 8.35, 8.11 [01:31:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.38, 7.59, 7.94 [01:43:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.67, 8.00, 7.84 [01:45:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.94, 7.91, 7.82 [01:47:26] !log [agent@jobrunner21] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/SemanticMediaWiki/maintenance/rebuildData.php --wiki=rosettacodewiki (END - exit=0) [01:47:28] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [01:49:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.35, 8.58, 8.10 [02:15:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.14, 7.46, 7.90 [02:21:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.23, 7.38, 7.65 [02:23:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.73, 7.62, 7.72 [02:33:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.02, 7.22, 7.36 [02:35:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.52, 6.92, 7.23 [02:39:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.11, 7.33, 7.31 [02:42:40] !log [agent@jobrunner21] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/SemanticMediaWiki/maintenance/rebuildData.php --wiki=rosettacodewiki (END - exit=0) [02:42:42] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [02:43:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.75, 7.84, 7.56 [02:45:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.16, 8.16, 7.69 [02:49:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.64, 7.77, 7.68 [02:55:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.53, 7.81, 7.69 [02:59:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.15, 7.86, 7.77 [03:07:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.95, 8.01, 7.84 [03:13:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.93, 7.82, 7.89 [03:17:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.32, 7.89, 7.83 [03:19:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.29, 7.86, 7.84 [03:21:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.14, 8.20, 7.98 [03:31:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 5.35, 7.15, 7.74 [03:37:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.20, 7.35, 7.60 [03:41:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.86, 7.85, 7.78 [03:45:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.31, 7.85, 7.76 [03:47:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.75, 7.88, 7.79 [03:51:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.56, 7.87, 7.79 [03:53:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.69, 7.57, 7.67 [04:11:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.28, 7.56, 7.23 [04:21:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.71, 7.96, 7.72 [04:37:31] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 5.72, 6.13, 6.74 [04:41:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.14, 6.85, 6.88 [04:43:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.57, 6.73, 6.83 [04:47:31] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 5.81, 6.32, 6.65 [04:53:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.85, 7.87, 7.20 [04:55:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.79, 7.52, 7.16 [05:01:31] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 5.40, 6.41, 6.78 [05:03:11] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [05:14:05] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.87, 6.92, 6.82 [05:17:54] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 6.16, 6.63, 6.75 [05:23:39] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.31, 7.45, 7.05 [05:25:34] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.49, 7.23, 7.03 [05:39:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.30, 7.55, 7.27 [05:41:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.69, 7.52, 7.30 [06:03:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.53, 8.11, 7.60 [06:05:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.25, 7.47, 7.44 [06:15:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.08, 7.33, 7.21 [06:21:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.45, 7.73, 7.45 [06:25:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.72, 7.80, 7.50 [06:27:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.75, 7.81, 7.55 [06:33:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.36, 7.90, 7.66 [06:45:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.31, 7.81, 7.91 [06:59:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.71, 7.60, 7.47 [07:01:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.72, 7.30, 7.39 [07:17:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.20, 7.45, 7.24 [07:33:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.49, 7.74, 7.95 [07:39:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 10.09, 8.40, 8.08 [08:02:19] PROBLEM - cloud3 MegaRAID on cloud3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:02:34] PROBLEM - cloud3 Puppet on cloud3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:03:54] PROBLEM - cloud3 get_raid_status_megacli on cloud3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:06:20] PROBLEM - cloud3 Current Load on cloud3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:06:26] PROBLEM - cloud3 Disk Space on cloud3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:06:45] PROBLEM - cloud3 ferm_active on cloud3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:06:51] PROBLEM - cloud3 conntrack_table_size on cloud3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:06:52] PROBLEM - cloud3 PowerDNS Recursor on cloud3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:07:09] PROBLEM - cloud3 MegaRAID on cloud3 is UNKNOWN: NRPE: Unable to read output [08:07:29] PROBLEM - cloud3 Puppet on cloud3 is UNKNOWN: UNKNOWN: Failed to check. Reason is: failed_to_parse_summary_file [08:08:16] RECOVERY - cloud3 Current Load on cloud3 is OK: LOAD OK - total load average: 7.43, 12.16, 6.36 [08:08:22] RECOVERY - cloud3 Disk Space on cloud3 is OK: DISK OK - free space: / 86699MiB (95% inode=98%); [08:08:42] RECOVERY - cloud3 ferm_active on cloud3 is OK: OK ferm input default policy is set [08:08:42] PROBLEM - cloud3 get_raid_status_megacli on cloud3 is UNKNOWN: NRPE: Unable to read output [08:08:49] RECOVERY - cloud3 conntrack_table_size on cloud3 is OK: OK: nf_conntrack is 0 % full [08:08:51] RECOVERY - cloud3 PowerDNS Recursor on cloud3 is OK: DNS OK: 0.269 seconds response time. wikitide.org returns 2607:5300:205:200::2aa8,51.79.55.151 [08:53:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.11, 7.21, 7.81 [08:57:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.33, 7.00, 7.52 [08:59:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.93, 7.29, 7.56 [09:09:31] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 6.00, 6.05, 6.79 [09:51:13] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.41, 7.25, 6.80 [09:53:08] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.03, 7.27, 6.87 [09:56:57] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 5.33, 6.59, 6.70 [10:04:37] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.49, 7.20, 6.93 [10:08:26] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.83, 7.73, 7.19 [10:10:21] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.74, 7.39, 7.13 [10:17:59] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 5.76, 6.24, 6.72 [10:34:15] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.34, 7.02, 6.82 [10:38:04] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 6.37, 6.63, 6.71 [10:44:47] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.39, 7.13, 6.88 [10:46:41] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.75, 6.94, 6.84 [10:52:25] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 6.00, 6.68, 6.79 [10:58:10] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.41, 6.81, 6.84 [11:03:54] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 5.88, 6.63, 6.79 [11:07:44] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.93, 6.82, 6.84 [11:09:39] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 6.42, 6.65, 6.78 [11:15:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.42, 7.07, 6.91 [11:17:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.60, 7.77, 7.20 [11:19:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.25, 7.64, 7.23 [11:25:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.07, 7.47, 7.24 [11:27:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.24, 7.22, 7.17 [11:29:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.75, 7.89, 7.42 [11:31:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.19, 7.50, 7.33 [11:33:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.07, 7.98, 7.52 [11:39:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.84, 7.75, 7.59 [11:45:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.65, 7.62, 7.53 [11:57:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.48, 7.79, 7.92 [12:07:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 10.59, 8.24, 7.85 [12:15:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.38, 7.95, 7.97 [12:17:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.48, 8.66, 8.23 [12:47:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.89, 7.39, 7.96 [12:53:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.81, 7.95, 8.00 [12:57:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.80, 7.66, 7.89 [13:05:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.18, 7.78, 7.76 [13:07:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.23, 7.63, 7.72 [13:11:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.67, 7.65, 7.65 [13:15:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.74, 8.00, 7.82 [13:19:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.12, 7.80, 7.75 [13:23:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.43, 7.82, 7.79 [13:27:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.90, 8.27, 7.97 [13:49:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 5.99, 7.33, 7.86 [13:53:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.67, 7.78, 7.91 [13:55:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.76, 7.76, 7.89 [14:15:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 10.19, 8.37, 7.76 [14:29:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.20, 7.99, 7.99 [14:35:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.06, 8.04, 7.94 [14:39:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.81, 7.80, 7.86 [14:43:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.38, 7.90, 7.86 [14:47:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.96, 7.95, 7.91 [14:49:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.15, 8.18, 8.00 [15:07:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.01, 7.59, 7.88 [15:13:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.45, 7.83, 7.85 [15:15:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.64, 7.70, 7.80 [15:17:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.32, 8.31, 8.01 [16:11:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.76, 7.17, 7.82 [16:16:28] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query nexttide.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [16:19:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.27, 7.24, 7.51 [16:23:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.33, 7.46, 7.56 [16:27:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.43, 7.86, 7.70 [16:31:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.78, 7.87, 7.74 [16:32:08] PROBLEM - cp5 Nginx Backend for mail21 on cp5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:33:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 10.75, 8.63, 8.01 [16:34:42] PROBLEM - cp5 NTP time on cp5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:35:17] PROBLEM - cp5 HTTPS on cp5 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 2611 bytes in 0.689 second response time [16:36:41] RECOVERY - cp5 NTP time on cp5 is OK: NTP OK: Offset 0.003420978785 secs [16:37:15] RECOVERY - cp5 HTTPS on cp5 is OK: HTTP OK: HTTP/2 200 - 2958 bytes in 1.757 second response time [16:37:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.80, 7.91, 7.85 [16:39:56] RECOVERY - cp5 Nginx Backend for mail21 on cp5 is OK: TCP OK - 0.000 second response time on localhost port 8202 [16:40:28] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw21 [16:45:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.11, 7.59, 7.67 [16:47:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.62, 7.73, 7.72 [16:47:56] PROBLEM - cp5 PowerDNS Recursor on cp5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:49:57] RECOVERY - cp5 PowerDNS Recursor on cp5 is OK: DNS OK: 2.211 seconds response time. wikitide.org returns 15.235.167.159,2402:1f00:8000:800::1b3d [16:51:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.48, 8.20, 7.91 [16:55:22] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 7 backends are healthy [16:59:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.34, 7.85, 7.89 [17:00:46] PROBLEM - cp5 HTTPS on cp5 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [17:01:16] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:01:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.53, 8.20, 8.01 [17:03:25] PROBLEM - cp5 Nginx Backend for mw22 on cp5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:03:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.48, 7.80, 7.88 [17:04:50] RECOVERY - cp5 HTTPS on cp5 is OK: HTTP OK: HTTP/2 200 - 2980 bytes in 1.361 second response time [17:05:22] RECOVERY - cp5 Nginx Backend for mw22 on cp5 is OK: TCP OK - 0.000 second response time on localhost port 8101 [17:06:44] PROBLEM - cp5 Nginx Backend for test21 on cp5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:07:27] PROBLEM - cp5 Disk Space on cp5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:08:40] RECOVERY - cp5 Nginx Backend for test21 on cp5 is OK: TCP OK - 0.000 second response time on localhost port 8180 [17:09:27] RECOVERY - cp5 Disk Space on cp5 is OK: DISK OK - free space: / 5802MiB (30% inode=94%); [17:13:10] PROBLEM - cp5 HTTPS on cp5 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [17:13:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.29, 8.13, 7.94 [17:15:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.74, 7.96, 7.90 [17:17:11] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 7 backends are healthy [17:17:14] RECOVERY - cp5 HTTPS on cp5 is OK: HTTP OK: HTTP/2 200 - 2980 bytes in 1.395 second response time [17:19:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.58, 8.18, 7.98 [17:24:42] PROBLEM - cp5 Nginx Backend for phorge21 on cp5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:26:38] RECOVERY - cp5 Nginx Backend for phorge21 on cp5 is OK: TCP OK - 0.000 second response time on localhost port 8200 [17:27:10] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:29:11] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 7 backends are healthy [17:30:55] PROBLEM - cp5 ferm_active on cp5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:31:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.52, 7.58, 7.87 [17:32:56] RECOVERY - cp5 ferm_active on cp5 is OK: OK ferm input default policy is set [17:40:46] PROBLEM - cp5 Nginx Backend for mw21 on cp5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:42:46] RECOVERY - cp5 Nginx Backend for mw21 on cp5 is OK: TCP OK - 0.000 second response time on localhost port 8100 [17:43:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.01, 6.93, 7.13 [17:45:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.56, 6.72, 7.02 [17:59:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.06, 7.59, 7.31 [18:01:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.89, 7.40, 7.28 [18:07:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.37, 7.80, 7.43 [18:14:19] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp2.wikitide.net - NS RECORDS OK [18:29:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 5.93, 7.42, 7.89 [18:33:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 10.07, 8.07, 7.98 [18:37:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.45, 7.98, 7.99 [18:41:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.85, 7.81, 7.88 [18:50:19] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp5.wikitide.net - NS RECORDS OK [19:03:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.43, 7.68, 8.00 [19:05:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.33, 8.30, 8.18 [19:31:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.52, 7.51, 7.94 [19:33:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.08, 8.05, 8.08 [19:41:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.56, 7.57, 7.94 [19:43:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.62, 7.81, 7.97 [19:47:46] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [19:53:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.78, 7.59, 7.93 [19:55:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.14, 7.91, 8.01 [20:17:03] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp2.wikitide.net - NS RECORDS OK [20:24:26] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [21:03:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.24, 7.07, 7.78 [21:19:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.60, 7.61, 7.38 [21:23:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.99, 7.47, 7.39 [21:50:55] PROBLEM - jobrunner21 APT on jobrunner21 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (4 critical updates). [21:53:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.35, 7.76, 7.25 [22:01:07] PROBLEM - mw21 APT on mw21 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (4 critical updates). [22:03:22] PROBLEM - mw22 APT on mw22 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (4 critical updates). [22:05:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.97, 7.71, 7.68 [22:11:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.93, 8.23, 7.90 [22:19:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.00, 7.90, 7.97 [22:21:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.92, 8.22, 8.07 [22:39:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 5.89, 7.12, 7.83 [22:49:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.80, 8.70, 8.12 [22:51:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.11, 8.00, 7.93 [22:53:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 7.59, 8.05, 7.96 [22:59:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.10, 7.72, 7.92 [23:20:26] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp5.wikitide.net - NS RECORDS OK [23:31:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.09, 7.31, 7.08 [23:33:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.21, 6.96, 6.99 [23:35:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.57, 7.45, 7.16 [23:57:23] PROBLEM - cp2 Current Load on cp2 is WARNING: LOAD WARNING - total load average: 1.65, 1.74, 1.25 [23:59:23] PROBLEM - cp2 Current Load on cp2 is CRITICAL: LOAD CRITICAL - total load average: 2.50, 2.06, 1.43