[00:01:43] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 10.54, 8.16, 7.33 [00:03:11] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query rosettacode.org. IN NS: Server 1.1.1.1 UDP port 53 answered SERVFAIL [00:08:58] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [00:11:31] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.68, 7.87, 7.77 [00:13:29] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.11, 7.92, 7.79 [00:15:27] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.86, 7.61, 7.70 [00:23:18] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.31, 7.87, 7.76 [00:25:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.24, 7.61, 7.68 [00:27:14] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.23, 7.77, 7.72 [00:29:11] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.97, 7.73, 7.71 [00:31:09] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.31, 7.80, 7.73 [00:32:42] PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org All nameservers failed to answer the query. [00:35:05] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.53, 7.64, 7.66 [00:42:56] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.70, 7.47, 7.47 [00:50:24] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mlrpgspeedruns.com. IN NS: Server 1.1.1.1 UDP port 53 answered SERVFAIL [00:50:47] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.90, 7.49, 7.54 [01:24:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.22, 7.36, 7.49 [01:30:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.57, 7.82, 7.66 [01:31:45] PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org All nameservers failed to answer the query. [01:32:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.42, 7.35, 7.51 [01:38:41] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [01:40:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.43, 7.57, 7.46 [01:44:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.80, 7.34, 7.41 [01:56:16] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 4.71, 6.04, 6.78 [02:01:17] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query rosettacode.org. IN NS: Server 1.1.1.1 UDP port 53 answered SERVFAIL [02:20:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.41, 6.95, 6.61 [02:22:16] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 5.88, 6.58, 6.52 [02:38:30] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [03:29:51] PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org All nameservers failed to answer the query. [03:57:08] PROBLEM - phorge21 issue-tracker.wikitide.org HTTPS on phorge21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 3906 bytes in 0.044 second response time [03:57:09] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.53, 6.71, 6.18 [03:57:38] PROBLEM - phorge21 phorge-static.wikitide.org HTTPS on phorge21 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [03:58:24] @agentisai ^ seems phorge is down [03:58:31] db is locked [03:58:46] hmm? [03:58:54] I'll look into it [03:59:07] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.91, 7.48, 6.52 [03:59:22] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query rosettacode.org. IN NS: Server 1.1.1.1 UDP port 53 answered SERVFAIL [03:59:26] it is static IE R2 I think, no? [03:59:50] no, it's the database [03:59:56] oh hmm [04:00:25] the alert was for phorge-static so I thought R2 lol [04:08:43] [1/9] ```sql [04:08:43] [2/9] +--------------------+ [04:08:43] [3/9] | @@global.read_only | [04:08:43] [4/9] +--------------------+ [04:08:44] [5/9] | 1 | [04:08:44] [6/9] +--------------------+ [04:08:44] [7/9] 1 row in set (0.000 sec) [04:08:45] [8/9] ``` [04:08:45] [9/9] Howd it get read only anyway @agentisai [04:18:45] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 5.31, 7.25, 7.54 [04:20:24] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com All nameservers failed to answer the query. [04:20:58] @agentisai as soon as I take it out of read only it goes back in [04:21:13] I think it's my fault [04:21:20] I'm backing up the database [04:21:45] Oh, wikitide-backups is designed to not do this so wonder why it is. [04:23:21] oh wikitide-backup isn't being used. [04:24:38] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 5.02, 5.60, 6.66 [04:28:31] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.22, 6.72, 6.84 [04:38:20] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.20, 7.65, 7.41 [04:46:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.99, 8.61, 7.82 [05:20:24] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mlrpgspeedruns.com. IN NS: Server 1.1.1.1 UDP port 53 answered SERVFAIL [05:37:59] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [05:40:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.73, 7.75, 7.99 [05:44:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.17, 7.91, 8.01 [05:52:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.09, 7.76, 7.94 [05:56:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.22, 7.84, 7.92 [06:00:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.24, 7.71, 7.86 [06:06:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.55, 7.95, 7.89 [06:07:54] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [06:10:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.78, 7.99, 7.94 [06:12:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.38, 8.20, 8.02 [06:20:24] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com All nameservers failed to answer the query. [06:28:01] PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org All nameservers failed to answer the query. [06:28:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.22, 7.56, 7.96 [06:30:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.23, 8.10, 8.13 [06:37:48] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [06:50:24] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mlrpgspeedruns.com. IN NS: Server 1.1.1.1 UDP port 53 answered SERVFAIL [07:04:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.36, 7.73, 7.99 [07:16:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.54, 8.02, 7.91 [07:28:02] RECOVERY - Host swiftac31 is UP: PING OK - Packet loss = 66%, RTA = 0.26 ms [07:28:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.79, 7.86, 7.97 [07:29:15] PROBLEM - swiftac31 ferm_active on swiftac31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:29:30] PROBLEM - swiftac31 NTP time on swiftac31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:30:30] PROBLEM - swiftac31 Puppet on swiftac31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:30:30] PROBLEM - swiftac31 Current Load on swiftac31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:30:30] PROBLEM - swiftac31 PowerDNS Recursor on swiftac31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:30:30] PROBLEM - swiftac31 APT on swiftac31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:30:50] PROBLEM - swiftac31 conntrack_table_size on swiftac31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:33:21] PROBLEM - Host swiftac31 is DOWN: PING CRITICAL - Packet loss = 100% [07:34:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.20, 7.83, 7.91 [07:36:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.72, 7.84, 7.92 [07:37:36] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [07:40:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.37, 8.09, 8.00 [07:42:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.82, 7.94, 7.95 [07:44:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.43, 8.10, 8.01 [09:49:36] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 1.82, 2.90, 4.83 [09:49:51] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com All nameservers failed to answer the query. [09:49:53] RECOVERY - phorge21 phorge-static.wikitide.org HTTPS on phorge21 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 58196 bytes in 0.512 second response time [09:50:11] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [09:50:23] PROBLEM - db21 Backups SQL on db21 is CRITICAL: connect to address 10.0.2.3 port 5666: Connection refusedconnect to host 10.0.2.3 port 5666: Connection refused [09:51:35] RECOVERY - phorge21 issue-tracker.wikitide.org HTTPS on phorge21 is OK: HTTP OK: HTTP/1.1 200 OK - 118685 bytes in 0.166 second response time [09:51:36] PROBLEM - db21 Backups SQL wtglobal on db21 is CRITICAL: connect to address 10.0.2.3 port 5666: Connection refusedconnect to host 10.0.2.3 port 5666: Connection refused [09:51:37] PROBLEM - db21 conntrack_table_size on db21 is CRITICAL: connect to address 10.0.2.3 port 5666: Connection refusedconnect to host 10.0.2.3 port 5666: Connection refused [09:51:40] PROBLEM - db21 Puppet on db21 is CRITICAL: connect to address 10.0.2.3 port 5666: Connection refusedconnect to host 10.0.2.3 port 5666: Connection refused [09:51:50] PROBLEM - db21 Disk Space on db21 is CRITICAL: connect to address 10.0.2.3 port 5666: Connection refusedconnect to host 10.0.2.3 port 5666: Connection refused [09:51:54] PROBLEM - db21 ferm_active on db21 is CRITICAL: connect to address 10.0.2.3 port 5666: Connection refusedconnect to host 10.0.2.3 port 5666: Connection refused [09:52:00] PROBLEM - db21 NTP time on db21 is CRITICAL: connect to address 10.0.2.3 port 5666: Connection refusedconnect to host 10.0.2.3 port 5666: Connection refused [09:52:06] PROBLEM - db21 PowerDNS Recursor on db21 is CRITICAL: connect to address 10.0.2.3 port 5666: Connection refusedconnect to host 10.0.2.3 port 5666: Connection refused [09:52:11] PROBLEM - db21 APT on db21 is CRITICAL: connect to address 10.0.2.3 port 5666: Connection refusedconnect to host 10.0.2.3 port 5666: Connection refused [09:52:12] PROBLEM - db21 Current Load on db21 is CRITICAL: connect to address 10.0.2.3 port 5666: Connection refusedconnect to host 10.0.2.3 port 5666: Connection refused [09:53:29] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.82, 6.38, 5.85 [09:55:27] RECOVERY - db21 Backups SQL wtglobal on db21 is OK: FILE_AGE OK: /var/log/backup-logs/sql-wtglobal-backup-weekly.log is 449716 seconds old and 1087 bytes [09:55:30] RECOVERY - db21 conntrack_table_size on db21 is OK: OK: nf_conntrack is 0 % full [09:55:40] RECOVERY - db21 Puppet on db21 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [09:55:42] RECOVERY - db21 ferm_active on db21 is OK: OK ferm input default policy is set [09:55:49] RECOVERY - db21 Disk Space on db21 is OK: DISK OK - free space: / 241604MiB (50% inode=99%); [09:55:54] RECOVERY - db21 NTP time on db21 is OK: NTP OK: Offset 0.001471430063 secs [09:55:58] RECOVERY - db21 PowerDNS Recursor on db21 is OK: DNS OK: 0.218 seconds response time. wikitide.org returns 2607:5300:205:200::2aa8,51.79.55.151 [09:56:00] RECOVERY - db21 APT on db21 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [09:56:00] RECOVERY - db21 Current Load on db21 is OK: LOAD OK - total load average: 8.70, 7.32, 6.29 [09:56:08] PROBLEM - db21 Backups SQL on db21 is WARNING: FILE_AGE WARNING: /var/log/backup-logs/sql-backup.log is 1141064 seconds old and 1356 bytes [09:59:22] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.46, 7.53, 6.58 [10:15:02] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.93, 7.81, 7.18 [10:17:00] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.31, 7.51, 7.14 [10:30:43] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 5.36, 6.25, 6.77 [11:18:37] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.93, 6.98, 6.34 [11:36:16] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 5.84, 6.76, 6.77 [11:53:08] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.88, 7.54, 6.83 [11:55:06] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.56, 7.06, 6.74 [11:57:03] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 5.88, 6.70, 6.65 [12:15:23] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [12:44:24] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [13:37:10] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.90, 6.11, 5.50 [13:39:07] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 6.45, 6.29, 5.65 [13:42:29] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [14:11:32] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [14:40:36] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [15:09:39] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [16:07:47] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [16:36:50] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [16:53:08] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.03, 6.96, 6.33 [16:55:06] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 6.51, 6.63, 6.28 [17:05:54] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [17:08:45] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.31, 6.99, 6.57 [17:10:41] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 6.29, 6.59, 6.46 [17:14:34] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.22, 6.99, 6.67 [17:16:32] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 5.76, 6.60, 6.57 [17:20:24] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.72, 7.43, 6.91 [17:22:22] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.00, 7.04, 6.82 [17:24:19] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.64, 7.48, 6.99 [17:26:17] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.32, 6.93, 6.85 [17:28:16] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 6.48, 6.64, 6.74 [17:34:57] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [17:36:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.19, 6.81, 6.71 [17:38:16] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 6.36, 6.67, 6.68 [17:43:15] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/05d07b199a80...426e3aebf81a [17:43:16] [02WikiTideOrg/puppet] 07Universal-Omega 03426e3ae - Update jobrunner-hi [18:04:01] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [18:11:08] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.20, 6.67, 6.39 [18:17:01] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.14, 7.68, 6.92 [18:18:59] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.83, 7.34, 6.89 [18:23:07] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/426e3aebf81a...a79e3528e33e [18:23:09] [02WikiTideOrg/puppet] 07Universal-Omega 03a79e352 - Absent rottenlinks cron [18:28:46] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 6.51, 6.68, 6.78 [18:44:24] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.50, 6.87, 6.71 [18:48:19] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.61, 7.77, 7.12 [18:49:32] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03mwdeploy-private-rsync [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/commit/29c04873fd93 [18:49:33] [02WikiTideOrg/puppet] 07Universal-Omega 0329c0487 - mwdeploy: support using -private in rsync command [18:49:35] [02puppet] 07Universal-Omega created branch 03mwdeploy-private-rsync - 13https://github.com/WikiTideOrg/puppet [18:49:39] [02puppet] 07Universal-Omega opened pull request 03#45: mwdeploy: support using -private in rsync command - 13https://github.com/WikiTideOrg/puppet/pull/45 [18:51:00] WikiTideOrg/puppet - Universal-Omega the build has errored. [18:51:29] [02puppet] 07Universal-Omega synchronize pull request 03#45: mwdeploy: support using -private in rsync command - 13https://github.com/WikiTideOrg/puppet/pull/45 [18:51:31] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03mwdeploy-private-rsync [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/29c04873fd93...efed1bd784e2 [18:51:32] [02WikiTideOrg/puppet] 07Universal-Omega 03efed1bd - Fix [18:52:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.74, 7.57, 7.18 [18:53:09] WikiTideOrg/puppet - Universal-Omega the build passed. [18:56:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.48, 7.68, 7.28 [19:02:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.01, 7.29, 7.26 [19:12:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.58, 7.07, 7.02 [19:14:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.37, 6.64, 6.86 [19:20:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.54, 7.51, 7.17 [19:22:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.62, 7.14, 7.08 [19:32:16] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 6.05, 6.15, 6.67 [19:39:08] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.00, 6.66, 6.71 [19:41:06] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 6.63, 6.72, 6.74 [19:48:54] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.38, 8.18, 7.31 [19:50:51] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.49, 7.91, 7.31 [20:06:32] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 5.16, 6.19, 6.70 [20:29:00] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.62, 6.34, 6.03 [20:30:33] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [20:34:53] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.31, 7.19, 6.48 [20:36:50] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 5.79, 6.60, 6.35 [20:42:40] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.82, 7.19, 6.70 [20:52:28] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 5.11, 6.39, 6.60 [21:00:34] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [21:30:33] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [23:18:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.60, 6.85, 6.25 [23:28:16] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 6.20, 6.78, 6.57 [23:30:34] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.