[00:45:46] RECOVERY - mon21 Check correctness of the icinga configuration on mon21 is OK: Icinga configuration is correct [00:46:33] PROBLEM - wiki.myehs.eu - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['fred.ns.cloudflare.com.', 'ivy.ns.cloudflare.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [00:47:06] PROBLEM - swiftproxy31 SSH on swiftproxy31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:47:15] PROBLEM - Host swiftproxy31 is DOWN: PING CRITICAL - Packet loss = 100% [01:38:56] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [04:29:57] PROBLEM - mlrpgspeedruns.com - LetsEncrypt on sslhost is CRITICAL: connect to address mlrpgspeedruns.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [04:30:25] these warnings are quite annoyinh [04:37:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.05, 7.40, 7.98 [04:39:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.26, 7.61, 7.98 [04:41:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.82, 7.54, 7.90 [04:43:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.20, 8.04, 8.06 [04:58:54] RECOVERY - mlrpgspeedruns.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mlrpgspeedruns.com' will expire on Mon 04 Mar 2024 02:14:22 PM GMT +0000. [05:57:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.67, 7.43, 7.99 [05:59:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.32, 7.51, 7.93 [06:01:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.02, 7.45, 7.87 [06:11:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.85, 7.64, 7.62 [06:13:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.11, 7.40, 7.54 [06:17:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.26, 7.61, 7.57 [06:19:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.14, 7.08, 7.38 [06:29:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.04, 6.96, 7.08 [06:31:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.00, 6.88, 7.03 [06:35:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.23, 7.56, 7.26 [06:37:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.68, 7.17, 7.16 [06:49:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.09, 7.41, 7.17 [06:51:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.55, 7.47, 7.23 [07:03:55] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 5.82, 6.30, 6.77 [07:13:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.46, 7.30, 6.92 [07:17:55] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 5.61, 6.46, 6.68 [07:32:27] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.90, 7.17, 6.85 [07:33:19] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp5.wikitide.net - NS RECORDS OK [07:38:12] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 5.02, 6.38, 6.67 [07:42:03] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.43, 6.80, 6.76 [07:53:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.09, 7.43, 7.12 [07:55:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.52, 7.52, 7.20 [08:03:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.28, 8.06, 7.45 [08:05:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.53, 7.75, 7.41 [08:13:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.01, 6.99, 7.05 [08:15:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.68, 7.32, 7.18 [08:17:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.32, 7.70, 7.33 [08:19:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.59, 7.45, 7.27 [08:31:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.87, 7.50, 7.23 [08:33:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.69, 7.58, 7.30 [08:45:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.80, 8.09, 7.66 [08:49:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.81, 7.96, 7.76 [08:57:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.35, 8.17, 7.83 [09:01:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.97, 7.60, 7.68 [09:05:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.17, 7.62, 7.64 [09:11:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 5.76, 7.59, 7.71 [09:17:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.78, 7.84, 7.72 [09:19:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.56, 7.72, 7.69 [09:21:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.68, 7.98, 7.78 [09:23:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.11, 7.68, 7.70 [09:51:55] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 6.26, 6.45, 6.73 [10:00:42] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.94, 6.80, 6.73 [10:02:37] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 4.62, 6.05, 6.47 [10:10:18] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.22, 7.35, 6.86 [10:12:13] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 5.98, 6.78, 6.71 [10:16:04] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.43, 7.54, 7.04 [10:17:59] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.77, 7.74, 7.18 [10:19:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.56, 8.17, 7.41 [10:23:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.27, 7.72, 7.45 [10:33:55] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 4.91, 5.87, 6.68 [10:43:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.26, 6.84, 6.71 [10:51:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.09, 7.49, 7.05 [10:53:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.75, 7.73, 7.21 [11:09:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.85, 7.83, 7.42 [11:13:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.58, 7.81, 7.50 [11:21:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.16, 7.60, 7.45 [11:25:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.80, 7.87, 7.61 [11:37:55] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 4.91, 5.94, 6.74 [11:44:46] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.41, 7.80, 7.29 [11:50:32] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.79, 7.82, 7.54 [12:17:55] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 5.38, 6.04, 6.73 [12:25:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.87, 6.97, 6.78 [12:33:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.38, 7.52, 7.04 [13:17:45] !log [@jobrunner21] starting deploy of {'config': True, 'versions': ['']} to [mw21, mw22, jobrunner21] [13:17:47] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [13:19:36] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [13:22:11] !log [@jobrunner21] finished deploy of {'config': True, 'versions': ['']} to [mw21, mw22, jobrunner21] - SUCCESS in 265s [13:22:13] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [13:48:54] RECOVERY - rosettacode.org - reverse DNS on sslhost is OK: SSL OK - rosettacode.org reverse DNS resolves to cp5.wikitide.net - NS RECORDS OK [14:09:41] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [16:07:44] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp5.wikitide.net - NS RECORDS OK [17:01:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.63, 7.47, 7.99 [17:03:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.19, 7.81, 8.05 [17:07:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.57, 7.69, 7.95 [17:10:01] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [17:11:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.66, 7.94, 7.98 [17:13:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.44, 7.56, 7.82 [17:15:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.27, 7.86, 7.90 [17:45:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.41, 7.70, 7.98 [17:47:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.00, 8.15, 8.10 [17:49:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.51, 7.84, 7.99 [17:57:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.54, 7.32, 7.55 [17:59:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 8.00, 7.53, 7.60 [18:01:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.23, 7.92, 7.72 [18:03:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.62, 7.85, 7.73 [18:07:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.14, 7.77, 7.71 [18:10:03] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp2.wikitide.net - NS RECORDS OK [18:11:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.46, 7.92, 7.82 [18:17:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.99, 8.10, 7.83 [18:37:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.06, 7.34, 7.89 [18:42:10] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [18:49:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.69, 7.88, 7.83 [19:41:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.56, 7.22, 7.94 [19:43:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 10.47, 8.41, 8.29 [20:01:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.28, 7.35, 8.00 [20:03:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.48, 7.85, 8.11 [20:07:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.60, 7.68, 7.99 [20:15:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.36, 7.82, 7.79 [20:17:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.86, 7.77, 7.78 [20:33:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.32, 6.93, 7.08 [20:37:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.62, 7.67, 7.40 [20:43:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.57, 8.05, 7.64 [20:49:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.14, 7.68, 7.72 [20:53:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.43, 8.10, 7.84 [20:59:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.02, 7.79, 7.83 [21:07:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.44, 7.41, 7.51 [21:09:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.31, 7.29, 7.45 [21:15:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 10.54, 8.62, 7.93 [21:20:20] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [21:21:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 5.64, 7.59, 7.79 [21:39:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.27, 7.48, 7.42 [21:41:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.24, 7.30, 7.36 [21:43:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.34, 7.71, 7.51 [22:01:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.67, 7.79, 8.00 [22:07:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.27, 7.54, 7.77 [22:09:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.23, 7.61, 7.78 [22:17:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.67, 7.76, 7.68 [22:18:12] RECOVERY - mlrpgspeedruns.com - reverse DNS on sslhost is OK: SSL OK - mlrpgspeedruns.com reverse DNS resolves to cp5.wikitide.net - NS RECORDS OK [22:23:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.88, 7.78, 7.80 [22:49:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.13, 7.24, 7.16 [22:51:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.38, 7.00, 7.09 [23:05:55] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 5.87, 6.42, 6.76 [23:09:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.90, 7.48, 7.09 [23:13:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.36, 7.90, 7.40 [23:31:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 10.54, 8.23, 7.57 [23:39:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 5.89, 7.82, 7.83 [23:43:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.67, 7.71, 7.74 [23:47:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.85, 7.91, 7.82 [23:49:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.02, 8.25, 7.95 [23:51:55] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.96, 7.78, 7.82 [23:57:55] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.48, 7.70, 7.72