[00:41:39] <icinga-wikitide>	 PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: LOAD WARNING - total load average: 7.10, 6.68, 6.31
[00:43:37] <icinga-wikitide>	 RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: LOAD OK - total load average: 6.12, 6.53, 6.30
[02:46:20] <icinga-wikitide>	 PROBLEM - cp1 PowerDNS Recursor on cp1 is CRITICAL: Domain 'wikitide.org' was not found by the server
[02:50:26] <icinga-wikitide>	 RECOVERY - cp1 PowerDNS Recursor on cp1 is OK: DNS OK: 3.398 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66
[03:20:48] <icinga-wikitide>	 PROBLEM - cp1 PowerDNS Recursor on cp1 is CRITICAL: Domain 'wikitide.org' was not found by the server
[03:26:55] <icinga-wikitide>	 RECOVERY - cp1 PowerDNS Recursor on cp1 is OK: DNS OK: 5.120 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66
[03:33:04] <icinga-wikitide>	 PROBLEM - cp1 PowerDNS Recursor on cp1 is CRITICAL: Domain 'wikitide.org' was not found by the server
[03:37:13] <icinga-wikitide>	 RECOVERY - cp1 PowerDNS Recursor on cp1 is OK: DNS OK: 7.432 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66
[03:47:24] <icinga-wikitide>	 PROBLEM - cp1 PowerDNS Recursor on cp1 is CRITICAL: Domain 'wikitide.org' was not found by the server
[03:51:33] <icinga-wikitide>	 RECOVERY - cp1 PowerDNS Recursor on cp1 is OK: DNS OK: 7.062 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66
[04:04:35] <icinga-wikitide>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mediawiki
[04:06:35] <icinga-wikitide>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy
[08:37:39] <icinga-wikitide>	 PROBLEM - test1 JobChron Service on test1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService'
[08:39:39] <icinga-wikitide>	 RECOVERY - test1 JobChron Service on test1 is OK: PROCS OK: 1 process with args 'redisJobChronService'
[14:06:40] <icinga-wikitide>	 PROBLEM - data.lophocmatngu.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'data.lophocmatngu.wiki' expires in 15 day(s) (Wed 31 Jan 2024 01:52:09 PM GMT +0000).
[20:03:24] <icinga-wikitide>	 PROBLEM - swiftproxy1 Current Load on swiftproxy1 is CRITICAL: LOAD CRITICAL - total load average: 21.91, 21.23, 9.54
[20:09:05] <icinga-wikitide>	 PROBLEM - swiftproxy1 Current Load on swiftproxy1 is WARNING: LOAD WARNING - total load average: 0.16, 6.88, 6.65
[20:10:59] <icinga-wikitide>	 RECOVERY - swiftproxy1 Current Load on swiftproxy1 is OK: LOAD OK - total load average: 0.08, 4.70, 5.88
[20:34:18] <botifico>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/e14f79527602...0a33c9162035
[20:34:21] <botifico>	 [02WikiTideOrg/puppet] 07Universal-Omega 030a33c91 - Use workspace for postfix relayhost
[20:35:25] <botifico>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/0a33c9162035...52593a00ebdc
[20:35:27] <botifico>	 [02WikiTideOrg/puppet] 07Universal-Omega 0352593a0 - Add base::mail
[20:35:43] <botifico>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/52593a00ebdc...0d16e26c01c2
[20:35:45] <botifico>	 [02WikiTideOrg/puppet] 07Universal-Omega 030d16e26 - Move
[20:37:29] <icinga-wikitide>	 PROBLEM - puppet1 APT on puppet1 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected.
[20:39:30] <icinga-wikitide>	 PROBLEM - puppet1 APT on puppet1 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (1 critical updates).
[20:47:15] <botifico>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/0d16e26c01c2...da3227850064
[20:47:16] <botifico>	 [02WikiTideOrg/puppet] 07Universal-Omega 03da32278 - Remove mail1
[20:47:56] <botifico>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/da3227850064...1d4b6eac3a80
[20:47:58] <botifico>	 [02WikiTideOrg/puppet] 07Universal-Omega 031d4b6ea - Remove mail1
[20:52:14] <icinga-wikitide>	 PROBLEM - ns1 APT on ns1 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected.
[20:54:15] <icinga-wikitide>	 PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 24 packages available for upgrade (1 critical updates).
[21:12:56] <icinga-wikitide>	 RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures
[21:13:37] <botifico>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/1d4b6eac3a80...4faad1aae736
[21:13:38] <botifico>	 [02WikiTideOrg/puppet] 07Universal-Omega 034faad1a - Use python3 in cron
[21:15:21] <icinga-wikitide>	 PROBLEM - mail1 webmail.wikitide.net HTTPS on mail1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:15:25] <icinga-wikitide>	 PROBLEM - mail1 IMAP on mail1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:15:25] <icinga-wikitide>	 PROBLEM - mail1 SMTP on mail1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:16:09] <icinga-wikitide>	 PROBLEM - mail1 HTTPS on mail1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10002 milliseconds
[21:34:53] <icinga-wikitide>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[21:36:42] <icinga-wikitide>	 PROBLEM - mem1 Puppet on mem1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[21:37:53] <icinga-wikitide>	 PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[21:38:29] <icinga-wikitide>	 PROBLEM - graylog1 Puppet on graylog1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[21:39:03] <icinga-wikitide>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[21:41:32] <icinga-wikitide>	 PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[21:42:49] <icinga-wikitide>	 PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[21:43:18] <icinga-wikitide>	 PROBLEM - phorge1 Puppet on phorge1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[21:43:31] <icinga-wikitide>	 PROBLEM - prometheus1 Puppet on prometheus1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[22:06:03] <icinga-wikitide>	 RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[22:06:42] <icinga-wikitide>	 RECOVERY - mem1 Puppet on mem1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[22:14:28] <icinga-wikitide>	 PROBLEM - ldap1 Puppet on ldap1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[22:25:00] <icinga-wikitide>	 PROBLEM - cp3 HTTPS on cp3 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 2628 bytes in 0.306 second response time
[22:25:04] <icinga-wikitide>	 PROBLEM - cp4 HTTPS on cp4 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 2628 bytes in 0.393 second response time
[22:26:35] <icinga-wikitide>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mediawiki
[22:26:49] <icinga-wikitide>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mediawiki
[22:30:35] <icinga-wikitide>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy
[22:30:49] <icinga-wikitide>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy
[22:30:55] <icinga-wikitide>	 RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/2 200 - 2974 bytes in 0.659 second response time
[22:30:57] <icinga-wikitide>	 RECOVERY - cp4 HTTPS on cp4 is OK: HTTP OK: HTTP/2 200 - 2996 bytes in 0.833 second response time
[22:38:15] <icinga-wikitide>	 PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[22:38:42] <icinga-wikitide>	 PROBLEM - mem1 Puppet on mem1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:03:20] <icinga-wikitide>	 PROBLEM - swiftac1 Puppet on swiftac1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:04:42] <icinga-wikitide>	 RECOVERY - mem1 Puppet on mem1 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures
[23:06:29] <icinga-wikitide>	 RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[23:06:29] <icinga-wikitide>	 RECOVERY - graylog1 Puppet on graylog1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[23:07:03] <icinga-wikitide>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures
[23:10:30] <icinga-wikitide>	 RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[23:16:56] <icinga-wikitide>	 PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:17:28] <icinga-wikitide>	 PROBLEM - swiftobject1 Puppet on swiftobject1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:18:54] <icinga-wikitide>	 PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:20:58] <icinga-wikitide>	 PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:21:11] <icinga-wikitide>	 PROBLEM - jobchron1 Puppet on jobchron1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:21:22] <icinga-wikitide>	 PROBLEM - bots1 Puppet on bots1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:21:37] <icinga-wikitide>	 PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:21:46] <icinga-wikitide>	 PROBLEM - matomo1 Puppet on matomo1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:22:58] <icinga-wikitide>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:23:55] <icinga-wikitide>	 PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:24:03] <icinga-wikitide>	 PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:24:52] <icinga-wikitide>	 PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:26:57] <icinga-wikitide>	 PROBLEM - db1 Puppet on db1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:28:11] <icinga-wikitide>	 PROBLEM - bast1 Puppet on bast1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:28:55] <icinga-wikitide>	 PROBLEM - cp1 Puppet on cp1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:31:37] <icinga-wikitide>	 PROBLEM - swiftproxy1 Puppet on swiftproxy1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:36:42] <icinga-wikitide>	 PROBLEM - mem1 Puppet on mem1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:38:29] <icinga-wikitide>	 PROBLEM - graylog1 Puppet on graylog1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:38:43] <icinga-wikitide>	 PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:39:03] <icinga-wikitide>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:41:33] <icinga-wikitide>	 PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.