[00:01:59] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/737085cae711...896d8cbca982 [00:02:00] [02WikiTideOrg/puppet] 07Universal-Omega 03896d8cb - Fix [00:02:01] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [00:02:26] RECOVERY - cp1 Disk Space on cp1 is OK: DISK OK - free space: / 6331MiB (11% inode=98%); [00:02:58] WikiTideOrg/puppet - Universal-Omega the build has errored. [00:21:46] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/896d8cbca982...8151a9a3815b [00:21:47] [02WikiTideOrg/puppet] 07Universal-Omega 038151a9a - Remove [00:21:49] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [00:22:38] WikiTideOrg/puppet - Universal-Omega the build has errored. [00:23:26] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/8151a9a3815b...43ec82ec0f40 [00:23:28] [02WikiTideOrg/puppet] 07Universal-Omega 0343ec82e - Fix [00:23:30] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [00:23:43] [02WikiTideOrg/puppet] 07Universal-Omega pushed 032 commits to 03fix-firewall [+0/-0/±2] 13https://github.com/WikiTideOrg/puppet/compare/43ec82ec0f40...4610e35ca342 [00:23:44] [02WikiTideOrg/puppet] 07Universal-Omega 034610e35 - Merge branch 'master' into fix-firewall [00:23:46] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [00:24:23] WikiTideOrg/puppet - Universal-Omega the build has errored. [00:24:37] WikiTideOrg/puppet - Universal-Omega the build has errored. [00:36:42] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [00:50:19] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:59:37] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/4610e35ca342...b68b4a127a94 [00:59:40] [02WikiTideOrg/puppet] 07Universal-Omega 03b68b4a1 - Fix [00:59:41] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [01:00:21] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:00:34] WikiTideOrg/puppet - Universal-Omega the build has errored. [01:05:43] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:06:00] PROBLEM - cp2 NTP time on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:06:02] PROBLEM - cp2 Disk Space on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:06:09] PROBLEM - cp2 APT on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:06:15] PROBLEM - cp2 Nginx Backend for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:06:33] PROBLEM - cp2 Current Load on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:06:34] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:06:44] PROBLEM - cp2 Nginx Backend for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:06:51] PROBLEM - cp2 Nginx Backend for puppet1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:07:03] PROBLEM - cp2 Nginx Backend for test1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:07:10] PROBLEM - cp2 Nginx Backend for phorge1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:07:11] PROBLEM - cp2 Nginx Backend for mail1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:07:18] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:07:32] PROBLEM - cp2 Nginx Backend for matomo1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:13:19] ^ not really down, just mon1 can't connect to cp2 right now (as far as I can see, I can still connect via cp2) [01:16:52] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [01:16:54] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/b68b4a127a94...2f65ab6dc3c1 [01:16:57] [02WikiTideOrg/puppet] 07Universal-Omega 032f65ab6 - Fix [01:19:37] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/2f65ab6dc3c1...2582a58f5c3a [01:19:40] [02WikiTideOrg/puppet] 07Universal-Omega 032582a58 - Fix cloud [01:19:41] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [01:23:27] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [01:23:28] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/2582a58f5c3a...af690046080e [01:23:30] [02WikiTideOrg/puppet] 07Universal-Omega 03af69004 - Fix 5666 [01:24:10] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:24:24] RECOVERY - cp2 Nginx Backend for matomo1 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8204 [01:24:29] RECOVERY - cp2 Nginx Backend for phorge1 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8200 [01:24:35] RECOVERY - cp2 Nginx Backend for mail1 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8202 [01:25:06] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 7 backends are healthy [01:25:15] RECOVERY - cp2 NTP time on cp2 is OK: NTP OK: Offset -0.003380864859 secs [01:25:37] RECOVERY - cp2 Disk Space on cp2 is OK: DISK OK - free space: / 6366MiB (33% inode=94%); [01:25:44] RECOVERY - cp2 APT on cp2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [01:25:47] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:25:51] RECOVERY - cp2 Nginx Backend for mw1 on cp2 is OK: TCP OK - 0.001 second response time on localhost port 8100 [01:25:59] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:26:10] RECOVERY - cp2 Current Load on cp2 is OK: LOAD OK - total load average: 0.33, 0.24, 0.14 [01:26:14] RECOVERY - cp2 Nginx Backend for test1 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8180 [01:26:17] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 3% [01:26:17] RECOVERY - cp2 Nginx Backend for mw2 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8101 [01:26:25] RECOVERY - cp2 Nginx Backend for puppet1 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8201 [01:29:10] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [01:30:21] RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [01:30:46] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:32:36] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [01:36:10] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [01:46:54] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/af690046080e...083a62ed8bdb [01:46:56] [02WikiTideOrg/puppet] 07Universal-Omega 03083a62e - Add base::firewall::bastion_public_ips [01:46:57] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [01:47:31] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/083a62ed8bdb...194aca496126 [01:47:34] [02WikiTideOrg/puppet] 07Universal-Omega 03194aca4 - base::firewall::use_public_bastion: true [01:47:36] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [01:47:45] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/194aca496126...64dd06c67297 [01:47:47] [02WikiTideOrg/puppet] 07Universal-Omega 0364dd06c - base::firewall::use_public_bastion: true [01:47:48] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [01:47:58] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/64dd06c67297...4b78189014f9 [01:48:01] [02WikiTideOrg/puppet] 07Universal-Omega 034b78189 - base::firewall::use_public_bastion: true [01:48:04] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [01:48:11] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/4b78189014f9...0f7ff3b70406 [01:48:12] [02WikiTideOrg/puppet] 07Universal-Omega 030f7ff3b - base::firewall::use_public_bastion: true [01:48:14] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [01:48:25] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/0f7ff3b70406...ae76ce5b3403 [01:48:26] [02WikiTideOrg/puppet] 07Universal-Omega 03ae76ce5 - base::firewall::use_public_bastion: true [01:48:28] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [01:48:45] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/ae76ce5b3403...262c4abc0bac [01:48:46] [02WikiTideOrg/puppet] 07Universal-Omega 03262c4ab - base::firewall::use_public_bastion: true [01:48:47] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [01:49:20] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [01:49:21] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/262c4abc0bac...b477dfa72b0c [01:49:22] [02WikiTideOrg/puppet] 07Universal-Omega 03b477dfa - use lookup [01:59:14] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [01:59:15] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/b477dfa72b0c...71a58f5ca0e2 [01:59:16] [02WikiTideOrg/puppet] 07Universal-Omega 0371a58f5 - Re-add but comment [05:44:37] RECOVERY - Host jobrunner1 is UP: PING OK - Packet loss = 0%, RTA = 0.53 ms [05:44:37] RECOVERY - valorantjp.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'valorantjp.wiki' will expire on Sat 20 Jan 2024 06:52:59 AM GMT +0000. [05:44:37] RECOVERY - Host puppet1 is UP: PING OK - Packet loss = 0%, RTA = 0.36 ms [05:44:37] RECOVERY - Host ldap1 is UP: PING OK - Packet loss = 0%, RTA = 0.48 ms [05:44:38] RECOVERY - realitylongterms.studio - reverse DNS on sslhost is OK: SSL OK - realitylongterms.studio reverse DNS resolves to cp1.wikitide.net - CNAME FLAT [05:44:41] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.rosettacode.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [05:44:41] RECOVERY - Host cp6 is UP: PING OK - Packet loss = 0%, RTA = 187.84 ms [05:44:42] RECOVERY - www.greatamerica.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'greatamerica.wiki' will expire on Sat 20 Jan 2024 06:37:56 AM GMT +0000. [05:44:42] PROBLEM - farthestfrontier.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - farthestfrontier.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [05:44:43] RECOVERY - Host cp5 is UP: PING OK - Packet loss = 0%, RTA = 206.07 ms [05:44:44] RECOVERY - mon1 NTP time on mon1 is OK: NTP OK: Offset 0.01923254132 secs [05:44:45] PROBLEM - puppet1 Puppet on puppet1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 1 hour ago with 3 failures [05:44:47] RECOVERY - cp6 HTTPS on cp6 is OK: HTTP OK: HTTP/2 200 - 2939 bytes in 1.450 second response time [05:44:47] RECOVERY - wiki.colleimadcat.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.colleimadcat.com' will expire on Sat 20 Jan 2024 06:42:51 AM GMT +0000. [05:44:48] RECOVERY - Host cp2 is UP: PING OK - Packet loss = 0%, RTA = 36.44 ms [05:44:50] RECOVERY - Host cloud1 is UP: PING OK - Packet loss = 0%, RTA = 0.19 ms [05:44:52] RECOVERY - Host services1 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms [05:44:52] PROBLEM - meta.sagan4.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - meta.sagan4.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [05:44:53] RECOVERY - wc.wikitide.org on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Tue 20 Feb 2024 12:20:10 PM GMT +0000. [05:44:54] RECOVERY - realitylongterms.studio - LetsEncrypt on sslhost is OK: OK - Certificate 'realitylongterms.studio' will expire on Thu 08 Feb 2024 10:21:17 PM GMT +0000. [05:44:54] RECOVERY - www.farthestfrontier.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'farthestfrontier.wiki' will expire on Sat 20 Jan 2024 05:08:31 AM GMT +0000. [05:44:54] PROBLEM - cp2 Puppet on cp2 is WARNING: WARNING: Puppet last ran 1 hour ago [05:44:54] RECOVERY - polandballwiki.com - reverse DNS on sslhost is OK: SSL OK - polandballwiki.com reverse DNS resolves to cp1.wikitide.net - CNAME FLAT [05:44:54] PROBLEM - lophocmatngu.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - lophocmatngu.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [05:44:56] RECOVERY - jobrunner1 JobRunner Service on jobrunner1 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [05:44:56] RECOVERY - Host mail1 is UP: PING OK - Packet loss = 0%, RTA = 0.36 ms [05:44:57] PROBLEM - www.greatamerica.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.greatamerica.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [05:44:59] RECOVERY - hsck.lophocmatngu.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'lophocmatngu.wiki' will expire on Sat 20 Jan 2024 06:40:34 AM GMT +0000. [05:44:59] RECOVERY - Host cp3 is UP: PING OK - Packet loss = 0%, RTA = 96.31 ms [05:45:01] PROBLEM - alpha.sagan4.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - alpha.sagan4.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [05:45:01] RECOVERY - Host mw1 is UP: PING OK - Packet loss = 0%, RTA = 0.33 ms [05:45:01] RECOVERY - cp2 Disk Space on cp2 is OK: DISK OK - free space: / 6325MiB (33% inode=94%); [05:45:04] RECOVERY - Host graylog1 is UP: PING OK - Packet loss = 0%, RTA = 0.32 ms [05:45:05] RECOVERY - cp6 Nginx Backend for puppet1 on cp6 is OK: TCP OK - 0.000 second response time on localhost port 8201 [05:45:05] RECOVERY - distrowiki.net - LetsEncrypt on sslhost is OK: OK - Certificate 'distrowiki.net' will expire on Sat 20 Jan 2024 06:36:29 AM GMT +0000. [05:45:05] PROBLEM - www.wikitide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.wikitide.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [05:45:06] RECOVERY - Host prometheus1 is UP: PING OK - Packet loss = 0%, RTA = 0.35 ms [05:45:06] RECOVERY - Host test1 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms [05:45:08] RECOVERY - sdiy.info - reverse DNS on sslhost is OK: SSL OK - sdiy.info reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [05:45:09] PROBLEM - ldap1 Puppet on ldap1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:45:09] RECOVERY - ldap1 NTP time on ldap1 is OK: NTP OK: Offset -3.889203072e-05 secs [05:45:09] PROBLEM - services1 mathoid on services1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:45:09] PROBLEM - graylog1 Puppet on graylog1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:45:11] RECOVERY - thegreatwar.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thegreatwar.uk' will expire on Sat 20 Jan 2024 06:49:55 AM GMT +0000. [05:45:11] RECOVERY - Host mw2 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [05:45:12] RECOVERY - lophocmatngu.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'lophocmatngu.wiki' will expire on Sat 20 Jan 2024 06:40:34 AM GMT +0000. [05:45:14] PROBLEM - cp3 Puppet on cp3 is WARNING: WARNING: Puppet last ran 1 hour ago [05:45:14] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:45:14] RECOVERY - mw1 Current Load on mw1 is OK: LOAD OK - total load average: 2.05, 0.80, 0.29 [05:45:14] RECOVERY - Host db1 is UP: PING OK - Packet loss = 0%, RTA = 0.35 ms [05:45:15] RECOVERY - Host mem1 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms [05:45:15] PROBLEM - hkrail.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - hkrail.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [05:45:16] RECOVERY - mason.sagan4.org - LetsEncrypt on sslhost is OK: OK - Certificate 'beta.sagan4.org' will expire on Sat 20 Jan 2024 06:35:08 AM GMT +0000. [05:45:16] RECOVERY - Host cp4 is UP: PING OK - Packet loss = 0%, RTA = 128.98 ms [05:45:17] PROBLEM - puppet1 HTTPS on puppet1 is WARNING: HTTP WARNING: HTTP/2 403 - 113 bytes in 0.027 second response time [05:45:17] PROBLEM - greatamerica.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - greatamerica.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [05:45:17] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 2.286 second response time [05:45:18] RECOVERY - www.rosettacode.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rosettacode.org' will expire on Sat 20 Jan 2024 06:52:33 AM GMT +0000. [05:45:18] RECOVERY - Host swiftac1 is UP: PING OK - Packet loss = 0%, RTA = 0.32 ms [05:45:19] RECOVERY - www.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'polandballwiki.com' will expire on Sat 20 Jan 2024 06:47:38 AM GMT +0000. [05:45:19] RECOVERY - puppet1 NTP time on puppet1 is OK: NTP OK: Offset -0.002998173237 secs [05:45:19] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/2 200 - 2916 bytes in 0.261 second response time [05:45:20] RECOVERY - www.wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Tue 20 Feb 2024 12:20:10 PM GMT +0000. [05:45:21] PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.406 second response time [05:45:22] RECOVERY - puppet1 Backups SSLKeys on puppet1 is OK: FILE_AGE OK: /var/log/sslkeys-backup.log is 603915 seconds old and 92 bytes [05:45:23] RECOVERY - Host jobchron1 is UP: PING OK - Packet loss = 0%, RTA = 0.70 ms [05:45:23] RECOVERY - Host ns1 is UP: PING OK - Packet loss = 0%, RTA = 0.35 ms [05:45:24] RECOVERY - mail1 HTTPS on mail1 is OK: HTTP OK: HTTP/2 301 - 227 bytes in 0.024 second response time [05:45:25] RECOVERY - distrowiki.net - reverse DNS on sslhost is OK: SSL OK - distrowiki.net reverse DNS resolves to cp1.wikitide.net - CNAME FLAT [05:45:28] RECOVERY - Host phorge1 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms [05:45:29] PROBLEM - mem1 Puppet on mem1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:45:58] RECOVERY - rosettacode.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rosettacode.org' will expire on Sat 20 Jan 2024 06:52:33 AM GMT +0000. [05:45:59] @cosmicalpha lol, look at this [05:45:59] @cosmicalpha lol, look at this [05:46:24] RECOVERY - cp2 NTP time on cp2 is OK: NTP OK: Offset -0.002871632576 secs [05:46:25] RECOVERY - services1 Disk Space on services1 is OK: DISK OK - free space: / 2279MiB (30% inode=71%); [05:46:29] PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.319 second response time [05:46:29] RECOVERY - mw2 NTP time on mw2 is OK: NTP OK: Offset 0.01555919647 secs [05:46:29] PROBLEM - prometheus1 Puppet on prometheus1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:46:29] RECOVERY - test1 HTTPS on test1 is OK: HTTP OK: HTTP/2 200 - 364 bytes in 0.028 second response time [05:46:29] RECOVERY - services1 APT on services1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [05:46:33] RECOVERY - polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'polandballwiki.com' will expire on Sat 20 Jan 2024 06:47:38 AM GMT +0000. [05:46:34] PROBLEM - db1 MariaDB on db1 is CRITICAL: Access denied for user 'icinga'@'10.0.0.111' (using password: YES) [05:46:34] RECOVERY - jobrunner1 NTP time on jobrunner1 is OK: NTP OK: Offset 0.00145944953 secs [05:46:34] RECOVERY - mw2 HTTPS on mw2 is OK: HTTP OK: HTTP/2 200 - 362 bytes in 0.021 second response time [05:46:34] RECOVERY - jobrunner1 HTTPS on jobrunner1 is OK: HTTP OK: HTTP/2 200 - 369 bytes in 0.037 second response time [05:46:35] RECOVERY - Host swiftobject1 is UP: PING OK - Packet loss = 0%, RTA = 0.49 ms [05:46:35] RECOVERY - jobchron1 NTP time on jobchron1 is OK: NTP OK: Offset 0.00202062726 secs [05:46:35] RECOVERY - Host cp1 is UP: PING OK - Packet loss = 0%, RTA = 0.40 ms [05:46:39] RECOVERY - mail1 IMAP on mail1 is OK: IMAP OK - 0.011 second response time on 10.0.0.110 port 143 [* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ STARTTLS LOGINDISABLED] Dovecot (Debian) ready.] [05:46:39] RECOVERY - mw1 HTTPS on mw1 is OK: HTTP OK: HTTP/2 200 - 362 bytes in 0.024 second response time [05:46:39] RECOVERY - test1 APT on test1 is OK: APT OK: 22 packages available for upgrade (0 critical updates). [05:46:39] RECOVERY - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is OK: OK - NGINX Error Rate is 10% [05:46:39] PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:46:39] RECOVERY - test1 Disk Space on test1 is OK: DISK OK - free space: / 18146MiB (45% inode=67%); [05:46:40] RECOVERY - mail1 NTP time on mail1 is OK: NTP OK: Offset 0.01866197586 secs [05:46:40] RECOVERY - Host bast1 is UP: PING OK - Packet loss = 0%, RTA = 0.38 ms [05:46:41] lol [05:46:41] lol [05:46:42] RECOVERY - graylog1 NTP time on graylog1 is OK: NTP OK: Offset 0.005156099796 secs [05:46:42] RECOVERY - swiftobject1 APT on swiftobject1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [05:46:43] RECOVERY - prometheus1 NTP time on prometheus1 is OK: NTP OK: Offset -0.001956194639 secs [05:46:44] RECOVERY - cp1 Varnish Backends on cp1 is OK: All 7 backends are healthy [05:46:44] RECOVERY - mail1 webmail.wikitide.net HTTPS on mail1 is OK: HTTP OK: HTTP/1.1 200 OK - 6205 bytes in 0.038 second response time RECOVERY - mail1 Current Load on mail1 is OK: LOAD OK - total load average: 0.09, 0.09, 0.04 [05:46:44] RECOVERY - test1 poolcounter process on test1 is OK: PROCS OK: 1 process with UID = 996 (poolcounter), command name 'poolcounterd' [05:46:44] RECOVERY - test1 memcached on test1 is OK: TCP OK - 0.000 second response time on 10.0.0.107 port 11211 [05:46:44] RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/2 200 - 2894 bytes in 0.632 second response time [05:46:45] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:46:46] RECOVERY - Host os1 is UP: PING OK - Packet loss = 0%, RTA = 0.43 ms [05:46:47] uh [05:46:47] uh [05:46:49] RECOVERY - mw1 NTP time on mw1 is OK: NTP OK: Offset 0.007549375296 secs [05:46:49] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset -0.001036167145 secs [05:46:49] RECOVERY - mail1 SMTP on mail1 is OK: SMTP OK - 0.035 sec. response time [05:46:49] RECOVERY - bast1 Current Load on bast1 is OK: LOAD OK - total load average: 0.24, 0.23, 0.09 [05:46:49] RECOVERY - cp1 Nginx Backend for puppet1 on cp1 is OK: TCP OK - 0.000 second response time on localhost port 8201 [05:46:49] RECOVERY - test1 NTP time on test1 is OK: NTP OK: Offset -0.0007225871086 secs [05:46:50] PROBLEM - puppet1 puppetserver on puppet1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:46:50] RECOVERY - test1 Current Load on test1 is OK: LOAD OK - total load average: 0.03, 0.04, 0.01 [05:46:51] RECOVERY - db1 NTP time on db1 is OK: NTP OK: Offset 0.0008256435394 secs [05:46:51] oh [05:46:51] oh [05:46:53] RECOVERY - Host swiftproxy1 is UP: PING OK - Packet loss = 0%, RTA = 0.54 ms [05:46:54] RECOVERY - cp1 Nginx Backend for test1 on cp1 is OK: TCP OK - 0.000 second response time on localhost port 8180 [05:46:54] RECOVERY - cp1 Nginx Backend for phorge1 on cp1 is OK: TCP OK - 0.000 second response time on localhost port 8200 [05:46:59] RECOVERY - swiftobject1 Current Load on swiftobject1 is OK: LOAD OK - total load average: 1.21, 0.68, 0.27 [05:46:59] two competing instances [05:47:00] two competing instances [05:47:00] RECOVERY - services1 mathoid on services1 is OK: TCP OK - 0.001 second response time on 10.0.0.115 port 10044 [05:47:02] RECOVERY - swiftproxy1 APT on swiftproxy1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [05:47:04] RECOVERY - phorge1 NTP time on phorge1 is OK: NTP OK: Offset -0.0005395710468 secs [05:47:04] RECOVERY - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is OK: OK - NGINX Error Rate is 5% [05:47:04] RECOVERY - cp1 Nginx Backend for mw2 on cp1 is OK: TCP OK - 0.000 second response time on localhost port 8101 [05:47:04] RECOVERY - mem1 NTP time on mem1 is OK: NTP OK: Offset 0.001430213451 secs [05:47:04] RECOVERY - phorge1 APT on phorge1 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [05:47:06] RECOVERY - services1 NTP time on services1 is OK: NTP OK: Offset 0.002727419138 secs [05:47:09] RECOVERY - swiftac1 NTP time on swiftac1 is OK: NTP OK: Offset 0.004939496517 secs [05:47:09] RECOVERY - cp1 APT on cp1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). RECOVERY - phorge1 Current Load on phorge1 is OK: LOAD OK - total load average: 0.22, 0.19, 0.09 [05:47:09] RECOVERY - phorge1 conntrack_table_size on phorge1 is OK: OK: nf_conntrack is 0 % full [05:47:09] RECOVERY - cp1 NTP time on cp1 is OK: NTP OK: Offset -0.0002613663673 secs [05:47:11] RECOVERY - services1 Current Load on services1 is OK: LOAD OK - total load average: 0.92, 0.72, 0.30 [05:47:14] PROBLEM - swiftac1 Puppet on swiftac1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. PROBLEM - db1 Puppet on db1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:47:14] RECOVERY - cp1 Nginx Backend for mail1 on cp1 is OK: TCP OK - 0.000 second response time on localhost port 8202 [05:47:19] RECOVERY - bast1 NTP time on bast1 is OK: NTP OK: Offset -0.0006265938282 secs [05:47:24] RECOVERY - swiftobject1 Disk Space on swiftobject1 is OK: DISK OK - free space: / 9451MiB (76% inode=88%); [05:47:24] RECOVERY - swiftproxy1 Current Load on swiftproxy1 is OK: LOAD OK - total load average: 0.36, 0.32, 0.13 [05:47:24] RECOVERY - os1 NTP time on os1 is OK: NTP OK: Offset 0.0007748007774 secs [05:47:24] RECOVERY - phorge1 Disk Space on phorge1 is OK: DISK OK - free space: / 14031MiB (82% inode=91%); [05:47:24] PROBLEM - swiftobject1 Puppet on swiftobject1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:47:26] RECOVERY - Host bots1 is UP: PING OK - Packet loss = 0%, RTA = 0.48 ms [05:47:29] RECOVERY - swiftobject1 NTP time on swiftobject1 is OK: NTP OK: Offset 0.001702666283 secs [05:47:35] RECOVERY - nexttide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nexttide.org' will expire on Sun 14 Jan 2024 05:21:30 PM GMT +0000. [05:48:24] PROBLEM - os1 Puppet on os1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:48:24] RECOVERY - bast1 APT on bast1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [05:48:24] RECOVERY - cp1 Nginx Backend for matomo1 on cp1 is OK: TCP OK - 0.001 second response time on localhost port 8204 [05:48:24] RECOVERY - bast1 ferm_active on bast1 is OK: OK ferm input default policy is set [05:48:29] RECOVERY - swiftobject1 Swift Object Service on swiftobject1 is OK: TCP OK - 0.000 second response time on 10.0.0.121 port 6000 [05:48:29] RECOVERY - bast1 conntrack_table_size on bast1 is OK: OK: nf_conntrack is 0 % full [05:48:29] RECOVERY - cp1 Nginx Backend for mw1 on cp1 is OK: TCP OK - 0.000 second response time on localhost port 8100 [05:48:29] RECOVERY - cp1 HTTPS on cp1 is OK: HTTP OK: HTTP/2 200 - 2894 bytes in 0.026 second response time [05:48:31] RECOVERY - beta.sagan4.org - LetsEncrypt on sslhost is OK: OK - Certificate 'beta.sagan4.org' will expire on Sat 20 Jan 2024 06:35:08 AM GMT +0000. [05:48:34] RECOVERY - bots1 Disk Space on bots1 is OK: DISK OK - free space: / 12703MiB (74% inode=91%); [05:48:34] RECOVERY - bots1 IRC Log Bot on bots1 is OK: PROCS OK: 1 process with args 'adminlogbot.py' [05:48:39] RECOVERY - swiftproxy1 Disk Space on swiftproxy1 is OK: DISK OK - free space: / 23766MiB (89% inode=95%); [05:48:49] RECOVERY - bots1 IRC Log Server Bot on bots1 is OK: PROCS OK: 1 process with args 'irclogserverbot.py' [05:48:51] RECOVERY - www.lgbtqia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lgbtqia.wiki' will expire on Sat 20 Jan 2024 06:55:06 AM GMT +0000. [05:48:54] RECOVERY - bots1 Current Load on bots1 is OK: LOAD OK - total load average: 0.10, 0.20, 0.09 [05:48:54] RECOVERY - swiftproxy1 Swift Proxy on swiftproxy1 is OK: TCP OK - 0.000 second response time on 10.0.0.122 port 80 [05:48:54] RECOVERY - hkrail.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'hkrail.wiki' will expire on Sat 20 Jan 2024 06:39:37 AM GMT +0000. [05:48:59] RECOVERY - swiftproxy1 NTP time on swiftproxy1 is OK: NTP OK: Offset 0.00344979763 secs [05:49:09] RECOVERY - bots1 IRC-Discord Relay Bot on bots1 is OK: PROCS OK: 2 processes with args 'relaybot' [05:49:19] RECOVERY - bots1 NTP time on bots1 is OK: NTP OK: Offset 0.002240568399 secs [05:49:26] RECOVERY - wikitide.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.com' will expire on Fri 02 Feb 2024 09:51:46 PM GMT +0000. [05:50:38] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 1 minute ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_srv-ssl],Exec[git_pull_puppet],Exec[git_pull_ssl] [05:50:50] RECOVERY - puppet1 puppetserver on puppet1 is OK: TCP OK - 0.001 second response time on 10.0.0.100 port 8140 [05:51:10] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [05:52:44] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:52:58] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:54:34] RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:56:07] RECOVERY - os1 Puppet on os1 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [05:56:26] RECOVERY - cp1 Puppet on cp1 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [05:56:29] RECOVERY - bast1 Puppet on bast1 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [05:56:47] RECOVERY - db1 Puppet on db1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:57:21] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [05:58:19] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [05:59:46] RECOVERY - wikitide.net - reverse DNS on sslhost is OK: SSL OK - wikitide.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [05:59:59] RECOVERY - thegreatwar.uk - reverse DNS on sslhost is OK: SSL OK - thegreatwar.uk reverse DNS resolves to cp1.wikitide.net - CNAME FLAT [06:00:05] PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org All nameservers failed to answer the query. [06:00:22] RECOVERY - wiki.burning.art - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.burning.art' will expire on Tue 20 Feb 2024 06:37:54 AM GMT +0000. [06:00:32] RECOVERY - swiftproxy1 Puppet on swiftproxy1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:00:52] RECOVERY - sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Tue 13 Feb 2024 01:26:39 PM GMT +0000. [06:01:28] RECOVERY - wikitide.org - reverse DNS on sslhost is OK: SSL OK - wikitide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [06:01:33] RECOVERY - farthestfrontier.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'farthestfrontier.wiki' will expire on Sat 20 Jan 2024 05:08:31 AM GMT +0000. [06:01:50] RECOVERY - wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Tue 20 Feb 2024 12:20:10 PM GMT +0000. [06:02:09] RECOVERY - greatamerica.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'greatamerica.wiki' will expire on Sat 20 Jan 2024 06:37:56 AM GMT +0000. [06:02:31] RECOVERY - swiftac1 Puppet on swiftac1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:02:36] RECOVERY - wiki.burning.art - reverse DNS on sslhost is OK: SSL OK - wiki.burning.art reverse DNS resolves to cp1.wikitide.net - CNAME OK [06:02:38] RECOVERY - test1 JobChron Service on test1 is OK: PROCS OK: 1 process with args 'redisJobChronService' [06:02:47] PROBLEM - wikitide.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikitide.net' expires in 8 day(s) (Mon 04 Dec 2023 03:06:37 PM GMT +0000). [06:03:15] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:04:53] RECOVERY - graylog1 Puppet on graylog1 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [06:05:14] RECOVERY - graylog1 HTTPS on graylog1 is OK: HTTP OK: HTTP/2 200 - 646 bytes in 0.027 second response time [06:06:24] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [06:06:59] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [06:07:02] RECOVERY - mem1 Puppet on mem1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:08:47] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [06:10:08] RECOVERY - prometheus1 Puppet on prometheus1 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [06:37:32] PROBLEM - bots1 NTP time on bots1 is CRITICAL: connect to address 10.0.0.123 port 5666: No route to hostconnect to host 10.0.0.123 port 5666: No route to host PROBLEM - cp1 Nginx Backend for mw2 on cp1 is CRITICAL: connect to address 10.0.0.113 port 5666: No route to hostconnect to host 10.0.0.113 port 5666: No route to host [06:37:32] RECOVERY - jobrunner1 NTP time on jobrunner1 is OK: NTP OK: Offset -0.002444922924 secs [06:37:32] RECOVERY - mw2 HTTPS on mw2 is OK: HTTP OK: HTTP/2 200 - 362 bytes in 0.031 second response time [06:37:32] RECOVERY - phorge1 NTP time on phorge1 is OK: NTP OK: Offset 0.06947258115 secs [06:37:32] PROBLEM - cp1 NTP time on cp1 is WARNING: NTP WARNING: Offset 0.3858127892 secs [06:37:35] RECOVERY - Host cp6 is UP: PING OK - Packet loss = 0%, RTA = 187.58 ms [06:37:36] RECOVERY - Host cp5 is UP: PING OK - Packet loss = 0%, RTA = 205.74 ms [06:37:37] RECOVERY - db1 NTP time on db1 is OK: NTP OK: Offset 0.0025780797 secs [06:37:38] PROBLEM - prometheus1 NTP time on prometheus1 is CRITICAL: connect to address 10.0.0.118 port 5666: No route to hostconnect to host 10.0.0.118 port 5666: No route to host [06:37:38] PROBLEM - graylog1 NTP time on graylog1 is CRITICAL: connect to address 10.0.0.117 port 5666: No route to hostconnect to host 10.0.0.117 port 5666: No route to host [06:37:39] RECOVERY - mw1 NTP time on mw1 is OK: NTP OK: Offset 0.001809030771 secs [06:37:40] RECOVERY - cp6 Nginx Backend for matomo1 on cp6 is OK: TCP OK - 0.000 second response time on localhost port 8204 [06:37:41] PROBLEM - swiftproxy1 NTP time on swiftproxy1 is CRITICAL: connect to address 10.0.0.122 port 5666: No route to hostconnect to host 10.0.0.122 port 5666: No route to host [06:37:42] RECOVERY - Host cp2 is UP: PING OK - Packet loss = 0%, RTA = 36.60 ms [06:37:43] RECOVERY - mem1 NTP time on mem1 is OK: NTP OK: Offset -0.01504737139 secs [06:37:47] RECOVERY - mw1 HTTPS on mw1 is OK: HTTP OK: HTTP/2 200 - 362 bytes in 0.036 second response time [06:37:47] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.406 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [06:37:48] PROBLEM - www.polandballwiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.polandballwiki.com reverse DNS resolves to vps-d267e842.vps.ovh.ca [06:37:48] RECOVERY - cp1 Nginx Backend for phorge1 on cp1 is OK: TCP OK - 0.000 second response time on localhost port 8200 [06:37:49] RECOVERY - test1 NTP time on test1 is OK: NTP OK: Offset 0.0007200539112 secs [06:37:52] RECOVERY - mon1 NTP time on mon1 is OK: NTP OK: Offset 0.001302838326 secs [06:37:52] RECOVERY - wc.wikitide.org on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Tue 20 Feb 2024 12:20:10 PM GMT +0000. [06:37:53] RECOVERY - Host cp3 is UP: PING OK - Packet loss = 0%, RTA = 96.31 ms [06:37:53] RECOVERY - cp3 Nginx Backend for matomo1 on cp3 is OK: TCP OK - 0.000 second response time on localhost port 8204 [06:37:59] RECOVERY - ldap1 NTP time on ldap1 is OK: NTP OK: Offset -0.008193403482 secs [06:37:59] RECOVERY - cp6 Nginx Backend for puppet1 on cp6 is OK: TCP OK - 0.000 second response time on localhost port 8201 [06:38:00] RECOVERY - cp6 HTTPS on cp6 is OK: HTTP OK: HTTP/2 200 - 2939 bytes in 1.454 second response time [06:38:02] RECOVERY - cp6 Nginx Backend for mail1 on cp6 is OK: TCP OK - 0.000 second response time on localhost port 8202 [06:38:03] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 54% [06:38:03] PROBLEM - swiftac1 NTP time on swiftac1 is CRITICAL: connect to address 10.0.0.120 port 5666: No route to hostconnect to host 10.0.0.120 port 5666: No route to host [06:38:07] RECOVERY - cp6 Nginx Backend for phorge1 on cp6 is OK: TCP OK - 0.000 second response time on localhost port 8200 [06:38:07] PROBLEM - services1 NTP time on services1 is WARNING: NTP WARNING: Offset 0.1765293777 secs [06:38:09] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 20 minutes ago with 0 failures [06:38:10] RECOVERY - Host cp4 is UP: PING OK - Packet loss = 0%, RTA = 128.90 ms [06:38:10] RECOVERY - mail1 NTP time on mail1 is OK: NTP OK: Offset 0.001951545477 secs [06:38:11] PROBLEM - bots1 Puppet on bots1 is CRITICAL: connect to address 10.0.0.123 port 5666: No route to hostconnect to host 10.0.0.123 port 5666: No route to host [06:38:11] RECOVERY - bast1 NTP time on bast1 is OK: NTP OK: Offset 0.02172848582 secs [06:38:13] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 79% [06:38:13] RECOVERY - puppet1 NTP time on puppet1 is OK: NTP OK: Offset -0.004508823156 secs [06:38:14] RECOVERY - mw2 NTP time on mw2 is OK: NTP OK: Offset 0.003044784069 secs [06:38:16] RECOVERY - mail1 SMTP on mail1 is OK: SMTP OK - 0.070 sec. response time [06:38:17] RECOVERY - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is OK: OK - NGINX Error Rate is 11% [06:38:17] RECOVERY - cp1 Varnish Backends on cp1 is OK: All 7 backends are healthy [06:38:19] RECOVERY - cp4 Nginx Backend for puppet1 on cp4 is OK: TCP OK - 0.000 second response time on localhost port 8201 [06:38:22] PROBLEM - swiftobject1 NTP time on swiftobject1 is CRITICAL: NTP CRITICAL: Offset 1.175358266 secs [06:38:23] RECOVERY - os1 NTP time on os1 is OK: NTP OK: Offset 0.002530813217 secs [06:39:21] RECOVERY - cp1 Nginx Backend for mw2 on cp1 is OK: TCP OK - 0.000 second response time on localhost port 8101 [06:39:23] PROBLEM - db1 Puppet on db1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:39:28] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [06:39:28] RECOVERY - cp5 HTTPS on cp5 is OK: HTTP OK: HTTP/2 200 - 2894 bytes in 1.584 second response time [06:39:28] RECOVERY - cp6 Disk Space on cp6 is OK: DISK OK - free space: / 6401MiB (33% inode=94%); [06:39:28] RECOVERY - cp1 NTP time on cp1 is OK: NTP OK: Offset -0.003505110741 secs [06:39:29] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is CRITICAL: CRITICAL - NGINX Error Rate is 71% [06:39:31] RECOVERY - cp1 HTTPS on cp1 is OK: HTTP OK: HTTP/2 200 - 2916 bytes in 0.022 second response time [06:39:32] RECOVERY - bots1 NTP time on bots1 is OK: NTP OK: Offset 0.002213567495 secs [06:39:33] RECOVERY - cp4 Nginx Backend for test1 on cp4 is OK: TCP OK - 0.000 second response time on localhost port 8180 [06:39:38] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/2 200 - 2894 bytes in 0.250 second response time [06:39:38] RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/2 200 - 2895 bytes in 0.625 second response time [06:39:38] PROBLEM - cp1 Puppet on cp1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:39:38] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [06:39:38] PROBLEM - phorge1 Puppet on phorge1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:39:41] RECOVERY - prometheus1 NTP time on prometheus1 is OK: NTP OK: Offset 0.001915335655 secs [06:39:42] RECOVERY - swiftproxy1 NTP time on swiftproxy1 is OK: NTP OK: Offset 0.002323418856 secs [06:39:42] RECOVERY - graylog1 NTP time on graylog1 is OK: NTP OK: Offset 0.001744270325 secs [06:39:44] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:39:48] RECOVERY - cp4 HTTPS on cp4 is OK: HTTP OK: HTTP/2 200 - 2916 bytes in 0.785 second response time [06:39:58] RECOVERY - cp4 Nginx Backend for phorge1 on cp4 is OK: TCP OK - 0.000 second response time on localhost port 8200 [06:39:59] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 3.664 second response time [06:40:00] PROBLEM - os1 Puppet on os1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:40:02] PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.426 second response time [06:40:02] PROBLEM - puppet1 puppetserver on puppet1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:40:03] RECOVERY - swiftac1 NTP time on swiftac1 is OK: NTP OK: Offset 0.003044962883 secs [06:40:03] PROBLEM - thegreatwar.uk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - thegreatwar.uk reverse DNS resolves to vps-d267e842.vps.ovh.ca [06:40:05] RECOVERY - mail1 webmail.wikitide.net HTTPS on mail1 is OK: HTTP OK: HTTP/1.1 200 OK - 6205 bytes in 0.044 second response time [06:40:05] PROBLEM - bots1 Puppet on bots1 is UNKNOWN: NRPE: Unable to read output [06:40:07] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:40:09] PROBLEM - wikitide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikitide.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [06:40:10] PROBLEM - wiki.burning.art - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.burning.art reverse DNS resolves to vps-d267e842.vps.ovh.ca [06:40:11] PROBLEM - graylog1 HTTPS on graylog1 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 113 bytes in 0.014 second response time [06:40:12] RECOVERY - services1 NTP time on services1 is OK: NTP OK: Offset -0.0003235042095 secs [06:40:24] RECOVERY - swiftobject1 NTP time on swiftobject1 is OK: NTP OK: Offset 0.002226054668 secs [06:40:58] PROBLEM - db1 MariaDB on db1 is CRITICAL: Access denied for user 'icinga'@'10.0.0.111' (using password: YES) [06:41:27] RECOVERY - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is OK: OK - NGINX Error Rate is 6% [06:41:34] PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.322 second response time [06:42:00] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 18% [06:42:00] PROBLEM - bots1 Puppet on bots1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:42:04] PROBLEM - swiftac1 Puppet on swiftac1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:42:09]