[00:01:18] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:03:18] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:05:58] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:08:45] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:10:03] RECOVERY - prometheus1 Puppet on prometheus1 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [00:10:22] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:10:29] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [00:11:57] RECOVERY - phorge1 Puppet on phorge1 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [00:12:43] RECOVERY - ldap1 Puppet on ldap1 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [03:01:37] RECOVERY - mon1 Backups Grafana on mon1 is OK: FILE_AGE OK: /var/log/grafana-backup.log is 95 seconds old and 347 bytes [06:14:59] RECOVERY - wc.wikitide.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000. [06:18:59] PROBLEM - wc.wikitide.org on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 27 day(s) (Tue 20 Feb 2024 12:20:10 PM GMT +0000). [07:12:59] PROBLEM - wc.wikitide.org on sslhost is CRITICAL: connect to address wikitide.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [07:14:59] PROBLEM - wc.wikitide.org on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 27 day(s) (Tue 20 Feb 2024 12:20:10 PM GMT +0000). [12:14:59] RECOVERY - wc.wikitide.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000. [12:18:59] PROBLEM - wc.wikitide.org on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 27 day(s) (Tue 20 Feb 2024 12:20:10 PM GMT +0000). [19:10:45] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:12:04] PROBLEM - prometheus1 Puppet on prometheus1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:12:26] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:12:29] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:13:57] PROBLEM - phorge1 Puppet on phorge1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:14:43] PROBLEM - ldap1 Puppet on ldap1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:17:23] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [19:21:40] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:23:13] PROBLEM - matomo1 Puppet on matomo1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:23:38] PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:23:45] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:24:14] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:27:52] PROBLEM - db1 Puppet on db1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:28:08] PROBLEM - cp1 Puppet on cp1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:28:28] PROBLEM - bast1 Puppet on bast1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:31:54] PROBLEM - swiftproxy1 Puppet on swiftproxy1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:33:07] PROBLEM - swiftac1 Puppet on swiftac1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:34:28] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:37:58] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:37:59] PROBLEM - graylog1 Puppet on graylog1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:38:45] PROBLEM - mem1 Puppet on mem1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:46:31] PROBLEM - swiftobject1 Puppet on swiftobject1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:47:21] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [19:49:13] PROBLEM - jobchron1 Puppet on jobchron1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:49:51] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:49:59] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:52:30] PROBLEM - bots1 Puppet on bots1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:14:59] RECOVERY - wc.wikitide.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000. [21:18:59] PROBLEM - wc.wikitide.org on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 26 day(s) (Tue 20 Feb 2024 12:20:10 PM GMT +0000). [23:14:59] RECOVERY - wc.wikitide.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000. [23:18:59] PROBLEM - wc.wikitide.org on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 26 day(s) (Tue 20 Feb 2024 12:20:10 PM GMT +0000).