[00:09:57] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[01:05:46] <icinga-miraheze>	 PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query.
[01:35:04] <icinga-miraheze>	 RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK
[02:06:15] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[02:13:41] <icinga-miraheze>	 PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query.
[02:43:21] <icinga-miraheze>	 RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp36.wikitide.net - CNAME OK
[04:08:35] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[05:06:10] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[06:08:19] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[06:16:30] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[06:36:58] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[07:13:15] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures
[07:33:35] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[07:35:43] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:37:41] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001966059208 secs
[07:50:20] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[08:19:10] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK
[08:43:48] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[08:47:55] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -9.363889694e-05 secs
[10:47:06] <icinga-miraheze>	 PROBLEM - db181 Backups SQL on db181 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864193 seconds old and 130240 bytes
[11:47:17] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[12:08:21] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[12:37:04] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[13:08:17] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query.
[14:06:14] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[14:52:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[15:02:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[15:08:25] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[16:06:12] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:17:01] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns]
[16:45:06] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:56:36] <icinga-miraheze>	 PROBLEM - encyclopediarobotica.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'encyclopediarobotica.org' expires in 15 day(s) (Mon 01 Jul 2024 04:30:05 PM GMT +0000).
[16:56:47] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/47a422dc5f04...c38a00010803
[16:56:49] <botifico>	 [02ssl] 07WikiTideSSLBot 03c38a000 - Bot: Update SSL cert for encyclopediarobotica.org
[17:25:42] <icinga-miraheze>	 RECOVERY - encyclopediarobotica.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediarobotica.org' will expire on Fri 13 Sep 2024 03:56:42 PM GMT +0000.
[18:16:38] <icinga-miraheze>	 PROBLEM - starveil.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'starveil.wiki' expires in 15 day(s) (Mon 01 Jul 2024 06:05:08 PM GMT +0000).
[18:16:52] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c38a00010803...9b8516a072c7
[18:16:53] <botifico>	 [02ssl] 07WikiTideSSLBot 039b8516a - Bot: Update SSL cert for starveil.wiki
[18:31:36] <icinga-miraheze>	 PROBLEM - mw151 APT on mw151 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (9 critical updates).
[18:31:56] <icinga-miraheze>	 PROBLEM - mw152 APT on mw152 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (9 critical updates).
[18:33:53] <icinga-miraheze>	 PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 10 packages available for upgrade (9 critical updates).
[18:37:00] <icinga-miraheze>	 PROBLEM - mw162 APT on mw162 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (9 critical updates).
[18:38:24] <icinga-miraheze>	 PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 10 packages available for upgrade (9 critical updates).
[18:40:41] <icinga-miraheze>	 PROBLEM - mw161 APT on mw161 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (9 critical updates).
[18:41:39] <icinga-miraheze>	 PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (9 critical updates).
[18:41:56] <icinga-miraheze>	 PROBLEM - mw181 APT on mw181 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (9 critical updates).
[18:43:10] <icinga-miraheze>	 PROBLEM - mw172 APT on mw172 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (9 critical updates).
[18:44:23] <icinga-miraheze>	 PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (9 critical updates).
[18:44:54] <icinga-miraheze>	 PROBLEM - mw171 APT on mw171 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (9 critical updates).
[19:02:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[19:15:52] <icinga-miraheze>	 RECOVERY - starveil.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'starveil.wiki' will expire on Fri 13 Sep 2024 05:16:46 PM GMT +0000.
[19:22:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[20:02:20] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[20:07:20] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[20:38:20] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[21:36:18] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[22:35:47] <icinga-miraheze>	 PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[23:05:05] <icinga-miraheze>	 RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK