[01:45:52] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [01:49:13] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:19:15] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [02:43:49] RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 14 Jun 2024 04:28:50 PM GMT +0000. [02:57:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:12:31] PROBLEM - theagonian.com - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [03:33:14] PROBLEM - theagonian.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for theagonian.com could not be found [04:54:10] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 144.68 ms [04:56:12] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 142.46 ms [05:21:38] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [05:40:52] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 144.63 ms [05:42:53] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 141.28 ms [05:51:42] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:19:30] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 141.02 ms [06:21:32] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 141.26 ms [06:21:49] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [06:43:57] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 141.43 ms [06:45:58] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 137.50 ms [06:57:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:01:08] PROBLEM - mw162 NTP time on mw162 is WARNING: NTP WARNING: Offset -0.1031705141 secs [08:12:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:24:14] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:52:20] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:54:17] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [09:01:04] RECOVERY - mw162 NTP time on mw162 is OK: NTP OK: Offset -0.07320281863 secs [09:26:15] PROBLEM - with.cpt-ra.bid - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [09:54:53] RECOVERY - with.cpt-ra.bid - LetsEncrypt on sslhost is OK: OK - Certificate 'with.cpt-ra.bid' will expire on Tue 06 Aug 2024 02:45:02 PM GMT +0000. [09:57:20] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:58:43] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw182 [11:59:05] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw182 [12:01:04] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [12:01:09] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:02:43] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [12:31:15] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [12:57:20] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:03:24] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:15:51] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [13:33:28] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [13:39:16] PROBLEM - mystiverse.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mystiverse.wiki All nameservers failed to answer the query. [13:44:39] RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 14 Jun 2024 04:28:50 PM GMT +0000. [14:00:03] [1/2] https://issue-tracker.miraheze.org/T12123 [14:00:04] [2/2] This problem has seriously affected the use. Please pay attention and solve this problem as soon as possible [14:00:44] Jesus Christ why is it always your wiki with these issues [14:00:53] I’ll take a look but I’m not tech so [14:02:19] I don't know why that's happening [14:02:28] [1/2] > Please pay attention and solve this problem as soon as possible [14:02:29] [2/2] We are all volunteers there is no need to be rude about it. [14:03:31] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:09:11] PROBLEM - mystiverse.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mystiverse.wiki could not be found [14:15:08] Please disable the EditSubpages extension [14:15:10] https://www.mediawiki.org/wiki/Extension:EditSubpages [14:50:10] https://tno.wiki/wiki/File:Miraheze.png thouhts? [15:03:39] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [15:07:20] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [15:16:12] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 141.83 ms [15:16:40] [02mw-config] 07OAuthority closed pull request 03#5571: T12147: Custom Footer Icons for tnowiki - 13https://github.com/miraheze/mw-config/pull/5571 [15:16:42] [02mw-config] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a4040a4737bc...946e21cf3f36 [15:16:43] [02mw-config] 07pixDeVl 03946e21c - T12147: Custom Footer Icons for tnowiki (#5571) [15:17:38] miraheze/mw-config - OAuthority the build passed. [15:18:14] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 141.43 ms [15:22:08] @originalauthority uh, i guess i should have mentioned that pull request should have been drafted waiting an update of the image link as requested by rec 😅 [15:22:23] https://static.miraheze.org/tnowiki/4/49/Miraheze.png [15:22:37] they updated the original image [15:22:47] The image link still refers to the old one rigthV [15:23:10] Different versions have different static cdn links I thought [15:23:16] no https://static.miraheze.org/tnowiki/4/49/Miraheze.png [15:23:33] they're sub-dir'ed based on their name [15:23:40] since they have the same name its the same location [15:23:42] !log [@test151] starting deploy of {'config': True} to test151 [15:23:43] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [15:23:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:23:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:23:52] Ah [15:24:06] I was looking at the numbers on the mediawiki one! [15:24:11] they can have different cachekeys which would be `?somekeyhere` or something but I don't think miraheze does that [15:24:14] Thanks [15:32:06] Does anything on static even use parameters [15:32:57] !log [@mwtask181] starting deploy of {'config': True} to all [15:33:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:33:08] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 11s [15:33:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:37:26] !log [@mwtask171] starting deploy of {'config': True} to all [15:37:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:37:34] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [15:37:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:52:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [15:58:08] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 141.13 ms [16:00:09] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 141.17 ms [16:27:41] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 142.32 ms [16:29:42] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 140.67 ms [17:55:18] PROBLEM - nijigen.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nijigen.wiki could not be found [18:39:12] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:01:04] PROBLEM - kafka181 NTP time on kafka181 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o