[00:15:58] PROBLEM - marvelcinematicuniverse.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for marvelcinematicuniverse.wiki could not be found [00:26:06] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [00:42:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:28:05] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [01:28:54] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [01:57:12] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:58:19] RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 14 Jun 2024 04:28:50 PM GMT +0000. [02:22:21] [02mw-config] 07Universal-Omega closed pull request 03#5557: T12108: Please alter these DynamicPageList3 variables on metzo.miraheze.org - 13https://github.com/miraheze/mw-config/pull/5557 [02:22:23] [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/80ebe93ff1ea...fc83ff44befa [02:22:26] [02mw-config] 07brandon-wm 03fc83ff4 - T12108: Please alter these DynamicPageList3 variables on metzo.miraheze.org (#5557) [02:23:20] miraheze/mw-config - Universal-Omega the build passed. [02:23:47] [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/fc83ff44befa...940a3a48a182 [02:23:48] [02mw-config] 07Universal-Omega 03940a3a4 - Fix tab [02:24:05] !log [@test151] starting deploy of {'config': True} to test151 [02:24:06] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [02:24:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:24:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:24:28] [02mw-config] 07Universal-Omega commented on pull request 03#5557: T12108: Please alter these DynamicPageList3 variables on metzo.miraheze.org - 13https://github.com/miraheze/mw-config/pull/5557#issuecomment-2109154613 [02:24:44] miraheze/mw-config - Universal-Omega the build passed. [02:27:07] [02mw-config] 07brandon-wm synchronize pull request 03#5558: T12085: About “Extension:TabberNeue” - 13https://github.com/miraheze/mw-config/pull/5558 [02:27:25] [02mw-config] 07brandon-wm closed pull request 03#5558: T12085: About “Extension:TabberNeue” - 13https://github.com/miraheze/mw-config/pull/5558 [02:28:00] miraheze/mw-config - brandon-wm the build passed. [02:30:19] [02mw-config] 07brandon-wm commented on pull request 03#5559: T11757: Disable Upload Wizard Tutorial and Remove Creative Commons selection on aieseattle.miraheze.org - 13https://github.com/miraheze/mw-config/pull/5559#issuecomment-2109159499 [02:32:41] !log [@mwtask181] starting deploy of {'config': True} to all [02:32:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:32:52] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 10s [02:32:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:37:40] !log [@mwtask171] starting deploy of {'config': True} to all [02:37:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:37:48] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [02:37:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:39:47] [02mw-config] 07brandon-wm opened pull request 03#5560: T12085: About “Extension:TabberNeue” - 13https://github.com/miraheze/mw-config/pull/5560 [02:40:39] miraheze/mw-config - brandon-wm the build passed. [02:56:08] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:58:20] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [04:10:45] RECOVERY - marvelcinematicuniverse.wiki - reverse DNS on sslhost is OK: SSL OK - marvelcinematicuniverse.wiki reverse DNS resolves to cp36.wikitide.net - CNAME FLAT [04:27:29] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:30:59] !log [evalprime@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=wiki (END - exit=65280) [05:31:02] !log [evalprime@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=wiki (END - exit=65280) [05:31:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:31:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:31:14] !log [evalprime@mwtask181] starting deploy of {'files': '../mediawiki/1.41/extensions/SemanticMediaWiki/.smw.json'} to all [05:31:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:31:24] !log [evalprime@mwtask181] finished deploy of {'files': '../mediawiki/1.41/extensions/SemanticMediaWiki/.smw.json'} to all - SUCCESS in 9s [05:31:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:56:03] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [06:03:01] [02mw-config] 07Universal-Omega commented on pull request 03#5560: T12085: About “Extension:TabberNeue” - 13https://github.com/miraheze/mw-config/pull/5560#issuecomment-2109352467 [09:24:23] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 84.247.188.12/cpweb, 37.60.225.206/cpweb [09:26:23] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [09:46:51] [02CreateWiki] 07redbluegreenhat synchronize pull request 03#516: Add an API endpoint to post comments to wiki requests - 13https://github.com/miraheze/CreateWiki/pull/516 [09:55:12] PROBLEM - db171 Backups SQL on db171 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864220 seconds old and 124977 bytes [09:55:16] miraheze/CreateWiki - redbluegreenhat the build passed. [10:08:35] !log [amokwiki]> INSERT INTO content_models (model_id, model_name) VALUES (1, 'wikitext'); [10:08:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:46:23] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 37.60.225.206/cpweb [11:48:23] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:43:04] PROBLEM - cloud18 Puppet on cloud18 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [13:11:11] RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:28:12] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:57:16] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [14:26:22] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [16:03:04] PROBLEM - kafka181 NTP time on kafka181 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o