[00:26:05] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:23:30] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [01:24:32] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [02:21:28] RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Tue 27 Aug 2024 03:54:13 PM GMT +0000. [02:23:24] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [06:16:03] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [06:45:06] RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [07:24:03] PROBLEM - cloud18 APT on cloud18 is CRITICAL: APT CRITICAL: 79 packages available for upgrade (1 critical updates). [07:28:54] PROBLEM - cloud17 APT on cloud17 is CRITICAL: APT CRITICAL: 77 packages available for upgrade (1 critical updates). [07:30:36] PROBLEM - cloud15 APT on cloud15 is CRITICAL: APT CRITICAL: 83 packages available for upgrade (1 critical updates). [07:32:59] PROBLEM - cloud16 APT on cloud16 is CRITICAL: APT CRITICAL: 77 packages available for upgrade (1 critical updates). [08:23:29] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/ce25aa8ef880...db85b7b8e3cc [08:23:30] [02ssl] 07WikiTideSSLBot 03db85b7b - Bot: Add SSL cert for wiki.penguinmod.com [08:25:24] [02dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/43707f6dc8fe...36c13d36f710 [08:25:26] [02dns] 07Reception123 0336c13d3 - Create bonesword.wiki [08:30:15] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/db85b7b8e3cc...812b1971ade5 [08:30:16] [02ssl] 07WikiTideSSLBot 03812b197 - Bot: Add SSL cert for bonesword.wiki [08:31:22] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/812b1971ade5...6b01586c6ff7 [08:31:24] [02ssl] 07WikiTideSSLBot 036b01586 - Bot: Add SSL cert for capixabapedia.vipnunes.com [09:22:20] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:38:21] PROBLEM - db171 Backups SQL on db171 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209658 seconds old and 97655 bytes [09:53:52] PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 212.94 ms [09:59:55] RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 138.05 ms [10:20:28] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [10:51:37] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:20:42] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [13:15:51] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [13:19:05] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [13:44:39] RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Tue 27 Aug 2024 03:54:13 PM GMT +0000. [13:48:10] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:46:19] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [15:15:24] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [15:41:53] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [16:08:05] [02mw-config] 07songnguxyz synchronize pull request 03#5582: UploadWizard patch for lhmnwiki - 13https://github.com/miraheze/mw-config/pull/5582 [16:08:08] [02mw-config] 07coderabbitai[bot] edited a comment on pull request 03#5582: UploadWizard patch for lhmnwiki - 13https://github.com/miraheze/mw-config/pull/5582#issuecomment-2142485780 [16:09:00] miraheze/mw-config - songnguxyz the build passed. [16:09:31] [02mw-config] 07songnguxyz commented on pull request 03#5582: UploadWizard patch for lhmnwiki - 13https://github.com/miraheze/mw-config/pull/5582#issuecomment-2143500085 [16:09:35] [02mw-config] 07coderabbitai[bot] edited a comment on pull request 03#5582: UploadWizard patch for lhmnwiki - 13https://github.com/miraheze/mw-config/pull/5582#issuecomment-2142485780 [16:10:40] RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Tue 27 Aug 2024 03:54:13 PM GMT +0000. [16:11:01] [02mw-config] 07coderabbitai[bot] edited pull request 03#5582: UploadWizard patch for lhmnwiki - 13https://github.com/miraheze/mw-config/pull/5582 [16:11:03] [02mw-config] 07songnguxyz edited a comment on pull request 03#5582: UploadWizard patch for lhmnwiki - 13https://github.com/miraheze/mw-config/pull/5582#issuecomment-2143500085 [16:11:52] [02mw-config] 07coderabbitai[bot] edited a comment on pull request 03#5582: UploadWizard patch for lhmnwiki - 13https://github.com/miraheze/mw-config/pull/5582#issuecomment-2142485780 [16:12:05] [02mw-config] 07coderabbitai[bot] commented on pull request 03#5582: UploadWizard patch for lhmnwiki - 13https://github.com/miraheze/mw-config/pull/5582#issuecomment-2143500964 [16:12:08] [02mw-config] 07coderabbitai[bot] edited a comment on pull request 03#5582: UploadWizard patch for lhmnwiki - 13https://github.com/miraheze/mw-config/pull/5582#issuecomment-2142485780 [16:35:03] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/moveBatch.php --wiki=themanaworldwiki -r=Requested at T12165 /home/reception/tmw-wiki-rename-dev-to-development (END - exit=256) [16:35:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:38:20] [02dns] 07Reception123 reviewed pull request 03#520 commit - 13https://github.com/miraheze/dns/pull/520#discussion_r1623248656 [16:41:39] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=neavetvwiki --summary=Imported from https://neavetv.fandom.com --search-recursively /home/reception/neavetv (START) [16:41:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:42:46] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=neavetvwiki --summary=Imported from https://neavetv.fandom.com --search-recursively /home/reception/neavetv (END - exit=0) [16:42:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:45:49] !log [salt-user@mwtask181] Renamed ieytdwiki to zoraxipediawiki using renamewiki.py [16:45:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:48:12] !log [salt-user@mwtask181] Renamed hublotwiki to justleafywiki using renamewiki.py [16:48:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:04:46] !log [salt-user@mwtask181] Renamed thebatmanwiki to batmanwiki using renamewiki.py [17:04:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:34:58] [02mw-config] 07Reception123 reviewed pull request 03#5580 commit - 13https://github.com/miraheze/mw-config/pull/5580#discussion_r1623255480 [17:36:57] miraheze/mw-config - DrewIndigo the build passed. [17:38:43] [02mw-config] 07songnguxyz reviewed pull request 03#5580 commit - 13https://github.com/miraheze/mw-config/pull/5580#discussion_r1623255894 [18:20:28] [02mw-config] 07songnguxyz reviewed pull request 03#5580 commit - 13https://github.com/miraheze/mw-config/pull/5580#discussion_r1623261499 [18:21:53] [02mw-config] 07Reception123 closed pull request 03#5579: add custom restriction levels to hypotheticalweatherwiki - 13https://github.com/miraheze/mw-config/pull/5579 [18:21:55] [02mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a0308ca8c8b7...725f373f67e7 [18:21:57] [02mw-config] 07DrewIndigo 03725f373 - add custom restriction levels to hypotheticalweatherwiki (#5579) [18:22:48] miraheze/mw-config - Reception123 the build passed. [18:22:52] !log [reception@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [18:22:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:06] !log [reception@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 13s [18:23:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:27] !log [@test151] starting deploy of {'config': True} to test151 [18:23:28] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [18:23:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:37:40] !log [@mwtask171] starting deploy of {'config': True} to all [18:37:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:37:50] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 9s [18:37:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:06:45] [02MirahezeMagic] 07colleirose synchronize pull request 03#491: Add SEO tweaks - 13https://github.com/miraheze/MirahezeMagic/pull/491 [19:13:07] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:42:12] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [19:45:23] PROBLEM - wiki.lewisgill.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [20:14:47] RECOVERY - wiki.lewisgill.com - reverse DNS on sslhost is OK: SSL OK - wiki.lewisgill.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [20:18:26] PROBLEM - wiki.lewisgill.com - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [20:46:23] PROBLEM - wiki.lewisgill.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [20:47:46] RECOVERY - wiki.lewisgill.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lewisgill.com' will expire on Tue 23 Jul 2024 09:32:19 PM GMT +0000. [20:55:49] [02CreateWiki] 07redbluegreenhat commented on pull request 03#516: Add an API endpoint to post comments to wiki requests - 13https://github.com/miraheze/CreateWiki/pull/516#issuecomment-2143586999 [21:15:47] RECOVERY - wiki.lewisgill.com - reverse DNS on sslhost is OK: SSL OK - wiki.lewisgill.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [21:19:18] PROBLEM - wiki.lewisgill.com - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:40:31] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:47:17] [02dns] 07BlankEclair reviewed pull request 03#520 commit - 13https://github.com/miraheze/dns/pull/520#discussion_r1623287385 [21:48:39] RECOVERY - wiki.lewisgill.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lewisgill.com' will expire on Tue 23 Jul 2024 09:32:19 PM GMT +0000. [22:09:38] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [22:17:55] PROBLEM - wiki.lewisgill.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [22:20:06] PROBLEM - wiki.lewisgill.com - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [22:33:50] PROBLEM - db181 PowerDNS Recursor on db181 is CRITICAL: CRITICAL - Plugin timed out while executing system call [22:34:39] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw161 [22:34:53] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 47.65, 26.00, 10.72 [22:35:00] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw161 [22:35:33] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw161 [22:35:46] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw161 [22:35:49] RECOVERY - db181 PowerDNS Recursor on db181 is OK: DNS OK: 0.207 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [22:35:54] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw161 [22:36:38] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw161 [22:36:55] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [22:37:15] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5627 bytes in 0.013 second response time [22:38:38] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [22:39:12] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.461 second response time [22:39:33] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [22:39:40] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [22:39:54] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [22:40:38] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [22:40:51] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.35, 8.51, 7.76 [22:49:44] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.09, 19.17, 14.45 [22:51:44] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.85, 20.14, 15.37 [22:53:44] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.52, 18.66, 15.43 [22:58:17] !log [oa@mwtask181] starting deploy of {'l10n': True, 'versions': '1.41', 'upgrade_skins': 'Citizen'} to all [22:58:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:59:07] !log [oa@mwtask181] finished deploy of {'l10n': True, 'versions': '1.41', 'upgrade_skins': 'Citizen'} to all - SUCCESS in 50s [22:59:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:49] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.50, 19.35, 14.14 [23:02:49] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 9.89, 15.36, 13.30 [23:32:51] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 8.79, 12.93, 6.26 [23:34:51] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 1.36, 8.71, 5.52