[00:31:30] [02statichelp] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±190] 13https://github.com/miraheze/statichelp/compare/0bed3c60e9a6...d13350cbd8c8 [00:31:31] [02statichelp] 07Universal-Omega 03d13350c - Auto-update Tech namespace pages 2024-09-17 00:31:04.354406 [01:29:25] [02dns] 07Universal-Omega closed pull request 03#617: remove avas.wiki zone - 13https://github.com/miraheze/dns/pull/617 [01:29:28] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/e62b4e9b05b8...08cf9981d630 [01:29:31] [02dns] 07MacFan4000 0308cf998 - remove avas.wiki zone (#617) [01:33:11] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/b88b96e81a8b...464beb2259ee [01:33:13] [02ssl] 07WikiTideSSLBot 03464beb2 - Bot: Add SSL cert for dccomicswiki.com [01:33:47] [02ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/464beb2259ee...a5d210258136 [01:33:48] [02ssl] 07MacFan4000 03a5d2102 - Update certs.yaml [01:36:13] [02puppet] 07Universal-Omega created branch 03techdocs - 13https://github.com/miraheze/puppet [01:36:15] [02puppet] 07Universal-Omega pushed 031 commit to 03techdocs [+1/-0/±0] 13https://github.com/miraheze/puppet/commit/d39eea0f3792 [01:36:18] [02puppet] 07Universal-Omega 03d39eea0 - Add python script to update static tech docs [01:36:21] [02puppet] 07Universal-Omega opened pull request 03#3933: Add python script to update static tech docs - 13https://github.com/miraheze/puppet/pull/3933 [01:36:25] [02puppet] 07coderabbitai[bot] commented on pull request 03#3933: Add python script to update static tech docs - 13https://github.com/miraheze/puppet/pull/3933#issuecomment-2354334129 [01:37:36] miraheze/puppet - Universal-Omega the build has errored. [01:47:59] RECOVERY - mon181 Check correctness of the icinga configuration on mon181 is OK: Icinga configuration is correct [01:49:09] [02statichelp] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±13] 13https://github.com/miraheze/statichelp/compare/d13350cbd8c8...801a2dc7e2fb [01:49:10] [02statichelp] 07Universal-Omega 03801a2dc - Auto-update Tech namespace pages 2024-09-17 01:48:56.104915 [01:53:33] [02puppet] 07Universal-Omega synchronize pull request 03#3933: Add python script to update static tech docs - 13https://github.com/miraheze/puppet/pull/3933 [01:53:34] [02puppet] 07Universal-Omega pushed 031 commit to 03techdocs [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d39eea0f3792...124631dd0a2f [01:53:37] [02puppet] 07Universal-Omega 03124631d - Fix [01:54:48] miraheze/puppet - Universal-Omega the build has errored. [01:55:03] [02statichelp] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/miraheze/statichelp/compare/801a2dc7e2fb...afb1ea153d10 [01:55:04] [02statichelp] 07Universal-Omega 03afb1ea1 - Auto-update Tech namespace pages 2024-09-17 01:54:32.221674 [01:56:34] [02puppet] 07Universal-Omega synchronize pull request 03#3933: Add python script to update static tech docs - 13https://github.com/miraheze/puppet/pull/3933 [01:56:35] [02puppet] 07Universal-Omega pushed 031 commit to 03techdocs [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/124631dd0a2f...5541a26f374e [01:56:38] [02puppet] 07Universal-Omega 035541a26 - Fix [01:57:43] miraheze/puppet - Universal-Omega the build has errored. [01:58:51] [02puppet] 07Universal-Omega pushed 031 commit to 03techdocs [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5541a26f374e...3e4681d978a1 [01:58:52] [02puppet] 07Universal-Omega 033e4681d - Fix [01:58:55] [02puppet] 07Universal-Omega synchronize pull request 03#3933: Add python script to update static tech docs - 13https://github.com/miraheze/puppet/pull/3933 [02:00:11] miraheze/puppet - Universal-Omega the build passed. [02:16:51] [02puppet] 07Universal-Omega synchronize pull request 03#3933: Add python script to update static tech docs - 13https://github.com/miraheze/puppet/pull/3933 [02:16:53] [02puppet] 07Universal-Omega pushed 031 commit to 03techdocs [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3e4681d978a1...a1b4f2358206 [02:16:55] [02puppet] 07Universal-Omega 03a1b4f23 - Fix User-Agent [02:16:56] [02puppet] 07coderabbitai[bot] edited a comment on pull request 03#3933: Add python script to update static tech docs - 13https://github.com/miraheze/puppet/pull/3933#issuecomment-2354334129 [02:18:10] miraheze/puppet - Universal-Omega the build passed. [04:13:04] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 138.04 ms [04:15:04] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 138.13 ms [04:28:04] [02puppet] 07Universal-Omega commented on pull request 03#3933: Add python script to update static tech docs - 13https://github.com/miraheze/puppet/pull/3933#issuecomment-2354481026 [04:28:11] [02puppet] 07coderabbitai[bot] commented on pull request 03#3933: Add python script to update static tech docs - 13https://github.com/miraheze/puppet/pull/3933#issuecomment-2354481128 [04:28:13] [02puppet] 07coderabbitai[bot] edited a comment on pull request 03#3933: Add python script to update static tech docs - 13https://github.com/miraheze/puppet/pull/3933#issuecomment-2354334129 [04:29:00] [02puppet] 07coderabbitai[bot] edited pull request 03#3933: Add python script to update static tech docs - 13https://github.com/miraheze/puppet/pull/3933 [04:30:46] [02puppet] 07coderabbitai[bot] reviewed pull request 03#3933 commit - 13https://github.com/miraheze/puppet/pull/3933#discussion_r1762270755 [04:30:47] [02puppet] 07coderabbitai[bot] reviewed pull request 03#3933 commit - 13https://github.com/miraheze/puppet/pull/3933#discussion_r1762270752 [04:30:50] [02puppet] 07coderabbitai[bot] reviewed pull request 03#3933 commit - 13https://github.com/miraheze/puppet/pull/3933#discussion_r1762270757 [04:30:52] [02puppet] 07coderabbitai[bot] reviewed pull request 03#3933 commit - 13https://github.com/miraheze/puppet/pull/3933#discussion_r1762270756 [04:30:56] [02puppet] 07coderabbitai[bot] reviewed pull request 03#3933 commit - 13https://github.com/miraheze/puppet/pull/3933#discussion_r1762270759 [04:30:57] [02puppet] 07coderabbitai[bot] edited a comment on pull request 03#3933: Add python script to update static tech docs - 13https://github.com/miraheze/puppet/pull/3933#issuecomment-2354334129 [04:33:04] [02puppet] 07Universal-Omega synchronize pull request 03#3933: Add python script to update static tech docs - 13https://github.com/miraheze/puppet/pull/3933 [04:33:05] [02puppet] 07Universal-Omega pushed 031 commit to 03techdocs [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a1b4f2358206...4c3149303152 [04:33:08] [02puppet] 07Universal-Omega 034c31493 - Rename page_id to _ [04:33:19] [02puppet] 07coderabbitai[bot] edited a comment on pull request 03#3933: Add python script to update static tech docs - 13https://github.com/miraheze/puppet/pull/3933#issuecomment-2354334129 [04:34:23] miraheze/puppet - Universal-Omega the build passed. [05:13:48] [02puppet] 07Universal-Omega created branch 03techdocs-refactor - 13https://github.com/miraheze/puppet [05:13:50] [02puppet] 07Universal-Omega pushed 031 commit to 03techdocs-refactor [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/5c7b38c9429e [05:13:53] [02puppet] 07Universal-Omega 035c7b38c - techdocs: refactor [05:13:55] [02puppet] 07Universal-Omega opened pull request 03#3934: techdocs: refactor - 13https://github.com/miraheze/puppet/pull/3934 [05:13:59] [02puppet] 07coderabbitai[bot] commented on pull request 03#3934: techdocs: refactor - 13https://github.com/miraheze/puppet/pull/3934#issuecomment-2354538843 [05:15:10] miraheze/puppet - Universal-Omega the build has errored. [05:19:08] [02puppet] 07Universal-Omega synchronize pull request 03#3934: techdocs: refactor - 13https://github.com/miraheze/puppet/pull/3934 [05:19:10] [02puppet] 07Universal-Omega pushed 031 commit to 03techdocs-refactor [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5c7b38c9429e...a52e1ed43b09 [05:19:13] [02puppet] 07Universal-Omega 03a52e1ed - Fix [05:20:19] miraheze/puppet - Universal-Omega the build has errored. [05:22:59] [02puppet] 07Universal-Omega synchronize pull request 03#3934: techdocs: refactor - 13https://github.com/miraheze/puppet/pull/3934 [05:23:00] [02puppet] 07Universal-Omega pushed 031 commit to 03techdocs-refactor [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a52e1ed43b09...7b153471ace2 [05:23:02] [02puppet] 07Universal-Omega 037b15347 - Fix [05:24:15] miraheze/puppet - Universal-Omega the build has errored. [05:24:25] [02puppet] 07Universal-Omega synchronize pull request 03#3934: techdocs: refactor - 13https://github.com/miraheze/puppet/pull/3934 [05:24:26] [02puppet] 07Universal-Omega pushed 031 commit to 03techdocs-refactor [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7b153471ace2...37fa97a57c64 [05:24:27] [02puppet] 07Universal-Omega 0337fa97a - Fix [05:25:40] miraheze/puppet - Universal-Omega the build passed. [05:26:11] [02puppet] 07Universal-Omega closed pull request 03#3934: techdocs: refactor - 13https://github.com/miraheze/puppet/pull/3934 [05:26:15] [02puppet] 07Universal-Omega deleted branch 03techdocs-refactor - 13https://github.com/miraheze/puppet [05:26:18] [02puppet] 07Universal-Omega deleted branch 03techdocs-refactor [05:40:04] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 70%, RTA = 138.07 ms [05:44:06] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 138.06 ms [08:17:31] PROBLEM - eng.archiopedia.org - Cloudflare on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [08:46:44] RECOVERY - eng.archiopedia.org - Cloudflare on sslhost is OK: OK - Certificate 'eng.archiopedia.org' will expire on Thu 28 Nov 2024 01:27:12 PM GMT +0000. [08:52:05] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 138.08 ms [08:56:06] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 138.20 ms [10:29:35] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 138.04 ms [10:31:35] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 138.09 ms [10:52:31] [02CreateWiki] 07Reception123 created branch 03Reception123-patch-1 - 13https://github.com/miraheze/CreateWiki [10:52:34] [02CreateWiki] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/commit/4568708f656b [10:52:36] [02CreateWiki] 07Reception123 034568708 - update default Main Page with Raidarr version [10:52:38] [02CreateWiki] 07Reception123 opened pull request 03#549: update default Main Page with Raidarr version - 13https://github.com/miraheze/CreateWiki/pull/549 [10:52:42] [02CreateWiki] 07coderabbitai[bot] commented on pull request 03#549: update default Main Page with Raidarr version - 13https://github.com/miraheze/CreateWiki/pull/549#issuecomment-2355315261 [10:54:24] [02CreateWiki] 07github-actions[bot] pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/4568708f656b...7813a4913f85 [10:54:25] [02CreateWiki] 07github-actions 037813a49 - CI: lint code to MediaWiki standards [10:54:26] [02CreateWiki] 07github-actions[bot] synchronize pull request 03#549: update default Main Page with Raidarr version - 13https://github.com/miraheze/CreateWiki/pull/549 [10:56:05] miraheze/CreateWiki - Reception123 the build has errored. [10:57:07] [02CreateWiki] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/af014ed141b8...c7fb9d125d80 [10:57:09] [02CreateWiki] 07Reception123 03c7fb9d1 - update default Main Page with Raidarr version (#549) [10:57:10] [02CreateWiki] 07Reception123 closed pull request 03#549: update default Main Page with Raidarr version - 13https://github.com/miraheze/CreateWiki/pull/549 [10:57:11] [02CreateWiki] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://github.com/miraheze/CreateWiki [10:57:13] [02CreateWiki] 07Reception123 deleted branch 03Reception123-patch-1 [10:57:57] !log [reception@mwtask181] starting deploy of {'l10n': True, 'lang': 'en', 'versions': '1.42', 'upgrade_extensions': 'MirahezeMagic'} to all [10:58:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:58:30] !log [reception@mwtask181] finished deploy of {'l10n': True, 'lang': 'en', 'versions': '1.42', 'upgrade_extensions': 'MirahezeMagic'} to all - SUCCESS in 33s [10:58:32] !log [reception@mwtask181] starting deploy of {'l10n': True, 'lang': 'en', 'versions': '1.42', 'upgrade_extensions': 'MirahezeMagic'} to all [10:58:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:58:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:58:57] !log [reception@mwtask181] finished deploy of {'l10n': True, 'lang': 'en', 'versions': '1.42', 'upgrade_extensions': 'MirahezeMagic'} to all - SUCCESS in 25s [10:59:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:59:10] !log [reception@mwtask181] starting deploy of {'l10n': True, 'lang': 'en', 'versions': '1.42', 'upgrade_extensions': 'CreateWiki'} to all [10:59:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:59:57] !log [reception@mwtask181] finished deploy of {'l10n': True, 'lang': 'en', 'versions': '1.42', 'upgrade_extensions': 'CreateWiki'} to all - SUCCESS in 47s [11:00:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:00:27] miraheze/CreateWiki - Reception123 the build has errored. [11:21:34] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 138.26 ms [11:23:34] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 138.05 ms [11:51:32] [02mw-config] 07BlankEclair opened pull request 03#5663: Allow *.{miraheze,wikitide}.org and the custom domain for TemplateStyles - 13https://github.com/miraheze/mw-config/pull/5663 [11:52:30] miraheze/mw-config - BlankEclair the build passed. [12:55:36] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 90%, RTA = 138.14 ms [13:01:43] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 138.07 ms [13:11:58] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 138.07 ms [13:13:58] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 138.07 ms [13:20:21] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 90%, RTA = 138.18 ms [13:22:22] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 138.12 ms [13:24:56] PROBLEM - phorge171 php-fpm on phorge171 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2' [13:26:06] PROBLEM - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.009 second response time [13:26:52] PROBLEM - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 502 Bad Gateway [13:28:07] Oh ugget [13:28:10] Not that again [13:28:36] @Infrastructure Specialists can someone restart php on phorge171 [13:31:50] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 100% [13:35:14] ain't nobody gonna talk about ns2? [13:38:06] RECOVERY - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is OK: HTTP OK: HTTP/1.1 200 OK - 19630 bytes in 0.086 second response time [13:38:52] RECOVERY - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17718 bytes in 0.040 second response time [13:38:56] RECOVERY - phorge171 php-fpm on phorge171 is OK: PROCS OK: 9 processes with command name 'php-fpm8.2' [13:40:52] There is no war in ns2 [13:42:12] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 138.06 ms [14:03:03] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 138.09 ms [14:05:03] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 138.18 ms [14:06:29] BlankEclair: meh, most will use ns1 anyway. [14:25:48] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 138.07 ms [14:29:54] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 138.03 ms [14:36:13] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 80%, RTA = 138.13 ms [14:38:13] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 138.28 ms [15:12:17] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 138.12 ms [15:16:19] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 138.16 ms [15:23:38] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 138.13 ms [15:25:38] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 138.31 ms [15:36:03] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 90%, RTA = 138.18 ms [15:40:12] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 138.15 ms [15:45:32] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/deleteBatch.php --wiki=metawiki /home/reception/forcedelmeta.txt --r=issue with deletion (START) [15:45:34] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/deleteBatch.php --wiki=metawiki /home/reception/forcedelmeta.txt --r=issue with deletion (END - exit=0) [15:45:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:45:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:47:17] Advise the issue [15:47:21] @reception123 [15:49:02] [1/2] FAILED to delete article [15:49:03] [2/2] FAILED to delete article [15:49:04] https://issue-tracker.miraheze.org/T12593 [15:51:57] No exception? [15:52:40] nothing... [15:53:24] Not even in graylog ? [15:53:26] Weird [15:53:36] @reception123 did you WikiCache script finish? [15:53:48] I can look but I doubt that there is if there was nothing when running the script [15:53:51] yes [15:54:11] Did you run it twice? [15:55:06] yeah [15:55:20] I can try again if it didn't work? [15:55:38] When did it finish [15:55:43] Last report was an hour ago [15:55:52] 15:14 [15:55:53] it would've finished yesterday [15:55:56] It's 17;00 now [15:56:00] Oh [15:56:04] but I don't quite remember checking if the second one did finish for sure [15:56:14] @reception123 make sure it did cause it's a w wiki [15:56:19] So quite far down [15:56:27] I'll run again then to be safe [15:57:03] !log salt-ssh -E 'mw.*' cmd.run 'sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/MirahezeMagic/maintenance/resetWikiCaches.php' (again) [15:57:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:26:24] [02mw-config] 07Universal-Omega closed pull request 03#5663: Allow *.{miraheze,wikitide}.org and the custom domain for TemplateStyles - 13https://github.com/miraheze/mw-config/pull/5663 [16:26:26] [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/915c9873b6fa...22e0b1d1b3cd [16:26:27] [02mw-config] 07BlankEclair 0322e0b1d - Allow *.{miraheze,wikitide}.org and the custom domain for TemplateStyles (#5663) [16:27:22] miraheze/mw-config - Universal-Omega the build passed. [16:32:33] !log [@mwtask181] starting deploy of {'config': True} to all [16:32:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:33:00] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 27s [16:33:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:37:08] !log [@mwtask171] starting deploy of {'config': True} to all [16:37:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:37:22] PROBLEM - mem151 Current Load on mem151 is CRITICAL: LOAD CRITICAL - total load average: 7.44, 3.98, 1.81 [16:37:23] RECOVERY - mwtask181 Disk Space on mwtask181 is OK: DISK OK - free space: / 30882MiB (13% inode=93%); [16:37:36] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 28s [16:37:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:39:18] RECOVERY - mem151 Current Load on mem151 is OK: LOAD OK - total load average: 1.39, 2.82, 1.63 [16:43:23] PROBLEM - mwtask181 Disk Space on mwtask181 is CRITICAL: DISK CRITICAL - free space: / 0MiB (0% inode=93%); [16:45:23] RECOVERY - mwtask181 Disk Space on mwtask181 is OK: DISK OK - free space: / 43787MiB (19% inode=93%); [16:53:24] !log [@test151] starting deploy of {'config': True} to test151 [16:53:25] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [16:53:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:53:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:07:23] PROBLEM - mwtask181 Disk Space on mwtask181 is WARNING: DISK WARNING - free space: / 21567MiB (9% inode=93%); [17:17:23] PROBLEM - mwtask181 Disk Space on mwtask181 is CRITICAL: DISK CRITICAL - free space: / 12129MiB (5% inode=93%); [17:25:36] [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4bae14dd57c2...3bfc0e099637 [17:25:39] [02puppet] 07Universal-Omega 033bfc0e0 - T12607: add paladox icinga [17:32:34] PROBLEM - mem161 Current Load on mem161 is CRITICAL: LOAD CRITICAL - total load average: 9.17, 4.76, 2.07 [17:34:32] PROBLEM - mem161 Current Load on mem161 is WARNING: LOAD WARNING - total load average: 1.89, 3.60, 1.98 [17:35:17] [02mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/22e0b1d1b3cd...8e6c706949ed [17:35:20] [02mw-config] 07MacFan4000 038e6c706 - add paladox access to staffwiki [17:36:19] miraheze/mw-config - MacFan4000 the build passed. [17:36:33] RECOVERY - mem161 Current Load on mem161 is OK: LOAD OK - total load average: 0.66, 2.52, 1.78 [17:37:04] !log [@mwtask171] starting deploy of {'config': True} to all [17:37:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:37:29] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 25s [17:37:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:53:43] !log [@test151] starting deploy of {'config': True} to test151 [17:53:44] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [17:53:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:53:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:01:51] !log [@mwtask181] starting deploy of {'config': True} to all [18:01:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:02:19] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 28s [18:02:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:19:13] [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/6019ba439534 [18:19:15] [02puppet] 07paladox 036019ba4 - Add paladox [18:19:16] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [18:19:19] [02puppet] 07paladox opened pull request 03#3935: Add paladox - 13https://github.com/miraheze/puppet/pull/3935 [18:19:28] [02puppet] 07coderabbitai[bot] commented on pull request 03#3935: Add paladox - 13https://github.com/miraheze/puppet/pull/3935#issuecomment-2356607345 [18:19:56] [02puppet] 07paladox closed pull request 03#3935: Add paladox - 13https://github.com/miraheze/puppet/pull/3935 [18:19:57] [02puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3bfc0e099637...ff62800bacec [18:19:58] [02puppet] 07paladox 03ff62800 - Add paladox (#3935) [18:19:59] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [18:20:02] [02puppet] 07paladox deleted branch 03paladox-patch-1 [19:42:27] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [19:46:33] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 27.94 ms [20:00:55] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 80%, RTA = 27.85 ms [20:09:13] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 27.87 ms [20:19:38] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [20:21:38] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 27.87 ms [20:42:57] PROBLEM - mw173 APT on mw173 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:43:23] PROBLEM - swiftobject181 APT on swiftobject181 is CRITICAL: APT CRITICAL: 70 packages available for upgrade (1 critical updates). [20:43:43] PROBLEM - kafka181 APT on kafka181 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (1 critical updates). [20:44:22] PROBLEM - mattermost1 APT on mattermost1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:45:12] PROBLEM - mw174 APT on mw174 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:45:12] PROBLEM - swiftobject171 APT on swiftobject171 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [20:45:34] PROBLEM - eventgate181 APT on eventgate181 is CRITICAL: APT CRITICAL: 59 packages available for upgrade (1 critical updates). [20:45:43] PROBLEM - phorge171 APT on phorge171 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (1 critical updates). [20:46:18] PROBLEM - swiftac171 APT on swiftac171 is CRITICAL: APT CRITICAL: 57 packages available for upgrade (2 critical updates). [20:46:35] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:46:42] PROBLEM - reports171 APT on reports171 is CRITICAL: APT CRITICAL: 78 packages available for upgrade (1 critical updates). [20:47:12] PROBLEM - cloud17 APT on cloud17 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (1 critical updates). [20:47:23] PROBLEM - swiftobject161 APT on swiftobject161 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [20:47:29] PROBLEM - mw171 APT on mw171 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:47:46] PROBLEM - mw172 APT on mw172 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:47:51] PROBLEM - db151 APT on db151 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:47:52] PROBLEM - swiftobject151 APT on swiftobject151 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (1 critical updates). [20:47:58] PROBLEM - db181 APT on db181 is CRITICAL: APT CRITICAL: 75 packages available for upgrade (1 critical updates). [20:48:02] PROBLEM - cp36 APT on cp36 is CRITICAL: APT CRITICAL: 66 packages available for upgrade (1 critical updates). [20:48:09] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:48:27] PROBLEM - mw153 APT on mw153 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:48:32] PROBLEM - mon181 APT on mon181 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:49:00] PROBLEM - mwtask161 APT on mwtask161 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:49:02] PROBLEM - graphite151 APT on graphite151 is CRITICAL: APT CRITICAL: 60 packages available for upgrade (2 critical updates). [20:49:36] PROBLEM - mw183 APT on mw183 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:49:53] PROBLEM - mwtask151 APT on mwtask151 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:50:40] PROBLEM - rdb151 APT on rdb151 is CRITICAL: APT CRITICAL: 66 packages available for upgrade (1 critical updates). [20:50:41] PROBLEM - graylog161 APT on graylog161 is CRITICAL: APT CRITICAL: 70 packages available for upgrade (1 critical updates). [20:52:05] PROBLEM - cp37 APT on cp37 is CRITICAL: APT CRITICAL: 66 packages available for upgrade (1 critical updates). [20:53:41] PROBLEM - swiftproxy171 APT on swiftproxy171 is CRITICAL: APT CRITICAL: 71 packages available for upgrade (2 critical updates). [20:54:11] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:54:15] PROBLEM - bots171 APT on bots171 is CRITICAL: APT CRITICAL: 83 packages available for upgrade (2 critical updates). [20:54:25] Huh [20:54:32] PROBLEM - jobchron171 APT on jobchron171 is CRITICAL: APT CRITICAL: 76 packages available for upgrade (1 critical updates). [20:54:38] What you huh'ing at [20:55:01] PROBLEM - swiftproxy161 APT on swiftproxy161 is CRITICAL: APT CRITICAL: 57 packages available for upgrade (2 critical updates). [20:55:29] PROBLEM - cloud18 APT on cloud18 is CRITICAL: APT CRITICAL: 118 packages available for upgrade (1 critical updates). [20:55:30] PROBLEM - db161 APT on db161 is CRITICAL: APT CRITICAL: 74 packages available for upgrade (1 critical updates). [20:56:08] PROBLEM - ldap171 APT on ldap171 is CRITICAL: APT CRITICAL: 66 packages available for upgrade (1 critical updates). [20:56:15] PROBLEM - mw152 APT on mw152 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:56:16] PROBLEM - mw181 APT on mw181 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:56:37] PROBLEM - db171 APT on db171 is CRITICAL: APT CRITICAL: 74 packages available for upgrade (1 critical updates). [20:56:41] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:58:26] PROBLEM - db172 APT on db172 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [20:58:34] PROBLEM - matomo151 APT on matomo151 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:59:56] PROBLEM - bast181 APT on bast181 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [21:00:39] PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:00:43] PROBLEM - os161 APT on os161 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [21:01:03] PROBLEM - mem161 APT on mem161 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [21:01:15] PROBLEM - puppet181 APT on puppet181 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:01:26] PROBLEM - bast161 APT on bast161 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [21:01:40] PROBLEM - cloud16 APT on cloud16 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [21:02:59] PROBLEM - mw163 APT on mw163 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:04:19] PROBLEM - db182 APT on db182 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [21:04:27] PROBLEM - os151 APT on os151 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [21:04:37] PROBLEM - mem151 APT on mem151 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [21:04:45] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:05:59] PROBLEM - mw154 APT on mw154 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:06:57] PROBLEM - mw151 APT on mw151 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:07:11] PROBLEM - cloud15 APT on cloud15 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [21:08:29] PROBLEM - mw164 APT on mw164 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:09:48] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:10:03] PROBLEM - mw162 APT on mw162 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:10:24] PROBLEM - mw184 APT on mw184 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:10:52] PROBLEM - mw161 APT on mw161 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:11:43] [02statichelp] 07Universal-Omega pushed 031 commit to 03main [+2/-2/±26] 13https://github.com/miraheze/statichelp/compare/afb1ea153d10...16f0ca120c6a [21:11:44] [02statichelp] 07Universal-Omega 0316f0ca1 - Auto-update Tech namespace pages 2024-09-17 21:11:28.707445 [21:11:44] PROBLEM - os162 APT on os162 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [21:41:54] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 22810MiB (10% inode=99%); [21:42:24] PROBLEM - os161 Disk Space on os161 is WARNING: DISK WARNING - free space: / 23354MiB (10% inode=99%); [21:44:24] RECOVERY - os161 Disk Space on os161 is OK: DISK OK - free space: / 24948MiB (11% inode=99%); [21:47:54] RECOVERY - os151 Disk Space on os151 is OK: DISK OK - free space: / 26961MiB (12% inode=99%); [21:48:24] PROBLEM - os161 Disk Space on os161 is WARNING: DISK WARNING - free space: / 22660MiB (10% inode=99%); [21:50:24] RECOVERY - os161 Disk Space on os161 is OK: DISK OK - free space: / 26985MiB (12% inode=99%); [22:06:27] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [22:08:13] [02puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ff62800bacec...807f28c0ecac [22:08:16] [02puppet] 07paladox 03807f28c - graylog: upgrade to 5.2.11 [22:12:39] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 27.85 ms [22:27:04] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [22:31:13] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 27.80 ms [22:32:23] [02statichelp] 07Universal-Omega pushed 031 commit to 03main [+1/-5/±12] 13https://github.com/miraheze/statichelp/compare/16f0ca120c6a...5ed024a94b88 [22:32:25] [02statichelp] 07Universal-Omega 035ed024a - Auto-update Tech namespace pages 2024-09-17 22:29:12.679314 [22:39:35] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 27.87 ms [22:43:41] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 27.86 ms [22:46:21] [02puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/807f28c0ecac...a3a2d7b3e873 [22:46:22] [02puppet] 07paladox 03a3a2d7b - nginx: increase ssl_session_cache to 300m and timeout to 6h [22:59:21] PROBLEM - reports171 Puppet on reports171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [23:10:33] PROBLEM - phorge171 Puppet on phorge171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [23:14:46] PROBLEM - mattermost1 Puppet on mattermost1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [23:19:09] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [23:23:42] [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/bdadd1ec0fec [23:23:44] [02puppet] 07paladox 03bdadd1e - nginx: set ssl_session_tickets to off [23:23:46] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [23:23:48] [02puppet] 07paladox opened pull request 03#3936: nginx: set ssl_session_tickets to off - 13https://github.com/miraheze/puppet/pull/3936 [23:23:54] [02puppet] 07coderabbitai[bot] commented on pull request 03#3936: nginx: set ssl_session_tickets to off - 13https://github.com/miraheze/puppet/pull/3936#issuecomment-2357182567 [23:38:22] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [23:38:38] RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [23:46:40] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 27.88 ms [23:47:09] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [23:51:14] [02puppet] 07paladox closed pull request 03#3936: nginx: set ssl_session_tickets to off - 13https://github.com/miraheze/puppet/pull/3936 [23:51:15] [02puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a3a2d7b3e873...16c8f28602a7 [23:51:17] [02puppet] 07paladox 0316c8f28 - nginx: set ssl_session_tickets to off (#3936) [23:51:20] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [23:51:21] [02puppet] 07paladox deleted branch 03paladox-patch-1 [23:57:05] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [23:57:21] RECOVERY - reports171 Puppet on reports171 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures