[00:01:15] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/4d9e9a63d184048158169bfb34d1d4331249928e [00:01:15] 02statichelp/03main 07WikiTideBot 034d9e9a6 Bot: Auto-update Tech namespace pages 2026-02-20 00:01:12 [00:07:40] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php CreateWiki:SetContainersAccess --wiki=elysiamcwiki (END - exit=0) [00:07:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:22:42] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 74464MiB (16.4% inode=100%); [00:27:18] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/039e0b417e3dc8158d719ce27defdc285912a26a [00:27:18] 02puppet/03main 07CosmicAlpha 03039e0b4 ImageMagick: support trixie [00:30:23] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 73719MiB (16.3% inode=100%); [00:48:11] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 73808MiB (16.3% inode=100%); [01:21:52] [02puppet] 07Universal-Omega created 03cgroup-trixie (+1 new commit) 13https://github.com/miraheze/puppet/commit/838b4ff5e9c2 [01:21:52] 02puppet/03cgroup-trixie 07CosmicAlpha 03838b4ff cgroup: support trixie [01:21:58] [02puppet] 07Universal-Omega opened pull request #4772: cgroup: support trixie (03main...03cgroup-trixie) 13https://github.com/miraheze/puppet/pull/4772 [01:22:08] [02puppet] 07coderabbitai[bot] commented on pull request #4772:
[…] 13https://github.com/miraheze/puppet/pull/4772#issuecomment-3931048146 [01:22:46] [02puppet] 07Universal-Omega pushed 1 new commit to 03cgroup-trixie 13https://github.com/miraheze/puppet/commit/53157e4e5a8e8785ff76a6f9906780743af3f149 [01:22:46] 02puppet/03cgroup-trixie 07CosmicAlpha 0353157e4 Add package [01:27:22] [02puppet] 07Universal-Omega pushed 1 new commit to 03cgroup-trixie 13https://github.com/miraheze/puppet/commit/98cfb55c1fcbb7fd052e706b3c4c8185e765bc7c [01:27:22] 02puppet/03cgroup-trixie 07CosmicAlpha 0398cfb55 Auto remove [01:30:20] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/39d6b465eb9e7dfefb61b8cfa9de46f0690ae5c6 [01:30:20] 02ssl/03main 07WikiTideBot 0339d6b46 Bot: Auto-update domain lists [01:34:42] [02puppet] 07Universal-Omega merged pull request #4772: cgroup: support trixie (03main...03cgroup-trixie) 13https://github.com/miraheze/puppet/pull/4772 [01:34:42] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/d5cf36506d11e13544479baf32a789193a091201 [01:34:43] 02puppet/03main 07CosmicAlpha 03d5cf365 cgroup: support trixie [01:34:45] [02puppet] 07Universal-Omega 04deleted 03cgroup-trixie at 0398cfb55 13https://api.github.com/repos/miraheze/puppet/commit/98cfb55 [01:36:47] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [01:37:51] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/eb1f3a49517494269499aa7f8d517ea3b0e66d69 [01:37:51] 02puppet/03main 07CosmicAlpha 03eb1f3a4 Absent SYSTEMD_CGROUP_ENABLE_LEGACY_FORCE [01:39:31] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/ac45c1c7677bf81c0812584e1e6238409ab5c4fc [01:39:31] 02puppet/03main 07CosmicAlpha 03ac45c1c Remove now unnecessary grub::bootparam [02:50:39] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.55, 18.06, 13.07 [02:54:39] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 16.83, 19.21, 14.84 [03:20:19] [02puppet] 07Universal-Omega pushed 1 new commit to 03polkit 13https://github.com/miraheze/puppet/commit/eb95d46e33efa6da4e650f497953882fe95b5073 [03:20:19] 02puppet/03polkit 07CosmicAlpha 03eb95d46 Only on trixie [03:21:25] [02puppet] 07Universal-Omega merged pull request #4770: Add shellbox polkit rule for cgroup v2 (03main...03polkit) 13https://github.com/miraheze/puppet/pull/4770 [03:21:25] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/d7e52962081d688468109d6fae6ea0dc9881570d [03:21:27] 02puppet/03main 07CosmicAlpha 03d7e5296 Add shellbox polkit rule for cgroup v2… [03:21:28] [02puppet] 07Universal-Omega 04deleted 03polkit at 03eb95d46 13https://api.github.com/repos/miraheze/puppet/commit/eb95d46 [03:57:54] [02puppet] 07Universal-Omega created 03remove-php8.2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/805f56fb00de [03:57:54] 02puppet/03remove-php8.2 07CosmicAlpha 03805f56f Remove PHP 8.2 scripts if on PHP 8.4 [03:57:59] [02puppet] 07Universal-Omega opened pull request #4773: Remove PHP 8.2 scripts if on PHP 8.4 (03main...03remove-php8.2) 13https://github.com/miraheze/puppet/pull/4773 [03:58:10] [02puppet] 07coderabbitai[bot] commented on pull request #4773:
[…] 13https://github.com/miraheze/puppet/pull/4773#issuecomment-3931471354 [03:58:32] [02puppet] 07github-actions[bot] pushed 1 new commit to 03remove-php8.2 13https://github.com/miraheze/puppet/commit/ef9cf3929ee23e63fb755999c4f8d9b90e640f1b [03:58:32] 02puppet/03remove-php8.2 07github-actions 03ef9cf39 CI: lint puppet code to standards… [03:58:51] [02puppet] 07Universal-Omega merged pull request #4773: Remove PHP 8.2 scripts if on PHP 8.4 (03main...03remove-php8.2) 13https://github.com/miraheze/puppet/pull/4773 [03:58:52] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/86914291fd530d88d712340296675ac85501b839 [03:58:53] 02puppet/03main 07CosmicAlpha 038691429 Remove PHP 8.2 scripts if on PHP 8.4 [03:58:55] [02puppet] 07Universal-Omega 04deleted 03remove-php8.2 at 03ef9cf39 13https://api.github.com/repos/miraheze/puppet/commit/ef9cf39 [04:01:31] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/f00e943ae87e272ef3bd2ecbc9b7e6b930b28ec9 [04:01:31] 02puppet/03main 07CosmicAlpha 03f00e943 Fix path [05:12:46] I think logsalmsg has stopped working. [05:13:32] [1/6] There was another hiccup where the following failed on mwtask181 but succeeded in mwtask171. It may have something to do with caching. [05:13:32] [2/6] ``` [05:13:32] [3/6] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/extensions/TitleKey/maintenance/rebuildTitleKeys.php --wiki=thecrisiswarswiki [05:13:33] [4/6] Type 'Y' to confirm: Y [05:13:33] [5/6] The "TitleKey" extension must be installed for this script to run. Please enable it and then try again. [05:13:33] [6/6] ``` [05:28:35] !log test [05:28:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:28:46] it works [05:29:15] odd [05:29:25] Wasn't sending anything here when I ran mwscript. Maybe it recovered now. [05:31:56] irclogserverbot was down [05:32:02] so server -> IRC wasnt working [05:32:22] !log [universalomega@bots171] Restart irclogserverbot [05:32:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:32:28] There we go [05:34:30] Btw I'm starting to suspect that the `mwscript` option in ManageWikiExtensions doesn't actually do anything when a non-class option is supplied. It has been a persistent issue with TitleKey at least. [05:35:09] It should be accepting anything run.php accepts since its just a wrapper around that. [07:03:45] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [07:13:52] PROBLEM - llm191 Puppet on llm191 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[openwebui] [07:30:20] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/0d9b0a528f2ba019afcf0a7e20420913078cb67f [07:30:20] 02ssl/03main 07WikiTideBot 030d9b0a5 Bot: Auto-update domain lists [07:31:45] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [07:34:39] The script seems to be run sometimes. Works first try on another (very samll) wiki. [07:39:00] [02landing] 07renovate[bot] pushed 1 new commit to 03main 13https://github.com/miraheze/landing/commit/de48770d534e55d8f78399618fba6d19142d022c [07:39:00] 02landing/03main 07renovate[bot] 03de48770 Update dependency glob to ^13.0.6 [07:39:00] [02landing] 07renovate[bot] 04deleted 03renovate/all-deps at 03de48770 13https://api.github.com/repos/miraheze/landing/commit/de48770 [07:39:45] miraheze/landing - renovate[bot] the build passed. [07:41:52] RECOVERY - llm191 Puppet on llm191 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:03:45] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [11:31:45] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [12:05:56] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 50.78, 44.73, 22.24 [12:09:51] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 3.80, 22.32, 18.10 [12:11:11] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/openvox-keyring.gpg] [12:11:48] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 3.00, 15.93, 16.27 [12:35:22] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:44:24] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 23.27, 19.98, 15.51 [12:44:24] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 22.77, 17.62, 13.02 [12:44:33] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 23.07, 18.75, 14.34 [12:44:37] PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 23.73, 19.54, 14.28 [12:45:41] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.09, 20.07, 15.50 [12:46:24] RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 16.39, 16.96, 13.34 [12:46:33] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 19.06, 18.56, 14.81 [12:46:37] RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 14.27, 17.57, 14.21 [12:47:41] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.52, 17.85, 15.25 [12:48:24] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 11.23, 17.63, 15.76 [13:26:04] wtf? [14:33:45] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [14:47:10] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 20.85, 32.60, 19.12 [14:51:04] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 5.97, 18.33, 16.26 [14:56:52] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 3 backends are down. mw152 mw183 mw201 [14:58:51] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [15:01:45] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [16:03:45] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [16:37:30] [02mw-config] 07YTFGolf opened pull request #6322: T14515: add free art licence footer onto thechurchofthestatuewiki (07miraheze:03main...07YTFGolf:03patch-T14515) 13https://github.com/miraheze/mw-config/pull/6322 [16:42:15] miraheze/mw-config - YTFGolf the build passed. [16:42:53] miraheze/mw-config - YTFGolf the build has errored. [17:00:23] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/6d3be5a1b82c052fb6ccdb51c848a63516dbb367 [17:00:24] 02ssl/03main 07WikiTideBot 036d3be5a Bot: Auto-update domain lists [17:01:45] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [17:02:13] [02mw-config] 07pskyechology approved pull request #6322 13https://github.com/miraheze/mw-config/pull/6322#pullrequestreview-3833162844 [17:02:41] [02mw-config] 07pskyechology merged 07YTFGolf's pull request #6322: T14515: add free art licence footer onto thechurchofthestatuewiki (07miraheze:03main...07YTFGolf:03patch-T14515) 13https://github.com/miraheze/mw-config/pull/6322 [17:02:41] [02mw-config] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/ffbc862a641e1a1909e50b6c7f070aab5071d016 [17:02:41] 02mw-config/03main 07YTFGolf 03ffbc862 T14515: add free art licence footer onto thechurchofthestatuewiki (#6322)… [17:02:49] !log [skye@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [17:02:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:03:21] !log [skye@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 31s [17:03:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:03:54] miraheze/mw-config - pskyechology the build passed. [17:32:02] [02mw-config] 07pskyechology requested changes on pull request #6309 13https://github.com/miraheze/mw-config/pull/6309#pullrequestreview-3833236484 [17:32:04] [02mw-config] 07pskyechology left a file comment in pull request #6309 0365e467e: `beta` is not a valid option, see: https://github.com/wikimedia/mediawiki-skins-MinervaNeue/blob/5fa331e/skin.json#L107-L112 13https://github.com/miraheze/mw-config/pull/6309#discussion_r2834237735 [17:32:06] [02mw-config] 07pskyechology left a file comment in pull request #6309 0365e467e: Ditto per [R3739](https://github.com/miraheze/mw-config/pull/6309/changes#r2834237735) 13https://github.com/miraheze/mw-config/pull/6309#discussion_r2834237973 [17:37:15] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=alerawiki --new=letariswiki --user=Reception123 (END - exit=0) [17:37:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:41:35] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/maintenance/deleteBatch.php --wiki=ucroniaswiki /home/reception/gentemzx.txt --u=Miraheze maintenance script --r=T14864 (START) [17:41:36] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/maintenance/deleteBatch.php --wiki=ucroniaswiki /home/reception/gentemzx.txt --u=Miraheze maintenance script --r=T14864 (END - exit=256) [17:41:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:41:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:42:00] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/maintenance/deleteBatch.php --wiki=ucroniaswiki /home/reception/gentemzx.txt --u=Miraheze_maintenance_script --r=T14864 (START) [17:42:02] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/maintenance/deleteBatch.php --wiki=ucroniaswiki /home/reception/gentemzx.txt --u=Miraheze_maintenance_script --r=T14864 (END - exit=256) [17:42:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:42:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:42:35] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php CentralAuth:createLocalAccount --wiki=ucroniaswiki Miraheze maintenance script (END - exit=0) [17:42:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:42:44] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php CentralAuth:createLocalAccount --wiki=ucroniaswiki Miraheze_maintenance_script (END - exit=0) [17:42:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:43:05] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php CentralAuth:createLocalAccount --wiki=ucroniaswiki Miraheze maintenance script (END - exit=0) [17:43:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:43:09] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/maintenance/deleteBatch.php --wiki=ucroniaswiki /home/reception/gentemzx.txt --u=Miraheze_maintenance_script --r=T14864 (START) [17:43:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:52:32] miraheze/mw-config - YTFGolf the build passed. [17:52:45] [02mw-config] 07YTFGolf left a file comment in pull request #6309 0365e467e: Slightly embarassing I missed that lol 13https://github.com/miraheze/mw-config/pull/6309#discussion_r2834417662 [17:55:14] miraheze/mw-config - YTFGolf the build passed. [17:57:01] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/maintenance/importDump.php --wiki=savagepediawiki /home/reception/savagepediawiki.xml (START) [17:57:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:58:53] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/maintenance/importDump.php --wiki=savagepediawiki /home/reception/savagepediawiki.xml (END - exit=2) [17:58:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:58:58] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/maintenance/importDump.php --wiki=savagepediawiki /home/reception/savagepediawiki.xml (START) [17:59:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:01:21] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/maintenance/importDump.php --wiki=savagepediawiki /home/reception/savagepediawiki.xml (END - exit=0) [18:01:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:04:23] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 45231MiB (10.0% inode=100%); [18:14:42] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 45366MiB (10.0% inode=100%); [18:22:11] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 45256MiB (10.0% inode=100%); [18:42:56] [02TSPortal] 07Universal-Omega created 03fix-exception (+1 new commit) 13https://github.com/miraheze/TSPortal/commit/3e18bedfa017 [18:42:56] 02TSPortal/03fix-exception 07CosmicAlpha 033e18bed Fix exception in DPAPolicy [18:43:11] [02TSPortal] 07Universal-Omega opened pull request #57: Fix exception in DPAPolicy (03main...03fix-exception) 13https://github.com/miraheze/TSPortal/pull/57 [18:43:22] [02TSPortal] 07coderabbitai[bot] commented on pull request #57:
[…] 13https://github.com/miraheze/TSPortal/pull/57#issuecomment-3936505810 [18:53:48] [02TSPortal] 07Universal-Omega merged pull request #57: Fix exception in DPAPolicy (03main...03fix-exception) 13https://github.com/miraheze/TSPortal/pull/57 [18:53:48] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/TSPortal/commit/dac66adf4b8a8aba8a705e0ca4ffdcd4bb633487 [18:53:48] 02TSPortal/03main 07CosmicAlpha 03dac66ad Fix exception in DPAPolicy [18:53:50] [02TSPortal] 07Universal-Omega 04deleted 03fix-exception at 033e18bed 13https://api.github.com/repos/miraheze/TSPortal/commit/3e18bed [18:55:00] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/TSPortal/commit/7b44f02feed763bd3c5f3082baa741ff820da42d [18:55:00] 02TSPortal/03main 07CosmicAlpha 037b44f02 Bump to version 28 [19:02:33] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/TSPortal/commit/e6f937be5fe8b9dcb1ee23e7478dc03e7d7fc278 [19:02:33] 02TSPortal/03main 07CosmicAlpha 03e6f937b Add CHANGELOG for v28 [19:04:06] [02TSPortal] 07Universal-Omega released Version 28 13https://github.com/miraheze/TSPortal/releases/tag/v28 [19:04:06] [02TSPortal] 07Universal-Omega tagged 03v28 at 03main 13https://github.com/miraheze/TSPortal/compare/v28 [19:05:21] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/11687e0975e999c4326014a8b9ca191fe15065bd [19:05:21] 02puppet/03main 07CosmicAlpha 0311687e0 Update TSPortal to v28 [19:26:32] PROBLEM - bots171 APT on bots171 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (1 critical updates). [19:27:20] PROBLEM - swiftobject171 APT on swiftobject171 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [19:33:17] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [19:33:52] PROBLEM - swiftobject161 APT on swiftobject161 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [19:33:53] PROBLEM - swiftobject151 APT on swiftobject151 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [19:38:30] PROBLEM - swiftproxy161 APT on swiftproxy161 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [19:42:29] [02TSPortal] 07Universal-Omega created 03restore-listener (+1 new commit) 13https://github.com/miraheze/TSPortal/commit/dbaf2f8d4670 [19:42:29] 02TSPortal/03restore-listener 07CosmicAlpha 03dbaf2f8 Restore webhook event listener functionality [19:42:38] [02TSPortal] 07Universal-Omega opened pull request #58: Restore webhook event listener functionality (03main...03restore-listener) 13https://github.com/miraheze/TSPortal/pull/58 [19:42:49] PROBLEM - swiftproxy171 APT on swiftproxy171 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [19:42:50] [02TSPortal] 07coderabbitai[bot] commented on pull request #58:
[…] 13https://github.com/miraheze/TSPortal/pull/58#issuecomment-3936756020 [19:43:11] PROBLEM - swiftobject191 APT on swiftobject191 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [19:44:35] PROBLEM - swiftobject201 APT on swiftobject201 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [19:45:14] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03restore-listener 13https://github.com/miraheze/TSPortal/commit/20f313027da0abe6df458ce5ccbba7c5280172ae [19:45:14] 02TSPortal/03restore-listener 07CosmicAlpha 0320f3130 Bump [19:46:23] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03restore-listener 13https://github.com/miraheze/TSPortal/commit/637f43bf3bfe23b83103e24b9079379a916aebb9 [19:46:23] 02TSPortal/03restore-listener 07CosmicAlpha 03637f43b Update [19:46:32] PROBLEM - swiftac171 APT on swiftac171 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [19:46:44] PROBLEM - swiftobject181 APT on swiftobject181 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [19:46:47] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03restore-listener 13https://github.com/miraheze/TSPortal/commit/c7cd5a96ea31672cad73bcfaa04be053134bcfca [19:46:47] 02TSPortal/03restore-listener 07CosmicAlpha 03c7cd5a9 Update [19:49:01] [02TSPortal] 07Universal-Omega merged pull request #58: Restore webhook event listener functionality (03main...03restore-listener) 13https://github.com/miraheze/TSPortal/pull/58 [19:49:01] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/TSPortal/commit/cc087fe3c3cbd8c6e4015fde8d611ac275cae48a [19:49:02] 02TSPortal/03main 07CosmicAlpha 03cc087fe Restore webhook event listener functionality [19:49:04] [02TSPortal] 07Universal-Omega 04deleted 03restore-listener at 03c7cd5a9 13https://api.github.com/repos/miraheze/TSPortal/commit/c7cd5a9 [19:49:55] [02TSPortal] 07Universal-Omega released Version 29 13https://github.com/miraheze/TSPortal/releases/tag/v29 [19:49:55] [02TSPortal] 07Universal-Omega tagged 03v29 at 03main 13https://github.com/miraheze/TSPortal/compare/v29 [19:51:16] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/fe65fe5b3f774b2c64e17770eaa2b11e65f01308 [19:51:16] 02puppet/03main 07CosmicAlpha 03fe65fe5 Update TSPortal to v29 [19:52:38] PROBLEM - reports171 Puppet on reports171 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 21 minutes ago with 0 failures [20:10:08] [02puppet] 07Universal-Omega pushed 1 new commit to 03patch-mwpatches 13https://github.com/miraheze/puppet/commit/3a2e381ea0d2a6b465c73bc5a7d667293f035c85 [20:10:08] 02puppet/03patch-mwpatches 07CosmicAlpha 033a2e381 Merge branch 'main' into patch-mwpatches [20:10:38] RECOVERY - reports171 Puppet on reports171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:11:36] miraheze/puppet - Universal-Omega the build passed. [20:33:34] [02mediawiki-repos] 07Universal-Omega created 03limit.sh-patch (+1 new commit) 13https://github.com/miraheze/mediawiki-repos/commit/4109d39dc9ff [20:33:34] 02mediawiki-repos/03limit.sh-patch 07CosmicAlpha 034109d39 Add patch file for limit.sh [20:33:54] [02mediawiki-repos] 07Universal-Omega opened pull request #117: Add patch file for limit.sh (03main...03limit.sh-patch) 13https://github.com/miraheze/mediawiki-repos/pull/117 [20:34:05] [02mediawiki-repos] 07coderabbitai[bot] commented on pull request #117:
[…] 13https://github.com/miraheze/mediawiki-repos/pull/117#issuecomment-3937034557 [20:35:54] [02mediawiki-repos] 07Universal-Omega pushed 1 new commit to 03limit.sh-patch 13https://github.com/miraheze/mediawiki-repos/commit/55673647495f86f179eaa301a9f75108c256dadb [20:35:55] 02mediawiki-repos/03limit.sh-patch 07CosmicAlpha 035567364 Add [20:57:01] PROBLEM - mwtask181 Puppet on mwtask181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 23 minutes ago with 0 failures [20:59:04] [02mediawiki-repos] 07Universal-Omega merged pull request #117: Add patch file for limit.sh (03main...03limit.sh-patch) 13https://github.com/miraheze/mediawiki-repos/pull/117 [20:59:04] [02mediawiki-repos] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mediawiki-repos/commit/4757a7b8b6bec244a603f383f4afcc7118dcec30 [20:59:04] 02mediawiki-repos/03main 07CosmicAlpha 034757a7b Add patch file for limit.sh… [20:59:06] [02mediawiki-repos] 07Universal-Omega 04deleted 03limit.sh-patch at 035567364 13https://api.github.com/repos/miraheze/mediawiki-repos/commit/5567364 [21:03:45] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [21:04:05] [02puppet] 07Universal-Omega pushed 1 new commit to 03patch-mwpatches 13https://github.com/miraheze/puppet/commit/a512ac0108089cc59039a84bc67b733db11a630f [21:04:05] 02puppet/03patch-mwpatches 07CosmicAlpha 03a512ac0 Fix [21:05:37] miraheze/puppet - Universal-Omega the build passed. [21:13:51] [02puppet] 07Universal-Omega pushed 1 new commit to 03patch-mwpatches 13https://github.com/miraheze/puppet/commit/3fd67ba9eed8049d8b8776e5c1f4213e3f2cb560 [21:13:51] 02puppet/03patch-mwpatches 07CosmicAlpha 033fd67ba Fix as it's a dictionary [21:15:26] miraheze/puppet - Universal-Omega the build has errored. [21:18:53] [02puppet] 07Universal-Omega pushed 1 new commit to 03patch-mwpatches 13https://github.com/miraheze/puppet/commit/db97f687dd114b4cad83a7fe777d163ecaac94d1 [21:18:53] 02puppet/03patch-mwpatches 07CosmicAlpha 03db97f68 One more fix [21:20:09] miraheze/puppet - Universal-Omega the build has errored. [21:31:45] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [21:39:48] [02puppet] 07Universal-Omega pushed 1 new commit to 03patch-mwpatches 13https://github.com/miraheze/puppet/commit/f3a07ce205a4a75043c97fd7b9725d81fbe1a590 [21:39:48] 02puppet/03patch-mwpatches 07CosmicAlpha 03f3a07ce Fix support for versions filter [21:41:12] miraheze/puppet - Universal-Omega the build has errored. [21:42:13] [02puppet] 07Universal-Omega pushed 1 new commit to 03patch-mwpatches 13https://github.com/miraheze/puppet/commit/e6298703608d8b82ca1d89ecec9bb5c8844263a5 [21:42:13] 02puppet/03patch-mwpatches 07CosmicAlpha 03e629870 Fix [21:43:45] miraheze/puppet - Universal-Omega the build passed. [21:51:14] [02puppet] 07Universal-Omega approved pull request #3885: I have finished this up to a point where it at least works as expected, at least seems to for me when testing on beta. Future work can be done on this later though. Thanks for doing this! I am merging so that we can use this for patches now. 13https://github.com/miraheze/puppet/pull/3885#pullrequestreview-3834353717 [21:51:32] [02puppet] 07Universal-Omega merged 07The-Voidwalker's pull request #3885: mwdeploy.py: add support for local patches (03main...03patch-mwpatches) 13https://github.com/miraheze/puppet/pull/3885 [21:51:34] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/c895d5b85ca911bbf0463d5b11ef76819a233ca2 [21:51:36] [02puppet] 07Universal-Omega 04deleted 03patch-mwpatches at 03e629870 13https://api.github.com/repos/miraheze/puppet/commit/e629870 [21:51:38] 02puppet/03main 07The-Voidwalker 03c895d5b mwdeploy.py: add support for local patches (#3885)… [21:53:12] miraheze/puppet - Universal-Omega the build passed. [21:57:26] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [22:14:45] !log [universalomega@puppet181] Upgraded packages on swiftproxy161: python3-pil [22:14:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:15:00] !log [universalomega@puppet181] Upgraded packages on test151: python3-pil [22:15:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:15:02] RECOVERY - test151 APT on test151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:15:16] !log [universalomega@puppet181] Upgraded packages on swiftproxy171: python3-pil [22:15:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:15:30] !log [universalomega@puppet181] Upgraded packages on swiftobject171: python3-pil [22:15:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:15:44] !log [universalomega@puppet181] Upgraded packages on swiftobject191: python3-pil [22:15:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:16:00] !log [universalomega@puppet181] Upgraded packages on swiftobject161: python3-pil [22:16:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:16:11] RECOVERY - swiftproxy161 APT on swiftproxy161 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:16:15] !log [universalomega@puppet181] Upgraded packages on swiftac171: python3-pil [22:16:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:16:17] RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:16:25] RECOVERY - swiftobject161 APT on swiftobject161 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:16:25] RECOVERY - swiftobject171 APT on swiftobject171 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:16:30] !log [universalomega@puppet181] Upgraded packages on swiftobject181: python3-pil [22:16:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:16:39] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.96, 18.20, 11.43 [22:16:43] RECOVERY - swiftproxy171 APT on swiftproxy171 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:16:44] RECOVERY - swiftobject181 APT on swiftobject181 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:16:46] !log [universalomega@puppet181] Upgraded packages on swiftobject151: python3-pil [22:16:49] RECOVERY - swiftobject191 APT on swiftobject191 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:16:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:17:02] !log [universalomega@puppet181] Upgraded packages on swiftobject201: python3-pil [22:17:03] RECOVERY - swiftobject201 APT on swiftobject201 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:17:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:17:07] RECOVERY - swiftobject151 APT on swiftobject151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:17:41] !log [universalomega@puppet181] Upgraded packages on bots171: python3-pil [22:17:41] RECOVERY - bots171 APT on bots171 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [22:17:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:18:39] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 16.23, 17.61, 12.06 [22:19:24] [02mediawiki-repos] 07SomeMWDev created 03add-commentstreams-patch (+1 new commit) 13https://github.com/miraheze/mediawiki-repos/commit/99636e077009 [22:19:24] 02mediawiki-repos/03add-commentstreams-patch 07SomeRandomDeveloper 0399636e0 Add CommentStreams patch for T14969 [22:20:00] [02mediawiki-repos] 07SomeMWDev opened pull request #118: Add CommentStreams patch for T14969 (03main...03add-commentstreams-patch) 13https://github.com/miraheze/mediawiki-repos/pull/118 [22:20:11] [02mediawiki-repos] 07coderabbitai[bot] commented on pull request #118:
[…] 13https://github.com/miraheze/mediawiki-repos/pull/118#issuecomment-3937427265 [22:21:05] [02mediawiki-repos] 07SomeMWDev merged pull request #118: Add CommentStreams patch for T14969 (03main...03add-commentstreams-patch) 13https://github.com/miraheze/mediawiki-repos/pull/118 [22:21:05] [02mediawiki-repos] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mediawiki-repos/commit/a77c0a1cbe4978d2365ee6466c8d2df4362b33f5 [22:21:05] 02mediawiki-repos/03main 07SomeRandomDeveloper 03a77c0a1 Add CommentStreams patch for T14969 (#118) [22:21:06] [02mediawiki-repos] 07SomeMWDev 04deleted 03add-commentstreams-patch at 0399636e0 13https://api.github.com/repos/miraheze/mediawiki-repos/commit/99636e0 [22:24:35] !log [somerandomdeveloper@test151] starting deploy of {'versions': '1.45', 'upgrade_extensions': 'CommentStreams'} to test151 [22:24:36] !log [somerandomdeveloper@test151] finished deploy of {'versions': '1.45', 'upgrade_extensions': 'CommentStreams'} to test151 - SUCCESS in 0s [22:24:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:24:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:24:56] !log [somerandomdeveloper@test151] starting deploy of {'force_upgrade': True, 'versions': '1.45', 'upgrade_extensions': 'CommentStreams'} to test151 [22:24:57] !log [somerandomdeveloper@test151] finished deploy of {'force_upgrade': True, 'versions': '1.45', 'upgrade_extensions': 'CommentStreams'} to test151 - SUCCESS in 0s [22:24:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:24:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:29:20] !log [somerandomdeveloper@mwtask181] starting deploy of {'force_upgrade': True, 'versions': '1.45', 'upgrade_extensions': 'CommentStreams'} to all [22:29:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:29:52] !log [somerandomdeveloper@mwtask181] finished deploy of {'force_upgrade': True, 'versions': '1.45', 'upgrade_extensions': 'CommentStreams'} to all - SUCCESS in 32s [22:29:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:30:39] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.09, 20.71, 16.15 [22:36:39] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 17.94, 19.89, 17.41 [22:41:07] !log [universalomega@mwtask181] starting deploy of {'upgrade_vendor': True, 'versions': '1.45'} to all [22:41:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:41:26] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1771623650000&orgId=1&to=1771627286679 [22:42:15] !log [universalomega@mwtask181] finished deploy of {'upgrade_vendor': True, 'versions': '1.45'} to all - SUCCESS in 68s [22:42:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:47:23] !log disable "Review Status" messages in the Coderabbit settings for the github org [22:47:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:55:50] [02mediawiki-repos] 07SomeMWDev created 03lakeus-branch (+1 new commit) 13https://github.com/miraheze/mediawiki-repos/commit/f3cfc38b8546 [22:55:50] 02mediawiki-repos/03lakeus-branch 07SomeRandomDeveloper 03f3cfc38 Switch Lakeus branch to REL1_45 [22:56:03] [02mediawiki-repos] 07SomeMWDev opened pull request #119: Switch Lakeus branch to REL1_45 (03main...03lakeus-branch) 13https://github.com/miraheze/mediawiki-repos/pull/119 [22:57:01] [02mediawiki-repos] 07SomeMWDev merged pull request #119: Switch Lakeus branch to REL1_45 (03main...03lakeus-branch) 13https://github.com/miraheze/mediawiki-repos/pull/119 [22:57:03] [02mediawiki-repos] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mediawiki-repos/commit/070d9dae9d76f2ca5cd9a4184a3fb0dcb7ec326c [22:57:05] [02mediawiki-repos] 07SomeMWDev 04deleted 03lakeus-branch at 03f3cfc38 13https://api.github.com/repos/miraheze/mediawiki-repos/commit/f3cfc38 [22:57:07] 02mediawiki-repos/03main 07SomeRandomDeveloper 03070d9da Switch Lakeus branch to REL1_45 (#119) [22:59:29] !log remove Lakeus staging folders on test151 and mwtask181 and run puppet [22:59:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:05:43] PROBLEM - mwtask161 HTTPS on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [23:05:44] PROBLEM - mwtask161 MediaWiki Rendering on mwtask161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:05:57] PROBLEM - mwtask161 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [23:06:08] PROBLEM - mwtask161 jobrunner.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [23:06:11] PROBLEM - mwtask161 videoscaler.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [23:06:29] PROBLEM - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [23:06:34] PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [23:06:38] PROBLEM - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [23:06:50] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:06:51] PROBLEM - mwtask181 videoscaler.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [23:07:59] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.232 second response time [23:08:23] RECOVERY - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.007 second response time [23:08:28] RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4292 bytes in 0.178 second response time [23:08:36] RECOVERY - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.003 second response time [23:08:37] RECOVERY - mwtask161 HTTPS on mwtask161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4292 bytes in 2.519 second response time [23:08:44] RECOVERY - mwtask161 MediaWiki Rendering on mwtask161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.896 second response time [23:08:49] RECOVERY - mwtask181 videoscaler.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.003 second response time [23:09:09] RECOVERY - mwtask161 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 1.400 second response time [23:09:13] RECOVERY - mwtask161 jobrunner.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 169 bytes in 0.060 second response time [23:09:16] RECOVERY - mwtask161 videoscaler.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 170 bytes in 0.629 second response time [23:11:26] [Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?from=1771625270000&orgId=1&to=1771628870000 [23:13:27] was this chrs [23:20:49] coughing baby vs thermonuclear infobox [23:22:14] [02mw-config] 07YTFGolf opened pull request #6323: T15003: Set 'vector-theme' preference to 'os' on lazerpigeonwiki (07miraheze:03main...07YTFGolf:03patch-T15003) 13https://github.com/miraheze/mw-config/pull/6323 [23:23:25] miraheze/mw-config - YTFGolf the build passed. [23:25:35] !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php Cargo:cargoRecreateData --wiki=frackinuniversewiki --table troubleshoot_research_node (END - exit=0) [23:25:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:27:20] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1771626410000&orgId=1&to=1771630040010 [23:29:28] [02MirahezeMagic] 07BlankEclair created 03T14033 (+1 new commit) 13https://github.com/miraheze/MirahezeMagic/commit/6e7b08356b10 [23:29:28] 02MirahezeMagic/03T14033 07Claire Elaina 036e7b083 T14033: Show Miraheze-specific message for Special:EmailUser… [23:29:54] [02MirahezeMagic] 07BlankEclair opened pull request #640: T14033: Show Miraheze-specific message for Special:EmailUser (03main...03T14033) 13https://github.com/miraheze/MirahezeMagic/pull/640 [23:33:21] miraheze/MirahezeMagic - BlankEclair the build has errored. [23:42:20] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1771626410000&orgId=1&to=1771630940013[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1771627130000&orgId=1&to=1771630940013 [23:45:53] [02MirahezeMagic] 07BlankEclair pushed 1 new commit to 03T14033 13https://github.com/miraheze/MirahezeMagic/commit/dbc48147d8d4a9ba53779eaf677281a9e27a99b4 [23:45:53] 02MirahezeMagic/03T14033 07Claire Elaina 03dbc4814 Fix phan errors [23:46:55] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 28.83, 23.01, 15.90 [23:48:53] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.35, 22.25, 16.51 [23:49:22] miraheze/MirahezeMagic - BlankEclair the build has errored. [23:50:45] !log [petramagna@test151] starting deploy of {'config': True} to test151 [23:50:46] !log [petramagna@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [23:50:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:50:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:50:50] miraheze/mw-config - lihaohong6 the build passed. [23:50:50] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.57, 23.52, 17.64 [23:52:20] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1771626410000&orgId=1&to=1771631540014[Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?from=1771627730000&orgId=1&to=1771631330000 [23:53:58] [02MirahezeMagic] 07BlankEclair pushed 1 new commit to 03T14033 13https://github.com/miraheze/MirahezeMagic/commit/3e2e16b641acf25d33d689951bd2bce74f9ffdda [23:53:58] 02MirahezeMagic/03T14033 07Claire Elaina 033e2e16b Fix more phan errors [23:54:45] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.10, 23.60, 19.10 [23:55:27] [02MirahezeMagic] 07github-actions[bot] pushed 1 new commit to 03T14033 13https://github.com/miraheze/MirahezeMagic/commit/c6254857846eed1bee30537ab704ff4b68f0b45d [23:55:27] 02MirahezeMagic/03T14033 07github-actions 03c625485 CI: lint code to MediaWiki standards… [23:57:33] miraheze/MirahezeMagic - BlankEclair the build has errored.