[00:00:04] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:00:18] !log [universalomega@puppet181] Upgraded packages on reports171: pdns-recursor [00:00:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000018-4f9a7ca2 [00:00:26] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/399378f6c6358818da02e7408d3c32e143ae1844 [00:00:26] 02ssl/03main 07WikiTideBot 03399378f Bot: Auto-update domain lists [00:00:33] !log [universalomega@puppet181] Upgraded packages on prometheus151: pdns-recursor [00:00:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000033-c6633c4d [00:00:49] !log [universalomega@puppet181] Upgraded packages on puppet181: pdns-recursor [00:00:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000049-411fcff7 [00:00:54] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/7dcdd609217a91d2dd1e2466ab763b2dc51cfda0 [00:00:54] 02statichelp/03main 07WikiTideBot 037dcdd60 Bot: Auto-update Tech namespace pages 2026-11-15 00:00:51 [00:00:54] RECOVERY - mw213 APT on mw213 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:00:58] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:01:07] !log [universalomega@puppet181] Upgraded packages on os161: pdns-recursor [00:01:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000107-a427512e [00:01:10] RECOVERY - mw211 APT on mw211 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:01:25] RECOVERY - reports171 APT on reports171 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:01:33] !log [universalomega@puppet181] Upgraded packages on mwtask181: pdns-recursor [00:01:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000133-e25c0cf3 [00:01:56] !log [universalomega@puppet181] Upgraded packages on mwtask171: pdns-recursor [00:01:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000156-8f10562a [00:01:58] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:02:09] RECOVERY - puppet181 APT on puppet181 is OK: APT OK: 7 packages available for upgrade (0 critical updates). [00:02:11] !log [universalomega@puppet181] Upgraded packages on phorge171: pdns-recursor [00:02:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000212-ae73ec08 [00:02:27] !log [universalomega@puppet181] Upgraded packages on rdb191: pdns-recursor [00:02:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000227-4bedd281 [00:02:42] RECOVERY - os161 APT on os161 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:02:44] !log [universalomega@puppet181] Upgraded packages on os201: pdns-recursor [00:02:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000244-463f5852 [00:03:01] !log [universalomega@puppet181] Upgraded packages on os202: pdns-recursor [00:03:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000301-beb4b7bd [00:03:04] RECOVERY - os202 APT on os202 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:03:18] !log [universalomega@puppet181] Upgraded packages on os191: pdns-recursor [00:03:19] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:03:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000318-dd077708 [00:03:26] RECOVERY - rdb191 APT on rdb191 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:03:34] !log [universalomega@puppet181] Upgraded packages on os151: pdns-recursor [00:03:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000334-15608050 [00:03:40] RECOVERY - os151 APT on os151 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:03:42] RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:03:54] !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php MirahezeMagic:ResetWiki --wiki=loginwiki --dbname=republicofteapotswiki --requester=Deathhunter (END - exit=0) (T16344) [00:03:55] !log [universalomega@puppet181] Upgraded packages on os162: pdns-recursor [00:03:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000354-cd43efb5 [00:03:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000355-c9eccf17 [00:04:07] RECOVERY - os201 APT on os201 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:04:08] !log [universalomega@puppet181] Upgraded packages on swiftobject151: pdns-recursor [00:04:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000408-96f2718e [00:04:12] RECOVERY - os191 APT on os191 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:04:26] !log [universalomega@puppet181] Upgraded packages on speedscope211: pdns-recursor [00:04:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000426-cf59d56b [00:04:42] !log [universalomega@puppet181] Upgraded packages on swiftac171: pdns-recursor [00:04:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000442-269db162 [00:04:59] !log [universalomega@puppet181] Upgraded packages on swiftobject161: pdns-recursor [00:05:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000459-50188db1 [00:05:09] RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:05:16] !log [universalomega@puppet181] Upgraded packages on swiftobject171: pdns-recursor [00:05:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000516-72b72491 [00:05:22] RECOVERY - speedscope211 APT on speedscope211 is OK: APT OK: 8 packages available for upgrade (0 critical updates). [00:05:33] !log [universalomega@puppet181] Upgraded packages on swiftproxy161: pdns-recursor [00:05:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000533-5246b7eb [00:05:49] !log [universalomega@puppet181] Upgraded packages on swiftobject191: pdns-recursor [00:05:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000549-e4966ab0 [00:05:51] RECOVERY - os162 APT on os162 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:05:56] RECOVERY - swiftobject171 APT on swiftobject171 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:06:06] RECOVERY - swiftobject151 APT on swiftobject151 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:06:07] !log [universalomega@puppet181] Upgraded packages on test151: pdns-recursor [00:06:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000607-be33c896 [00:06:23] !log [universalomega@puppet181] Upgraded packages on swiftproxy171: pdns-recursor [00:06:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000623-26b7cce6 [00:06:42] !log [universalomega@puppet181] Upgraded packages on swiftobject181: pdns-recursor [00:06:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000642-ca8c4e25 [00:06:44] RECOVERY - swiftproxy171 APT on swiftproxy171 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:06:49] !log ran Miraheze\MatomoAnalytics\MatomoAnalytics::getSiteID("republicofteapotswiki", true); (T16344) [00:06:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000649-ada42743 [00:06:52] RECOVERY - swiftobject161 APT on swiftobject161 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:06:53] RECOVERY - swiftobject181 APT on swiftobject181 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:07:02] !log [universalomega@puppet181] Upgraded packages on swiftobject211: pdns-recursor [00:07:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000702-e2365207 [00:07:18] !log [universalomega@puppet181] Upgraded packages on swiftobject201: pdns-recursor [00:07:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115000718-6006bad3 [00:07:20] RECOVERY - swiftproxy161 APT on swiftproxy161 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:07:24] RECOVERY - swiftobject191 APT on swiftobject191 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:07:28] RECOVERY - swiftobject211 APT on swiftobject211 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:07:48] RECOVERY - test151 APT on test151 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:07:54] RECOVERY - swiftobject201 APT on swiftobject201 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:12:10] !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php MirahezeMagic:ResetWiki --wiki=loginwiki --dbname=mystiasizakayawiki --requester=Cypo (END - exit=0) (T16357) [00:12:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115001210-77de9e21 [00:14:27] !log Miraheze\MatomoAnalytics\MatomoAnalytics::getSiteID("mystiasizakayawiki", true); (T16357) [00:14:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115001427-54ff54fd [01:55:34] !log [petramagna@test151] Starting deploy of {'config': True, 'debug': True} to test151 (T16507) [01:55:35] !log [petramagna@test151] finished deploy of {'config': True, 'debug': True} to test151 - SUCCESS in 0s (T16507) [01:55:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115015534-8b4a44fa [01:55:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115015535-80027b72 [02:02:02] [02mw-config] 07lihaohong6 merged pull request #6644: Implement default extension and user config changes in RfF (07miraheze:03main...07lihaohong6:03T16507) 13https://github.com/miraheze/mw-config/pull/6644 [02:02:02] [02mw-config] 07lihaohong6 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/20c58373b822642fdc688a37f8a530b58b5aa33e [02:02:02] 02mw-config/03main 07Peter Li 0320c5837 Implement default extension and user config changes in RfF (#6644)… [02:10:42] !log [petramagna@mwtask181] Starting deploy of {'pull': 'config', 'config': True, 'debug': True} to all (T16507) [02:10:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115021042-080b8c64 [02:11:09] !log [petramagna@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'debug': True} to all - SUCCESS in 27s (T16507) [02:11:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115021109-48df9f43 [03:14:28] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/ec9fae7088bf1e18876aaa99836671e5e15648dd [03:14:29] 02puppet/03main 07CosmicAlpha 03ec9fae7 safe_service_restart: use fsslc.wtnet hostnames [04:00:26] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/16a63624ee8926396dfc47a50329b1b02e0d8f21 [04:00:26] 02ssl/03main 07WikiTideBot 0316a6362 Bot: Auto-update domain lists [05:30:24] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/6c8fcdf9fa2c7467158d861d7dc1f14ecfc1d924 [05:30:24] 02ssl/03main 07WikiTideBot 036c8fcdf Bot: Auto-update domain lists [05:31:32] !log [waki285@test151] Starting deploy of {'force_upgrade': True, 'versions': '1.47', 'upgrade_extensions': 'ThemeToggle'} to test151 [05:31:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115053132-7b3394e0 [05:31:40] !log [waki285@test151] finished deploy of {'force_upgrade': True, 'versions': '1.47', 'upgrade_extensions': 'ThemeToggle'} to test151 - SUCCESS in 8s [05:31:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115053140-bd747bfd [05:32:06] !log [waki285@test151] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php updateSpecialPages --wiki=exttestwikibeta --only=ThemeUsage (END - exit=0) [05:32:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115053206-031fcb46 [05:36:37] !log [waki285@test151] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php updateSpecialPages --wiki=exttestwikibeta --only=ThemeUsage (END - exit=0) [05:36:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115053637-031fcb46 [05:38:37] !log [waki285@test151] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php updateSpecialPages --wiki=exttestwikibeta --only=ThemeUsage (END - exit=0) [05:38:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115053837-031fcb46 [07:38:33] [02mw-config] 07Honoka55 opened pull request #6648: Fix T16524: add per-wiki csps (07miraheze:03main...07Honoka55:03patch-5) 13https://github.com/miraheze/mw-config/pull/6648 [07:39:41] miraheze/mw-config - Honoka55 the build passed. [09:15:08] [02mw-config] 07RhinosF1 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/4471ebeeffe7fd567a9fa4195f972909ea1af12a [09:15:08] 02mw-config/03main 07RhinosF1 034471ebe Update Database.php [09:15:37] !log [rhinos@mwtask181] Starting deploy of {'pull': 'config', 'config': True} to all [09:15:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115091537-48f2067f [09:16:04] !log [rhinos@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 26s [09:16:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115091604-857a4b69 [09:16:12] miraheze/mw-config - RhinosF1 the build passed. [11:18:44] [1/4] https://cdn.discordapp.com/attachments/808001911868489748/1571484588872568862/20261106_031719.png?ex=6afae692&is=6af99512&hm=9894107d414ed4327d57d12a6531f18c14fbe060b53d36f419bbde768f839fd4& [11:18:44] [2/4] https://cdn.discordapp.com/attachments/808001911868489748/1571484590139383899/20261115_000507.png?ex=6afae693&is=6af99513&hm=a4238eb7fd9982a8e34b5155f316cc7a8eae3d88d31972d58b9507914660e6ce& [11:18:44] [3/4] https://cdn.discordapp.com/attachments/808001911868489748/1571484590646759554/20261114_215909.png?ex=6afae693&is=6af99513&hm=4786d1763bc4dc5285b55645acc582c7fda478e70c101b636692441d276576ea& [11:18:45] [4/4] https://cdn.discordapp.com/attachments/808001911868489748/1571484591552864397/PXL_20261106_114613_502.png?ex=6afae693&is=6af99513&hm=a3f2c18698bb234cc478c4e6097d4aa94b8f945e86e6171ff9d86c2341d0e150& [11:33:26] PROBLEM - ping on mattermost2 is CRITICAL: PING CRITICAL - Packet loss = 100% [11:35:17] PROBLEM - Host mattermost2 is DOWN: PING CRITICAL - Packet loss = 100% [11:39:42] RECOVERY - Host mattermost2 is UP: PING OK - Packet loss = 0%, RTA = 30.63 ms [11:39:44] RECOVERY - ping on mattermost2 is OK: PING OK - Packet loss = 0%, RTA = 30.61 ms [11:56:36] [02mw-config] 07RhinosF1 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/057aa9a359933f373f5d162c02365eb0025a81e4 [11:56:36] 02mw-config/03main 07RhinosF1 03057aa9a Add a message about duration to RO text. [11:57:28] !log [rhinos@mwtask181] Starting deploy of {'pull': 'config', 'config': True} to all [11:57:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115115728-48f2067f [11:57:45] miraheze/mw-config - RhinosF1 the build passed. [11:57:56] !log [rhinos@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 27s [11:57:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115115755-fedfdd2c [12:14:27] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period http://grafana.wikitide.net/d/GtxbP1Xnk?from=1794741230000&orgId=1&to=1794744866680 [12:30:27] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/bab9c1ce467dbfeeedcb3dff2ee32a7821802eb2 [12:30:27] 02ssl/03main 07WikiTideBot 03bab9c1c Bot: Auto-update domain lists [13:08:43] [02MirahezeRequests] 07Reception123 closed pull request #5: finalize RequestAccount due to high priority (T14935) (03main...03requestaccount) 13https://github.com/miraheze/MirahezeRequests/pull/5 [13:09:42] [02mw-config] 07Honoka55 opened pull request #6649: Fix T16527: override wgCirrusSearchUseIcuTokenizer for xyywiki (07miraheze:03main...07Honoka55:03T16527) 13https://github.com/miraheze/mw-config/pull/6649 [13:29:50] miraheze/puppet - Reception123 the build has errored. [13:29:52] [02puppet] 07Reception123 pushed 1 new commit to 03Reception123-patch-2 13https://github.com/miraheze/puppet/commit/8da0a88d407c4a73e6cc404f9999fdece601bfac [13:29:52] 02puppet/03Reception123-patch-2 07Reception123 038da0a88 Update policy_checker.py [13:31:01] miraheze/puppet - Reception123 the build has errored. [13:31:18] [02puppet] 07Reception123 pushed 1 new commit to 03Reception123-patch-2 13https://github.com/miraheze/puppet/commit/cada48e3fa10216a64264e3ecca33e9ba92fe68b [13:31:18] 02puppet/03Reception123-patch-2 07Reception123 03cada48e Delete modules/role/manifests/mediawiki/bots.pp [13:32:19] miraheze/puppet - Reception123 the build has errored. [13:35:02] [02puppet] 07Reception123 pushed 1 new commit to 03Reception123-patch-2 13https://github.com/miraheze/puppet/commit/2f3ae351ff5e9d5e08ddc376af9d745c733d279e [13:35:02] 02puppet/03Reception123-patch-2 07Reception123 032f3ae35 replace cron with systemd [13:35:44] [02puppet] 07github-actions[bot] pushed 1 new commit to 03Reception123-patch-2 13https://github.com/miraheze/puppet/commit/b93feac9b1c8617c0dd5534396d66e549a1b4581 [13:35:44] 02puppet/03Reception123-patch-2 07github-actions 03b93feac CI: lint puppet code to standards… [13:36:03] miraheze/puppet - Reception123 the build has errored. [13:38:46] [02MirahezeMagic] 07lihaohong6 left a file comment in pull request #684 0367faf46: Appending to the talk page would work I think since it sends a notification to the affected user. […] 13https://github.com/miraheze/MirahezeMagic/pull/684#discussion_r4578608556 [13:48:06] [02mw-config] 07lihaohong6 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/ef39ce430e4cd9ab2747d1b56aaf298860ea66b2 [13:48:06] 02mw-config/03main 07Honoka55 03ef39ce4 Override wgCirrusSearchUseIcuTokenizer for xyywiki (#6649)… [13:48:06] [02mw-config] 07lihaohong6 merged 07Honoka55's pull request #6649: Fix T16527: override wgCirrusSearchUseIcuTokenizer for xyywiki (07miraheze:03main...07Honoka55:03T16527) 13https://github.com/miraheze/mw-config/pull/6649 [13:49:37] !log [petramagna@test151] Starting deploy of {'pull': 'config', 'config': True, 'debug': True} to test151 [13:49:38] !log [petramagna@test151] finished deploy of {'pull': 'config', 'config': True, 'debug': True} to test151 - SUCCESS in 1s [13:49:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115134937-a20a1ab7 [13:49:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115134938-9f20aa19 [13:49:40] [02mw-config] 07RhinosF1 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/f4e93356baf6c7b1961e9ffbb7a318b92096925b [13:49:40] 02mw-config/03main 07RhinosF1 03f4e9335 Update Database.php [13:50:08] !log [petramagna@mwtask181] Starting deploy of {'pull': 'config', 'config': True, 'debug': True} to all (T16527) [13:50:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115135008-defac9ba [13:50:09] !log [rhinos@mwtask181] Starting deploy of {'pull': 'config', 'config': True} to all [13:50:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115135009-48f2067f [13:50:10] !log [rhinos@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [13:50:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115135010-d8009374 [13:50:34] !log [petramagna@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'debug': True} to all - SUCCESS in 25s (T16527) [13:50:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115135034-d7ce9f8f [13:50:41] miraheze/mw-config - RhinosF1 the build passed. [13:50:46] !log [rhinos@mwtask181] Starting deploy of {'pull': 'config', 'config': True} to all [13:50:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115135046-48f2067f [13:51:10] !log [rhinos@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 24s [13:51:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115135110-c37a1cf2 [13:54:18] !log [petramagna@mwtask171] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php CirrusSearch:UpdateSearchIndexConfig --wiki=xyywiki --reindexAndRemoveOk --indexIdentifier now (END - exit=130) (T16527) [13:54:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115135418-6944e0de [13:54:43] [02mw-config] 07RhinosF1 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/c6cba98b9f64472207867ce028c1f81a06ee2a2b [13:54:43] 02mw-config/03main 07RhinosF1 03c6cba98 mark db161 as inactive so wikis don't get created there during maint [13:54:49] !log [rhinos@mwtask181] Starting deploy of {'pull': 'config', 'config': True} to all [13:54:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115135449-48f2067f [13:55:14] !log [rhinos@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s [13:55:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115135514-ecb3f61f [13:55:48] miraheze/mw-config - RhinosF1 the build passed. [13:56:17] [02mw-config] 07RhinosF1 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/b64874b05149855debd39a58cc5ba1c5e93a721e [13:56:17] 02mw-config/03main 07RhinosF1 03b64874b Update LocalSettings.php [13:56:20] !log [rhinos@mwtask181] Starting deploy of {'pull': 'config', 'config': True} to all [13:56:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115135620-48f2067f [13:56:46] !log [rhinos@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s [13:56:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115135645-ecb3f61f [13:57:19] miraheze/mw-config - RhinosF1 the build passed. [13:59:41] RECOVERY - mw151 Check unit status of php8.4-fpm_check_restart on mw151 is OK: OK: Status of the systemd unit php8.4-fpm_check_restart [14:00:58] !log [petramagna@mwtask171] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php CirrusSearch:UpdateSearchIndexConfig --wiki=xyywiki --reindexAndRemoveOk --indexIdentifier now (END - exit=0) (T16527) [14:00:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115140058-134a1a2b [14:04:28] !log [petramagna@mwtask171] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php CirrusSearch:ForceSearchIndex --wiki=xyywiki --deletes --from 2026-11-15T13:52:45Z (END - exit=0) (T16527) [14:04:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115140428-28a0c25f [14:04:53] !log [petramagna@mwtask171] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php CirrusSearch:ForceSearchIndex --wiki=xyywiki --from 2026-11-15T13:52:45Z (END - exit=0) (T16527) [14:04:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115140453-a1b3f6c4 [14:25:30] !log [petramagna@test151] Starting deploy of {'config': True, 'debug': True} to test151 [14:25:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115142530-bd538078 [14:25:31] !log [petramagna@test151] finished deploy of {'config': True, 'debug': True} to test151 - SUCCESS in 0s [14:25:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115142531-79883921 [14:29:39] !log [petramagna@test151] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php CirrusSearch:UpdateSearchIndexConfig --wiki=defaultexttestwikibeta --reindexAndRemoveOk --indexIdentifier now (END - exit=0) [14:29:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115142939-122e2004 [14:46:38] miraheze/MirahezeMagic - Honoka55 the build passed. [15:08:18] !log [petramagna@test151] Starting deploy of {'folders': '1.47/extensions/ThemeToggle', 'debug': True} to test151 [15:08:19] !log [petramagna@test151] finished deploy of {'folders': '1.47/extensions/ThemeToggle', 'debug': True} to test151 - SUCCESS in 0s [15:08:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115150818-7283466d [15:08:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115150819-2a861eaa [15:08:49] !log [petramagna@test151] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php updateSpecialPages --wiki=exttestwikibeta (END - exit=0) [15:08:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115150849-5af28bd3 [15:12:40] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.68, 3.30, 2.32 [15:16:40] RECOVERY - os151 Current Load on os151 is OK: LOAD OK - total load average: 3.18, 3.33, 2.58 [16:04:46] !log [petramagna@test151] Starting deploy of {'config': True} to test151 (T15624) [16:04:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115160446-7f75e6e4 [16:04:47] !log [petramagna@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s (T15624) [16:04:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115160447-6c9bc437 [16:05:17] !log [petramagna@test151] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php ManageWiki:PopulateWikiSettings --wiki=metawikibeta --setting=wgUnlinkedWikibaseImitationMode --sourcelist=/tmp/ulwb-true.txt --task=T15624 (END - exit=1) [16:05:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115160517-602e973d [16:05:38] !log [petramagna@test151] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php ManageWiki:PopulateWikiSettings --wiki=metawikibeta -- --setting=wgUnlinkedWikibaseImitationMode --sourcelist=/tmp/ulwb-true.txt (END - exit=1) (T15624) [16:05:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115160539-c53fb09c [16:07:14] !log [petramagna@test151] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php ManageWiki:PopulateWikiSettings --wiki=metawikibeta --setting=wgUnlinkedWikibaseStatementsParserFunc --remove --all-wikis --execute (START) (T15624) [16:07:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115160714-104e0bdd [16:07:16] !log [petramagna@test151] sudo -u www-data php /srv/mediawiki/1.47/maintenance/run.php ManageWiki:PopulateWikiSettings --wiki=metawikibeta --setting=wgUnlinkedWikibaseStatementsParserFunc --remove --all-wikis --execute (END - exit=0; time=1s) (T15624) [16:07:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115160716-ae376fbc [16:08:40] [02mw-config] 07lihaohong6 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/8934834f7d80ee292ed3f3ae7ad28b75796afed9 [16:08:40] 02mw-config/03main 07Honoka55 038934834 Switch UnlinkedWikibase deprecated config (#6627)… [16:08:42] [02mw-config] 07lihaohong6 merged 07Honoka55's pull request #6627: Task T15624: switch UnlinkedWikibase deprecated config (07miraheze:03main...07PleasantGoatWiki:03ulwbconf) 13https://github.com/miraheze/mw-config/pull/6627 [16:09:05] !log [petramagna@mwtask181] Starting deploy of {'pull': 'config', 'config': True, 'debug': True} to all (T15624) [16:09:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115160905-2eb32d4c [16:09:32] !log [petramagna@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'debug': True} to all - SUCCESS in 27s (T15624) [16:09:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115160932-f852452d [16:10:13] miraheze/mw-config - lihaohong6 the build passed. [16:11:51] !log [petramagna@mwtask171] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php ManageWiki:PopulateWikiSettings --wiki=metawiki --setting=wgUnlinkedWikibaseImitationMode --sourcelist=/tmp/ulwb-true.txt (END - exit=0) (T15624) [16:11:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115161151-e373d0a0 [16:14:27] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period http://grafana.wikitide.net/d/GtxbP1Xnk?from=1794741230000&orgId=1&to=1794759266710 [16:16:14] !log [petramagna@mwtask171] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php ManageWiki:PopulateWikiSettings --wiki=metawiki --setting=wgUnlinkedWikibaseStatementsParserFunc --remove --all-wikis --execute (START) (T15624) [16:16:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115161614-05ca4e6c [16:20:48] !log [petramagna@mwtask171] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php ManageWiki:PopulateWikiSettings --wiki=metawiki --setting=wgUnlinkedWikibaseStatementsParserFunc --remove --all-wikis --execute (END - exit=0; time=273s) (T15624) [16:20:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115162048-7322eb7c [16:40:55] !log [somerandomdeveloper@test151] Starting deploy of {'pr': 653, 'pr_repo': 'MirahezeMagic', 'folders': '1.47/MirahezeMagic'} to test151 [16:40:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115164055-121d33bc [16:41:09] !log [somerandomdeveloper@test151] Starting deploy of {'pr': 653, 'pr_repo': 'extensions/MirahezeMagic', 'folders': '1.47/extensions/MirahezeMagic'} to test151 [16:41:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115164109-be6895ed [16:41:28] !log [somerandomdeveloper@test151] Starting deploy of {'pr': 653, 'pr_repo': '1.47/extensions/MirahezeMagic', 'folders': '1.47/extensions/MirahezeMagic'} to test151 [16:41:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115164128-681e8f3c [16:41:33] how does this work lol [16:42:39] !log [somerandomdeveloper@test151] Starting deploy of {'pr': 653, 'pr_repo': 'extensions/MirahezeMagic', 'folders': '1.47/extensions/MirahezeMagic'} to test151 [16:42:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115164239-be6895ed [16:43:34] !log [somerandomdeveloper@test151] Starting deploy of {'pr': 653, 'pr_repo': 'extensions/MirahezeMagic'} to test151 [16:43:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115164334-e5e86ad0 [16:43:49] nvm ig I'm doing it the manual way [16:46:49] !log [somerandomdeveloper@test151] Starting deploy of {'folders': '1.47/extensions/MirahezeMagic'} to test151 [16:46:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115164649-e804f18f [16:46:50] !log [somerandomdeveloper@test151] finished deploy of {'folders': '1.47/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [16:46:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115164650-971c46c2 [16:52:58] miraheze/MirahezeMagic - SomeMWDev the build passed. [17:11:41] [02MirahezeMagic] 07SomeMWDev commented on pull request #653: Sorry about the delay. I've tested this on beta, and because CentralAuth is loaded after MirahezeMagic, the hook handler doesn't work. I've pushed a commit to fix this and register the hook handler when the MediaWikiServices hook is called (as an […] 13https://github.com/miraheze/MirahezeMagic/pull/653#issuecomment-6898192377 [17:12:25] !log [somerandomdeveloper@test151] Starting deploy of {'folders': '1.47/extensions/MirahezeMagic'} to test151 [17:12:26] !log [somerandomdeveloper@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [17:12:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115171225-e804f18f [17:12:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115171226-fe2f2fb1 [17:12:37] !log [somerandomdeveloper@test151] Starting deploy of {'force': True, 'folders': '1.47/extensions/MirahezeMagic'} to test151 [17:12:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115171237-a94e3f09 [17:12:38] !log [somerandomdeveloper@test151] finished deploy of {'force': True, 'folders': '1.47/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [17:12:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115171238-b01dea25 [17:13:04] !log [somerandomdeveloper@test151] Starting deploy of {'l10n': True, 'versions': '1.46'} to test151 [17:13:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115171304-73a63a25 [17:13:13] !log [somerandomdeveloper@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [17:13:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115171313-99aba97f [17:13:17] !log [somerandomdeveloper@test151] Starting deploy of {'l10n': True, 'versions': '1.47'} to test151 [17:13:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115171317-bcb3bc4d [17:13:38] !log [somerandomdeveloper@test151] finished deploy of {'l10n': True, 'versions': '1.47'} to test151 - SUCCESS in 21s [17:13:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115171338-0311768c [17:18:24] miraheze/MirahezeMagic - SomeMWDev the build passed. [17:22:52] [02MirahezeMagic] 07SomeMWDev approved pull request #653 13https://github.com/miraheze/MirahezeMagic/pull/653#pullrequestreview-5911918486 [17:23:10] [02MirahezeMagic] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/2b3d0baf515399dc781c87f883dca7650227fe9a [17:23:10] 02MirahezeMagic/03main 07Luka Simić 032b3d0ba Disable Special:GlobalVanishRequest (#653)… [17:23:12] [02MirahezeMagic] 07SomeMWDev merged 07KockaAdmiralac's pull request #653: Disable Special:GlobalVanishRequest (07miraheze:03main...07KockaAdmiralac:03globalvanishrequest) 13https://github.com/miraheze/MirahezeMagic/pull/653 [17:23:48] !log [somerandomdeveloper@test151] Starting deploy of {'l10n': True, 'versions': 'all', 'upgrade_extensions': 'MirahezeMagic'} to test151 [17:23:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115172348-cc4ee00c [17:24:30] !log [somerandomdeveloper@test151] finished deploy of {'l10n': True, 'versions': 'all', 'upgrade_extensions': 'MirahezeMagic'} to test151 - SUCCESS in 41s [17:24:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115172429-7c8595c5 [17:26:37] !log [somerandomdeveloper@mwtask181] Starting deploy of {'l10n': True, 'versions': 'all', 'upgrade_extensions': 'MirahezeMagic'} to mwtask181 [17:26:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115172637-ba00a2f9 [17:28:38] !log [somerandomdeveloper@mwtask181] finished deploy of {'l10n': True, 'versions': 'all', 'upgrade_extensions': 'MirahezeMagic'} to mwtask181 - SUCCESS in 121s [17:28:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115172838-025d37f1 [17:30:01] miraheze/MirahezeMagic - SomeMWDev the build passed. [17:31:34] !log [somerandomdeveloper@mwtask181] Starting deploy of {'l10n': True, 'versions': 'all', 'upgrade_extensions': 'MirahezeMagic'} to all [17:31:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115173134-3ce6e1fc [17:33:30] !log [somerandomdeveloper@mwtask181] finished deploy of {'l10n': True, 'versions': 'all', 'upgrade_extensions': 'MirahezeMagic'} to all - SUCCESS in 116s [17:33:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115173330-2fff9ece [17:34:05] !log [somerandomdeveloper@mwtask181] Starting deploy of {'force_upgrade': True, 'versions': 'all', 'upgrade_extensions': 'MirahezeMagic'} to all [17:34:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115173405-3e40cc62 [17:35:04] !log [somerandomdeveloper@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw173 [17:35:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115173504-5b0755bc [17:35:09] opcache likely [17:35:20] (expected) [17:39:44] will try to deploy a version where the hook isn't registered yet first, wait a minute for opcache to expire, and then deploy the full commit [17:41:03] !log [somerandomdeveloper@mwtask181] Starting deploy of {'folders': '1.46/extensions/MirahezeMagic,1.47/extensions/MirahezeMagic'} to mwtask181 [17:41:04] !log [somerandomdeveloper@mwtask181] finished deploy of {'folders': '1.46/extensions/MirahezeMagic,1.47/extensions/MirahezeMagic'} to mwtask181 - SUCCESS in 0s [17:41:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115174103-6d794baa [17:41:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115174104-547edf74 [17:41:42] !log [somerandomdeveloper@mwtask181] Starting deploy of {'folders': '1.46/extensions/MirahezeMagic,1.47/extensions/MirahezeMagic'} to all [17:41:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115174142-0976001c [17:42:13] !log [somerandomdeveloper@mwtask181] finished deploy of {'folders': '1.46/extensions/MirahezeMagic,1.47/extensions/MirahezeMagic'} to all - SUCCESS in 31s [17:42:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115174213-ad5d1b76 [17:43:45] !log [somerandomdeveloper@mwtask181] Starting deploy of {'folders': '1.46/extensions/MirahezeMagic,1.47/extensions/MirahezeMagic'} to all [17:43:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115174345-0976001c [17:44:14] !log [somerandomdeveloper@mwtask181] finished deploy of {'folders': '1.46/extensions/MirahezeMagic,1.47/extensions/MirahezeMagic'} to all - SUCCESS in 28s [17:44:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115174414-a3e8f053 [17:59:39] !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php MirahezeMagic:ClearUserRealNames --wiki=vocaloidlyricswiki --dry-run (END - exit=0) [17:59:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115175939-b8f05950 [18:00:51] !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php MirahezeMagic:ClearUserRealNames --wiki=vocaloidlyricswiki (END - exit=0) (T16475) [18:00:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115180051-8721a851 [18:06:51] [02mw-config] 07SomeMWDev created 03T16249 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/3038ec728fc0 [18:06:51] 02mw-config/03T16249 07SomeRandomDeveloper 033038ec7 Set default thumb size for tfukwiki to 300px… [18:07:06] [02mw-config] 07SomeMWDev opened pull request #6650: Set default thumb size for tfukwiki to 300px (03main...03T16249) 13https://github.com/miraheze/mw-config/pull/6650 [18:08:24] [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/8aefe6023dc51439b780a836438d6a64e9d8d524 [18:08:26] 02mw-config/03main 07SomeRandomDeveloper 038aefe60 Set default thumb size for tfukwiki to 300px (#6650)… [18:08:28] [02mw-config] 07SomeMWDev merged pull request #6650: Set default thumb size for tfukwiki to 300px (03main...03T16249) 13https://github.com/miraheze/mw-config/pull/6650 [18:08:29] [02mw-config] 07SomeMWDev 04deleted 03T16249 at 033038ec7 13https://api.github.com/repos/miraheze/mw-config/commit/3038ec7 [18:08:37] !log [somerandomdeveloper@mwtask181] Starting deploy of {'pull': 'config', 'config': True} to all (T16249) [18:08:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115180837-aeb93d7b [18:09:03] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s (T16249) [18:09:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115180903-9634bbb8 [18:16:47] PROBLEM - mwtask181 Check unit status of mediawiki_job_update-statistics on mwtask181 is CRITICAL: CRITICAL: Status of the systemd unit mediawiki_job_update-statistics [18:18:14] ^ cause by db161 being readonly, I saw the errors in the log earlier [18:20:47] Ack, not a lot we can do [18:21:47] [02mw-config] 07SomeMWDev created 03T16528 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/e7fd8555252c [18:21:47] 02mw-config/03T16528 07SomeRandomDeveloper 03e7fd855 Set default vector theme for slotboundwiki to 'night'… [18:21:54] [02mw-config] 07SomeMWDev opened pull request #6651: Set default vector theme for slotboundwiki to 'night' (03main...03T16528) 13https://github.com/miraheze/mw-config/pull/6651 [18:22:47] [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/a88d818f701e339728b3cf95f6ebeb6aca16c4c3 [18:22:48] 02mw-config/03main 07SomeRandomDeveloper 03a88d818 Set default vector theme for slotboundwiki to 'night' (#6651)… [18:22:50] [02mw-config] 07SomeMWDev merged pull request #6651: Set default vector theme for slotboundwiki to 'night' (03main...03T16528) 13https://github.com/miraheze/mw-config/pull/6651 [18:22:52] [02mw-config] 07SomeMWDev 04deleted 03T16528 at 03e7fd855 13https://api.github.com/repos/miraheze/mw-config/commit/e7fd855 [18:22:56] !log [somerandomdeveloper@mwtask181] Starting deploy of {'pull': 'config', 'config': True} to all (T16528) [18:22:57] miraheze/mw-config - SomeMWDev the build passed. [18:22:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115182256-ab0ce358 [18:23:24] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 28s (T16528) [18:23:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115182324-5eff8f70 [18:23:55] miraheze/mw-config - SomeMWDev the build passed. [18:32:41] [02mw-config] 07SomeMWDev requested changes on pull request #6646 13https://github.com/miraheze/mw-config/pull/6646#pullrequestreview-5912196916 [18:32:42] [02mw-config] 07SomeMWDev left a file comment in pull request #6646 034978b4c: This config option was removed in https://github.com/wikimedia/mediawiki-extensions-MultimediaViewer/commit/f308afb (which is part of 1.47) 13https://github.com/miraheze/mw-config/pull/6646#discussion_r4579648535 [18:36:08] [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/38ec3de70af810da1811f5c4fff4da02fcf52ac2 [18:36:08] 02mw-config/03main 07dependabot[bot] 0338ec3de Update phpunit/phpunit requirement from 13.4.5 to 13.4.6 (#6625)… [18:36:10] [02mw-config] 07SomeMWDev merged 07dependabot[bot]'s pull request #6625: Update phpunit/phpunit requirement from 13.4.5 to 13.4.6 (03main...03dependabot/composer/phpunit/phpunit-13.4.6) 13https://github.com/miraheze/mw-config/pull/6625 [18:36:12] [02mw-config] 07SomeMWDev 04deleted 03dependabot/composer/phpunit/phpunit-13.4.6 at 0387d452d 13https://api.github.com/repos/miraheze/mw-config/commit/87d452d [18:37:10] miraheze/mw-config - SomeMWDev the build passed. [18:39:07] [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/5640953351ee18c52280a9ce6aaf270634690a14 [18:39:07] 02mw-config/03main 07Winston Sung 035640953 Update $wgHooks for TKU Wiki (tkuwiki) (#6604)… [18:39:07] [02mw-config] 07SomeMWDev merged 07winstonsung's pull request #6604: Update $wgHooks for TKU Wiki (tkuwiki) (07miraheze:03main...07winstonsung:03dev-6604) 13https://github.com/miraheze/mw-config/pull/6604 [18:39:13] !log [somerandomdeveloper@mwtask181] Starting deploy of {'pull': 'config', 'config': True} to all [18:39:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115183913-4d715c65 [18:39:39] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s [18:39:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115183939-24b2b936 [18:40:02] miraheze/mw-config - SomeMWDev the build passed. [18:45:47] [02mw-config] 07SomeMWDev left a file comment in pull request #6503 033984f61: Isn't the interwiki called `bucketwiki`? 13https://github.com/miraheze/mw-config/pull/6503#discussion_r4579694617 [18:46:19] [02mw-config] 07SomeMWDev left a file comment in pull request #6503 033984f61: nvm, I can't read 13https://github.com/miraheze/mw-config/pull/6503#discussion_r4579696675 [18:47:11] [02mw-config] 07SomeMWDev 04force-pushed 03YTFGolf-patch-1 from 033984f61 to 031b8c270 13https://api.github.com/repos/miraheze/mw-config/commits/YTFGolf-patch-1 [18:47:11] 02mw-config/03YTFGolf-patch-1 07WWR 03cec9430 Add `bucketwiki` as an import source if Bucket enabled… [18:47:11] 02mw-config/03YTFGolf-patch-1 07WWR 031b8c270 Update LocalSettings.php [18:47:44] !log [somerandomdeveloper@test151] Starting deploy of {'pr': 6503, 'pr_repo': 'config'} to test151 [18:47:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115184744-7f1d6326 [18:47:45] !log [somerandomdeveloper@test151] finished deploy of {'pr': 6503, 'pr_repo': 'config'} to test151 - SUCCESS in 1s [18:47:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115184745-36847c0c [18:48:21] miraheze/mw-config - SomeMWDev the build passed. [18:49:27] !log [somerandomdeveloper@test151] Starting deploy of {'config': True} to test151 [18:49:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115184927-81577ee6 [18:49:28] !log [somerandomdeveloper@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [18:49:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115184928-83ceb510 [18:50:41] !log [somerandomdeveloper@test151] Starting deploy of {'config': True} to test151 [18:50:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115185041-81577ee6 [18:50:42] !log [somerandomdeveloper@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [18:50:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115185042-83ceb510 [18:51:05] !log [somerandomdeveloper@test151] Starting deploy of {'pr': 6503, 'pr_repo': 'config'} to test151 [18:51:06] !log [somerandomdeveloper@test151] finished deploy of {'pr': 6503, 'pr_repo': 'config'} to test151 - SUCCESS in 0s [18:51:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115185105-7f1d6326 [18:51:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115185106-50bd031e [18:51:49] [02mw-config] 07MacFan4000 created 03MacFan4000-patch-1 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/1a8f68a95afd [18:51:49] 02mw-config/03MacFan4000-patch-1 07MacFan4000 031a8f68a Temp disable Renames/Vanishes for db161 maintenance [18:51:55] [02mw-config] 07MacFan4000 opened pull request #6652: Temp disable Renames/Vanishes for db161 maintenance (03main...03MacFan4000-patch-1) 13https://github.com/miraheze/mw-config/pull/6652 [18:53:19] [02mw-config] 07SomeMWDev left a review on pull request #6503: I've tried this on beta and it did not work 13https://github.com/miraheze/mw-config/pull/6503#pullrequestreview-5912275913 [18:53:28] miraheze/mw-config - MacFan4000 the build passed. [18:54:37] [02mw-config] 07RhinosF1 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/cb7d42937446ae6e82185d9a0c67611d5304c9c6 [18:54:37] 02mw-config/03main 07RhinosF1 03cb7d429 Update Database.php [18:55:34] !log [rhinos@mwtask181] Starting deploy of {'pull': 'config', 'config': True} to all [18:55:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115185534-48f2067f [18:55:41] miraheze/mw-config - RhinosF1 the build passed. [18:55:54] [02mw-config] 07MacFan4000 pushed 1 new commit to 03MacFan4000-patch-1 13https://github.com/miraheze/mw-config/commit/2ed283c900cf33340e47ddbd3f669457c4764dec [18:55:54] 02mw-config/03MacFan4000-patch-1 07MacFan4000 032ed283c Merge branch 'main' into MacFan4000-patch-1 [18:56:00] !log [rhinos@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 26s [18:56:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115185600-857a4b69 [18:56:08] [02mw-config] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/756886b99e4670afae77e3b0797eda4caf4410f9 [18:56:08] 02mw-config/03main 07MacFan4000 03756886b Temp disable Renames/Vanishes for db161 maintenance (#6652) [18:56:08] [02mw-config] 07MacFan4000 merged pull request #6652: Temp disable Renames/Vanishes for db161 maintenance (03main...03MacFan4000-patch-1) 13https://github.com/miraheze/mw-config/pull/6652 [18:56:09] [02mw-config] 07MacFan4000 04deleted 03MacFan4000-patch-1 at 032ed283c 13https://api.github.com/repos/miraheze/mw-config/commit/2ed283c [18:56:23] [02mw-config] 07SomeMWDev closed pull request #6361: T15065: Fix 3D extension not attempting to generate thumbnails (03main...03T15065) 13https://github.com/miraheze/mw-config/pull/6361 [18:56:24] [02mw-config] 07SomeMWDev commented on pull request #6361: Done in https://github.com/miraheze/mw-config/commit/6924e3f 13https://github.com/miraheze/mw-config/pull/6361#issuecomment-6899546313 [18:56:27] [02mw-config] 07SomeMWDev 04deleted 03T15065 at 03ad04349 13https://api.github.com/repos/miraheze/mw-config/commit/ad04349 [18:57:10] miraheze/mw-config - MacFan4000 the build passed. [18:58:01] !log [macfan@mwtask181] Starting deploy of {'pull': 'config', 'config': True} to all [18:58:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115185801-09a64ea4 [18:58:26] !log [macfan@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 24s [18:58:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115185826-2871944e [18:59:47] [02mw-config] 07SomeMWDev left a file comment in pull request #6652 032ed283c: Wrong array key 13https://github.com/miraheze/mw-config/pull/6652#discussion_r4579750980 [19:00:02] [02mw-config] 07SomeMWDev left a file comment in pull request #6652 032ed283c: This will likely not work because of the way CentralAuth registers the pages: https://github.com/wikimedia/mediawiki-extensions-CentralAuth/blob/a60ae57/includes/Hooks/Handlers/SpecialPageInitHookHandler.php#L29-L48 13https://github.com/miraheze/mw-config/pull/6652#discussion_r4579751779 [19:01:07] [02mw-config] 07SomeMWDev left a review on pull request #6652: The error messages will show up weirdly because DisabledSpecialPage expects message keys, would recommend using RawMessage 13https://github.com/miraheze/mw-config/pull/6652#pullrequestreview-5912313031 [19:05:17] [02mw-config] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/b8fa6eaae7e6415f767c45e61bd5a280a89fba7a [19:05:17] 02mw-config/03main 07MacFan4000 03b8fa6ea fix [19:05:21] !log [macfan@mwtask181] Starting deploy of {'pull': 'config', 'config': True} to all [19:05:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115190521-09a64ea4 [19:05:46] !log [macfan@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s [19:05:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115190546-17778d91 [19:06:18] miraheze/mw-config - MacFan4000 the build passed. [19:19:29] !log [macfan@test151] Starting deploy of {'pull': 'config', 'config': True} to test151 [19:19:30] !log [macfan@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s [19:19:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115191929-53ab8a0a [19:19:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115191930-c58d8793 [19:33:30] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 8191 bytes in 0.081 second response time [19:36:03] [02mw-config] 07YTFGolf commented on pull request #6503: hmm that's annoying. I'll see if `bucketwiki` works though (assuming I don't completely forget about this again) 13https://github.com/miraheze/mw-config/pull/6503#issuecomment-6900098951 [19:37:31] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.672 second response time [19:38:10] PROBLEM - test151 Check unit status of mediawiki_job_mediamoderation-scan on test151 is CRITICAL: CRITICAL: Status of the systemd unit mediawiki_job_mediamoderation-scan [20:02:52] [02mw-config] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/638fac7e9a4f1e4817cfee01915caa54b0f1aeac [20:02:52] 02mw-config/03main 07MacFan4000 03638fac7 Update GlobalSettings.php [20:14:26] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period http://grafana.wikitide.net/d/GtxbP1Xnk?from=1794741230000&orgId=1&to=1794773666739 [20:15:14] [02mw-config] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/7ffd42d32ec81a24f8ea4ec17e941d668716bb10 [20:15:15] 02mw-config/03main 07MacFan4000 037ffd42d temp revoke centralauth-rename [20:15:29] !log [macfan@mwtask181] Starting deploy of {'pull': 'config', 'config': True} to all [20:15:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115201529-09a64ea4 [20:15:57] !log [macfan@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 27s [20:15:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115201557-92c52b59 [20:21:35] [02mw-config] 07RhinosF1 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/15fed4920b0cc46cbdb9ba32d6663d68a52fbc7e [20:21:35] 02mw-config/03main 07RhinosF1 0315fed49 Update Database.php [20:22:01] !log [rhinos@mwtask181] Starting deploy of {'pull': 'config', 'config': True} to all [20:22:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115202201-48f2067f [20:22:29] !log [rhinos@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 28s [20:22:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115202229-a78fb9fa [20:22:38] miraheze/mw-config - RhinosF1 the build passed. [21:27:56] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/9fc36d4b7955d67daa441540aa5f485db3a67197 [21:27:56] 02puppet/03main 07CosmicAlpha 039fc36d4 renovate: automerge python test tooling and github actions [21:29:29] miraheze/puppet - Universal-Omega the build passed. [21:31:11] [02puppet] 07Universal-Omega created 03renovate-mongodb (+1 new commit) 13https://github.com/miraheze/puppet/commit/02a04c82935b [21:31:11] 02puppet/03renovate-mongodb 07CosmicAlpha 0302a04c8 renovate: add mongodb updates… [21:31:17] [02puppet] 07Universal-Omega opened pull request #5172: renovate: add mongodb updates (03main...03renovate-mongodb) 13https://github.com/miraheze/puppet/pull/5172 [21:32:44] miraheze/puppet - Universal-Omega the build passed. [21:48:12] [02puppet] 07Universal-Omega created 03pin-mwclient (+1 new commit) 13https://github.com/miraheze/puppet/commit/e56f485cc5a0 [21:48:13] 02puppet/03pin-mwclient 07CosmicAlpha 03e56f485 Pin mwclient commit [21:48:19] [02puppet] 07Universal-Omega opened pull request #5173: Pin mwclient commit (03main...03pin-mwclient) 13https://github.com/miraheze/puppet/pull/5173 [21:49:24] [02puppet] 07Universal-Omega pushed 1 new commit to 03pin-mwclient 13https://github.com/miraheze/puppet/commit/bb9087c1280cc65e94939ace0e0b1897849caf53 [21:49:24] 02puppet/03pin-mwclient 07CosmicAlpha 03bb9087c Fix [21:54:11] [02puppet] 07Universal-Omega pushed 1 new commit to 03renovate-mongodb 13https://github.com/miraheze/puppet/commit/4223ee38704f097397fca53a5dec7684a4dd4226 [21:54:12] 02puppet/03renovate-mongodb 07CosmicAlpha 034223ee3 8.4 [21:58:00] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/6aa0fe830096a13fd59714828f8afa19b42a8acd [21:58:00] 02puppet/03main 07CosmicAlpha 036aa0fe8 renovate: add mongodb updates… [21:58:00] [02puppet] 07Universal-Omega merged pull request #5172: renovate: add mongodb updates (03main...03renovate-mongodb) 13https://github.com/miraheze/puppet/pull/5172 [21:58:01] [02puppet] 07Universal-Omega 04deleted 03renovate-mongodb at 034223ee3 13https://api.github.com/repos/miraheze/puppet/commit/4223ee3 [21:58:52] [02puppet] 07Universal-Omega pushed 1 new commit to 03pin-mwclient 13https://github.com/miraheze/puppet/commit/ca8b4200558db96e65804e49ead7245a9eeaaecc [21:58:52] 02puppet/03pin-mwclient 07CosmicAlpha 03ca8b420 Merge branch 'main' into pin-mwclient [21:58:54] [02puppet] 07renovate[bot] created 03renovate/mongodb (+1 new commit) 13https://github.com/miraheze/puppet/commit/05437b3a1d6c [21:58:54] 02puppet/03renovate/mongodb 07renovate[bot] 0305437b3 chore(deps): update dependency mongodb/mongo to v8.3.17 [21:58:59] [02puppet] 07renovate[bot] added the label 'dependencies' to pull request #5174 (chore(deps): update dependency mongodb/mongo to v8.3.17) 13https://github.com/miraheze/puppet/pull/5174 [21:58:59] [02puppet] 07renovate[bot] added the label 'server-software' to pull request #5174 (chore(deps): update dependency mongodb/mongo to v8.3.17) 13https://github.com/miraheze/puppet/pull/5174 [21:58:59] [02puppet] 07renovate[bot] opened pull request #5174: chore(deps): update dependency mongodb/mongo to v8.3.17 (03main...03renovate/mongodb) 13https://github.com/miraheze/puppet/pull/5174 [22:00:24] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/83fe4583808b923f8da9fa4bc2529608ef5c9ff8 [22:00:25] 02ssl/03main 07WikiTideBot 0383fe458 Bot: Auto-update domain lists [22:02:34] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/87a97d3c3b5982b990f99af694fff96b987b250b [22:02:34] 02puppet/03main 07CosmicAlpha 0387a97d3 Pin mwclient commit [22:02:34] [02puppet] 07Universal-Omega merged pull request #5173: Pin mwclient commit (03main...03pin-mwclient) 13https://github.com/miraheze/puppet/pull/5173 [22:02:36] [02puppet] 07Universal-Omega 04deleted 03pin-mwclient at 03ca8b420 13https://api.github.com/repos/miraheze/puppet/commit/ca8b420 [22:21:25] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/9832a8d857e8990503d62cf16f7d1615b3732122 [22:21:25] 02puppet/03main 07CosmicAlpha 039832a8d renovate: add mwclient [22:31:23] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/750c0b5c4b2193359f53878b8bcd89d3ca3d4003 [22:31:23] 02mw-config/03main 07whostacking 03750c0b5 T15986: share files from doorswiki with the DOORS language wikis (#6641)… [22:31:25] [02mw-config] 07Universal-Omega merged 07whostacking's pull request #6641: T15986: share files from doorswiki with the DOORS language wikis (07miraheze:03main...07whostacking:03doors-shared-media) 13https://github.com/miraheze/mw-config/pull/6641 [22:32:36] miraheze/mw-config - Universal-Omega the build passed. [22:32:44] !log [universalomega@mwtask181] Starting deploy of {'pull': 'config', 'config': True} to all [22:32:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115223244-4d1b4212 [22:33:11] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 26s [22:33:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115223311-d7a95ade [22:34:27] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period http://grafana.wikitide.net/d/GtxbP1Xnk?from=1794778250000&orgId=1&to=1794782066757 [22:34:28] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period http://grafana.wikitide.net/d/GtxbP1Xnk?from=1794741230000&orgId=1&to=1794782066757 [22:42:14] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php sql --wiki=doorswiki /srv/mediawiki/1.46/extensions/GlobalUsage/sql/mysql/tables-generated.sql (END - exit=0) [22:42:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115224214-97cf35c1 [22:54:27] [Grafana] RESOLVED: MediaWiki JobQueue is stalled http://grafana.wikitide.net/d/GtxbP1Xnk?from=1794778250000&orgId=1&to=1794783170000 [22:54:28] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period http://grafana.wikitide.net/d/GtxbP1Xnk?from=1794741230000&orgId=1&to=1794783266760 [23:13:58] !log sudo mongosh --eval 'db.adminCommand({ setFeatureCompatibilityVersion: "8.0", confirm: true })' on graylog161 [23:13:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115231358-ffe64b9f [23:15:31] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/c2aff7c3b5ddf6912b4cb365aa06651bca80b61a [23:15:31] 02puppet/03main 07renovate[bot] 03c2aff7c chore(deps): update dependency mongodb/mongo to v8.3.17 (#5174)… [23:15:31] [02puppet] 07Universal-Omega merged 07renovate[bot]'s pull request #5174: chore(deps): update dependency mongodb/mongo to v8.3.17 (03main...03renovate/mongodb) 13https://github.com/miraheze/puppet/pull/5174 [23:15:32] [02puppet] 07Universal-Omega 04deleted 03renovate/mongodb at 0305437b3 13https://api.github.com/repos/miraheze/puppet/commit/05437b3 [23:18:41] PROBLEM - graylog161 Puppet on graylog161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 2 minutes ago with 1 failures [23:27:36] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/0524e1affd319926973957db544192a108bac717 [23:27:36] 02puppet/03main 07CosmicAlpha 030524e1a mongodb: use trixie repo [23:28:09] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/a03c076fe05a90182320add126428e76d71de5e6 [23:28:09] 02puppet/03main 07CosmicAlpha 03a03c076 mongodb: use 8.0 key [23:28:42] PROBLEM - graylog161 Puppet on graylog161 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mongodb-server-8.3.gpg] [23:30:41] PROBLEM - graylog161 Puppet on graylog161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 1 minute ago with 1 failures [23:35:28] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/99739ae837d384de3ec4c4f5a4e1529e1e4e5234 [23:35:28] 02puppet/03main 07CosmicAlpha 0399739ae Revert [23:36:42] RECOVERY - graylog161 Puppet on graylog161 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [23:39:57] [02puppet] 07Universal-Omega created 03mongodb-apt (+1 new commit) 13https://github.com/miraheze/puppet/commit/b68ae2a05024 [23:39:57] 02puppet/03mongodb-apt 07CosmicAlpha 03b68ae2a mongodb: rename apt key [23:40:05] [02puppet] 07Universal-Omega opened pull request #5175: mongodb: rename apt key (03main...03mongodb-apt) 13https://github.com/miraheze/puppet/pull/5175 [23:40:47] [02puppet] 07Universal-Omega pushed 1 new commit to 03mongodb-apt 13https://github.com/miraheze/puppet/commit/c53dcc86693ed3dbe96ee77baade1b1177b19ed8 [23:40:47] 02puppet/03mongodb-apt 07CosmicAlpha 03c53dcc8 Rename [23:41:16] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/6e946529de618a370134f749f2df7e5e0222bf18 [23:41:17] 02puppet/03main 07CosmicAlpha 036e94652 mongodb: rename apt key [23:41:18] [02puppet] 07Universal-Omega merged pull request #5175: mongodb: rename apt key (03main...03mongodb-apt) 13https://github.com/miraheze/puppet/pull/5175 [23:41:20] [02puppet] 07Universal-Omega 04deleted 03mongodb-apt at 03c53dcc8 13https://api.github.com/repos/miraheze/puppet/commit/c53dcc8 [23:52:47] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.72, 3.11, 2.07 [23:53:06] !log sudo mongosh --eval 'db.adminCommand({ setFeatureCompatibilityVersion: "8.3", confirm: true })' on graylog161 [23:53:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log/2026-11#sal-20261115235306-fe5f31f9 [23:53:55] [02landing] 07renovate[bot] fast-forwarded 03main from 03732ccc2 to 03cf17f8a 13https://github.com/miraheze/landing/compare/732ccc2c6e30...cf17f8a4e97d [23:53:56] [02landing] 07renovate[bot] 04deleted 03renovate/all-deps at 03cf17f8a 13https://api.github.com/repos/miraheze/landing/commit/cf17f8a [23:54:44] miraheze/landing - renovate[bot] the build passed. [23:54:47] RECOVERY - os151 Current Load on os151 is OK: LOAD OK - total load average: 2.28, 2.81, 2.09 [23:59:27] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period http://grafana.wikitide.net/d/GtxbP1Xnk?from=1794783350000&orgId=1&to=1794787166770 [23:59:28] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period http://grafana.wikitide.net/d/GtxbP1Xnk?from=1794741230000&orgId=1&to=1794787166770