[00:00:59] <MacFanBot>	 [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/7d5a7c04cf10b81c60d4c6d4cfd4032a3539455d
[00:01:00] <MacFanBot>	 02statichelp/03main 07WikiTideBot 037d5a7c0 Bot: Auto-update Tech namespace pages 2026-04-09 00:00:57
[02:25:28] <MacFanBot>	 [02mw-config] 07lihaohong6 commented on pull request #6366: There are some changes related to temporary accounts in https://github.com/wikimedia/mediawiki-extensions-Echo/commit/dc03a1a#diff-67493371a84598f281d87ee7e0f2484R804 which won't matter for us for now. […] 13https://github.com/miraheze/mw-config/pull/6366#issuecomment-4211043048
[03:36:31] <icinga-miraheze>	 PROBLEM - bast181 Current Load on bast181 is CRITICAL: LOAD CRITICAL - total load average: 10.60, 4.69, 1.85
[03:36:41] <icinga-miraheze>	 PROBLEM - bast181 PowerDNS Recursor on bast181 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[03:38:24] <icinga-miraheze>	 PROBLEM - bast181 Puppet on bast181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[03:38:42] <icinga-miraheze>	 RECOVERY - bast181 PowerDNS Recursor on bast181 is OK: DNS OK: 0.652 seconds response time. bast181.wikitide.net returns 2602:294:0:b12::102,38.46.223.202
[03:40:19] <icinga-miraheze>	 RECOVERY - bast181 Puppet on bast181 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures
[03:41:09] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of git_pull_ssl on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_ssl
[03:41:21] <icinga-miraheze>	 PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.45],Exec[git_pull_3d2png]
[03:41:40] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of git_pull_puppet on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_puppet
[03:42:34] <icinga-miraheze>	 PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.45],Exec[git_pull_3d2png]
[03:44:32] <icinga-miraheze>	 RECOVERY - bast181 Current Load on bast181 is OK: LOAD OK - total load average: 0.21, 2.93, 2.66
[03:49:09] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of git_pull_ssl on puppet181 is OK: OK: Status of the systemd unit git_pull_ssl
[03:49:40] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of git_pull_puppet on puppet181 is OK: OK: Status of the systemd unit git_pull_puppet
[04:07:40] <icinga-miraheze>	 RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[04:10:43] <icinga-miraheze>	 RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[12:13:03] <MacFanBot>	 [02DiscordNotifications] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/DiscordNotifications/commit/c974f46a421db17f01356a8759df0c30fce30a3b
[12:13:03] <MacFanBot>	 02DiscordNotifications/03main 07translatewiki.net 03c974f46 Localisation updates from https://translatewiki.net.
[12:13:04] <MacFanBot>	 [02CreateWiki] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/CreateWiki/commit/77384446ecae622ebeab871f9d112b9be08e4f0c
[12:13:04] <MacFanBot>	 02CreateWiki/03main 07translatewiki.net 037738444 Localisation updates from https://translatewiki.net.
[12:13:06] <MacFanBot>	 [02RemovePII] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/RemovePII/commit/8eb4023bad35d4ac3592d365b191e0da45ea88cb
[12:13:06] <MacFanBot>	 02RemovePII/03main 07translatewiki.net 038eb4023 Localisation updates from https://translatewiki.net.
[12:13:08] <MacFanBot>	 [02ManageWiki] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/214be48b12dd5929074d454f7d3eec97c2ab5e00
[12:13:10] <MacFanBot>	 [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/835276dcc82dde38f21800b352ffb05739885096
[12:13:12] <MacFanBot>	 02ManageWiki/03main 07translatewiki.net 03214be48 Localisation updates from https://translatewiki.net.
[12:13:13] <MacFanBot>	 02MirahezeMagic/03main 07translatewiki.net 03835276d Localisation updates from https://translatewiki.net.
[12:18:00] <miraheze-github>	 miraheze/RemovePII - translatewiki the build passed.
[12:19:12] <miraheze-github>	 miraheze/ManageWiki - translatewiki the build passed.
[12:20:05] <miraheze-github>	 miraheze/MirahezeMagic - translatewiki the build passed.
[12:20:48] <miraheze-github>	 miraheze/DiscordNotifications - translatewiki the build passed.
[12:21:08] <miraheze-github>	 miraheze/CreateWiki - translatewiki the build passed.
[12:27:53] <icinga-miraheze>	 PROBLEM - mwtask181 Disk Space on mwtask181 is WARNING: DISK WARNING - free space: / 21997MiB (9.9% inode=95%);
[13:15:39] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 35.92, 16.90, 7.76
[13:17:57] <icinga-miraheze>	 PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 3 backends are down. mw172 mw182 mw183
[13:19:57] <icinga-miraheze>	 RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy
[13:29:57] <icinga-miraheze>	 PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 9 backends are down. mw162 mw171 mw182 mw153 mw173 mw183 mw191 mw193 mw201
[13:30:43] <icinga-miraheze>	 PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 7 backends are down. mw162 mw172 mw163 mw173 mw183 mw202 mw203
[13:30:46] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/27bb63018f9184bac2ed94f0f7228e5f476b8e4a
[13:30:46] <MacFanBot>	 02ssl/03main 07WikiTideBot 0327bb630 Bot: Auto-update domain lists
[13:30:49] <icinga-miraheze>	 PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 5 backends are down. mw162 mw182 mw163 mw191 mw203
[13:31:17] <icinga-miraheze>	 PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 10 backends are down. mw171 mw172 mw182 mw153 mw163 mw173 mw183 mw191 mw193 mw201
[13:31:32] <icinga-miraheze>	 PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[13:31:32] <RhinosF1>	 Huh
[13:31:33] <RhinosF1>	 No
[13:31:39] <RhinosF1>	 Oh
[13:31:42] <RhinosF1>	 A database
[13:31:43] <RhinosF1>	 Fuck
[13:32:15] <icinga-miraheze>	 PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received
[13:32:17] <icinga-miraheze>	 PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[13:33:07] <icinga-miraheze>	 PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[13:33:57] <icinga-miraheze>	 RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy
[13:34:13] <icinga-miraheze>	 RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.080 second response time
[13:34:17] <icinga-miraheze>	 RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 5.334 second response time
[13:34:43] <icinga-miraheze>	 RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy
[13:34:49] <icinga-miraheze>	 RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy
[13:35:06] <icinga-miraheze>	 RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.809 second response time
[13:35:17] <icinga-miraheze>	 RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy
[13:35:29] <icinga-miraheze>	 RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 4.567 second response time
[13:36:02] <MirahezeRelay>	 <abaddriverlol> @rhinosf1 still an issue?
[13:36:19] <MirahezeRelay>	 <rhinosf1, replying to abaddriverlol> I haven't seen the db recover so probably
[13:36:27] <MirahezeRelay>	 <rhinosf1> load on db171 alerted first
[13:36:33] <MirahezeRelay>	 <abaddriverlol> which cluster again
[13:36:45] <MirahezeRelay>	 <rhinosf1> c3
[13:36:57] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'config': True, 'force': True} to all
[13:37:19] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 22s
[13:37:23] <MirahezeRelay>	 <abaddriverlol> !log depool c3
[13:37:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:37:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:37:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:38:10] <MirahezeRelay>	 <abaddriverlol> load average: 30.19, 48.36, 37.38
[13:38:12] <MirahezeRelay>	 <abaddriverlol> on db171
[13:38:39] <MirahezeRelay>	 <rhinosf1> @abaddriverlol is it coming down or has it stalled?
[13:38:48] <MirahezeRelay>	 <abaddriverlol> 14.3
[13:38:49] <MirahezeRelay>	 <abaddriverlol> going down
[13:39:18] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 53%
[13:39:34] <MirahezeRelay>	 <rhinosf1> @abaddriverlol any sign of anything running that could have caused it?
[13:39:39] <MirahezeRelay>	 <rhinosf1> If not, let it go back to normal and repool and hope it's calmed
[13:39:48] <MirahezeRelay>	 <abaddriverlol> no idea, I forgot to dump a proclist
[13:39:56] <MirahezeRelay>	 <abaddriverlol> and I think we have slowlogs disabled
[13:40:09] <MirahezeRelay>	 <rhinosf1> We do
[13:40:24] <MirahezeRelay>	 <rhinosf1> Let load go back to normal levels then and repool
[13:40:46] <MirahezeRelay>	 <abaddriverlol> load is at 2.12 in the last minute
[13:40:54] <MirahezeRelay>	 <abaddriverlol> 1.79
[13:40:59] <MirahezeRelay>	 <rhinosf1> That sounds good
[13:41:06] <MirahezeRelay>	 <abaddriverlol> will repool
[13:41:07] <MirahezeRelay>	 <rhinosf1> Cross your fingers then
[13:41:27] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'config': True} to all
[13:41:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:41:49] <MirahezeRelay>	 <abaddriverlol> !log repool c3
[13:41:52] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 24s
[13:42:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:42:08] <MirahezeRelay>	 <abaddriverlol> load is going up again it seems
[13:42:44] <MirahezeRelay>	 <abaddriverlol> got a proclist
[13:43:18] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 89%
[13:43:23] <MirahezeRelay>	 <rhinosf1> Shit
[13:43:23] <MirahezeRelay>	 <rhinosf1> Can you get a proc list?
[13:43:24] <MirahezeRelay>	 <rhinosf1> That means it's probably traffic related
[13:43:34] <MirahezeRelay>	 <abaddriverlol> going down again from 40 to 34
[13:43:45] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'config': True, 'force': True} to all
[13:43:47] <MirahezeRelay>	 <abaddriverlol> nvm
[13:43:49] <MirahezeRelay>	 <rhinosf1, replying to abaddriverlol> Anything stand out
[13:43:54] <MirahezeRelay>	 <abaddriverlol> hold on
[13:44:05] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 20s
[13:44:11] <MirahezeRelay>	 <abaddriverlol> !log depool c3 again
[13:44:48] <MirahezeRelay>	 <abaddriverlol> https://issue-tracker.miraheze.org/P588
[13:45:07] <MirahezeRelay>	 <abaddriverlol> lets continue on MM
[13:45:18] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 46%
[13:45:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:45:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:45:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:45:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:51:46] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'config': True} to all
[13:51:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:52:09] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 23s
[13:52:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:52:14] <MirahezeRelay>	 <abaddriverlol> !log repool c3 again
[13:52:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:53:18] <icinga-miraheze>	 RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 10%
[14:01:39] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 0.41, 1.36, 10.76
[14:03:39] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 0.53, 1.10, 9.52
[14:11:40] <icinga-miraheze>	 PROBLEM - swiftproxy161 Current Load on swiftproxy161 is CRITICAL: LOAD CRITICAL - total load average: 9.20, 8.07, 4.58
[14:13:40] <icinga-miraheze>	 RECOVERY - swiftproxy161 Current Load on swiftproxy161 is OK: LOAD OK - total load average: 2.73, 6.00, 4.24
[17:12:31] <MacFanBot>	 [02mediawiki-repos] 07SomeMWDev created 03remove-bucket-migration-patch (+1 new commit) 13https://github.com/miraheze/mediawiki-repos/commit/028ae9f88e63
[17:12:31] <MacFanBot>	 02mediawiki-repos/03remove-bucket-migration-patch 07SomeRandomDeveloper 03028ae9f T14235: Remove bucket migration maintenance script patch
[17:12:47] <MacFanBot>	 [02mediawiki-repos] 07SomeMWDev drafted pull request #140: T14235: Remove bucket migration maintenance script patch (03main...03remove-bucket-migration-patch) 13https://github.com/miraheze/mediawiki-repos/pull/140
[17:12:50] <MacFanBot>	 [02mediawiki-repos] 07SomeMWDev readied pull request #140: T14235: Remove bucket migration maintenance script patch (03main...03remove-bucket-migration-patch) 13https://github.com/miraheze/mediawiki-repos/pull/140
[17:13:14] <MacFanBot>	 [02mediawiki-repos] 07SomeMWDev merged pull request #140: T14235: Remove bucket migration maintenance script patch (03main...03remove-bucket-migration-patch) 13https://github.com/miraheze/mediawiki-repos/pull/140
[17:13:15] <MacFanBot>	 [02mediawiki-repos] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mediawiki-repos/commit/fa2abc6d584e377b428f684f9c36ddc980f4f429
[17:13:17] <MacFanBot>	 02mediawiki-repos/03main 07SomeRandomDeveloper 03fa2abc6 T14235: Remove bucket migration maintenance script patch (#140)
[17:13:18] <MacFanBot>	 [02mediawiki-repos] 07SomeMWDev 04deleted 03remove-bucket-migration-patch at 03028ae9f 13https://api.github.com/repos/miraheze/mediawiki-repos/commit/028ae9f
[17:18:09] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'versions': '1.45', 'upgrade_extensions': 'Bucket'} to test151
[17:18:10] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'versions': '1.45', 'upgrade_extensions': 'Bucket'} to test151 - SUCCESS in 0s
[17:18:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:18:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:18:17] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'force_upgrade': True, 'versions': '1.45', 'upgrade_extensions': 'Bucket'} to test151
[17:18:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:19:50] <MirahezeRelay>	 <abaddriverlol> [1/7] ```
[17:19:51] <MirahezeRelay>	 <abaddriverlol> [2/7] Upgrading Bucket
[17:19:51] <MirahezeRelay>	 <abaddriverlol> [3/7] Execute: sudo -u www-data git -C /srv/mediawiki-staging/1.45/extensions/Bucket apply --check /srv/mediawiki-staging/patches/public/bucket-avoid-grantall-T14235.patch
[17:19:51] <MirahezeRelay>	 <abaddriverlol> [4/7] error: patch failed: includes/BucketDatabase.php:151
[17:19:52] <MirahezeRelay>	 <abaddriverlol> [5/7] error: includes/BucketDatabase.php: patch does not apply
[17:19:52] <MirahezeRelay>	 <abaddriverlol> [6/7] ```
[17:19:52] <MirahezeRelay>	 <abaddriverlol> [7/7] did we ever upstream this?
[17:56:33] <MirahezeRelay>	 <posix_memalign, replying to abaddriverlol> [1/2] I don't think so. The last conversation was asking WG maintainers about converting the grant all to a grant of about 7 or 8 permissions, which are Miraheze defaults and worked for our use case. The response was that it sounds like an acceptable solution.
[17:56:33] <MirahezeRelay>	 <posix_memalign, replying to abaddriverlol> [2/2] No issue or PR was submitted afterwards.
[17:57:11] <MirahezeRelay>	 <abaddriverlol> Then we should probably do that
[17:57:16] <MirahezeRelay>	 <abaddriverlol> Since the patch doesn't apply anymore for some reason
[18:02:15] <MirahezeRelay>	 <posix_memalign> Yeah. Really odd that the patch fails on an unchanged file.
[18:03:20] <MirahezeLSBot>	 !log [petramagna@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php CirrusSearch:ForceSearchIndex --wiki=robeatswiki (END - exit=0)
[18:03:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:03:39] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'force_upgrade': True, 'versions': '1.45', 'upgrade_extensions': 'Bucket'} to test151
[18:03:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:03:50] <MirahezeRelay>	 <posix_memalign> I'll create a task and whoever claims it gets the fun task of upstreaming I guess.
[18:04:33] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'force_upgrade': True, 'versions': '1.45', 'upgrade_extensions': 'Bucket'} to test151
[18:04:34] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'force_upgrade': True, 'versions': '1.45', 'upgrade_extensions': 'Bucket'} to test151 - SUCCESS in 0s
[18:04:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:04:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:04:44] <MirahezeRelay>	 <abaddriverlol> a git reset --hard fixed it
[18:04:47] <MirahezeRelay>	 <abaddriverlol> still should be upstreamed
[18:06:58] <MirahezeRelay>	 <posix_memalign, replying to abaddriverlol> I think mwdeploy does not know that it should revert local patches (which are applied as local file changes instead of commits) before deploying/upgrading, so it's always necessary to clean up the repo manually.
[18:07:35] <MirahezeRelay>	 <abaddriverlol> Oh right, that would make sense
[18:08:23] <MirahezeRelay>	 <posix_memalign> Task: https://issue-tracker.miraheze.org/T15221
[18:08:40] <MirahezeRelay>	 <abaddriverlol> Thx
[18:24:19] <MirahezeRelay>	 <posix_memalign> [1/2] I'm reindexing CirrusSearch for `bolanqiuwiki`, `holidayswiki`, and `blobwiki` in light of the issues in https://issue-tracker.miraheze.org/T14896.
[18:24:19] <MirahezeRelay>	 <posix_memalign> [2/2] It's not necessary for `bolanqiuwiki` because they had a different error (the more well-known one well search completely errors), so I already reindexed yesterday, but the script has already started so I'll let it finish. IIRC it's safe to interrupt but doesn't hurt to reindex.
[18:25:29] <MirahezeLSBot>	 !log [petramagna@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php CirrusSearch:ForceSearchIndex --wiki=bolanqiuwiki (END - exit=0)
[18:25:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:28:55] <MirahezeLSBot>	 !log [petramagna@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php CirrusSearch:ForceSearchIndex --wiki=blobwiki (END - exit=256)
[18:28:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:30:31] <MirahezeLSBot>	 !log [petramagna@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php ManageWiki:ResetWikiCaches --wiki=blobwiki (END - exit=0)
[18:30:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:31:27] <MirahezeRelay>	 <mudscape, replying to abaddriverlol> I'm doing bucket stuff today anyway, if you can create an issue on GitHub with a link to the patch file I can do it
[18:31:38] <MirahezeRelay>	 <posix_memalign, replying to MirahezeLSBot> [1/4] ```
[18:31:38] <MirahezeRelay>	 <posix_memalign, replying to MirahezeLSBot> [2/4] The "CirrusSearch" extension must be installed for this script to run. Please enable it and then try again.
[18:31:38] <MirahezeRelay>	 <posix_memalign, replying to MirahezeLSBot> [3/4] ```
[18:31:38] <MirahezeRelay>	 <posix_memalign, replying to MirahezeLSBot> [4/4] That's crazy. Cache not caching up on mwtask servers more than 10 hours after an extension is enabled.
[18:32:31] <MirahezeRelay>	 <abaddriverlol, replying to mudscape> The patch remove all grants <https://github.com/miraheze/mediawiki-repos/blob/main/patches/public/bucket-avoid-grantall-T14235.patch> which I assume wouldn't work with WG's setup, somebody needs to write a new one that applies specific grants
[18:32:33] <MirahezeLSBot>	 !log [petramagna@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php CirrusSearch:ForceSearchIndex --wiki=blobwiki (END - exit=0)
[18:32:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:46:15] <MirahezeRelay>	 <mudscape, replying to abaddriverlol> hmmmm I see. I'll let one of y'all handle it then
[19:08:07] <MacFanBot>	 [02mw-config] 07SomeMWDev commented on pull request #6366: @lihaohong6 this was backported in https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Echo/+/1269536 13https://github.com/miraheze/mw-config/pull/6366#issuecomment-4216821670
[19:08:29] <MirahezeLSBot>	 !log [petramagna@mwtask161] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importDump --wiki=maillewiki --no-updates --username-prefix=wgg:maille --uploads maille_20260409_rv2.xml (START)
[19:08:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:08:44] <MirahezeLSBot>	 !log [petramagna@mwtask161] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importDump --wiki=maillewiki --no-updates --username-prefix=wgg:maille --uploads maille_20260409_rv2.xml (END - exit=256)
[19:08:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:12:03] <MirahezeLSBot>	 !log [petramagna@mwtask161] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importDump --wiki=maillewiki --no-updates --username-prefix=wgg:maille --uploads maille_20260409_rv2.xml (START)
[19:12:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:12:13] <MirahezeLSBot>	 !log [petramagna@mwtask161] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importDump --wiki=maillewiki --no-updates --username-prefix=wgg:maille --uploads maille_20260409_rv2.xml (END - exit=256)
[19:12:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:13:15] <MirahezeLSBot>	 !log [petramagna@mwtask161] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importDump --wiki=maillewiki --no-updates --username-prefix=wgg:maille --uploads maille_20260409_rv2.xml (START)
[19:13:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:14:48] <MirahezeLSBot>	 !log [petramagna@mwtask161] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importDump --wiki=maillewiki --no-updates --username-prefix=wgg:maille --uploads maille_20260409_rv2.xml (END - exit=0)
[19:14:51] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:16:33] <MirahezeLSBot>	 !log [petramagna@mwtask161] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php rebuildall --wiki=maillewiki (START)
[19:16:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:17:38] <MirahezeLSBot>	 !log [petramagna@mwtask161] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php rebuildall --wiki=maillewiki (END - exit=0)
[19:17:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:18:07] <MirahezeLSBot>	 !log [petramagna@mwtask161] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initEditCount --wiki=maillewiki (END - exit=0)
[19:18:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:18:17] <MirahezeLSBot>	 !log [petramagna@mwtask161] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initSiteStats --wiki=maillewiki (END - exit=0)
[19:18:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:17:08] <MirahezeRelay>	 <zipppee, replying to posix_memalign> I wouldn't expect it to given that it doesn't have web requests
[20:18:32] <MirahezeRelay>	 <abaddriverlol, replying to zipppee> there are also other issues which cause stale data to be kept in the cache for longer
[20:18:47] <MirahezeRelay>	 <abaddriverlol> e.g. stuff like <https://github.com/miraheze/mw-config/pull/6360>
[20:18:48] <MacFanBot>	 [GitHub] [miraheze/mw-config #6360] open PR by SomeMWDev, created 4 days, 5 minutes ago: MirahezeFunctions: Don't write stale extension data back into the config cache | (empty comment)
[20:19:14] <MirahezeRelay>	 <abaddriverlol> (assuming that's the main cause, not sure)
[20:19:24] <MirahezeRelay>	 <zipppee> I should probably merge that into WO
[20:20:00] <MirahezeRelay>	 <zipppee> but that would probably make sense
[20:20:21] <MirahezeRelay>	 <abaddriverlol, replying to zipppee> if you want to test it for me in WO prod, sure :P
[20:20:31] <MirahezeRelay>	 <zipppee, replying to abaddriverlol> gladly
[20:20:35] <MirahezeRelay>	 <zipppee> it's more stable than what I test in prod so
[20:20:42] <MirahezeRelay>	 <abaddriverlol> how many appservers do you have
[20:22:02] <MirahezeRelay>	 <zipppee> 1
[20:22:07] <MirahezeRelay>	 <zipppee> 💀 wait I merged it with the /srv paths
[20:22:10] <MirahezeRelay>	 <zipppee> and just pushed that to prod
[20:22:11] <MirahezeRelay>	 <zipppee> wtf am I doing
[20:22:23] <MirahezeRelay>	 <zipppee> a quick git reset --hard never hurt anyone
[20:22:54] <MirahezeRelay>	 <zipppee> but tbf I am looking to pool an extra 2 appservers and then a staging one
[20:23:07] <MirahezeRelay>	 <zipppee> and replace the existing one because I left that around from when I moved from the old monolith
[20:25:06] <MirahezeRelay>	 <abaddriverlol, replying to zipppee> how are you gonna test a patch which fixes cache desyncing then
[20:25:19] <MirahezeRelay>	 <zipppee, replying to abaddriverlol> we have a jobrunner server lmao
[20:25:54] <MirahezeRelay>	 <abaddriverlol> yeah but ideally this change shouldn't negatively affect caching on appservers, which we can't test this way however
[20:26:10] <MirahezeRelay>	 <abaddriverlol> but ig we can test how many maintenance script runs it takes to invalidate the cache
[20:26:12] <MirahezeRelay>	 <zipppee> well I was planning to setup new appservers tonight anyway
[20:33:59] <MirahezeRelay>	 <zipppee> Yeah I'm not sure about the patch
[21:03:19] <MirahezeLSBot>	 !log [petramagna@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php CirrusSearch:ForceSearchIndex --wiki=holidayswiki (END - exit=0)
[21:03:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:13:34] <MirahezeRelay>	 <abaddriverlol, replying to zipppee> I don't see any way for that to happen in Miraheze's version of MirahezeFunctions
[21:14:01] <MirahezeRelay>	 <zipppee, replying to abaddriverlol> Ours is not SUL3 setup yet which may have been why
[21:14:03] <MirahezeRelay>	 <abaddriverlol> oh
[21:14:04] <MirahezeRelay>	 <abaddriverlol> gg
[21:14:13] <MirahezeRelay>	 <zipppee> I need to setup SUL3 tbh
[21:14:34] <MirahezeRelay>	 <zipppee> It’s an easy(ish) change afaict but I haven’t bothered yet
[21:56:57] <MirahezeRelay>	 <kellybrooksj326> discordapp.com/invite/hcQ4R7mP6x
[21:57:18] <MirahezeRelay>	 <Dyno#3861> @kellybrooksj326 No invite links.
[22:51:53] <MirahezeRelay>	 <kellybrooksj326> discordapp.com/invite/hcQ4R7mP6x
[22:51:54] <MirahezeRelay>	 <Dyno#3861> @kellybrooksj326 No invite links.
[23:00:57] <MacFanBot>	 [02puppet] 07MacFan4000 opened pull request #4838: beta: add mw 1.46 (07miraheze:03main...07MacFan4000:03patch-77) 13https://github.com/miraheze/puppet/pull/4838
[23:04:40] <MacFanBot>	 [02RottenLinks] 07squirrelslair commented on pull request #105: I pulled the newest version (2.2.0 (62e7614)) and tried with run.php: […] 13https://github.com/miraheze/RottenLinks/pull/105#issuecomment-4218329446
[23:06:32] <MacFanBot>	 [02RottenLinks] 07MacFan4000 commented on pull request #105: you need to capitalize the U in Update 13https://github.com/miraheze/RottenLinks/pull/105#issuecomment-4218340934