[05:18:01] You know when things report 200 you think they will work, but apparently in MediaWiki status code 200 means do not deploy... [05:18:35] wha [05:18:42] you mean mwdeploy? [05:19:09] context: MediaWiki rendering reported 200 so I deployed but that 200 was actually an error, and it should have been reporting 410 apparently [05:19:30] icinga and mwdeploy, nothing indicated that should have gone wrong. [05:19:39] [1/2] mmmmmmmmmmmmmm [05:19:39] [2/2] https://cdn.discordapp.com/attachments/1006789349498699827/1564508857689186414/image.png?ex=6ae185ea&is=6ae0346a&hm=fffb06a0ac18c28ae3528bf237a34c70fc59965396d885f088914afb60766fdb& [05:20:02] it said error: status code 200, I took that as meaning error with error not that the status code 200 would actually bring things down lol [05:20:45] Anyway, we now have 3 new MediaWiki servers in production. [05:21:58] And I spent the last 5 hours rebuilding mcrouter from source because apparently we never had a version compatible with Trixie and we're still using bookwork version on our current mw servers, so new servers I had to correct that... [05:22:11] mcrouter? [05:22:17] huh.. [05:23:27] [1/2] The thing that hasn't had an actual release in 7 years but they still push changes, just not for trixie and with no release it's just for Facebook... [05:23:27] [2/2] https://github.com/facebook/mcrouter [05:23:46] what is mcrouter [05:24:10] we use it for memcached to route requests to the 4 different memcached server. It stands for memcached router. [05:26:09] Even non tech folks should recognize what a bananas undertaking this is. Hats off to CA for doing this. [05:27:03] how have we been using an incompatible version [05:27:15] Since 2023 [05:27:44] not just recently [05:27:48] I think the question was how on earth was it working, not how long we were bandaiding [05:28:05] I know, was getting to that lol [05:28:59] We should be reinstalling servers not doing in place upgrades for Debian major upgrades because that would have prevented this. It was still using packages already installed on bookworm but that don't work when first installed on Trixie. [05:29:30] wouldnt that have [05:29:32] caused it to not work [05:29:35] and then [05:29:36] well [05:29:38] 💥 [05:30:00] Yes but that would have been better to find out then then now when I thought I would launch servers in 30 minutes not 5 and a half hours lol [05:30:01] Ahhh, so in-place installs aren't checking package compatibility then... Hm [05:32:35] ManageWiki still uses OOUI not Codex right [06:06:49] Simple footer icon addition PR if anyone wants to merge, feel free: https://github.com/miraheze/mw-config/pull/6611 [06:16:27] Merged. Deploy in progress [06:16:34] Thanks. [06:16:41] No problem [06:17:47] 24 servers to deploy to now lol. [06:17:57] Epic. [06:18:03] Will be 25 tomorrow [06:18:13] Adding more? [06:18:44] One more, will be splitting mwdeploy out of mwtask181 to its own + probably launching some more bets infrastructure. [06:19:20] Any luck on that docker image for a beta infra? Or that's lower on the ladder rungs rn? [06:19:49] Lower right now. I have to many things to do lol [06:21:46] I am working on reducing our downtime which is very ironic because two changes I did to prevent it so much today ended up causing some.... [06:22:41] Well, sometimes uptime requires a little bit of downtime. [06:22:45] At least it's not like php2shell. [06:23:21] yeah... [06:34:21] That was a wild time we don't want to happen again. [06:43:05] @cosmicalpha CI is failing for `mw-config`, PKSA advisory for phpnit `13.4.3`, it appears to be resolved in 13.4.5, can I just bump it to 13.4.5 in a PR? [06:43:34] Sure [06:44:26] I am done with things for the night but I can merge that since it doesn't need to be deployed. Ping me when you have PR [06:45:02] PR made: https://github.com/miraheze/mw-config/pull/6612 [06:45:30] Hopefully the tests will pass. [06:45:41] Yep, lint succeeded. [06:46:34] Thanks! [06:46:45] And night. I am going to sleep earlier tonight lol [06:47:05] It's lucky that I just happened to notice CI failing with my latest change. [06:47:13] Good night, and sleep well. Thanks for all your work! [06:49:27] That advisory was only published three hours ago FYI, lmao. [14:20:36] Tried moving a TemplateStyles stylesheet and it caused a Fatal Error: `[7f053b0ee2e84bbdd7f11e54] 2026-10-27 14:20:01: Fatal exception of type "Wikimedia\NormalizedException\NormalizedException` [14:22:31] graylog log anything challenge (impossible) [14:40:56] [1/16] ``` [14:40:56] [2/16] [7f053b0ee2e84bbdd7f11e54] /wiki/Special:MovePage?action=submit Wikimedia\NormalizedException\NormalizedException: Failed to create dummy revision on Main Page/styles.css (page ID 677) [14:40:56] [3/16] from /srv/mediawiki/1.46/includes/Storage/PageUpdater.php(737) [14:40:57] [4/16] #0 /srv/mediawiki/1.46/includes/Page/MovePage.php(965): MediaWiki\Storage\PageUpdater->saveDummyRevision(string, int) [14:40:57] [5/16] #1 /srv/mediawiki/1.46/includes/Page/MovePage.php(665): MediaWiki\Page\MovePage->moveToInternal(MediaWiki\User\User, MediaWiki\Title\Title, string, bool, array) [14:40:57] [6/16] #2 /srv/mediawiki/1.46/includes/Page/MovePage.php(495): MediaWiki\Page\MovePage->moveUnsafe(MediaWiki\User\User, string, bool, array) [14:40:58] [7/16] #3 /srv/mediawiki/1.46/includes/Specials/SpecialMovePage.php(1011): MediaWiki\Page\MovePage->moveIfAllowed(MediaWiki\User\User, string, bool) [14:40:58] [8/16] #4 /srv/mediawiki/1.46/includes/Specials/SpecialMovePage.php(190): MediaWiki\Specials\SpecialMovePage->doSubmit() [14:40:58] [9/16] #5 /srv/mediawiki/1.46/includes/SpecialPage/SpecialPage.php(730): MediaWiki\Specials\SpecialMovePage->execute(null) [14:40:59] [10/16] #6 /srv/mediawiki/1.46/includes/SpecialPage/SpecialPageFactory.php(1711): MediaWiki\SpecialPage\SpecialPage->run(null) [14:40:59] [11/16] #7 /srv/mediawiki/1.46/includes/Actions/ActionEntryPoint.php(505): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, MediaWiki\Context\RequestContext) [14:41:00] [12/16] #8 /srv/mediawiki/1.46/includes/Actions/ActionEntryPoint.php(145): MediaWiki\Actions\ActionEntryPoint->performRequest() [14:41:00] [13/16] #9 /srv/mediawiki/1.46/includes/MediaWikiEntryPoint.php(180): MediaWiki\Actions\ActionEntryPoint->execute() [14:41:01] [14/16] #10 /srv/mediawiki/config/initialise/entrypoints/index.php(76): MediaWiki\MediaWikiEntryPoint->run() [14:41:01] [15/16] #11 {main}``` [14:41:02] [16/16] seems like graylog was just delayed [14:41:07] anyway what to do about that idk [16:34:10] use sentry instead [16:34:25] actually sentry is worse because I can't search request IDs tbh [16:34:30] but that's more of a me problem [20:46:16] Will be working on this later today btw. [20:58:06] @cosmicalpha you up? [20:58:13] Yes [20:58:31] Do you seen missing i18n messages if you click thanks on the edit on https://dandysworld.miraheze.org/wiki/Main_Page?diff=prev&oldid=15905 [20:58:37] I don't see meta affected though [20:58:40] @cosmicalpha [20:59:46] Is the message modified (even English modified)? [21:00:20] Nope [21:00:21] No [21:00:25] https://dandysworld.miraheze.org/wiki/MediaWiki:Thanks-confirmation2 [21:00:30] It sees it there [21:01:09] Does thanks use the allmessages API? That might explain it. We did a patch for 1.46 to fix similar issues, but maybe it didn't work on the API. It could somewhat related even if it's not the same issue though. [21:02:41] It shouldn't do [21:02:59] I see no reason it wouldn't just do the same as everything else [21:03:09] It's a standard part of the interface [21:03:33] I don't know but this is very odd. It may have something to do with me rebuilding the localization cache yesterday to launch mw211, mw212, and mw213... [21:03:53] @cosmicalpha is it worth forcing a redeploy [21:04:08] I can do it now if you want [21:04:08] And just deploying mediawiki with --l10n --ignore-time [21:04:21] @cosmicalpha ye, just redeploy everything [21:04:27] make sure to do --ignore-time [21:04:38] which is a terribly named parameter [21:04:58] I think we've had this before with new servers [21:04:58] Doing [21:05:02] Stuff has to be done twice [21:05:14] I did it three times yesterday [21:05:27] Maybe four times is the charm then [21:05:45] But I just deployed what was on mwtask181 with --ignore-time to the new servers maybe it needed a full rebuild. [21:06:29] Didn't seem to work @rhinosf1 [21:06:34] still same issue [21:07:12] is it a cache [21:07:46] It could be. I really have no idea... [21:07:52] I feel like we've had this before [21:08:11] @paladox is your memory better than mine? [21:09:01] @cosmicalpha is there anything kept in /srv/mediawiki/cache that keeps the i18n information that hasn't been reset by your deploy [21:09:16] There should not be, no. [21:09:42] regenerate the i18n files i guess? [21:09:45] delete [21:09:58] Oh wait yeah let me try that. [21:10:09] --ignore-time should have forced whatever was there to have been overridden though [21:10:18] We can try but if that works then rsync is lying [21:10:37] it could be the extension message php file (i forgot it off heart) is out of date or something [21:10:45] i know it doesn't seem to regenerate [21:10:51] unless you delete it manually [21:11:14] That's a bug then [21:11:20] @cosmicalpha ^ [21:11:39] I will look at that. [21:16:04] Seems to be fixed. [21:16:53] I didn't regenerate extension message files though, just deleted everything in /srv/mediawiki/cache/1.46/l10n and rebuilt [21:16:58] @rhinosf1 [21:18:03] Ok [21:19:30] Now I should probably get to writing new docs for the lb-ctl tool I wrote... [21:19:33] Still seems broken on https://dandysworld.miraheze.org/wiki/Main_Page?diff=prev&oldid=15905 for me [21:20:01] Worked when I added another url param so it must be cache. [21:20:05] And https://dandysworld.miraheze.org/wiki/Bassie%27s_Store?diff=prev&oldid=15596 [21:20:23] I just tried a completely random page so it's a fun cache @cosmicalpha [21:21:04] https://cdn.discordapp.com/attachments/1006789349498699827/1564750808858230945/Screenshot_20261027_152027_Kiwi_Browser.png?ex=6ae26740&is=6ae115c0&hm=478135238b86fec536d85a031482b5b38d153842de267674c7979fe46aeb7088& [21:21:12] Weird [21:21:18] What cache is doing that then [21:21:28] resource loader probably [21:21:40] Might need to purge it then [21:22:35] It might be fixed [21:23:56] Seems fixed on my end! Thank you :d_minoriheart: [21:52:12] Yeah I've seen this bug before. Deploy with `--l10n` but still see missing messages because RL uses a separate cache. There is a small window of time where i18n string misses will be cached by RL which can persist for hours if not more. [21:58:37] Do we plan to create another server for processing logs or simply sample nginx logs (e.g. log 1 out of every 5 requests) or do something else? [21:58:52] Yes I plan to create another Graylog server [21:59:21] Redundancy is better anyway. There is a chance we will have to do both [22:02:40] I sent a comment. I can change it myself and merge if you'd like.