[00:09:28] also the new pygments version seems to be faster: https://wikis.world/@legoktm/110297130332484253 [00:09:50] which is a very nice bonus [00:37:32] legoktm: I just realized that the base container there buster. It would likely be even faster if we switched to bullseye with python3.9 instead of 3.7. [00:37:45] :o [00:38:30] do we have bullseye+PHP containers already? [00:39:19] I'm not sure. Possibly not yet. [00:40:25] https://docker-registry.wikimedia.org/dev/buster-php81-fpm/tags/ [00:40:54] so yes, we do have one, but not quite the "right" one [00:41:08] oh. that's still buster [00:43:02] anyway, it's possible but not trivial yet. would need to bootstrap the php first. For this use case we shouldn't need to worry about fancy things like our custom modules and icu version matches though. [00:51:23] mhm. I think it would be valuable to stick with what MediaWiki is running on just for consistency [15:51:09] https://blog.packagist.com/packagist-org-maintainer-account-takeover/ password reuse :/ [20:36:21] legoktm: hi, you there? [20:37:20] hauskater: in about 10 minutes I will be [20:37:32] legoktm: ack :) [20:42:19] hauskater: okay hi :) [20:43:11] legoktm: hi - I think it was you that created the logstash dashboard called 'global rename logs', but it's no longer working after March 16 - any idea where this may come from? [20:44:43] :o if that was me that was a loooong time ago, but let's see [20:46:29] oh I think that was when Renameuser moved into core [20:46:35] so maybe the log channel names changed [20:48:50] hauskater: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/812477 is probably what broke it [20:49:02] but that's much older [20:49:42] yeah, the most recent log is from March 16, 2023 [20:50:22] cc Krinkle [20:51:03] my guess is that the entries are still being logged, just that the dashboard needs tweaking [20:51:31] * Krinkle looks [20:53:00] using `channel` instead of `channel.keyword` doesn't return any results either :( [20:56:34] * Krinkle is editing [21:00:16] * Krinkle saved [21:00:23] yeah just needds channel changing to CentralAuth [21:00:37] but.. that's a lot more messages than before so I've added a normalized_message filter for what seems to be relevant to renames [21:01:30] so no longer using channel = Renameuser? [21:05:28] hauskater: channel=CentralAuthRename is now part of channel=CentralAuth, generally our messages go by component and CentralAuth is a component. It's not really meant as symbolic name or subgroup of a few messags, that's usually what we have grafana for but yeah it's totally doable using normalized_message "is one of", which is what I did. [21:05:38] I don't know about channel=Renameuser. [21:05:47] That sounds like the non-glboal extension, which is now a core component. [21:05:54] That probably still exists as its own software component indeed [21:06:09] as its own log channel* [21:06:28] although with my edit, it'll no longer match messages in that channel [21:07:41] that channel is still being used apparently: https://codesearch.wmcloud.org/core/?q=LoggerFactory.*Renameuser&files=&excludeFiles=&repos= [21:07:50] Krinkle: and you left some quoted search on top too [21:08:11] not sure if that affects the output, I'm new to the stuff [21:08:16] There are no entries in channel=Renameuser last 7 days [21:08:25] I don't think that's used much in production mode, mostly debug-only so not in prod [21:08:42] hauskater: ack, fixed [21:09:22] It's had to really capture anything that could in theory be related to a rename. fatal errors can ocurr anywhere, so you'd want to capture e.g. exception.trace as well and filter that by code paths that may overlap with renames, etc. [21:09:22] Krinkle: hmm, it was used daily until March 16, then no entries at all :) [21:10:25] Mar 16: channel=Renameuser, message="GlobalRename: Finished rename for …… to ………" [21:10:58] https://gerrit.wikimedia.org/g/mediawiki/core/+/32ded76880f86359a0a9dc6999ed2113c8bbb9c1/includes/RenameUser/RenameuserSQL.php#366 [21:11:08] $this->logger = LoggerFactory::getInstance( 'Renameuser' ); [21:11:11] Looks pretty straight forward [21:11:30] https://gerrit.wikimedia.org/g/operations/mediawiki-config/+/58e6be6d9c88e242ad22427c8e8edc6a1211ec0e/wmf-config/InitialiseSettings.php#2144 [21:11:30] 'Renameuser' => 'debug', [21:15:43] I'm fairly sure this isn't related to my commit anyway, since that's all in core [21:16:22] Krinkle: aye, but I mean, for some reason even the GlobalRename: Finished etc messages are not being logged at all apparently? [21:16:39] "GlobalRename: Finished rename for" is a message from core [21:16:41] I linked it above [21:16:48] "GlobalRename" is injected as prefix [21:17:20] You can go on the "mediawiki" dashboard (no filters) and search `normalized_message:"Finished rename"` [21:17:25] It shows the same ones stopping March 16 [21:17:31] so it hasn't gone to a different channel either [21:17:37] nor has it gotten a different prefix [21:18:39] https://gerrit.wikimedia.org/g/mediawiki/extensions/CentralAuth/+/c8471fe7293408f0a337cc792f43274efa855a48/includes/GlobalRename/LocalRenameJob/LocalRenameUserJob.php#78 [21:19:16] https://logstash.wikimedia.org/goto/22fa99884cdfc2db91f478f48603316d [21:19:56] exactly [21:20:10] So I guess a next question could be - is the code running at all? [21:20:15] It's called by LocalRenameUserJob [21:20:30] you could check mwlog1002 and grep through job logs to see if those still run as normal [21:21:05] I don't think I have access to mwlog1002 :) [21:21:57] it should be part of the same access as deployment host / maintenance host [21:22:06] anyway, there's a few bits of it in logstash as well [21:22:14] I'm only ldap/nda :) [21:22:15] it seems a few of them at least fail with `Deadlock found when trying to get lock; try restarting transaction Function: WikiPage::insertOn Query: INSERT IGNORE INTO `page` [21:23:21] `channel: JobExecutor normalized_message:Exception executing job: LocalPageMoveJob` [21:25:01] hauskater: I suggest filing a task about the Renameuser messages, maybe try to find when the code moved into core, and which train went out on/around March 16 when it stopped and check that weeks Changelog on mediawiki.org for anything else that might stand out [21:25:17] cc myself and Tim if the core-ification of Renameuser ext went out that week [21:27:55] lots of patches at https://phabricator.wikimedia.org/T27482 :) [21:28:03] I'll file the task as suggested [21:28:17] surely these messages should be going somewhere [21:31:40] it got merged in 1.40.0-wmf.26 per https://phabricator.wikimedia.org/T330204 [21:31:50] which is before March 16, checking next trains [21:40:07] Krinkle: I see a patch that changed wfDebugLog( 'Renameuser', ... to $this->logger->info( [21:40:48] I'll file a task as instructed and link to the previous logstash data as well [22:25:32] All debug entries from RenameuserSQL.php ( $this->debug( ... ); ) no longer appear after that date. Noted on task.