[06:21:00] Hi, can somebody do a sanity check here? https://phabricator.wikimedia.org/T137382#2367087 [12:02:44] just noticed that the reload button for the pre-authmanager captcha implementation is broken [12:03:01] yay for involuntarily fixed bugs [12:34:40] bd808: how do I backfill SAL? will elasticsearch pick up page edits? [13:18:25] tgr: Did you hear that we accidentally disabled AuthManager on testwiki and test2wiki when we enabled it for group 1 yesterday? They're included in "wikipedia", sigh. [13:18:52] yeah, I saw the irc logs [13:18:57] confusing [14:38:37] tgr: the process for backfilling my https://tools.wmflabs.org/sal/ is manual and undocumented :/ [14:39:05] But it wouldn't have been broken by and authn changes I don't think [14:39:49] indeed, it has no gap [18:53:50] anomie: we had a crazy login spike today at 6:00 UTC exactly [18:54:19] graphite claims there were 2500 tries in a minute or two, all failed [18:54:24] tgr: Hmm. Sounds like someone's cron job [19:28:28] tgr: Looks like the spike was login attempts for a user "Not registered username" on stewardwiki. 11240 lines between 05:59:44 and 06:04:28. Account isn't registered, obviously. I wonder if it's some kind of thing trying to test invalid logins that got broken by AuthManager's changed error statuses. [19:31:26] that's like 30 req/s, pretty intense testing [19:32:31] I suspect it's the same old "retry with no backoff" behavior we saw in some bots when SessionManager broke stuff, usually it'd probably do one failed login then go on to whatever else it does. [19:33:27] * anomie doesn't feel like spending another half hour waiting for a zcat-and-grep on archived api.log files to check that, though [20:40:13] AaronSchulz: hi! [20:40:22] AaronSchulz: got question to you about MultiHttClient [20:43:47] sure [21:10:33] SMalyshev: maybe you figured it out already :) [21:10:51] AaronSchulz: I wanted to ask why run() is final there [21:11:06] it makes it hard to create test mocks [21:11:19] it's just a wrapper which should never do anything else...but it's fine to make it public for mocking [21:12:21] right. So if it's final I can't mock it and since everybody calls it it makes it hard [21:12:27] so maybe remove final [21:12:28] ? [21:12:42] i.e. I can remove it [21:13:44] I can mock runMulti of course but that means I need to create more complex structure inside willReturn() which is unnecessary complication [21:18:11] final / private / protected are worthless language features, IMO [21:19:59] you can signal to other developers which APIs are public and which should be considered internal using a naming convention, like prefixing private/internal methods with an underscore [21:22:48] Raymond Hettinger, a well-known figure in the Python community, put it this way: Python has no locked doors; it's a consenting adults language. If you open the door you're responsible for what you see. [21:25:24] SMalyshev: removing it is fine, yes [21:25:30] since the introduction of Reflection in PHP 5, PHP doesn't really have locked doors either, just a way to throw some barbed wire and place obstacles in the way of another developer [21:25:36] AaronSchulz: ok, cool [21:26:32] ori: I agree, I rarely find final useful... there are people that disagree about "consenting adults" principle though and prefer more B&D approach :) [21:28:20] they must have a very low opinion of the people who use their code [21:32:24] bd808: About? [21:32:38] Reedy: 0/ [21:32:45] User complaining of a stuck rename [21:32:52] They're renamed on mw.org [21:32:55] The queue doesn't agree [21:32:56] So can't login [21:33:02] Any ideas how to kick it? :) [21:33:33] https://en.wikipedia.org/wiki/Special:GlobalRenameProgress/AlessandroDiFrancesco [21:33:37] tgr might know. He was fixing some global rename things yesterday [21:33:40] https://www.mediawiki.org/wiki/Special:CentralAuth/AlessandroDiFrancesco [21:33:53] I thought I'd seen something about possibly being broken [21:34:22] there was some AuthManager difficulty I think [21:34:31] yeah, that was T135656 [21:34:31] T135656: GlobalRename is broken, presumably due to authmanager changes - https://phabricator.wikimedia.org/T135656 [21:34:52] not AuthManager specifically, but caused by some of the preparations [21:35:07] The rename looks to have worked [21:35:12] Just the rename process doesn't think so [21:35:30] maybe we should just poke the db row then? [21:35:56] ru_status enum ('queued', 'inprogress', 'failed') [21:36:35] what's it do when it's finished? [21:36:38] you are going to make me remember how this works aren't you? ;) [21:36:54] * @copyright © 2014 Bryan Davis and Wikimedia Foundation. [21:36:58] * Reedy goes to ask the Wikimedia Foundation [21:37:08] I can do by example [21:37:14] Let me have a look at the users row in the db [21:37:59] Looks like the answer is, the row disappears from _status [21:38:04] | Alessandrodifrancesco | AlessandroDiFrancesco | mediawikiwiki | inprogress | [21:38:21] if you delete the row, that will stop CA from preventing the login [21:38:35] no idea if it will screw things up somewhere else [21:38:40] heh [21:39:12] the renames looked successful to me, but I only have vague ideas what exactly they involve [21:39:33] row is deleted on success -- https://github.com/wikimedia/mediawiki-extensions-CentralAuth/blob/master/includes/GlobalRename/GlobalRenameUserStatus.php#L172-L186 [21:39:42] ok, I'll kill it hten [21:41:14] looking at the code, I thing the rename code itself has run successfully but RenameUserComplete didn't [21:41:30] which probably means extension user ids are not updated? [21:41:49] The user in question has no edits or anything on mworg [21:41:53] (usernames, because RenameUser doesn't change the ID, I think?) [21:42:14] oh, in that case just drop it [21:42:23] "There are no renames in progress for AlessandroDiFrancesco. They may have already finished." [21:42:28] there is a list of all the affected users somewhere in the bug [21:42:43] they should probably get the same treatment [21:42:46] [22:42:38] Yeeees!!! It works !!! [21:42:59] I guess, for users with no edits, we most certainly can [21:43:11] Others might need some more massaging [21:43:28] There's 120 in progress rows [21:43:30] well, 119 now [21:43:36] I think edits are updated by RenameUser itself, not the hook [21:43:38] The job that may have broken is https://github.com/wikimedia/mediawiki-extensions-CentralAuth/blob/master/includes/LocalRenameJob/LocalRenameUserJob.php [21:44:09] Is it failing in job queue? [21:44:20] 'page_namespace IN (' . NS_USER . ',' . NS_USER_TALK . ')' [21:44:24] / This should never happen! [21:44:24] / If it does happen, the user will be locked out of their account [21:44:24] / until a sysadmin intervenes... [21:44:26] Y U NO MAKELIST? [21:44:38] bd808: yes, first by RenameuserSQL line 200-ish erroring out and after that saveSettings at line 353 when it tried to update centralauth localnames [21:44:41] That should log [21:45:11] so I think the only things that did not happen were the hook call and publishing to RC [21:45:18] and cleaning up the job table [21:46:08] * bd808 blames legoktm and hoo for doing all the actual business logic for this [21:46:15] When I looked at it yesterday: For most of the users in the list, what happened was that somehow the localnames table got rows added for the new name on mediawikiwiki before Renameuser ran the hook to make CA update the row, causing unique constraint failures on the update. If anything else hooks Renameuser's hook, it might not have gotten run either. There was also one guy that had like 100 rows in the table, I have no idea what happened for [21:46:15] that one. The other day I deleted the conflicting localnames rows. [21:46:55] Should we disable global renames? [21:46:57] At the time it had looked like we stopped getting those mediawikiwiki rows. [21:48:30] Doesn't look like we have any new problematic mediawikiwiki rows now, either. Or at least if it happened it didn't stop Renameuser from deleting the renameuser_status rows. [21:48:56] after deploying the fix I made two test renames and those worked [21:49:03] I think new renames are OK [21:49:52] So it's just some in limbo? [21:50:33] yes, twenty users or so [21:51:17] https://phabricator.wikimedia.org/T135656#2364325 [21:56:10] Shame we don't have the edit count in the centralauth tables [22:13:48] anomie: wanna review https://gerrit.wikimedia.org/r/#/c/293620 while the train is getting delayed? it's simple and tested both ways [22:51:48] anomie: want me to deploy https://gerrit.wikimedia.org/r/#/c/293502 as well? [22:52:24] tgr: If you want. We don't really need it at the moment, but it could be handy next time something like this comes up. [23:21:41] ostriches: the most recent set of MW core tags weren't gpg signed? [23:23:15] Ugh did I forget that? [23:23:17] Fml [23:31:49] ostriches: :/ relatedly, could I get https://gerrit.wikimedia.org/r/#/q/owner:legoktm+status:open+topic:file-perms,n,z merged and backported into 1.27 so I don't have to do that manually in the debian package? [23:32:37] legoktm, reviewing [23:32:47] ty :) [23:36:08] also https://github.com/firebase/php-jwt/pull/96 is funny