[00:46:08] 10Release-Engineering-Team, 10Scap, 10Operations, 10Performance-Team (Radar): mwscript emits warning "grep: GREP_OPTIONS is deprecated; please use an alias or script" - https://phabricator.wikimedia.org/T198775 (10Krinkle) [01:19:58] (03PS1) 10Yaron Koren: Add MintyDocs extension to layout.yaml [integration/config] - 10https://gerrit.wikimedia.org/r/443756 [01:22:37] (03PS2) 10Yaron Koren: Add MintyDocs extension to layout.yaml [integration/config] - 10https://gerrit.wikimedia.org/r/443756 [02:16:49] Krenair: Do you know whether the -tin and -mira deployment-prep hosts are still used or whether they have plan for removal? [02:16:57] It seems odd to currently have *three* deploy hosts in beta [02:17:02] tin, mira and deploy01 [04:42:41] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on 'https://en.m.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 2015 bytes in 0.036 second response time [04:42:44] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 1343 bytes in 0.003 second response time [04:42:52] PROBLEM - Host deployment-mediawiki06 is DOWN: CRITICAL - Host Unreachable (10.68.19.241) [04:47:45] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 36085 bytes in 1.068 second response time [04:47:45] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 46869 bytes in 0.855 second response time [04:48:03] Project beta-scap-eqiad build #214497: 04FAILURE in 4 min 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/214497/ [04:57:40] Project beta-scap-eqiad build #214498: 04STILL FAILING in 3 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/214498/ [05:01:50] Project beta-scap-eqiad build #214499: 04STILL FAILING in 4 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/214499/ [05:07:41] Project beta-scap-eqiad build #214500: 04STILL FAILING in 4 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/214500/ [05:17:51] Project beta-scap-eqiad build #214501: 04STILL FAILING in 4 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/214501/ [05:27:47] Project beta-scap-eqiad build #214502: 04STILL FAILING in 4 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/214502/ [05:31:28] !log beta-scap-eqiad failing due to mediawiki06 still being a scap target (despite being shutdown per T192996) [05:31:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:31:33] T192996: Delete deployment-mediawiki06 - https://phabricator.wikimedia.org/T192996 [05:31:51] !log Fix beta-scap-eqiad by cherry-picking https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/443767/ T192996 [05:31:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:37:08] Yippee, build fixed! [05:37:09] Project beta-scap-eqiad build #214503: 09FIXED in 4 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/214503/ [07:42:22] (03PS3) 10Hashar: Add MintyDocs extension to layout.yaml [integration/config] - 10https://gerrit.wikimedia.org/r/443756 (https://phabricator.wikimedia.org/T198488) (owner: 10Yaron Koren) [07:42:42] (03CR) 10Hashar: [C: 032] "I have just edited the commit message to link to T198488 :]" [integration/config] - 10https://gerrit.wikimedia.org/r/443756 (https://phabricator.wikimedia.org/T198488) (owner: 10Yaron Koren) [07:44:18] (03Merged) 10jenkins-bot: Add MintyDocs extension to layout.yaml [integration/config] - 10https://gerrit.wikimedia.org/r/443756 (https://phabricator.wikimedia.org/T198488) (owner: 10Yaron Koren) [07:49:11] (03CR) 10Sebastian Berlin (WMSE): "Thanks for fixing this. I had recently noticed that this was something that would become an issue at some point, but had no plan for solvi" [integration/quibble] - 10https://gerrit.wikimedia.org/r/441544 (https://phabricator.wikimedia.org/T197687) (owner: 10Hashar) [07:57:44] (03CR) 10Hashar: [C: 032] "You are welcome :] It is unlikely to benefit any other extension but at least we have it around now!" [integration/quibble] - 10https://gerrit.wikimedia.org/r/441544 (https://phabricator.wikimedia.org/T197687) (owner: 10Hashar) [08:41:26] (03CR) 10Hashar: [C: 032] "There is also a parser test text file for Poem. So maybe another change can also clone the Poem extension :]" [integration/config] - 10https://gerrit.wikimedia.org/r/442004 (owner: 10Arlolra) [08:43:17] (03Merged) 10jenkins-bot: Run Parsoid's langParserTests against MediaWiki [integration/config] - 10https://gerrit.wikimedia.org/r/442004 (owner: 10Arlolra) [09:24:02] hashar: I'm just brainstorming a bit, but maybe we should split the browser tests into structure / regular, and only have the structure ones run against extensions [09:40:58] legoktm: yup I had something like that in mind. Ie in core add a tests/selenium/smoke/ that would run for all extensions [09:41:03] typically targetting the Special:Blank page [09:41:09] gotta speak about it with zeljkof :] [09:41:17] I am off / duties [09:42:20] +1 from me :) [12:13:31] 10Continuous-Integration-Config, 10Patch-For-Review: New extension, MintyDocs, needs to be added to zuul/layout.yaml - https://phabricator.wikimedia.org/T198488 (10Yaron_Koren) @hashar - thanks for your help with this. [14:50:41] 10Release-Engineering-Team, 10MediaWiki-extensions-WikimediaIncubator, 10Epic, 10I18n: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585 (10Amire80) [16:10:30] (03CR) 10Arlolra: "> There is also a parser test text file for Poem. So maybe another change can also clone the Poem extension :]" [integration/config] - 10https://gerrit.wikimedia.org/r/442004 (owner: 10Arlolra) [16:55:51] heds up I'm going to do an extended test of something that uses the conduit api; there's sleeps of a second every so often, and the queries shuold be short enough (except fo r'give me all transactions for a task', can't sk for only some) [16:56:02] but someone poke me if it annoys [16:56:15] all requests are reads, n edits; and they are all one at a tmie [16:58:32] apergos: are you whitelisted against the rate limiter? [16:58:37] You may find your self not being able to load phab if the rate limiter kicks in [16:58:43] I'm not editing [16:58:46] it's only reads [16:58:52] no edits [16:59:05] paladox: [16:59:13] Ok [16:59:49] I've been doing tiny little tests so far but this one I want to let go for some time [17:04:33] well it told me too many requests [17:04:42] so I'll slow it down some; I wonder what the read limit is [17:36:18] well I see. queries still not working as I like... [17:36:31] and really that's because fields in the query are utterly undocumented [17:36:40] and there's no way to know what to put in there [17:36:44] joy [17:42:28] users on mediawiki who use the vidual editor instead of editing the wikitext do not get shown the right stuff for the https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests page. [17:42:34] *visual [17:42:55] they get shown the message about how your entry has been added to blah and go visit it to see ... [17:43:09] before of course being able to edit anything [18:43:23] 10Release-Engineering-Team, 10GitHub-Mirrors, 10Security-Team: Enforce 2FA for GitHub members - https://phabricator.wikimedia.org/T198810 (10Reedy) [18:44:32] 10Release-Engineering-Team, 10GitHub-Mirrors, 10Security-Team: Enforce 2FA for GitHub members - https://phabricator.wikimedia.org/T198810 (10Reedy) [18:44:52] greg-g: Reckon I can JFDI that with an email to wikitech-l and mediawiki-l? [18:46:36] and/or give a weeks notice and then do it then [19:06:53] Why does discourse tell me "Sorry you cannot post a link to that host." for phab? [19:07:16] Wtf [19:07:17] "Your post was flagged as spam: the community feels it is an advertisement, something that is overly promotional in nature instead of being useful or relevant to the topic as expected." [19:07:35] https://discourse-mediawiki.wmflabs.org/t/database-structure-change-for-use-in-mysql-cluster/595 [19:09:37] Reedy, we should probably at least remove the ownership and teams of people without MFA [19:10:01] Checking the box is more fun and enforces it [19:10:31] Reedy, also, slight problem [19:10:42] Reedy, the wmfgerrit and wmfphab (?) users [19:11:08] enable 2fa and store the seed? :P [19:11:24] do they talk to github over SSH? [19:11:29] Does phab and gerrit replication support 2fa? [19:11:40] I'm guessing they use ssh [19:11:48] Phab uses https [19:11:57] Gerrit im not sure [19:12:36] so we've gotta do the personal access token thing and change the credentials phab uses to connect? [19:13:04] But does phab support that? [19:13:18] Tokens in addition to password for mirroring to a external source [19:13:18] if it can do https pushing then yes [19:13:23] That has 2fa [19:13:39] the personal access tokens work as passwords IIRC paladox [19:13:42] no need for any extra field [19:13:47] Oh [19:14:27] think someone would just need to update https://phabricator.wikimedia.org/K13 [19:16:32] https://github.com/wikimedia/puppet/blob/production/modules/gerrit/manifests/replication_key.pp [19:17:43] https://github.com/wikimedia/puppet/blob/b5a3581793afe2516bab6e4013873203674dff19/hieradata/role/common/gerrit.yaml#L20 [19:17:54] Krenair: gerrit uses ssh for replication ^^ [19:33:09] my testing has been less than satisfactory. now I have the random HTTPSConnectionPool(host='phabricator.wikimedia.org', port=443): Read timed out. [19:33:11] bleah [19:33:47] apergos: you hit rate limit! [19:34:00] when I hit the rate limit it notifies me [19:34:06] tather than just timing out [19:34:09] apergos: you should be able to request that your ip be whitelisted [19:34:32] read timed out? [19:34:32] I don't need it [19:34:34] honestly [19:34:39] yeah just now [19:34:51] I didn't retry in case there might be something going on [19:34:52] that thing is going through misc-web-lb [19:35:23] really? every request? ugh [19:35:36] phabricator.wikimedia.org is behind misc-web-lb [19:35:47] well I can add retry logic I guess [19:36:38] I'm pretty sure you shouldn't be getting read timed out when talking to misc-web-lb [19:36:45] is this running from the wikimedia network? [19:37:05] nah, I'm at home [19:37:17] I'm trying to get a reader in shape for some conduit stuff [19:37:23] alright so potentially crappy ISPs in the middle [19:37:27] the lack of decent docs is kicking my *ss tbh [19:37:50] I only got it once [19:37:57] maybe try running it from within the wikimedia network and see if it still gets read timed out then [19:37:58] in who knows how many queries [19:38:34] but more than once yes? [19:39:10] because it's late and I'm snarky (sorry) [19:39:14] (10:37:50 μμ) apergos: I only got it once [19:39:53] if I just get this dang one param on the feed query working I can stop for today [19:40:58] sorry, missed that line [19:41:05] I fugured [19:41:14] read above that it was "the random [...] read timed out" [19:41:22] I'm mostly just really really crabby at the phab folks for how sorry-ass their conduit docs are [19:41:33] and what a mess the mix of old and new methods you have to use it [19:41:35] *is [19:41:54] you and me too, though for me it's rather far down the list of things wrong with phabricator [19:41:58] lolol [19:42:09] well it's only at the top right now because I'm fighting with it [19:42:15] :D [19:42:54] nd after a bunch of queries another read timeout [19:42:59] not an error, just meh [19:43:18] yeah 'll have to fortify he connection logic here... BLAH [19:43:38] meh [19:44:00] OR [19:44:08] I can increase the timeout from 5 secs to 10 [19:44:16] to be fair though [19:44:17] these should be short little queries but you never know [19:44:39] before blaming this one on phab we should remove the various networks in the middle from the equation [19:44:53] oh, the read timeout? I don't blame that on phab necessarily [19:45:03] and actually, if phab decided not to complete an HTTP request or TCP connection [19:45:05] I blame everything else squarely on it, there's plenty of actual blame to go around [19:45:12] varnish would be expected to return something [19:45:29] yeah well I have code that says 'don't wait forever' so I'll just up that a bit [19:46:04] hmmm yeah I think varnish will likely be allowing much longer than that, so maybe it is just your timeout value [19:46:29] we'l try 10, if that's not good enough I'll increase some more [19:47:04] in fact actually if you set it that low you're probably asking to get more errors when dealing with something like phab :p [19:52:06] well I have it at 10 and enough passes through the main loop to see that it looks to be working [19:52:08] AT LAST [19:52:30] which means I can call it done for the day! woo hoo [19:55:10] ok I'm just gonna let it run til it dies or finishes, if phab was going to be unhappy we'd know about it by now [19:55:22] i got 1 second sleeps in between a lot of the stuff so... [20:11:00] What's up with beta? [20:11:13] auto deploy looks broken [20:15:19] !log beta unbroke beta code autodeploy T72597 [20:15:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:15:24] T72597: Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597 [20:37:32] 10Release-Engineering-Team, 10GitHub-Mirrors, 10Security-Team: Enforce 2FA for GitHub members - https://phabricator.wikimedia.org/T198810 (10Krenair) We discussed this in #wikimedia-releng and realised that we'll need to sort out the bots, wmfgerrit and wmfphab. wmfgerrit uses SSH for actual automated action... [20:42:19] 10Gerrit, 10Security, 10Upstream: Integrate MFA into Gerrit - https://phabricator.wikimedia.org/T198813 (10Krenair) https://groups.google.com/forum/#!topic/repo-discuss/tkCL3-MsXGM [22:02:46] 10Release-Engineering-Team, 10GitHub-Mirrors, 10Security-Team: Enforce 2FA for GitHub members - https://phabricator.wikimedia.org/T198810 (10Reedy) [22:18:18] 10Gerrit, 10Release-Engineering-Team: Enable 2FA on wmfgerrit github account - https://phabricator.wikimedia.org/T198822 (10Reedy)