[00:00:09] 10Release-Engineering-Team, 10Developer Productivity: FY201819 TEC12:O1 Outcome – Local development is unified with testing and production - https://phabricator.wikimedia.org/T222237 (10Jdforrester-WMF) [00:00:11] 10Release-Engineering-Team, 10Epic: Complete first round of the Release Engineering Developer Satisfaction survey - https://phabricator.wikimedia.org/T197635 (10Jdforrester-WMF) [00:00:13] 10Release-Engineering-Team, 10Developer Productivity, 10Epic: FY201819 TEC12 Program – Developer Productivity - https://phabricator.wikimedia.org/T212449 (10Jdforrester-WMF) [00:56:21] PROBLEM - Citoid on deployment-sca02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:16:11] RECOVERY - Citoid on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.029 second response time [01:25:16] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Community-Tech, 10LDAP-Access-Requests, 10Toolforge: Phab/Wikitech/Gerrit accounts for Harumi Monroy - https://phabricator.wikimedia.org/T222110 (10Dzahn) Everything is done besides the toolforge approval. I now also see the request in toolsadmin. It... [01:53:58] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Community-Tech, 10LDAP-Access-Requests, 10Toolforge: Phab/Wikitech/Gerrit accounts for Harumi Monroy - https://phabricator.wikimedia.org/T222110 (10Krenair) 05Open→03Resolved [01:54:23] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Community-Tech, 10LDAP-Access-Requests, 10Toolforge: Phab/Wikitech/Gerrit accounts for Harumi Monroy - https://phabricator.wikimedia.org/T222110 (10Krenair) (Bryan did the Toolforge approval bit) [02:28:36] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Community-Tech, 10LDAP-Access-Requests, 10Toolforge: Phab/Wikitech/Gerrit accounts for Harumi Monroy - https://phabricator.wikimedia.org/T222110 (10Dzahn) thanks @Krenair and @bd808 ! here you go @MaxSem ! welcome @HMonroy ! [04:25:51] 10Continuous-Integration-Infrastructure, 10Operations: Jessie rsyslog_8.1901.0-1~bpo8+wmf1_amd64.deb package fails to upgrade - https://phabricator.wikimedia.org/T222166 (10hashar) The workaround kind of make sense, however whenever we provision a new instance we would end up with a broken apt upgrade due to t... [04:32:24] (03CR) 10Hashar: [C: 03+2] "Seems the container was all broken previously :)" [integration/config] - 10https://gerrit.wikimedia.org/r/507513 (owner: 10Thcipriani) [04:34:03] (03CR) 10Hashar: "Sorry I have missed +2 ing this when you followed up in April 10th :-(" [integration/config] - 10https://gerrit.wikimedia.org/r/494778 (https://phabricator.wikimedia.org/T96390) (owner: 10Thcipriani) [04:34:18] (03Merged) 10jenkins-bot: release-notes: TOOLS_RELEASE_DIR -> RELEASE_TOOLS_DIR [integration/config] - 10https://gerrit.wikimedia.org/r/507513 (owner: 10Thcipriani) [04:37:42] (03CR) 10Hashar: [C: 03+2] "Successfully tagged docker-registry.discovery.wmnet/releng/release-notes:0.0.3" [integration/config] - 10https://gerrit.wikimedia.org/r/507513 (owner: 10Thcipriani) [04:38:17] (03CR) 10Hashar: [C: 03+1] "Successfully tagged docker-registry.discovery.wmnet/releng/release-notes:0.0.3" [integration/config] - 10https://gerrit.wikimedia.org/r/507514 (owner: 10Thcipriani) [05:14:52] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<40.00%) [07:09:54] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [09:23:09] PROBLEM - Citoid on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 1970: Connection refused [09:28:06] RECOVERY - Citoid on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.028 second response time [10:25:04] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [10:30:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.026 second response time [11:21:34] (03PS3) 10Umherirrender: [WikimediaEvents] Add dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/506914 [11:30:44] 10Release-Engineering-Team (Kanban), 10MW-1.32-release: Release 1.32.1 as a maintenance release - https://phabricator.wikimedia.org/T213595 (10Ciencia_Al_Poder) It hasn't been listed in https://www.mediawiki.org/ :( [12:42:20] 10Release-Engineering-Team (Kanban), 10MW-1.32-release: Release 1.32.1 as a maintenance release - https://phabricator.wikimedia.org/T213595 (10Reedy) >>! In T213595#5149657, @Ciencia_Al_Poder wrote: > It hasn't been listed in https://www.mediawiki.org/ :( {{sofixit}} ;) [13:40:46] (03PS1) 10Umherirrender: [UniversalLanguageSelector] Add dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/507565 [13:46:40] 10Release-Engineering-Team, 10MediaWiki-extensions-LocalisationUpdate, 10Security: Reenable l10update in production - https://phabricator.wikimedia.org/T220481 (10Aklapper) [13:46:43] 10Deployments, 10Release-Engineering-Team, 10TechCom-RFC, 10Core Platform Team (Code Health (TEC13)), and 2 others: RFC: Reevaluate LocalisationUpdate extension for WMF - https://phabricator.wikimedia.org/T158360 (10Aklapper) [14:08:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [14:08:14] (03PS1) 10Umherirrender: [TemplateStyles] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/507575 [14:14:14] (03PS1) 10Umherirrender: [WikiLove] Add dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/507577 [14:14:53] (03PS2) 10Umherirrender: [WikiLove] Add dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/507577 [14:31:07] Reedy: thcipriani want to huddle up a bit and see if things work? I think https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/506587/ needs to merge [14:32:50] Can do... I have a 1:1 in about half an hour [14:38:49] we can do it later if you're short on time or even tomorrow to my mind, we didn't actually schedule anything for today as it happens. [14:39:41] it'll only take 5 minutes or so to get the patch live [14:40:14] kk [14:40:21] What happens if we restart gerrit and someone is indef? [14:40:31] ? [14:41:12] If someone is blocked indef during a Gerrit’s maint window (for what ever reason) would someone manually inactivate a account? [14:42:00] Yeah [14:42:15] Ok [14:42:31] Plausible, but shouldn't happen regularly [14:43:06] Yup (hopefully) we could always get it so gerrit syncs with the backend (ldap) [14:43:13] Automatically [14:44:27] PROBLEM - Host deployment-ms-be03 is DOWN: CRITICAL - Host Unreachable (172.16.5.51) [14:45:19] PROBLEM - Host deployment-ms-be04 is DOWN: CRITICAL - Host Unreachable (172.16.4.129) [14:45:31] chasemp: Reedy ohahai -- I am around now -- am I still needed? [14:45:53] thcipriani: I think reedy is going to push out the main patch and then we can maybe do a bit of poking [14:45:59] k [14:46:01] Yeah, can do :) [14:47:14] thcipriani: I finagled all teh puppet things so afaik teh block on wikitech spider out to phab and gerrit is actionable, I figure let's register an account or two and test it [14:47:22] if it's not working we go backwards and circle up [14:47:33] if it's working we sit on our hands and let a few folks trickle in? [14:48:01] WFM [14:48:10] chasemp: sounds like a plan [14:50:47] Could use the gerrit health check account (I think) to test [14:51:50] for gerrit at least, yeah, GerritHealthCheckBot. Never tied that account to phab tho. [14:53:02] one thing worth noting is that while blocks matriculate...unblocks do not I think :D [14:53:14] so that's bound to create some confusion somehwere down teh line but that's life [14:53:30] heh [14:53:36] I imagine the amount being unblocked to be small... [14:53:42] Compromised accounts mostly [14:53:42] It’s easy to unblock gerrits side [14:53:49] Reedy: agreed [14:53:58] paladox: the point is it's not automatic, not that it's easy or hard [14:54:12] Yup [14:54:27] thcipriani: chasemp Code is live [14:56:18] Reedy: https://wikitech.wikimedia.org/w/index.php?title=Special:CreateAccount [14:56:24] Developer account creation temporarily disabled. See wikitech-l for more information. [14:56:40] That's a different patch ;P [14:56:43] well shoot [14:56:52] riiiight [14:56:58] Did Bryan make a patch for that? [14:56:59] * Reedy looks [14:57:36] Doesn't look like he has [14:57:38] I have the powers to do that temporarily [14:57:57] I think bawolf had the original patch maybe [14:58:28] I forget does setting a account as inactive block ssh access [14:58:56] yes [14:59:11] Good [15:00:06] do we have an account to test teh block thing on off hand without enable registration thcipriani ? [15:00:19] He's going to create one :P [15:00:37] * thcipriani does create one [15:02:20] ThciprianiBlockAccount1 [15:04:33] ok I'll block it... [15:04:43] chasemp: hold on, lemme login to phab [15:04:49] ok yep [15:05:41] thcipriani: gotta approve it too there [15:05:59] ok, logged into phab, email verified and approved, logged into gerrit added an ssh key there. [15:06:14] thcipriani: nice [15:06:31] https://phabricator.wikimedia.org/p/ThciprianiBlockAccount1/ [15:06:41] blocking... [15:06:47] ThciprianiBlockAccount1 has been blocked. [15:06:54] phab is disabled [15:07:09] want to check gerrit thcipriani ? [15:07:13] I'm not sure how to test [15:07:33] https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/507594/ will re-enable account creation [15:07:43] > Authentication failed for WikitechGerritAccountBot: account inactive or not provisioned in Gerrit [15:07:52] so no joy [15:08:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.025 second response time [15:08:32] Hmm. So if they haven't logged into gerrit yet, they can't be blocked pre-emptively? [15:09:02] thcipriani: I'm not sure I understand, isn't hat what we want to see after the block? [15:09:30] chasemp: that's the account that does the blocking [15:09:36] ohhhh [15:09:49] I'm slow on the draw this morning I guess [15:09:55] Reedy yes [15:11:40] Because no user would be created in gerrit if the user does not exist. [15:11:58] ie it needs to have a id (which means All-Users) [15:12:33] I'm testing, user is active (can login) [15:17:03] hrm, works in my testing, but I don't know that I change anything about that account...: https://phabricator.wikimedia.org/P8466 [15:17:47] huh [15:18:48] the initial error above doesn't seem like bad pass or anything so that's extra wonky [15:18:54] thcipriani: I notice you are using all small case [15:18:57] and the configuraiton is WikitechGerritAccountBot [15:19:02] just making sure that doesn't matter [15:19:03] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [15:19:08] chasemp: lemme test that, it shouldn't matter [15:20:06] chasemp: what do I know, it does matter and lowercase is the only thing that works. [15:20:18] well, very interesting [15:20:31] so I'll change the config puppet side to be all lower case and we try again? [15:21:07] I'll make a blockaccount2 I guess? or is it trivial to unblock in phab and wikitech? (I can ensure unblocked in gerrit) [15:21:29] please make a second account bc otherwise the test is poluted I fee like [15:21:35] even if we think it's all ok [15:21:43] k [15:21:46] thanks [15:21:59] Note that gerrit has a feature that can sync all accounts in ldap with gerrit. [15:23:45] thcipriani: labweb now says: $wmfGerritApiUser = "wikitechgerritaccountbot"; [15:23:51] assuming I don't need to restart apache or anything [15:23:55] I guess I could [15:23:57] nope, shouldn't... [15:24:04] might be some bytecode caching stuff [15:24:09] PROBLEM - Citoid on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 1970: Connection refused [15:24:26] thcipriani: ok ready for another crack at it [15:25:43] note that the lowercase thing is so you can use any case [15:25:49] also [15:26:00] using basic auth has to match the exact case [15:26:06] the lowercase thing is just for logins [15:26:09] not for git/ssh [15:26:17] thcipriani ^^ [15:29:33] chasemp: hang on just one second, I got a phone call from the vets office that I've been waiting on, sorry -- should be done in a sec :( [15:29:53] thcipriani: no worries to me, circle back when you've got time [15:31:45] 10Release-Engineering-Team (Kanban), 10MW-1.32-release: Release 1.32.1 as a maintenance release - https://phabricator.wikimedia.org/T213595 (10Jdforrester-WMF) @Ciencia_Al_Poder added to News: https://www.mediawiki.org/w/index.php?diff=3217222&oldid=3192363&title=Template:MediaWiki_News&diffmode=source I adde... [15:36:41] chasemp: ok, off the phone, I am now seeing that you have blocked account creation on wikitech from my IP :) [15:39:16] for channel completeness, I addressed the above :) [15:42:38] thcipriani: I'm going to block ThciprianiBlockAccount2 now? [15:42:56] chasemp: yes please, logged into gerrit and phab. Added ssh key to gerrit. [15:44:45] thcipriani: phab disable happened [15:45:38] http://tyler.zone/gerrit-disable-happened.png [15:45:47] \o/ [15:45:54] thcipriani: can ThciprianiBlockAccount2 log in to wikitech? [15:45:57] RIP [15:46:04] * thcipriani tries [15:46:21] I think blocked accounts should be able to login [15:46:27] I thought we disabled that [15:46:36] fuzzy memories [15:47:36] > Incorrect username or password entered. [15:47:44] kk [15:48:02] pw is coming from pw manager...so I *think* all info is correct [15:48:07] Reedy: would you feel up to reverting bawolfs disabling of password reset? [15:48:15] then we can test what happens with that too! [15:48:22] Can do, yeah [15:48:45] I recall that there is some setting to disallow login on block, and it USED to be not set pre-march but it was changed [15:49:42] https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/507604/ [15:50:02] lol, conflicts [15:50:33] "// Temporarily disable password resets. Revisit in 2 weeks" [15:50:48] :) [15:51:04] temporary [15:51:21] git being stupid [15:52:52] chasemp: Just the password reset? [15:52:57] please [15:54:10] RECOVERY - Citoid on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.025 second response time [15:56:05] would it be accurate to say Reedy thcipriani that these two groups combined equal the users who can block other users on wikitech? https://wikitech.wikimedia.org/w/index.php?title=Special:ListUsers&group=contentadmin and https://wikitech.wikimedia.org/w/index.php?title=Special:ListUsers&group=cloudadmin [15:56:33] I don't know the answer to that [15:56:35] https://wikitech.wikimedia.org/wiki/Special:UserGroupRights [15:56:41] (block) [15:56:43] Anyone with (block) [15:56:49] 10Release-Engineering-Team (Kanban), 10MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), 10Patch-For-Review, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4), and 2 others: Enable a MobileFrontend daily Selenium build - https://phabricator.wikimedia.org/T220659 (10Jdlrobson) https://gerrit.wikimedia.... [15:56:51] https://wikitech.wikimedia.org/w/index.php?title=Special:ListUsers&group=sysop [15:56:58] https://wikitech.wikimedia.org/w/index.php?title=Special:ListUsers&group=contentadmin [15:58:37] thcipriani: chasemp password reset should be back [15:58:57] thcipriani: want to try to reset password for ThciprianiBlockAccount2 to see how far you get? [15:59:02] I'm just being cautious here [15:59:04] * thcipriani does [16:00:08] PROBLEM - Citoid on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 1970: Connection refused [16:00:54] thcipriani: the other thing I want to test is if that account can login via any other ldap endpoint but I'm not thikning of one that should work w/ no permissions [16:00:56] striker? [16:01:07] looking at https://office.wikimedia.org/wiki/Security/LDAP [16:01:20] Disable authentication to LDAP using the associated account (phab:T168692) [16:01:20] Disable ssh public key authentication to Cloud VPS instances using the associated account (phab:T168692) [16:01:20] Search for and disable an associated Phabricator account (phab:T218654) [16:01:21] T168692: Blocking an account on wikitech should disable LDAP logins - https://phabricator.wikimedia.org/T168692 [16:02:24] tried "forgot password" in wikitech with ThciprianiBlockAccount2 [16:02:31] got a reset email with a temp password [16:02:42] does it work? :) [16:02:45] could not login to any of Gerrit Phabricator or Wikitech [16:02:53] nice [16:03:44] thcipriani: can you see if you can login w/ that account to /toolsadmin.wikimedia.org [16:04:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.041 second response time [16:05:03] thcipriani: reason I'm asking is if iiuc T168692 ldap should no longer allow auth even from an endpoint taht doesn't know better like toolsadmin [16:05:08] RECOVERY - Citoid on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.024 second response time [16:05:11] chasemp: cannot login there with temp password or old password, proper case or lowercase. [16:05:16] oh nice [16:10:03] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [16:11:46] Reedy: question for you on https://office.wikimedia.org/wiki/Security/LDAP [16:11:55] asking folks to do mwscript invalidateUserSessions.php --wiki=labswiki --user= [16:12:03] makes it sound like a block does not invalidate a users session on wikitech [16:12:13] which is weird if it also forbids them from logging in [16:12:25] wondering what the current state of that behavior is [16:12:44] Because MW is weird and lets blocked users do things like edit their own talk pages to request unblocking... [16:13:05] but if that's not allowed it still doesn't invalidate their session? [16:13:30] totally understood there, but I would think w/ current setup it would only make sense to nuke the session [16:13:44] Mmm [16:13:58] I think it might make more sense to add an "invalidate sessions" option to the block page for wikitech [16:14:04] And maybe default it to be checked [16:14:33] So they don't have to have shell access to do it [16:14:50] yeah it's the only part of things now that requires shell is why I was clarifying mainly [16:14:55] and then also not-intuitive I guess [16:15:03] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.033 second response time [16:15:51] Reedy: I'm not sure how difficult it is to make that happen :) [16:16:01] Not too bad, I don't think [16:16:11] Just the question is where the code should live [16:16:16] Just gonna file a task for it [16:17:29] so right now a blocked user doesn't have their session invalidated on wikitech but also...can't edit anything? [16:17:40] and can't re-login [16:17:51] pretty much [16:18:05] ok [16:18:06] tx [16:18:19] https://phabricator.wikimedia.org/T222281 [16:20:34] chasemp: Though, we can do it pretty easily as a hack in wikitech.php [16:20:44] Not making it an option in Special:Block, just do it unconditionally [16:21:07] I'm headscratching over whether the current situation is actually a problem or just confusing [16:21:19] It's an odd edge case [16:21:40] But if there's some other exploit they can still do by having their account (at least as is in wikitech)... It seems easier to just kick them out [16:21:46] true [16:21:54] You could argue this is useful for compromised accounts too [16:22:03] on the pedias [16:22:34] thcipriani: Reedy do we want to hold off on that? [16:23:19] chasemp: nice thing to make a ticket for, but not strictly necessary afaiu [16:24:38] thcipriani: Reedy my feeling is someone should send an email saying we are gonig to turn things back on, pointing to https://office.wikimedia.org/wiki/Security/LDAP and explaining a bit to SRE, releng, probably all of engineering? [16:24:45] and I bet they would appreciate tha temail before we actually do it [16:24:55] maybe we send the email today and then flip teh switch tomorrow morning? [16:26:30] bc I think we are 1 patch away from things being live again [16:26:32] I'll make a mw-config patch later to do it one way [16:26:52] yes please send an email about it [16:27:02] Yeah, I think https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/507594/ is the only thing left (re-enabling CreateAccount for anons [16:27:08] folks definitely ought to have a heads up [16:27:20] yeah ^ [16:27:31] yep, I agree [16:27:49] also taking the opportunity to call "not it" on explaining everything since I only understand my small part. I'd be able to help edit an email/talk about the gerrit bit. [16:28:01] heh [16:28:06] I can send the email sure, I was going to ask if you want to so you can get credit [16:28:17] I feel like you got hte short end of hte stick here on much of this [16:28:19] wikicoins! [16:28:28] more stock options for thcipriani [16:28:35] aw man [16:28:53] those will be valuable when the end times come [16:29:02] oh wait: I read that as "no more stock options" [16:29:07] hahah [16:29:19] "huzzah!" then [16:29:34] * chasemp breaks glass and puts on ill-fitting tie [16:29:42] ok, sounds like we are making an executive decision to wait till tomorrow to give a heads up to everyone [16:29:47] We can gdoc/etherpad writing up a response [16:30:02] And need to decide when to deploy https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/507594/ [16:31:34] plan on tomorrow as long as no one yells 'nooooooooooOOOoooo'? [16:33:23] http://www.nooooooooooooooo.com/ [16:40:28] PROBLEM - Host deployment-ms-fe02 is DOWN: CRITICAL - Host Unreachable (172.16.5.66) [16:40:45] PROBLEM - Host deployment-poolcounter04 is DOWN: CRITICAL - Host Unreachable (172.16.5.58) [17:30:08] 10Release-Engineering-Team, 10MobileFrontend, 10Browser-Tests, 10MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), and 3 others: AssertionError: false === true at thereShouldBeALinkToCreateMyUserPage on wmf-quibble PHP jobs - https://phabricator.wikimedia.org/T221860 (10Jdlrobson) This will be talked about tomorro... [17:31:19] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MediaWiki-extensions-MultimediaViewer, 10MobileFrontend, and 9 others: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature - https://phabricator.wikimedia.org/T199939 (10Jdlrobson) Will talk about this tomorrow... [18:02:23] (03PS2) 10Thcipriani: train-deploy-notes: update sha1; src dir; image [integration/config] - 10https://gerrit.wikimedia.org/r/507514 [18:04:13] (03PS3) 10Thcipriani: train-deploy-notes: update sha1; src dir; image [integration/config] - 10https://gerrit.wikimedia.org/r/507514 [18:07:35] (03CR) 10Thcipriani: [C: 03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/507514 (owner: 10Thcipriani) [18:07:42] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10Patch-For-Review, 10User-zeljkofilipin: Upgrade webdriverio to version 5 - https://phabricator.wikimedia.org/T213268 (10Jdforrester-WMF) [18:07:46] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown, 10Patch-For-Review, 10User-zeljkofilipin: `npm install` fails for mediawiki/core with EPEERINVALID when running on Node 11 - https://phabricator.wikimedia.org/T210506 (10Jdforrester-WMF) [18:10:26] (03Merged) 10jenkins-bot: train-deploy-notes: update sha1; src dir; image [integration/config] - 10https://gerrit.wikimedia.org/r/507514 (owner: 10Thcipriani) [18:14:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [18:29:07] 10Continuous-Integration-Config, 10OOUI: Speed up oojs/ui CI job/tests - https://phabricator.wikimedia.org/T189055 (10Jdforrester-WMF) [18:39:06] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.035 second response time [18:41:49] 10Continuous-Integration-Config, 10OOUI: Speed up oojs/ui CI job/tests - https://phabricator.wikimedia.org/T189055 (10Jdforrester-WMF) [18:42:16] Well this is the second day gerrit has had low threads! Yay! [18:43:49] PROBLEM - Content Translation Server on deployment-sca01 is CRITICAL: connect to address 172.16.5.13 and port 8080: Connection refused [18:45:04] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [18:46:28] and improvements in the heap!! [18:48:50] RECOVERY - Content Translation Server on deployment-sca01 is OK: HTTP OK: HTTP/1.1 200 OK - 904 bytes in 0.026 second response time [18:51:14] 10Continuous-Integration-Config, 10OOUI: Speed up oojs/ui CI job/tests - https://phabricator.wikimedia.org/T189055 (10Jdforrester-WMF) [18:51:17] 10Continuous-Integration-Config, 10OOUI: Speed up oojs/ui CI job/tests - https://phabricator.wikimedia.org/T189055 (10Jdforrester-WMF) Nowadays this runs significantly faster (though almost twice as fast on my local machine as on CI, sadly). I think we can deem this Resolved? [18:55:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.025 second response time [19:02:07] 10Phabricator, 10Release-Engineering-Team (Kanban): Calendar shows "Unhandled exception error: Expected a single result!" - https://phabricator.wikimedia.org/T220241 (10RhinosF1) @mmodell @20after4: I note there's a phab deployment tommorow that you're doing. Will it be in that? [20:40:13] 10Release-Engineering-Team (Kanban), 10MW-1.34-notes (1.34.0-wmf.4; 2019-05-07), 10Patch-For-Review, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4), and 2 others: Enable a MobileFrontend daily Selenium build - https://phabricator.wikimedia.org/T220659 (10Jdlrobson) 05Open→03Resolved Web... [20:51:52] (03PS1) 10Umherirrender: [CollaborationKit] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/507706 [20:54:23] (03PS2) 10Umherirrender: [CollaborationKit] Add dependency [integration/config] - 10https://gerrit.wikimedia.org/r/507706 [21:01:20] (03PS1) 10Umherirrender: [EventLogging] Add dependency [integration/config] - 10https://gerrit.wikimedia.org/r/507707 [21:06:33] 10Release-Engineering-Team (Backlog), 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4), 10Spike, 10User-zeljkofilipin: [Spike] Have a discussion around Minerva selenium browser test architecture - https://phabricator.wikimedia.org/T220755 (10Jdlrobson) [21:06:35] 10Release-Engineering-Team (Backlog), 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4), 10Spike, 10User-zeljkofilipin: [Spike] Have a discussion around Minerva selenium browser test architecture - https://phabricator.wikimedia.org/T220755 (10Jdlrobson) a:03Jdlrobson [21:06:37] 10Release-Engineering-Team (Backlog), 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4), 10Spike, 10User-zeljkofilipin: [Spike] Have a discussion around Minerva selenium browser test architecture - https://phabricator.wikimedia.org/T220755 (10Jdlrobson) (conversation happeni... [21:06:39] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Readers-Web-Backlog (Tracking): CI: Minerva PHPUnit tests should be included in shared extension gate job - https://phabricator.wikimedia.org/T202030 (10Jdlrobson) [22:03:09] https://www.theregister.co.uk/2019/04/30/kata_containers_zuul/ [22:03:16] zuul's no longer a pilot.