[00:00:10] <wikibugs>	 06Release-Engineering-Team, 15User-zeljkofilipin: Add a European mid-day SWAT window - https://phabricator.wikimedia.org/T137970#2456318 (10greg)
[00:00:11] <wikibugs>	 06Release-Engineering-Team, 15User-zeljkofilipin: Update new SWAT member process and deploying documentation in prep for adding European SWAT window - https://phabricator.wikimedia.org/T139545#2456315 (10greg) 05Open>03Resolved a:03thcipriani I think the combination of the mw1017/X-Wikimedia-Debug header...
[00:01:39] <wikibugs>	 05Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review: Update all references to git.wikimedia.org and replace them with the Phabricator equivalent - https://phabricator.wikimedia.org/T137353#2456324 (10Dzahn) https://meta.wikimedia.org/w/index.php?title=System_administrators&diff=prev&oldid=15761467  https://...
[00:04:43] <wikibugs>	 06Release-Engineering-Team, 15User-zeljkofilipin: Identify inaugural SWAT members for the European SWAT window - https://phabricator.wikimedia.org/T139544#2456332 (10greg) @hashar @Dereckson @JanZerebecki: When would you all be available to start doing this window? Note that Antoine will be on vacation startin...
[00:06:29] <wikibugs>	 06Release-Engineering-Team, 15User-zeljkofilipin: Identify inaugural SWAT members for the European SWAT window - https://phabricator.wikimedia.org/T139544#2456334 (10Dereckson) I'm available next week.
[00:21:12] <wikibugs>	 05Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review: Update all references to git.wikimedia.org and replace them with the Phabricator equivalent - https://phabricator.wikimedia.org/T137353#2456376 (10Danny_B) @Dzahn  Done.
[04:18:10] <wmf-insecte>	 Project selenium-MultimediaViewer » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #72: 04FAILURE in 22 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/72/
[05:09:03] <wikibugs>	 05Gerrit-Migration, 06Release-Engineering-Team, 10releng-201516-q3, 10ArchCom-RfC, and 4 others: [RfC]: Migrate code review / management from Gerrit to Phabricator - https://phabricator.wikimedia.org/T119908#2456668 (10RobLa-WMF)
[05:50:23] <wmf-insecte>	 Project mediawiki-core-code-coverage build #2132: 04STILL FAILING in 2 hr 50 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2132/
[08:10:20] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Parsoid, 06Services: Remove automatic beta updates for Parsoid - https://phabricator.wikimedia.org/T140218#2456870 (10mobrovac)
[08:12:33] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Parsoid, 06Services: Remove automatic beta updates for Parsoid - https://phabricator.wikimedia.org/T140218#2456884 (10mobrovac)
[08:13:40] <paladox>	 hashar https://gerrit-new.wikimedia.org/r/ :)
[08:13:45] <paladox>	 and hi
[08:14:30] <paladox>	 https://lists.wikimedia.org/pipermail/wikitech-l/2016-July/086047.html
[08:18:07] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Parsoid, 06Services: Remove automatic beta updates for Parsoid - https://phabricator.wikimedia.org/T140218#2456899 (10mobrovac) I removed `role::ci::slave::labs` and `role::parsoid::beta` from `deployment-parsoid0[56]`.
[08:22:09] <wikibugs>	 06Release-Engineering-Team, 15User-zeljkofilipin: Identify inaugural SWAT members for the European SWAT window - https://phabricator.wikimedia.org/T139544#2435541 (10Ladsgroup) I would like to help for deployment if there will be such time slot :) Also a training would be awesome. I did some deployments with s...
[08:22:28] <wikibugs>	 06Release-Engineering-Team, 15User-zeljkofilipin: Identify inaugural SWAT members for the European SWAT window - https://phabricator.wikimedia.org/T139544#2456905 (10Ladsgroup)
[08:26:58] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Parsoid, 06Services: Remove automatic beta updates for Parsoid - https://phabricator.wikimedia.org/T140218#2456908 (10hashar) So if I understand properly Parsoid is going to move to the generic service runner system. For beta we would:  * remove deployment-parsoid as a slave...
[08:28:11] <grrrit-wm>	 (03PS1) 10Mobrovac: Remove beta-parsoid-update-eqiad from mediawiki/services/parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/298701 (https://phabricator.wikimedia.org/T140218) 
[08:31:35] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Parsoid, 06Services, 13Patch-For-Review: Remove automatic beta updates for Parsoid - https://phabricator.wikimedia.org/T140218#2456915 (10mobrovac) >>! In T140218#2456908, @hashar wrote: > So if I understand properly Parsoid is going to move to the generic service runner sy...
[08:32:08] <grrrit-wm>	 (03PS2) 10Hashar: Remove beta-parsoid-update-eqiad from mediawiki/services/parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/298701 (https://phabricator.wikimedia.org/T140218) (owner: 10Mobrovac)
[08:33:03] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] "Removed the job definition in JJB and a special filter in zuul to tweak the message reported to Gerrit." [integration/config] - 10https://gerrit.wikimedia.org/r/298701 (https://phabricator.wikimedia.org/T140218) (owner: 10Mobrovac)
[08:33:49] <hashar>	 !log removing deployment-parsoid05 from the Jenkins slaves T140218
[08:33:50] <stashbot>	 T140218: Remove automatic beta updates for Parsoid - https://phabricator.wikimedia.org/T140218
[08:33:53] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[08:35:28] <grrrit-wm>	 (03Merged) 10jenkins-bot: Remove beta-parsoid-update-eqiad from mediawiki/services/parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/298701 (https://phabricator.wikimedia.org/T140218) (owner: 10Mobrovac)
[08:35:55] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Parsoid, 06Services, 13Patch-For-Review: Remove automatic beta updates for Parsoid - https://phabricator.wikimedia.org/T140218#2456919 (10hashar) 05Open>03Resolved a:03mobrovac > The Parsoid folks are ok with manually updating beta for now. We do need to find a generi...
[08:45:40] <paladox>	 hashar wow gerrit supports downloading from browser now. Same functionality as gitblit
[08:46:31] <mobrovac>	 hashar: what's that wikitech page where i can add a a puppet module to deployment-prep so that i can assign it to a host?
[08:55:30] <wikibugs>	 10Beta-Cluster-Infrastructure: "UnderflowException from ... FancyCaptcha.class.php: Ran out of captcha images" at Special:CreateAccount - https://phabricator.wikimedia.org/T140220#2456940 (10zeljkofilipin)
[08:59:53] <mobrovac>	 k, found it - https://wikitech.wikimedia.org/wiki/Special:NovaPuppetGroup
[09:17:43] <shinken-wm>	 PROBLEM - Host deployment-parsoid07 is DOWN: CRITICAL - Host Unreachable (10.68.17.24)
[09:24:26] <mobrovac>	 i recreated this one ^
[09:34:21] <paladox>	 hashar You can set signed by now. 
[09:37:48] <shinken-wm>	 RECOVERY - Host deployment-parsoid07 is UP: PING OK - Packet loss = 0%, RTA = 0.69 ms
[10:03:59] <wikibugs>	 06Release-Engineering-Team, 15User-zeljkofilipin: Identify inaugural SWAT members for the European SWAT window - https://phabricator.wikimedia.org/T139544#2457055 (10hashar) @hashar can do until end of July.  Lets open the window starting next week (week of of July 18th), announce it and get patches candidates...
[10:14:07] <wikibugs>	 06Release-Engineering-Team, 03releng-201617-q1, 15User-greg, 15User-zeljkofilipin: Perform a technical debt analysis of software and services maintained by WMF Release Engineering - https://phabricator.wikimedia.org/T138225#2457068 (10zeljkofilipin)
[10:17:44] <shinken-wm>	 PROBLEM - Host deployment-parsoid07 is DOWN: CRITICAL - Host Unreachable (10.68.20.187)
[10:36:47] <wikibugs>	 06Release-Engineering-Team, 15User-zeljkofilipin: Identify inaugural SWAT members for the European SWAT window - https://phabricator.wikimedia.org/T139544#2457118 (10zeljkofilipin) >>! In T139544#2456332, @greg wrote: > would you be able to start attending this window (6am Pacific/13:00 UTC) the week of July 2...
[10:37:45] <shinken-wm>	 RECOVERY - Host deployment-parsoid07 is UP: PING OK - Packet loss = 0%, RTA = 0.73 ms
[10:41:06] <mobrovac>	 hashar: why is the parsoid user available on every beta instance?
[10:41:11] <mobrovac>	 we need to get rid of that
[10:41:29] <hashar>	 mobrovac: it is in LDAP to get a fixed UID on all instances?
[10:41:43] <mobrovac>	 possibly, yes, that's what i'm thinking
[10:41:45] <hashar>	 yeah the account is in ldap
[10:41:49] <hashar>	 with UID 605
[10:41:56] <mobrovac>	 can you remove it?
[10:42:26] <hashar>	 mobrovac: context is https://phabricator.wikimedia.org/T65329
[10:43:01] <hashar>	 the justification was to have all parsoid instances to be able to write to /data/project/parsoid  
[10:43:03] <hashar>	 apparently
[10:43:06] <hashar>	 so yeah no more needed. 
[10:43:12] <hashar>	 I can't edit LDAP though
[10:43:21] <hashar>	 and not sure how to freeup the UID 605
[10:43:38] <mobrovac>	 so ops need to do it?
[10:43:53] <hashar>	 mobrovac: and theoretically you can have a 'parsoid' user  local to the instance which would kind of override the one from LDAP
[10:43:55] <hashar>	 (iirc)
[10:44:06] <mobrovac>	 it doesn't work though
[10:44:08] <hashar>	 fill a task / refers to T65329
[10:44:08] <stashbot>	 T65329: create a parsoid user in labs LDAP - https://phabricator.wikimedia.org/T65329
[10:44:14] <mobrovac>	 yes
[10:44:15] <hashar>	 and there must be a project related to LDAP labs edit
[10:44:19] <mobrovac>	 will do
[10:44:26] <hashar>	 looking it up
[10:44:50] <hashar>	 https://phabricator.wikimedia.org/project/view/1564/
[10:45:05] <hashar>	 #LDAP-Access-Requests looks like a good candidate
[10:45:20] <hashar>	 + #operations
[10:49:29] <mobrovac>	 hashar: https://phabricator.wikimedia.org/T140227
[11:13:27] <wikibugs>	 10Continuous-Integration-Config, 06WMDE-Analytics-Engineering, 03TCB-Team-Sprint-2016-06-29, 15User-Addshore: Turn on phplint for analytics/wmde/scripts repo - https://phabricator.wikimedia.org/T140067#2457216 (10Addshore) 05Open>03Resolved
[11:37:35] <hashar>	 !log apt-get upgrade on deployment-mediawiki02
[11:37:38] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[11:59:39] <paladox>	 hashar are we going to do the zuul upgrade?
[11:59:45] <paladox>	 please
[12:02:04] <wikibugs>	 10Deployment-Systems, 10scap, 10MediaWiki-API, 03Scap3 (Scap3-MediaWiki-MVP), and 2 others: Create a script to run test requests for the MediaWiki service - https://phabricator.wikimedia.org/T136839#2457283 (10jayvdb) In case it was useful, @legoktm and I had {T100903} working briefly.  It shouldnt take mu...
[12:04:46] <paladox>	 :)
[12:07:44] <wikibugs>	 10Beta-Cluster-Infrastructure: "UnderflowException from ... FancyCaptcha.class.php: Ran out of captcha images" at Special:CreateAccount - https://phabricator.wikimedia.org/T140220#2457300 (10Gilles) Is this really meant to be a child task to {T64835}?
[12:29:16] <paladox>	 hashar :)
[12:29:43] <paladox>	 hashar https://gerrit.wikimedia.org/r/#/c/298710/ 
[12:29:47] <paladox>	 it is a gerrit
[12:29:50] <paladox>	 change in puppet
[12:31:00] <paladox>	 and it is still running in ssh 
[12:31:01] <paladox>	 :)
[12:31:06] <paladox>	 https://gerrit-zuul.wmflabs.org/
[12:36:08] <hashar>	 !log T137525 Upgrading Zuul 2.1.0-95-g66c8e52-wmf1precise1 ... zuul_2.1.0-151-g30a433b-wmf1precise1_amd64.deb
[12:36:09] <stashbot>	 T137525: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525
[12:36:10] <hashar>	 paladox: ^^^
[12:36:17] <hashar>	 paladox: I have been distracted this morning
[12:37:07] <paladox>	 Oh :0
[12:37:13] <paladox>	 :)
[12:37:17] <paladox>	 thanks
[12:37:33] <paladox>	 hashar gerrit-new.wikimedia.org is up
[12:37:50] <hashar>	 seen tht
[12:38:04] <paladox>	 Yep
[12:38:15] <paladox>	 I found one of the bugs ostriches descrbied
[12:38:20] <paladox>	 described
[12:38:36] <paladox>	 I can edit in the browser now which is very quick now
[12:39:03] <grrrit-wm>	 (03CR) 10Hashar: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/291730 (https://phabricator.wikimedia.org/T128569) (owner: 10Hashar)
[12:39:13] <grrrit-wm>	 (03PS1) 10Paladox: Testing: Do not merge [integration/config] - 10https://gerrit.wikimedia.org/r/298718 
[12:40:31] <paladox>	 Yep looks like zuul stop proccessing ssh
[12:40:39] <paladox>	 I wonder if anything is in the logs
[12:42:21] <paladox>	 hashar should python-apscheduler that be commented out in precise?
[12:42:26] <paladox>	 since it isent in jessie
[12:42:32] <paladox>	 https://phabricator.wikimedia.org/diffusion/CIZU/browse/debian%252Fprecise-wikimedia/debian/control
[12:42:38] <paladox>	 https://phabricator.wikimedia.org/diffusion/CIZU/browse/debian%252Fjessie-wikimedia/debian/control
[12:43:42] <paladox>	 and shoulden python-gear that be cross out since it is # in jessie
[12:44:11] <paladox>	 https://phabricator.wikimedia.org/rCIZUee775170dfe657d7a8d102aec7ca622b9e197232
[12:44:20] <paladox>	 Crossed out python-apscheduler 
[12:44:38] <grrrit-wm>	 (03CR) 10Paladox: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/298718 (owner: 10Paladox)
[12:45:15] <grrrit-wm>	 (03CR) 10Hashar: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/291730 (https://phabricator.wikimedia.org/T128569) (owner: 10Hashar)
[12:46:35] <paladox>	 hashar maybe it is a good idea we can test with gerrit-new to see if it was a gerrit ssh bug
[12:47:28] <paladox>	 hashar it seems you applied wmf precise 2
[12:47:33] <hashar>	   (key.get_name(), hostname, hexlify(key.get_fingerprint())))
[12:47:34] <paladox>	 i mean precise wmf 1
[12:47:37] <hashar>	  /usr/lib/python2.7/dist-packages/paramiko/client.py:95: UserWarning: Unknown ssh-rsa host key for [ytterbium.wikimedia.org]:29418: dce9687b991b27d0f9fdce6a2ebf92e1
[12:47:37] <hashar>	   (key.get_name(), hostname, hexlify(key.get_fingerprint())))
[12:47:41] <paladox>	 Oh
[12:47:51] <paladox>	 So we found the problem :)
[12:48:01] <grrrit-wm>	 (03CR) 10Paladox: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/298718 (owner: 10Paladox)
[12:48:20] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Testing: Do not merge [integration/config] - 10https://gerrit.wikimedia.org/r/298718 (owner: 10Paladox)
[12:48:25] <paladox>	 Yay it works
[12:48:27] <paladox>	 hashar ^
[12:48:43] <grrrit-wm>	 (03CR) 10Paladox: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/298718 (owner: 10Paladox)
[12:48:53] <paladox>	 Oh it says not registered
[12:48:59] <paladox>	 integration-config-tox-jessie NOT_REGISTERED
[12:48:59] <paladox>	 integration-zuul-layoutdiff NOT_REGISTERED (non-voting)
[12:48:59] <paladox>	 integration-zuul-layoutvalidation NOT_REGISTERED
[12:49:32] <grrrit-wm>	 (03CR) 10Hashar: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/291730 (https://phabricator.wikimedia.org/T128569) (owner: 10Hashar)
[12:50:18] <wikibugs>	 10Beta-Cluster-Infrastructure: "UnderflowException from ... FancyCaptcha.class.php: Ran out of captcha images" at Special:CreateAccount - https://phabricator.wikimedia.org/T140220#2457533 (10zeljkofilipin) Not sure, @hashar suggested it.
[12:51:41] <grrrit-wm>	 (03CR) 10Hashar: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/291730 (https://phabricator.wikimedia.org/T128569) (owner: 10Hashar)
[12:52:04] <paladox>	 hashar yay it works
[12:52:05] <paladox>	 :)
[12:52:15] <hashar>	 paladox: yeah I forgot to reenable the gearman client
[12:52:22] <paladox>	 Oh
[12:52:56] <paladox>	 thankyou for fixing the problem
[12:53:08] <paladox>	 hashar but should you apply the wmf 2 patch
[12:53:12] <hashar>	 fixing what?
[12:53:22] <paladox>	 Well you found a ssh problem
[12:53:31] <paladox>	 Your running
[12:53:31] <hashar>	 not at all
[12:53:32] <paladox>	 2.1.0-151-g30a433b-wmf1precise1
[12:53:36] <paladox>	 but should it be
[12:53:37] <paladox>	 2.1.0-151-g30a433b-wmf1precise2
[12:53:41] <paladox>	 please
[12:53:44] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2457551 (10hashar) ``` $ sudo -u zuul /usr/share/python/zuul/bin/python /usr/bin/zuul-server -c /etc/zuul/zuul-server.conf -d /usr/lib/python2.7/d...
[12:54:08] <paladox>	 https://phabricator.wikimedia.org/rCIZU9fa7dc7ad49185ee3e5c425f4e3bcee1c367058e
[12:54:12] <paladox>	 hashar that one ^^
[12:55:59] <hashar>	 paladox: yeah that can presumably fix the issue
[12:56:04] <hashar>	 but I want to know about the issue first :D
[12:56:08] <paladox>	 Oh
[12:56:19] <paladox>	 thankyou for taking a look at zuul :)
[12:57:12] <hashar>	 https://gerrit.wikimedia.org/r/#/c/292363/1/debian/patches/0007-GerritWatcher-add-poll_timeout.patch,cm
[12:57:17] <hashar>	 that one might a mitigation
[12:57:22] <wikibugs>	 10Continuous-Integration-Config, 15User-zeljkofilipin: Refactor Jenkins job configuration for UploadWizard-api-commons.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T140076#2457575 (10zeljkofilipin)
[12:57:24] <wikibugs>	 10Browser-Tests-Infrastructure, 15User-zeljkofilipin: There should be a way to run custom Rake task in selenium* jobs - https://phabricator.wikimedia.org/T133542#2457576 (10zeljkofilipin)
[12:57:26] <wikibugs>	 10Browser-Tests-Infrastructure, 07Upstream, 07WorkType-NewFunctionality: JJB should support YAML axis - https://phabricator.wikimedia.org/T128462#2457577 (10zeljkofilipin)
[12:57:29] <wikibugs>	 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review, 15User-zeljkofilipin: Merge tests/browser/environments.yml and tests/browser/config/config.yml in WikidataBrowserTests - https://phabricator.wikimedia.org/T128097#2457578 (10zeljkofilipin)
[12:57:31] <wikibugs>	 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Migration of browsertests* Jenkins jobs to selenium* jobs - https://phabricator.wikimedia.org/T128190#2457574 (10zeljkofilipin)
[12:57:32] <paladox>	 Oh
[12:57:35] <hashar>	 but that is just to be able to reload while Gerrit is idling
[12:57:43] <paladox>	 Oh
[12:57:56] <hashar>	 the GerritWatcher threads is blocking / polling the Gerrit stream-events command
[12:57:59] <hashar>	 so the main loop waits for that to complete
[12:58:07] <paladox>	 Oh
[12:58:16] <paladox>	 So that will be fixed with that :)
[12:58:18] <hashar>	 and thus the configuration reload does not occur until an event is received from Gerrit
[12:58:21] <hashar>	 yeah
[12:58:27] <paladox>	 :_
[12:58:28] <hashar>	 but that is not the issue I had :D
[12:58:28] <paladox>	 :)
[12:58:31] <paladox>	 Oh
[12:58:36] <paladox>	 What other issues?
[12:58:43] <paladox>	 do you have with zuul
[12:58:56] <hashar>	 beside https://phabricator.wikimedia.org/T137525 ?
[12:58:57] <hashar>	 none
[12:59:05] <paladox>	 Oh
[12:59:13] <hashar>	 the thing is that it works just fine
[12:59:17] <hashar>	 and ends up deadlocking at some point
[12:59:23] <paladox>	 hashar do we need to update zuul to work with lead
[12:59:42] <paladox>	 I say we can close this as resolved now https://phabricator.wikimedia.org/T137525
[12:59:49] <wikibugs>	 10Browser-Tests-Infrastructure, 15User-zeljkofilipin: Migration of browsertests* Jenkins jobs to selenium* jobs cleanup and optional task - https://phabricator.wikimedia.org/T140235#2457586 (10zeljkofilipin)
[13:00:35] <grrrit-wm>	 (03Abandoned) 10Paladox: Testing: Do not merge [integration/config] - 10https://gerrit.wikimedia.org/r/298718 (owner: 10Paladox)
[13:00:42] <grrrit-wm>	 (03CR) 10Paladox: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/298718 (owner: 10Paladox)
[13:01:22] <wikibugs>	 10Continuous-Integration-Config, 15User-zeljkofilipin: Refactor Jenkins job configuration for UploadWizard-api-commons.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T140076#2457616 (10zeljkofilipin)
[13:01:24] <wikibugs>	 10Browser-Tests-Infrastructure, 15User-zeljkofilipin: There should be a way to run custom Rake task in selenium* jobs - https://phabricator.wikimedia.org/T133542#2457617 (10zeljkofilipin)
[13:01:26] <wikibugs>	 10Browser-Tests-Infrastructure, 15User-zeljkofilipin: Migration of browsertests* Jenkins jobs to selenium* jobs cleanup and optional task - https://phabricator.wikimedia.org/T140235#2457615 (10zeljkofilipin)
[13:01:31] <wikibugs>	 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review, 15User-zeljkofilipin: Merge tests/browser/environments.yml and tests/browser/config/config.yml in WikidataBrowserTests - https://phabricator.wikimedia.org/T128097#2457619 (10zeljkofilipin)
[13:01:43] <wikibugs>	 10Browser-Tests-Infrastructure, 07Tracking, 15User-zeljkofilipin: Migration of browsertests* Jenkins jobs to selenium* jobs cleanup and optional task - https://phabricator.wikimedia.org/T140235#2457586 (10zeljkofilipin)
[13:02:22] <wikibugs>	 10Browser-Tests-Infrastructure, 07Tracking, 15User-zeljkofilipin: Migration of browsertests* Jenkins jobs to selenium* jobs cleanup and optional task - https://phabricator.wikimedia.org/T140235#2457586 (10zeljkofilipin)
[13:06:15] <paladox>	 hashar maybe we can try the new gerrit-new
[13:06:17] <paladox>	 with zuul
[13:06:18] <paladox>	 :)
[13:06:36] <hashar>	 to do that I first need a more up to date Zuul
[13:06:43] <hashar>	 which has been blocked on https://phabricator.wikimedia.org/T137525
[13:06:56] <hashar>	 and I would like to move it forward a bit more
[13:06:57] <hashar>	 then
[13:07:12] <hashar>	 I think the Zuul we have on gallium  would be able to connect to BOTH Gerrit servers :D
[13:08:40] <paladox>	 Oh
[13:09:04] <paladox>	 hashar but isent  https://phabricator.wikimedia.org/T137525 now resolved
[13:09:13] <paladox>	 zuul works and ssh hasent stopped working now
[13:09:32] <paladox>	 hashar oh did upstream update to support mutiple gerrit's
[13:11:26] <hashar>	 yeah
[13:11:35] <hashar>	 the whole part of how Zuul handles connections has been reworked
[13:11:40] <hashar>	 which is what the -151- version is about
[13:11:49] <paladox>	 Oh
[13:12:00] <hashar>	 in the sense that from -95- to -151-  most of the patches are related to the huge refactoring
[13:12:10] <paladox>	 Oh
[13:12:15] <hashar>	 so one can http://docs.openstack.org/infra/zuul/connections.html#connections
[13:12:25] <paladox>	 But is https://phabricator.wikimedia.org/T137525 that still blocking us from doing any more updates
[13:12:25] <hashar>	 eg define multiple connections
[13:12:25] <paladox>	 ?
[13:12:28] <paladox>	 Oh
[13:12:28] <paladox>	 :)
[13:13:02] <hashar>	 and maybe we will end up with something like:
[13:13:07] <hashar>	 [connection WMFPHabricator]
[13:13:07] <paladox>	 Oh
[13:13:11] <hashar>	 driver=Differential
[13:13:11] <paladox>	 :)
[13:13:20] <hashar>	 server=iridium.eqiad.wmnet
[13:13:27] <paladox>	 :)
[13:13:30] <hashar>	 conduit_url=whatever
[13:13:30] <hashar>	 etc
[13:13:36] <paladox>	 Oh yep
[13:13:37] <hashar>	 and another one 
[13:13:47] <hashar>	 [connection GitHub-wikimedia]
[13:13:49] <hashar>	 driver=GitHub
[13:13:51] <paladox>	 Oh yep
[13:13:51] <hashar>	 ;-}
[13:13:53] <paladox>	 :)
[13:14:09] <hashar>	 which I am pretty sure someone has completed a serie of patches to bring up GitHub pull requests inside zuul
[13:14:13] <paladox>	 hashar but gerrit i think will be changed to the new lead server really soon
[13:14:19] <paladox>	 Oh
[13:14:21] <paladox>	 yep
[13:14:52] <hashar>	 paladox: https://review.openstack.org/#/q/project:openstack-infra/zuul+branch:master+topic:github-integration
[13:15:03] <paladox>	 Thanks
[13:15:07] <hashar>	 paladox: that is OpenStack repo. The serie is under the topic:github-integration
[13:15:12] <paladox>	 Oh
[13:15:20] <paladox>	 yep
[13:15:20] <paladox>	 :)
[13:16:01] <paladox>	 hashar https://github.com/wikimedia/operations-puppet/blob/30ffadcb6d8e6925affd55958981dce8c2c448d9/modules/role/manifests/zuul/configuration.pp#L8
[13:16:30] <hashar>	 yeah
[13:16:34] <hashar>	 we will deal with that dont worry
[13:16:39] <paladox>	 Ok
[13:16:45] <paladox>	 gerrit is much faster
[13:16:49] <paladox>	 in gerrit 2.12
[13:16:56] <paladox>	 loads instantly for me including diffs
[13:17:06] <hashar>	 ostriches and I are attending in at least two one hours meetings each week 
[13:17:14] <paladox>	 also in ios 10 beta i can now quickly view patches in integration/config
[13:17:17] <hashar>	 + the mailing lists
[13:17:24] <paladox>	 Yep
[13:17:26] <hashar>	 + the tasks we are both subscribed to
[13:17:26] <hashar>	 ;-D
[13:17:30] <paladox>	 Oh
[13:17:53] <paladox>	 can i close https://phabricator.wikimedia.org/T137525 as resolved please
[13:17:57] <paladox>	 hashar ^^
[13:20:38] <paladox>	 :)
[13:21:51] <hashar>	 NO
[13:22:13] <hashar>	 for at least a couple days
[13:22:24] <hashar>	 till  Iam 101% sure it is fine 
[13:24:14] <paladox>	 Oh :)
[13:25:20] <paladox>	 Does this mean we can continue with the zuul updates
[13:25:21] <paladox>	 :)
[13:25:46] <paladox>	 hashar could you deploy version  2.1.0-151-g30a433b-wmf2precise1
[13:25:48] <paladox>	 please
[13:26:04] <paladox>	 Which fixes two things
[13:26:04] <paladox>	 https://phabricator.wikimedia.org/rCIZU9fa7dc7ad49185ee3e5c425f4e3bcee1c367058e
[13:26:07] <paladox>	 please
[13:26:24] <hashar>	 paladox: look at those pathces
[13:26:29] <hashar>	 I have replied to that earlier
[13:26:33] <hashar>	 basically:  we dont need them
[13:26:36] <paladox>	 Oh
[13:26:38] <paladox>	 Sorry
[13:26:42] <hashar>	 ;-D
[13:26:56] <paladox>	 :)
[13:27:10] <hashar>	 and I am pretty sure the issue is a weird deadlock / thread race in paramiko
[13:27:25] <paladox>	 Yep
[13:28:03] <hashar>	 "/usr/lib/python2.7/dist-packages/Crypto/Random/_UserFriendlyRNG.py"
[13:28:04] <hashar>	 self._lock.acquire()   
[13:28:20] <paladox>	 Oh
[13:31:34] <hashar>	 having a quick break
[13:32:27] <paladox>	 Ok
[13:32:38] <paladox>	 hashar looks like skype are bringing skype back to linux
[13:32:39] <paladox>	 http://blogs.skype.com/2016/07/08/exciting-news-for-linux-users/
[13:32:44] <paladox>	 :)
[13:35:18] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2457746 (10hashar) The new Zuul has been running for 40 minutes now and seems to be all fine.   Maybe it was some random rare deadlock ?  I am kee...
[13:36:09] <hashar>	 paladox: the neat thing is you get Zuul working in Gerrit 2.12.x ;-}
[13:36:14] <hashar>	 be back in a bit
[13:36:25] <paladox>	 Ok yep :)
[13:36:33] <hashar>	 paladox: https://ring.cx/ !
[13:36:36] <hashar>	 which is FOSS
[13:36:40] <paladox>	 Oh
[13:36:41] <paladox>	 :)
[13:36:45] <hashar>	 and does not depend on a central service
[13:36:49] <paladox>	 Oh
[13:36:52] <paladox>	 It dosent work for
[13:36:53] <paladox>	 me
[13:37:08] <paladox>	 We could try running zuul on gerrit new and old gerrit at the same time
[13:37:11] <paladox>	 after your break
[13:37:12] <hashar>	 I guess it is forbidden in UK  / rejected by the great firewall
[13:37:13] <paladox>	 if you want
[13:37:16] <paladox>	 Oh
[13:37:22] <paladox>	 I doint have a great firewall
[13:37:38] <paladox>	 Oh it worked
[13:37:41] <paladox>	 it added .com
[13:37:43] <paladox>	 at the end
[13:37:53] <paladox>	 Stupid internet explorer
[13:39:00] <paladox>	 hashar We could try running zuul on gerrit new and old gerrit at the same time <!--- that would be really good if we do that
[13:39:02] <paladox>	 :)
[13:39:45] <paladox>	 Oh your on ring
[13:39:48] <paladox>	 hashar ^
[13:42:30] <hashar>	 tomorrow
[13:42:33] <hashar>	 it is too late to experiment
[13:43:12] <paladox>	 Oh
[13:43:13] <paladox>	 Ok
[13:43:43] <hashar>	 bah Gearman no more reports statsd metrics https://graphite.wikimedia.org/render/?from=-8hours&height=180&width=400&target=alias(color(zuul.geard.queue.running,'blue'),'Running')&target=alias(color(zuul.geard.queue.waiting,'red'),'Waiting')&target=alias(color(zuul.geard.queue.total,'888888'),'Total')&title=Gearman%20job%20queue%20(8%20hours)
[13:43:44] <hashar>	 :(
[13:45:29] <paladox>	 Oh
[13:46:36] <hashar>	 !log T137525 Stopped zuul that ran in a terminal (with -d). Started it with the init script.
[13:46:37] <stashbot>	 T137525: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525
[13:46:51] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[13:47:06] <paladox>	 hashar oh so could it be because it was running in terminal
[13:47:14] <paladox>	 and that broke gearman
[13:48:00] <hashar>	 maybe
[13:48:07] <grrrit-wm>	 (03CR) 10Paladox: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/298718 (owner: 10Paladox)
[13:48:11] <paladox>	 Ok
[13:48:33] <grrrit-wm>	 (03CR) 10Paladox: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/291730 (https://phabricator.wikimedia.org/T128569) (owner: 10Hashar)
[13:48:46] <hashar>	 the statsd config is handled by environment variable
[13:48:56] <hashar>	 and yeah running it from my terminal they would not be set
[13:48:58] <hashar>	 that must be it
[13:49:08] <hashar>	 it is back
[13:49:10] <paladox>	 Oh
[13:49:21] <paladox>	 :)
[13:50:47] <hashar>	 it is broken
[13:51:03] <paladox>	 Oh
[13:51:08] <paladox>	 It's broken?
[13:51:26] <wikibugs>	 10Beta-Cluster-Infrastructure: "UnderflowException from ... FancyCaptcha.class.php: Ran out of captcha images" at Special:CreateAccount - https://phabricator.wikimedia.org/T140220#2457813 (10AlexMonk-WMF) Yes, captchas are stored in swift in production but there doesn't appear to be a script to create the contai...
[13:51:57] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2457814 (10hashar) Zuul did not properly connect to Gerrit / stream-events: ``` + ssh -p 29418 hashar@gerrit.wikimedia.org 'gerrit show-connection...
[13:52:03] <hashar>	 zuul can't connect to Gerrit :/
[13:52:56] <paladox>	 Oh
[13:53:03] <paladox>	 Did it shows an error
[13:53:05] <paladox>	 hashar ^^
[13:53:06] <paladox>	 ?
[13:55:20] <paladox>	 hashar https://github.com/openstack-infra/zuul/commit/30a433b202a65dd7ddcc6eebf4d34041f6743740
[13:58:24] <hashar>	 !log  T137525 reverted Zuul back to zuul_2.1.0-95-g66c8e52-wmf1precise1_amd64.deb  . It could not connect to Gerrit reliably
[13:58:25] <stashbot>	 T137525: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525
[13:58:29] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[13:59:20] <paladox>	 hashar do you know the error
[14:01:13] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2457835 (10hashar) Thread dump of Zuul just after it started and with Gerrit showing a connection from gallium but without a username is P3415....
[14:01:14] <hashar>	 got yet another trace
[14:01:26] <hashar>	 previous one was https://phabricator.wikimedia.org/P3204
[14:01:33] <hashar>	 the one from minutes ago is https://phabricator.wikimedia.org/P3415
[14:01:39] <paladox>	 Oh
[14:01:44] <paladox>	 hashar would this fix it
[14:01:45] <paladox>	 https://github.com/openstack-infra/zuul/commit/ccef05dede36d3645939199f79f8e7e198f03abe
[14:01:49] <hashar>	   File "/usr/lib/python2.7/dist-packages/Crypto/Random/_UserFriendlyRNG.py", line 154, in reinit
[14:01:49] <hashar>	     self._lock.acquire()
[14:02:29] <paladox>	 Oh
[14:03:49] <hashar>	 will be for later
[14:03:54] <paladox>	 Ok
[14:04:04] <paladox>	 It dosent show the error in there?
[14:04:12] <hashar>	 at least
[14:04:27] <hashar>	 it runs fine when running it in foreground with "-d"
[14:04:34] <hashar>	 it at least ran for 45 minutes or so
[14:04:40] <paladox>	 Yep
[14:04:43] <hashar>	 just have to find out why it fails otherwise
[14:04:49] <hashar>	 I am having an audio with thcipriani now
[14:05:05] <paladox>	 Ok
[14:10:16] <paladox>	 hashar We could pin paramiko to just 1.8.0 since there are many more 1.x releases and seems paramiko 1.8.0 works for us or which version are we using
[14:10:43] <paladox>	 since *9* zuul version forgot the full version but the one we reverted to what version are we running
[14:10:51] <paladox>	 since that woulden have the paramiko  pinning
[14:15:32] <paladox>	 hashar i think the issue may be fixed upstream in 1.17.1
[14:15:33] <paladox>	 http://www.paramiko.org/changelog.html
[14:15:51] <paladox>	 We may need to re package that zuul version to include 1.17.1
[14:16:14] <paladox>	 https://github.com/paramiko/paramiko/issues/537
[14:18:27] <hashar>	 paladox: both our Zuul versions uses the same paramiko version
[14:18:35] <paladox>	 Oh
[14:18:57] <paladox>	 hashar would pinning it to just 1.8 fix it
[14:23:16] <paladox>	 hashar could it be because https://phabricator.wikimedia.org/rCIZU89d8d5c44f1b1e719c759020a29dc3289bb00e62
[14:23:21] <paladox>	 we cross out python-apscheduler (>= 2.1.1),
[14:23:31] <paladox>	 when i doint think we should since jessie dosent
[14:24:36] <yurik>	 hashar, https://gerrit.wikimedia.org/r/#/c/298738/  :(
[14:26:03] <paladox>	 hashar 3.2.0 has been released of https://github.com/agronholm/apscheduler/releases
[14:26:15] <paladox>	 which fixes https://github.com/agronholm/apscheduler/commit/f559affa716345157044243ac042625bcfcb9079
[14:26:33] <paladox>	 and is pined to 3.0 at https://phabricator.wikimedia.org/diffusion/CIZU/change/master/debian/patches/0001-wmf-soften-requirements.patch;89d8d5c44f1b1e719c759020a29dc3289bb00e62
[14:26:50] <paladox>	 pbr is also pined to pbr>=0.5.21
[14:26:54] <paladox>	 even though 1.8.0 works
[14:27:03] <paladox>	 0.8.2
[14:27:05] <paladox>	 i mean
[14:27:26] <paladox>	 paramiko<2.0.0
[14:27:26] <paladox>	  seems to be said like that
[14:28:05] <paladox>	 Even though it is like paramiko>=1.8.0,<2.0.0
[14:28:09] <paladox>	 https://phabricator.wikimedia.org/diffusion/CIZU/browse/debian%252Fprecise-wikimedia/requirements.txt
[14:29:05] <paladox>	 Oh wait it is said twice
[14:34:56] <hasharAway>	 yurik: bug fill it please. I am busy ;(
[14:35:09] <yurik>	 ok
[14:46:53] <paladox>	 hasharAway http://www.winbeta.org/news/skype-announces-linux-client-alpha-features-coming-chrome-chromebook
[14:46:54] <paladox>	 :)
[14:56:26] <James_F>	 twentyafterfour: What's happening with the train? AFAICS wmf.10 isn't cut yet, but Greg implied we'd proceed with an accelerated wmf.10 train today?
[14:56:44] <Reedy>	 HS1
[14:56:51] <hasharAway>	 James_F: yeah probably group0 and group1 today in one go
[14:59:04] <James_F>	 hasharAway: Kk. Thanks.
[14:59:08] <James_F>	 Reedy: Whoosh.
[14:59:16] * Reedy cries in a corner
[14:59:30] <James_F>	 Mostly I'm worried about where the wmf.10 cut point will be and what might go out early.
[15:01:10] <hasharAway>	 well maybe we can  cut the wmf branch with  "master@{yesterday}"
[15:01:11] <hasharAway>	 :-}
[15:01:29] <Reedy>	 THAT IS UNPOSSIBLE CANNOT BE DONE
[15:03:49] <hasharAway>	 Reedy: you are underestimating gitrevisions(7)  :D
[15:03:58] <hasharAway>	 <refname>@{<date>}, e.g. master@{yesterday}, HEAD@{5 minutes ago}
[15:03:58] <hasharAway>	            A ref followed by the suffix @ with a date specification enclosed in a brace pair
[15:04:01] <Reedy>	 does make-wmf-branch work? ;)
[15:04:07] <Reedy>	 *work with that
[15:04:13] <Reedy>	 hackhackhack
[15:04:18] <ostriches>	 Why would I cut it early?
[15:04:47] <greg-g>	 James_F: oops, I forgot to update [[deployments]] it is ostriches this week
[15:04:50] <ostriches>	 I don't want to cut as of yesterday or we'll have to backport a ton.
[15:05:00] <ostriches>	 (for the user probs)
[15:05:27] <hasharAway>	 Reedy: maybe;   make-wmf-branch -o 'master@{yesterday}' -n 1.27.0-wmf.10
[15:06:09] <ostriches>	 I doubt -o with fancy things will work tbh
[15:06:17] <ostriches>	 (Although yes, I know I can with git branch lol)
[15:06:48] <greg-g>	 ostriches: yo Chad, did you know about this git thing? It has some fancy commands. :)
[15:07:20] <ostriches>	 I think I used this git thing once or twice, ya
[15:07:35] <hasharAway>	         $oldVersion = $this->oldVersion == 'master' ? 'master' : $this->branchPrefix . $this->oldVersion;
[15:07:50] <hasharAway>	 so it would try cloning   wmf/master@{yesterday}   and bails out
[15:08:06] <wikibugs>	 10Beta-Cluster-Infrastructure: "UnderflowException from ... FancyCaptcha.class.php: Ran out of captcha images" at Special:CreateAccount - https://phabricator.wikimedia.org/T140220#2458050 (10AlexMonk-WMF) It'd also be a good idea to check whether the Score and Math extensions are functioning in beta.
[15:08:34] <ostriches>	 hasharAway: Yep!
[15:09:56] <hasharAway>	 ostriches: meanwhile I tried to upgrade Zuul in prod again and it fails connecting to Gerrit stream-events :(
[15:09:58] <hasharAway>	 gotta repro
[15:10:06] <hasharAway>	 and I just had a 1 hour chat with Tyler
[15:10:10] <ostriches>	 Upgrade zuul pointing at existing gerrit?
[15:10:19] <hasharAway>	 seems we can set up a new jenkins/zuul setup on the machine that will replace gallium
[15:10:26] <hasharAway>	 and point that 2nd CI installation to lead / gerrit-new
[15:10:33] <ostriches>	 Yeah I'd rather test that.
[15:10:38] <hasharAway>	 so when we switch Gerrit we would also switch to the new CI infra
[15:10:39] <ostriches>	 Rather than new zuul : old gerrit
[15:10:40] <ostriches>	 :D
[15:10:41] <hasharAway>	 and done™
[15:10:44] <hasharAway>	 yeah
[15:10:59] <hasharAway>	 and potentially newer zuul versions  are able to connect to multiple Gerrit
[15:11:00] <ostriches>	 How soon will the rest of the new CI infra be ready? Gerrit's on a timeline of 'any day now really'
[15:11:09] <hasharAway>	 well
[15:11:20] <hasharAway>	 we only clarified the open questions we had last thursday
[15:11:28] <hasharAway>	 been preparing a bunch of various document to clarify
[15:11:37] <hasharAway>	 and now it is "on which machine to put the crap on"
[15:11:51] <hasharAway>	 given the new host will be in labs support host network  (and behind misc varnish)
[15:12:06] <hasharAway>	 it boils down to   (A)  use scandium  but that might not have enough disk 
[15:12:14] <hasharAway>	 (B) move contint1001 to labs support networr
[15:12:32] <hasharAway>	 given contint1001 already has the puppet manifests applied, it would just need the Jenkins config files and history to be rsynced to from gallium
[15:12:56] <hasharAway>	 then bunch of puppet work to accommodate for us having two set of (Jenkins/Zuul)
[15:13:13] <ostriches>	 puppet is easy :D
[15:13:13] <greg-g>	 what kind of turn around does chris usually have for physically moving machines (is my understanding for getting it into labs support)
[15:13:21] <hasharAway>	 ostriches: yup
[15:13:29] <ostriches>	 Speaking of: I've been having dreams about puppet.
[15:13:33] <ostriches>	 This is unhealthy probably :p
[15:13:44] <hasharAway>	 greg-g: yeah physically moving the machine was my understanding as well
[15:13:45] <James_F>	 Yes. Very.
[15:13:56] <hasharAway>	 but then apparently it just about switching a VLAN on a switch
[15:13:57] <hasharAway>	 (speculation)
[15:14:27] <hasharAway>	 so for example one would go to the Switch on which contint1001 is plugged in, then do a magic command to change the Ethernet VLAN / change IP  and done (hopefully)
[15:14:41] <hasharAway>	 then rsync the crap of data +  puppet changes
[15:15:17] <hasharAway>	 ostriches: I have been dreaming last night of Python / Paramiko / /dev/urandom for entropy :-(
[15:15:31] <chasemp>	 I'm waiting to talk during our sync up
[15:15:40] <chasemp>	 but there is no need to move the machine, it's just a re-vlan and IP
[15:15:43] <ostriches>	 hasharAway: Dreams of paramiko? You might want to see a doctor :p
[15:15:52] <chasemp>	 contint1001 is in teh same row as scandium now
[15:15:53] <chasemp>	 etc
[15:16:01] <hasharAway>	 James_F: also I have seen your task about decoupling  oojs/ooui stuff in Zuul.  I am afraid it is in the backlog, at least at top of it
[15:16:19] <hashar>	 chasemp: if you have spare time we can chat now.  Tyler not attending tomorrow
[15:16:30] <paladox>	 ostriches i fixed your bug you found
[15:16:42] <James_F>	 hashar: No worries at all. The gerrit upgrade and related stuff obviously trumps everything else.
[15:16:46] <paladox>	 ostriches https://gerrit.wikimedia.org/r/#/c/298709/
[15:16:57] <ostriches>	 I saw.
[15:16:58] <hashar>	 chasemp: seems using contint1001 is the straightforward way.  Given ops budget agree to lend the machine to releng
[15:17:07] <paladox>	 But must not be merged until the day we move to the new gerrit install
[15:17:13] <chasemp>	 I wanted to talk to you in hangout as I"m not sure how to wrestle what (to me) is a maze of tickets into a new direction combined with it seems like whenever I propose keeping contint1001 you poo-poo it by saying it's too overprovisioned
[15:17:20] <chasemp>	 which is ok but then I'm lost on what you want to do
[15:17:21] <ostriches>	 paladox: Nah, we can parameterize it :)
[15:17:27] <paladox>	 Oh
[15:17:28] <paladox>	 ostriches also this one https://gerrit.wikimedia.org/r/#/c/298710/ fixes the links not showing 
[15:17:49] <hashar>	 chasemp: yeah I am overthinking :\   
[15:17:51] <ostriches>	 James_F, greg-g: I could start the wmf.10 branch *now* though
[15:18:09] <ostriches>	 So we're <24h behind, albeit still ahead of where people expected originally.
[15:18:12] <chasemp>	 if you want to keep contint1001 then stop arguing against it :)
[15:18:16] <chasemp>	 that's the gist 
[15:18:24] <hashar>	 chasemp: my stance on contint1001 is that could potentially be used on another project that need that horse power. Otherwise I am fine with using contint1001 since it exceeds the CI requirement
[15:18:27] <hashar>	 hehe
[15:18:41] * James_F has a quick look.
[15:18:43] <greg-g>	 ostriches: branch, yeah
[15:18:43] <ostriches>	 hashar: You can never have too many hardware electrons :P
[15:18:49] <chasemp>	 sure but no one is going to say it's a reasonable move if you don't since it's your deal
[15:18:57] <chasemp>	 so I've just been holding off to talk
[15:19:03] <hashar>	 OHH
[15:19:05] <greg-g>	 hashar: just take it :)
[15:19:19] <hashar>	 so lets do contint1001 , move it to labs support vlan
[15:19:22] <ostriches>	 greg-g, hashar: Something something gift horse :p
[15:19:33] <paladox>	 hashar that could future proof ci
[15:19:50] <ostriches>	 What paladox said ^. Spare ram and cpu is better than not enough :D
[15:20:01] <chasemp>	 I was going to write up basically we have room for perf improvement w/ jenkins (to me) and we know we are going to stack this host to the gills with functionality
[15:20:02] <James_F>	 ostriches: The main things that are normally worrying (breaking change new version of OOUI, worrying changes in /vendor or /resources) seem OK.
[15:20:08] <chasemp>	 so let's take the chance 
[15:20:16] <chasemp>	 but I can't do that against your wishes as it has seemed :)
[15:20:19] <chasemp>	 so cool
[15:20:26] <ostriches>	 James_F: Ok. Then I'll start now before it becomes worrying :)
[15:20:32] <James_F>	 ostriches: Good idea. :-)
[15:20:33] <chasemp>	 right, scandium now is in teh same row as row a has a labs-support vlan
[15:20:39] <hashar>	 it is more about "lets not waste precious resources that could be better used by ops on another project"
[15:20:51] <ostriches>	 hashar: Shush you. We'll take the resources :D
[15:20:55] <chasemp>	 you are killing me smalls
[15:20:56] <hashar>	 I think I am / was worried about stealing resource from others
[15:21:01] <ostriches>	 Don't be!
[15:21:20] <hashar>	 I should channel my thoughts via ostriches. He is much more pragmatic than I am
[15:22:41] <hashar>	 chasemp: I guess I can get rid of the tasks related to adding disk on scandium?
[15:22:45] <paladox>	 hashar but if we future proof ci we would be able to handle more traffic
[15:23:04] <hashar>	 paladox: what do you mean by future proof ?
[15:23:10] <chasemp>	 hashar: let's make the new flesh out the new path and then yeah cut out historical cruft
[15:23:22] <chasemp>	 contint1001 is in the private vlan yes now, but it's in row a
[15:23:26] <paladox>	 hashar well with more resources comes more space for dealing with traffic
[15:23:34] <chasemp>	 conventional wisdom is that "labs things" go in row b, I'm assumign that was the thought on physical move
[15:23:38] <paladox>	 from zuul to jenkins if jenkins is hosted on gallium
[15:23:39] <chasemp>	 but row a has a labs-support vlan
[15:23:45] <chasemp>	 see scandium or labstore1003
[15:23:56] <chasemp>	 so I would propose as task which addresses all of these points
[15:23:56] <hashar>	 so potentially it is acceptable to keep contint1001 in row A ?
[15:23:59] <paladox>	 It will also i thinkd decrease chances of hardware failure
[15:24:07] <hashar>	 chasemp: and move all three in one go?
[15:24:18] <chasemp>	 (points to follow)
[15:24:30] <chasemp>	 * contint1001 was not formally transitioned though it is officialy now shown that way so we need to ask for definite ok 
[15:24:41] <hashar>	 paladox: the idea is to have the new zuul/jenkins to point on lead/gerrit-new.  And when ready we switch both gerrit and CI to their new equivalents
[15:24:47] <chasemp>	 arguing it's basically already allocated and not being missed and we need a box and this is here fulfilling the requirements
[15:24:47] <hashar>	 paladox: having them run in parallel 
[15:24:57] <paladox>	 Oh
[15:25:00] <chasemp>	 and it's very dififcult to find a box that has the storage needs that isn't going to be overprovisioned
[15:25:20] <chasemp>	 but finessed in nice reasonable lanuage
[15:25:29] <hashar>	 chasemp: what is the process to have it assigned to Releng officially ? Should I fill a backfilling procurement ticket ? 
[15:25:36] <chasemp>	 * contint1001 moved to labs-support in row a and reimaged
[15:25:39] <paladox>	 But with the new server we could start getting what ever services needing migrating from gallium to the new one
[15:25:42] <paladox>	 including doc.wm.org
[15:25:43] <chasemp>	 make a task and have mark say "ok"
[15:25:49] <chasemp>	 but let's have it be one task so it's all clear
[15:25:49] <paladox>	 could be hosted there do to more power
[15:25:54] <chasemp>	 too much task spread atm is confusing
[15:26:03] <hashar>	 yeah agreed
[15:26:18] <hashar>	 I have used phabricator to brain dump bunch of unrelated things
[15:26:59] <chasemp>	 if both of those points were addressed, mark says great on a task that outlines we need to keep contint1001 and why including language to move it to labs-support as a drop in replacement for scandium
[15:27:04] <chasemp>	 I will do the move and a reimage
[15:27:51] <hashar>	 do we have to reimage ?
[15:28:20] <chasemp>	 typically a server is reimaged on a vlan move here (or so I have seen) but also
[15:28:30] <chasemp>	 idk whwat has been done to that server in that emergency allocation
[15:28:34] <chasemp>	 and I don't want to keep any of it
[15:28:44] <hashar>	 sounds good
[15:28:46] <ostriches>	 paladox: Amended https://gerrit.wikimedia.org/r/#/c/298709/ to vary based on where we're installing
[15:28:53] <ostriches>	 Should allow us to land now instead of later
[15:28:56] <paladox>	 ostriches thankyou
[15:29:01] <ostriches>	 (gotta run through puppet compiler first)
[15:29:05] <paladox>	 Oh :)
[15:29:24] <James_F>	 ostriches: BTW gerrit-new for me seems noticeably faster than gerrit; hopefully it's because the new version is faster, but I guess it might be due to the load. :-) In particular https://gerrit-new.wikimedia.org/r/#/projects/mediawiki/extensions/VisualEditor,dashboards/default loads in just over a second, as compared to ~7 in prod.
[15:30:01] <ostriches>	 James_F: Yeah, things do seem snappier to me as well. It's making me kinda worried that the slowdown people are seeing is in GWT/JS/browserland, about which there's little I can do to improve :\
[15:30:06] <wikibugs>	 10Continuous-Integration-Infrastructure: Add a second Zuul merger instance on scandium.eqiad.wmnet - https://phabricator.wikimedia.org/T136711#2458098 (10hashar) 05stalled>03declined Declining for now. Might revisit later on but really I have filled too many random tasks.  We have been with a single zuul-mer...
[15:30:23] <James_F>	 ostriches: We can tell them to buy a faster box.
[15:30:32] * James_F grins.
[15:30:42] <ostriches>	 Lol. I'm pretty sure it's a browser thing. GWT is notoriously mean to browsers it doesn't like.
[15:30:47] <ostriches>	 (ie: anything but latest Chrome really lol)
[15:30:50] <paladox>	 ostriches i found it was 1 sec to load a page
[15:30:57] <James_F>	 And the new feature that lets me do https://gerrit-new.wikimedia.org/r/#/q/owner:self+-is:mergeable+status:open+-message:WIP is particularly lovely, for instance.
[15:30:59] <paladox>	 instead of a few secs for gerrit.wikimedia.org
[15:31:09] <wikibugs>	 10Continuous-Integration-Infrastructure: Add a second Zuul merger instance on scandium.eqiad.wmnet - https://phabricator.wikimedia.org/T136711#2458104 (10hashar)
[15:31:15] <hashar>	 chasemp: thanks man :)
[15:31:19] <ostriches>	 James_F: Yay lucene indexing :D
[15:31:19] <paladox>	 diffs loading quicker too but that is because they now do responsive fast
[15:31:27] <James_F>	 ostriches: +1
[15:32:20] <paladox>	 ostriches could we also try https://gerrit.wikimedia.org/r/#/c/298710/ please
[15:32:29] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Parsoid, 06Services, 13Patch-For-Review: Remove automatic beta updates for Parsoid - https://phabricator.wikimedia.org/T140218#2458112 (10mobrovac)
[15:32:35] <paladox>	 Im not sure if it will break gerrit 2.8 but it will fix gerrit 2.12 as i tested
[15:32:41] <paladox>	 please
[15:33:41] <paladox>	 hashar that machine can also be used for doc.wm.org
[15:33:58] <ostriches>	 paladox: It should be tested against 2.8 then too.
[15:34:06] <ostriches>	 (or not applied)
[15:34:08] <paladox>	 ostriches im not sure how to
[15:34:16] <ostriches>	 Set up a gerrit 2.8 install and test it
[15:34:20] <paladox>	 Oh
[15:34:29] <ostriches>	 Or do like I did and make it configurable via hiera.
[15:34:33] <paladox>	 Yep
[15:34:36] <paladox>	 i was thinking that
[15:34:37] <paladox>	 :)
[15:34:41] <ostriches>	 I'm trying to keep the "migrate" patches to like 1 with a bunch of hiera.
[15:34:46] <ostriches>	 Trying to avoid actual manifest changes :)
[15:34:55] <paladox>	 How would i do it so some configs are empty
[15:34:56] <paladox>	 please
[15:35:03] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Technical-Debt: Relocate CI generated docs and coverage reports - https://phabricator.wikimedia.org/T137890#2458122 (10hashar) From a discussion I had with @thcipriani I have been overthinking the requirement to move the doc hosting out of gallium/ its replacement....
[15:35:11] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Technical-Debt: Relocate CI generated docs and coverage reports - https://phabricator.wikimedia.org/T137890#2458124 (10hashar) p:05High>03Normal
[15:35:52] <ostriches>	 paladox: You can use conditionals in erb templates. Basically if the value is unset (or empty) you can not print that bit.
[15:35:54] <ostriches>	 Otherwise do
[15:35:58] <paladox>	 Oh
[15:36:00] <paladox>	 thanks
[15:36:23] <paladox>	 So i set the default valid and in the old gerrit server i just leave it empty
[15:36:45] <ostriches>	 Old gerrit set it to an empty string in ytterbium.yaml
[15:36:50] <ostriches>	 Then you can do your conditional like this:
[15:37:09] <ostriches>	 <%- if @foo -%> .... <%- end -%>
[15:37:14] <mobrovac>	 is jenkins clogged or something?
[15:37:35] <paladox>	 Ok thanks
[15:37:38] <ostriches>	 mobrovac: Looks like it? 17 Wikibase jobs just sitting there
[15:37:39] <paladox>	 mobrovac seems it is
[15:37:46] <paladox>	 one has been running for 40mins
[15:37:51] <mobrovac>	 :S
[15:38:09] <ostriches>	 E_TOOMANYTABS
[15:38:16] <ostriches>	 *kernel panic*
[15:38:30] <paladox>	 Oh, it causes mine to slow down
[15:38:59] <paladox>	 But could be just internet explorer
[15:39:05] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Technical-Debt: Jenkins: Figure out a long term solution to maintaining Jenkins configuration - https://phabricator.wikimedia.org/T69027#2458132 (10hashar) Moving this out of the phase out gallium project. Having part of Jenkins unpuppetized is definitely an issue but...
[15:43:14] <paladox>	 ostriches i found a syntax error
[15:43:19] <paladox>	 in the patch you uploaded
[15:43:25] <paladox>	 You did : instead of ::
[15:43:34] <paladox>	 gerrit:jetty::changeidlink: '#q,$1,n,z'
[15:43:42] <paladox>	 gerrit:jetty::commitlink: '#q,$2,n,z' 
[15:44:13] <ostriches>	 Wherps
[15:44:36] <ostriches>	 Fixed
[15:44:47] <paladox>	 Thanks :)
[15:45:15] <ostriches>	 hashar: Possibly dumb question. Do we have a test on ops/puppet to make sure hiera files are valid yaml?
[15:47:10] * ostriches kicks make-wmf-branch, hard.
[15:47:14] <ostriches>	 Stupid script, who wrote you?
[15:47:18] <ostriches>	 Oh yeah, I wrote most of you
[15:47:30] <wikibugs>	 10Browser-Tests-Infrastructure, 15User-zeljkofilipin: Migration of browsertests* Jenkins jobs to selenium* jobs cleanup and optional task - https://phabricator.wikimedia.org/T140235#2458189 (10Danny_B)
[15:48:17] <paladox>	 ostriches i will enable linkdrafts in gerrit 2.12 due to some big changes
[15:48:35] <paladox>	 I found without enabling that it dosent show the diffusion link for the commit
[15:48:59] <paladox>	 due to the fact anything created in web is now presumed to be refs/drafts until it is published
[15:49:20] <paladox>	 ostriches But i will leave it disabled it in gerrit 2.8
[15:49:57] <ostriches>	 I would pay $50 to make those damn links to diffusion/gitblit/etc go away.
[15:50:11] <ostriches>	 Bane of my existence those.
[15:50:12] <paladox>	 Oh
[15:50:24] <paladox>	 gitblit have gone
[15:50:29] <paladox>	 just diffusion now
[15:50:44] <ostriches>	 I know. I'm saying I want them gone generally. I think they're a terrible feature.
[15:50:49] <ostriches>	 Doesn't matter *what* they link to
[15:51:01] <ostriches>	 Anyway, back to work
[15:51:09] <ostriches>	 James_F, greg-g: Branch in progress.
[15:51:11] <paladox>	 Ok
[15:51:31] <James_F>	 Ta.
[15:51:55] <James_F>	 ostriches: They'll be gone when we kill gerrit entirely for Phab. ;-)
[15:53:24] <paladox>	 ostriches do i do something like file = <%= @gitwebfile %>
[15:54:03] <paladox>	 Oh wait
[15:54:04] <grrrit-wm>	 (03PS1) 10Chad: make-wmf-branch: Don't use SSH, it's lame [tools/release] - 10https://gerrit.wikimedia.org/r/298784 
[15:54:05] <paladox>	 i get it now
[15:54:18] <paladox>	 doing <%- if @foo -%> and then setting true or false will cause it to not work
[15:54:24] <paladox>	 great idea and the easy way
[15:54:26] <paladox>	 :)
[15:54:27] <paladox>	 thanks
[15:54:28] <ostriches>	 James_F: Yeah in 2023 or so :p
[15:54:50] <ostriches>	 thcipriani: https://gerrit.wikimedia.org/r/#/c/298784/ aka I hate ssh.
[15:54:51] <James_F>	 ostriches: Optimistic much?
[15:56:36] <ostriches>	 Yeah I'm pretty optimistic today. For example: I'm thinking today will be 29 hour workday instead of a 34  hour workday!
[15:57:26] <thcipriani>	 ostriches: lgtm, will need to update the docs for that change: https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys#Create_the_new_branch_in_gerrit
[15:57:40] <greg-g>	 ostriches: go ahead and leave 2 hours early tonight, you know, at 1am
[15:58:06] <ostriches>	 Yay, that means I'll get 2 hours of sleep instead of none! :D
[15:58:22] <paladox>	 ostriches https://gerrit.wikimedia.org/r/298710 
[15:58:25] <paladox>	 please
[15:58:27] <ostriches>	 thcipriani: Yeah. Mainly that you'll want to use .netrc or some environment variables for the duration.
[15:59:09] <thcipriani>	 ostriches: yarp. Seems easier to me to use https, I just switched to ssh since that's what's in the docs :P
[15:59:34] <thcipriani>	 documentation Nuremberg defense
[15:59:57] <ostriches>	 Upside is it means you can do it from basically anywhere.
[16:00:02] <ostriches>	 (ie: I do it from tin lol)
[16:00:31] <paladox>	 ostriches does that https://gerrit.wikimedia.org/r/298710  work
[16:00:34] <paladox>	 and is that valid
[16:00:39] <paladox>	 or did i do it wrong
[16:00:41] <paladox>	 please
[16:00:49] <ostriches>	 I have no idea, I haven't had time to test yet.
[16:00:55] <paladox>	 Oh
[16:01:09] <ostriches>	 I've got about 15 things on my todo list *before* lunch :D
[16:01:15] <paladox>	 Oh
[16:02:30] <ostriches>	 James_F: extensions/skins/vendor done, doing core now
[16:02:39] <paladox>	 hashar we could use some of the new server resources for ci purposes like eg testing
[16:02:49] <paladox>	 setting up a docker with nodepool and using the server
[16:03:37] <paladox>	 ostriches ^^
[16:04:09] <paladox>	 It seems ci is being overloaded
[16:04:25] <paladox>	 too too many tests and not enough instances being created
[16:05:21] <urandom>	 !log Installing Cassandra 2.2.6-wmf1 on deployment-restbase0[1-2].deployment-prep.eqiad.wmflabs : T126629
[16:05:22] <stashbot>	 T126629: Cassandra 2.2.6 - https://phabricator.wikimedia.org/T126629
[16:05:26] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[16:06:49] <hashar>	 chasemp: I have filled the contint1001 assignment/move to labs support as https://phabricator.wikimedia.org/T140257 
[16:07:16] <paladox>	 hashar :)
[16:12:48] <hashar>	 paladox: yeah we are down to 10 instances
[16:13:16] <paladox>	 hashar yep
[16:13:34] <hashar>	 the gearman job queue at the bottom of https://integration.wikimedia.org/zuul/ is a good indication
[16:13:40] <paladox>	 Yep
[16:13:54] <hashar>	 and the two others graphs are test/ gate pipeline
[16:14:01] <paladox>	 Yep
[16:14:01] <hashar>	 the gate being high priority it is usually flat
[16:14:12] <hashar>	 with rest piling up being usually in the "test" pipeline
[16:14:18] <paladox>	 Oh
[16:14:49] <hashar>	 not sure why we have 700 jobs though
[16:15:09] <wikibugs>	 10scap, 03Scap3 (Scap3-Adoption-Phase1), 07WorkType-NewFunctionality: Need a way to see config diffs in Scap - https://phabricator.wikimedia.org/T118206#2458347 (10mmodell)
[16:15:11] <paladox>	 700 jobs?
[16:15:14] <wikibugs>	 10scap, 07Documentation, 03Scap3 (Scap3-Adoption-Phase1), 15User-mobrovac: Document Scap3 config-deploy - https://phabricator.wikimedia.org/T116634#2458349 (10mmodell)
[16:15:22] * paladox following the news of the new pm of uk.
[16:15:24] <hashar>	 oh no
[16:15:31] <hashar>	 was looking at https://grafana.wikimedia.org/dashboard/db/releng-zuul?panelId=10&fullscreen
[16:15:35] <hashar>	 metrics are confusing I guess
[16:15:38] <paladox>	 Oh
[16:17:30] <hashar>	 merger:merge                                   437  1  1
[16:17:53] <paladox>	 Oh
[16:18:42] <hashar>	 $ zuul-gearman.py status|grep ^merger
[16:18:42] <hashar>	 merger:merge	416	1	1
[16:18:44] <hashar>	 merger:update	169	0	1
[16:18:47] <paladox>	 Oh
[16:18:56] <hashar>	 it has troubles keeping up with all the Wikibase changes I guess
[16:19:01] <paladox>	 yep
[16:19:06] * paladox http://news.sky.com/story/live-cameron-prepares-to-hand-over-to-may-10500005
[16:22:17] <hashar>	 I am off
[16:22:24] <hashar>	 CI is catching up
[16:23:12] <paladox>	 Ok
[16:24:31] <paladox>	 ostriches we want linkdrafts
[16:24:39] <ostriches>	 No we don't.
[16:24:41] <paladox>	 otherwise in gerrit 2.12 commits wont work
[16:24:52] <ostriches>	 Of course they will.
[16:25:03] <paladox>	 Yep but we want them to be linked for refs/changes
[16:25:06] <paladox>	 instead of not
[16:25:58] <ostriches>	 refs/changes/* are linked. This is about refs/drafts/*
[16:26:00] <paladox>	 ostriches i doint know why upstream did it this way but without setting that true it dosent work in commits who are uploaded via web without setting that
[16:26:07] <paladox>	 actually nope
[16:26:19] <ostriches>	 Because they're put in refs/drafts/* when done via the web afaict.
[16:26:21] <paladox>	 refs/drafts is when they are created via web too but when you click public 
[16:26:23] <paladox>	 public
[16:26:26] <paladox>	 publish
[16:26:32] <paladox>	 then they are refs/changes
[16:26:39] <paladox>	 but it still dosent show the link 
[16:26:44] <ostriches>	 At which point it will link, because it's not a draft anymore...
[16:26:48] <paladox>	 since it thinks it is refs/drafts/
[16:26:53] <paladox>	 actually it does
[16:26:57] <paladox>	 i tested on gerrit-test
[16:27:09] <ostriches>	 Seriously?!?
[16:27:15] <paladox>	 Yep
[16:27:16] <ostriches>	 Reason #1982109129021809 I hate those stupid links to diffusion
[16:27:23] <paladox>	 Oh ha
[16:27:49] <ostriches>	 Meh, we might as well enable it for 2.8 too
[16:27:51] <ostriches>	 Nbd.
[16:28:11] <ostriches>	 Main reason was we weren't replicating refs/drafts/* to gitblit. But if they're in diffusion it's nbd.
[16:28:24] <ostriches>	 It's not like anyone should use them for security things anyway considering how insecure they are.
[16:28:32] <wikibugs>	 10Deployment-Systems, 06Release-Engineering-Team (Long-Lived-Branches): Take heat off day before the weekly branch-cut? - https://phabricator.wikimedia.org/T118212#2458401 (10mmodell)
[16:28:45] <ostriches>	 If you're expecting refs/drafts/* to be hidden you've already shot yourself in the foot :p
[16:28:47] <wikibugs>	 10Deployment-Systems, 06Release-Engineering-Team (Long-Lived-Branches): Convert old wmf/* deployment branches to tags (recurring chore) - https://phabricator.wikimedia.org/T1288#2458403 (10mmodell)
[16:28:56] <wikibugs>	 10MediaWiki-Releasing, 06Release-Engineering-Team (Long-Lived-Branches), 10MediaWiki-General-or-Unknown: Remove EOL MediaWiki release branches - https://phabricator.wikimedia.org/T92503#2458406 (10mmodell)
[16:29:01] <wikibugs>	 10Deployment-Systems, 06Release-Engineering-Team (Long-Lived-Branches): thoroughly document the new branch cutting plan / strategy - https://phabricator.wikimedia.org/T136015#2458409 (10mmodell)
[16:29:08] <wikibugs>	 10Deployment-Systems, 06Release-Engineering-Team (Long-Lived-Branches), 03releng-201617-q1, 07Epic: Merge to deployed branches instead of cutting a new deployment branch every week. - https://phabricator.wikimedia.org/T89945#2458411 (10mmodell)
[16:29:23] <wikibugs>	 06Release-Engineering-Team (Long-Lived-Branches), 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 07Spike: Spike: Plan reforms of the CentralNotice deployment branch - https://phabricator.wikimedia.org/T136904#2458413 (10mmodell)
[16:29:30] <wikibugs>	 10Deployment-Systems, 06Release-Engineering-Team (Long-Lived-Branches): Convert old wmf/* deployment branches to tags (recurring chore) - https://phabricator.wikimedia.org/T1288#2458415 (10demon) Actually, I'd like to repurpose this as "Just kill old branches." The tags are useless and clutter shit up which is...
[16:29:35] <wikibugs>	 10Browser-Tests-Infrastructure, 06Release-Engineering-Team (Long-Lived-Branches): Track state of browser tests before new wmfXX branch cut - https://phabricator.wikimedia.org/T437#2458416 (10mmodell)
[16:29:40] <paladox>	 ostriches ok
[16:29:41] <paladox>	 thanks
[16:29:50] <wikibugs>	 10Deployment-Systems, 06Release-Engineering-Team (Long-Lived-Branches): Convert old wmf/* deployment branches to tags (recurring chore) - https://phabricator.wikimedia.org/T1288#2458431 (10demon) Also we should prune old branches for extensions/skins/vendor too, I only do it for core right now.
[16:30:19] <paladox>	 ostriches done
[16:32:22] <ostriches>	 twentyafterfour, James_F, others: Minor naming convention nit. We should us -wmf.0n for the <10 point branches. Proper sorting is nice :)
[16:32:36] <ostriches>	 *use
[16:32:56] <James_F>	 ostriches: I'm happy to use that if you want.
[16:33:39] <ostriches>	 Eh it won't matter again until 1.29.0 now since we hit 10 today :p
[16:33:42] <ostriches>	 But it was just a thought
[16:33:57] * paladox oh my god know pm now for a few mins
[16:34:17] * paladox http://news.sky.com/story/theresa-may-heads-to-palace-to-become-next-pm-10500731
[16:47:06] * paladox we have a new pm (Theresa May has been officially appointed as Prime Minister.
[16:47:06] <paladox>	 )
[16:47:36] * paladox https://d2kmm3vx031a1h.cloudfront.net/GEM7OOExRR2WXDlzBLu8_12615267-1.jpg
[16:50:04] * paladox http://news.sky.com/story/theresa-may-heads-to-palace-to-become-next-pm-10500731
[16:51:04] <greg-g>	 paladox: we get it :) it's off-topic here :)
[16:52:55] <paladox>	 sorry
[16:53:04] <ostriches>	 twentyafterfour: I tweaked the train guide a bit. I basically moved everything except "do the wikiversions stuff" and "update roadmap" to prior to "WAIT"
[16:53:20] <ostriches>	 Couple of things can definitely go before and makes the actual window shorter.
[16:55:25] <ostriches>	 Goal today is to get everything done *other* than the actual wikiversions swapped prior to our window since we're running abbreviated.
[16:59:00] <greg-g>	 +1
[17:19:16] <wikibugs>	 06Release-Engineering-Team, 05Release: Fix make-deploy-notes to use proper API formats - https://phabricator.wikimedia.org/T140268#2458699 (10demon)
[17:19:25] <wikibugs>	 06Release-Engineering-Team, 05Release: Fix make-deploy-notes to use proper API formats - https://phabricator.wikimedia.org/T140268#2458711 (10demon) p:05Triage>03Lowest
[17:35:59] <paladox>	 ostriches would you be able to https://gerrit.wikimedia.org/r/#/c/298710/ and https://gerrit.wikimedia.org/r/#/c/298709/ so mutante can merge please.
[17:36:04] <paladox>	 either +1 or -1.
[17:36:06] <paladox>	 please
[17:37:14] <ostriches>	 709 I'm testing then will +1 if I like it
[17:37:21] <ostriches>	 710 I won't have time to test for a bit
[17:38:22] <paladox>	 Ok
[17:38:25] <paladox>	 thanks
[17:55:17] <wikibugs>	 06Release-Engineering-Team: Repositories dashboard - https://phabricator.wikimedia.org/T112259#1630317 (10mmodell) Maybe helpful: https://phabricator.wikimedia.org/r/
[18:02:17] <wmf-insecte>	 Project mediawiki-core-code-coverage build #2133: 04STILL FAILING in 3 hr 2 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2133/
[18:05:54] <wikibugs>	 10Beta-Cluster-Infrastructure: Setup poolcounter daemon in Beta Cluster - https://phabricator.wikimedia.org/T38891#2458977 (10greg)
[18:06:20] <wikibugs>	 10Beta-Cluster-Infrastructure: Setup poolcounter daemon in Beta Cluster - https://phabricator.wikimedia.org/T38891#418990 (10greg) 05declined>03Open I'm reopening this to explore if we can set it up in Beta Cluster now.
[18:08:44] <wikibugs>	 10Beta-Cluster-Infrastructure: Setup poolcounter daemon in Beta Cluster - https://phabricator.wikimedia.org/T38891#2458987 (10demon) p:05Lowest>03Normal I don't see why we can't. Sure, it's not necessary for beta operations, but it's a prod:beta delta and caused an outage today because of config mismatches.
[18:09:59] <wikibugs>	 10Beta-Cluster-Infrastructure: Setup poolcounter daemon in Beta Cluster - https://phabricator.wikimedia.org/T38891#2458993 (10hashar) According to T112501 , deployment-poolcounter01.deployment-prep.eqiad.wmflabs is the Poolcounter instance since September 2015.  ``` wmf-config/LabsServices.php:$wmfAllServices['e...
[18:10:36] <hashar>	 ostriches: greg-g: we got a poolcounter daemon :)
[18:10:48] <greg-g>	 hah, then we should just enable it? ;)
[18:10:50] <ostriches>	 Then we should turn it on :p
[18:10:51] <ostriches>	 lol
[18:11:10] <hashar>	 it has been enabled back in September 2015
[18:11:17] <hashar>	 if that is no more the case, it would be a regression
[18:11:53] <wikibugs>	 10Beta-Cluster-Infrastructure: Setup poolcounter daemon in Beta Cluster - https://phabricator.wikimedia.org/T38891#2459000 (10greg) And then:  ```lang=irc 18:03 <   legoktm> '-wmgUsePoolCounter' => [ 18:03 <   legoktm> 'default' => false, // T38891 18:03 <   legoktm> ],  ``` :)
[18:12:16] <hashar>	 loh looooo
[18:12:17] <paladox>	 hashar im just wondering if we should test zuul with gerrit 2.12 (I want to see if it works with a new host and a new gerrit version)
[18:12:26] <paladox>	 and hi
[18:12:35] <hashar>	 paladox: what is your bait ? :)
[18:12:52] <paladox>	 hashar What do you mean by that?
[18:12:54] <paladox>	 :)
[18:14:55] <hashar>	 paladox: well you get a test setup with Zuul and gerrit 2.12 which works
[18:15:00] <paladox>	 Yep
[18:15:15] <hashar>	 and yeah we will get a zuul  hooked with gerrit-new / lead
[18:15:17] <paladox>	 But i mean the new zuul version
[18:15:29] <paladox>	 which we found didnt work well today with gerrit 2.8
[18:15:44] <hashar>	 greg-g: ostriches: ah so deployed but not used/enabled bah :(
[18:15:52] <greg-g>	 someone forgot a step ;)
[18:16:50] <paladox>	 the zuul version  2.1.0-151-g30a433b
[18:16:52] <paladox>	 *
[18:16:58] <paladox>	 hashar ^^
[18:17:30] <paladox>	 hashar it seems nodepool has gone down again
[18:17:35] <paladox>	 Oh never mind
[18:17:43] <paladox>	 just really slow
[18:19:05] <hashar>	 paladox: a single CR+2 on a mediawiki/core change  cause ALL the stand-by pool to be consumed
[18:19:12] <paladox>	 Oh
[18:19:28] <hashar>	 then it takes like 40-50 seconds to get new instances to spawn and get added in Jenkins for later consumption
[18:19:33] <paladox>	 Yep
[18:20:24] <shinken-wm>	 PROBLEM - Host deployment-upload is DOWN: CRITICAL - Host Unreachable (10.68.16.189)
[18:20:43] <Luke081515>	 hashar: I guess nodepool is a bit slower then a few week before, because we now have 10 instead of 20 instances as limit?
[18:20:43] <paladox>	 hashae even though i tested zuul and gerrit 2.12
[18:20:49] <paladox>	 i havent tested stats
[18:20:57] <paladox>	 the gearman thing 
[18:22:05] <paladox>	 hashar maybe if you want / have time you could help me to setup stats so i can test to make sure it works for jessie (Gerrit 2.12) please.
[18:22:13] <paladox>	 I will be having dinner in a min
[18:22:34] <paladox>	 so i carn't do it right at the second. :)
[18:23:37] <Krenair>	 I feel like gerrit is much slower than it was a few days ago :/
[18:24:07] <wikibugs>	 06Release-Engineering-Team, 06Developer-Relations, 06Team-Practices: Consider alternative processes for Unbreak Now bugs, especially those which cross-cut components - https://phabricator.wikimedia.org/T140207#2459108 (10ksmith) We in TPG don't feel like this falls directly within our current scope. We're wi...
[18:24:08] <Krenair>	 git-upload-pack '/operations/mediawiki-config' seems to take forever :(
[18:24:19] <paladox>	 Krenair try https://gerrit-new.wikimedia.org
[18:24:55] <paladox>	 It is much faster and should be quicker but only test it there.
[18:25:31] <ostriches>	 Krenair: Ran a repack on it. Doubt it'll help, but can't hurt.
[18:25:43] <wikibugs>	 10MediaWiki-Codesniffer, 07Easy: Add sniff to detect usage of deprecated $wgUser - https://phabricator.wikimedia.org/T50963#2459121 (10Nikerabbit)
[18:25:45] <Krenair>	 thanks ostriches 
[18:33:40] <wikibugs>	 06Release-Engineering-Team, 06Developer-Relations, 06Team-Practices: Consider alternative processes for Unbreak Now bugs, especially those which cross-cut components - https://phabricator.wikimedia.org/T140207#2456573 (10greg) There are 16 current UBN! tasks in Phabricator: https://phabricator.wikimedia.org/...
[18:34:33] <wmf-insecte>	 Project beta-code-update-eqiad build #112704: 04FAILURE in 1 min 31 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/112704/
[18:36:29] <paladox>	 I got thanked for doing https://wikitech.wikimedia.org/w/index.php?title=How_to_install_zuul,_gerrit_and_jenkins_on_instances&oldid=prev&diff=731056 by zeljkof :)
[18:38:09] <wikibugs>	 06Release-Engineering-Team, 06Developer-Relations, 06Team-Practices: Consider alternative processes for Unbreak Now bugs, especially those which cross-cut components - https://phabricator.wikimedia.org/T140207#2459188 (10greg) >>! In T140207#2457777, @Aklapper wrote: > https://www.mediawiki.org/wiki/Phabrica...
[18:40:28] <wikibugs>	 06Release-Engineering-Team, 06Developer-Relations, 10Phabricator, 06Team-Practices: Consider alternative processes for Unbreak Now bugs, especially those which cross-cut components - https://phabricator.wikimedia.org/T140207#2459195 (10greg)
[18:44:27] <wmf-insecte>	 Yippee, build fixed!
[18:44:28] <wmf-insecte>	 Project beta-code-update-eqiad build #112705: 09FIXED in 1 min 26 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/112705/
[18:47:57] <zeljkof>	 paladox: you are welcome ;)
[18:48:05] <paladox>	 :)
[18:50:19] <paladox>	 zeljkof i still need to improve the doc but once that is done then anyone will be able to replicate the ci in labs
[18:50:21] <paladox>	 :)
[18:58:02] <wikibugs>	 06Release-Engineering-Team, 06Developer-Relations: Decision on Wikimedia Foundation service-level agreement on code review times - https://phabricator.wikimedia.org/T113707#2459259 (10RobLa-WMF)
[18:58:32] <wikibugs>	 10Deployment-Systems, 07Technical-Debt: deleteMediaWiki should delete /var/lib/l10nupdate/caches/cache-$wmgVersionNumber - https://phabricator.wikimedia.org/T119747#2459262 (10mmodell)
[19:00:06] <wikibugs>	 06Release-Engineering-Team, 06ArchCom, 06Developer-Relations, 10Phabricator, 06Team-Practices: Consider alternative processes for Unbreak Now bugs, especially those which cross-cut components - https://phabricator.wikimedia.org/T140207#2459264 (10ksmith) Adding ArchCom, in case they want to participate i...
[19:00:16] <wikibugs>	 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Setup a Swift cluster on beta-cluster to match production - https://phabricator.wikimedia.org/T64835#2459268 (10AlexMonk-WMF) @fgiunchedi got me more information about those swift containers (P3419) and I managed to recreate them in `mwscript eval.php commonsw...
[19:16:47] <wikibugs>	 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Setup a Swift cluster on beta-cluster to match production - https://phabricator.wikimedia.org/T64835#2459306 (10AlexMonk-WMF) root container is actually easy, assuming the file list in T130709 is still complete: ```root@deployment-ms-fe01:~# swift post root -r...
[19:23:14] <wikibugs>	 06Release-Engineering-Team, 06Developer-Relations: Decision on Wikimedia Foundation service-level agreement on code review times - https://phabricator.wikimedia.org/T113707#1673927 (10Nikerabbit) Language team has https://www.mediawiki.org/wiki/Wikimedia_Language_engineering/Code_review_statement_of_intent to...
[19:32:16] <James_F>	 Hmm. Beta Cluster code updates seem to have stalled? MobileFrontend is on ea9d5c4 but master is 848f091 and has been for over an hour…
[19:32:40] <Reedy>	 https://integration.wikimedia.org/ci/job/beta-scap-eqiad/
[19:34:24] <James_F>	 Yeah, hence my confusion.
[19:35:02] <James_F>	 Hmm. "Job ['/usr/bin/scap', 'pull', '--no-update-l10n', 'deployment-tin.deployment-prep.eqiad.wmflabs'] called with an empty host list." Is that a problem?
[19:35:20] <Reedy>	 early on, no
[19:35:31] <Reedy>	 it should just move from staging to live, on tin
[19:36:02] <Reedy>	 20:36:26 20:36:26 scap failed: LockFailedError Failed to lock /var/lock/scap: [Errno 11] Resource temporarily unavailable (duration: 00m 00s)
[19:36:25] <Reedy>	 oh, ignore that
[19:36:50] <Reedy>	 it seemed to sync all the apaches
[19:36:59] <ostriches>	 James_F: Probably because jenkins isn't doing anything right now
[19:38:30] <James_F>	 ostriches: Hmm. It's run a few times since the merge…
[19:38:47] <ostriches>	 It's sitting there saying do de doooooo right now at least.
[19:39:11] <wikibugs>	 06Release-Engineering-Team, 06ArchCom, 06Developer-Relations, 10Phabricator, 06Team-Practices: Consider alternative processes for Unbreak Now bugs, especially those which cross-cut components - https://phabricator.wikimedia.org/T140207#2459368 (10Luke081515) If we use one of this proposals, we have to no...
[19:39:16] <Reedy>	 Date:   Wed Jul 13 17:59:12 2016 +0000
[19:39:16] <Reedy>	     Merge "Inline and fix micro.autosize.js for iOS9"
[19:39:18] <ostriches>	 James_F: I'm thinking Zuul is British. It really likes queuing 
[19:39:22] <Reedy>	 that's in staging
[19:39:34] <Luke081515>	 greg-g: ^ maybe a interesting proposal for you at that task
[19:40:36] <hasharAway>	 so in short
[19:40:37] <Reedy>	 non staging looks upto date
[19:41:13] <hasharAway>	 a long serie of Wikibase patches is causing a denial of service on zuul-merger
[19:41:24] <hasharAway>	 and until the patches are merged
[19:41:27] <hasharAway>	 the jobs are NOT Scheduled
[19:41:35] <ostriches>	 Ok I have a better idea
[19:41:39] <ostriches>	 `service zuul restart`
[19:41:40] <hasharAway>	 that and we are on limited capacity of 10 nodes
[19:41:43] <hasharAway>	 no
[19:41:44] <ostriches>	 Then they're all done
[19:41:47] <hasharAway>	 that is not going to change anything
[19:41:56] <ostriches>	 It'll empty the queues....
[19:42:01] * James_F grins.
[19:43:18] <Reedy>	 James_F: the files on deployment-mediawiki03 look uptodate for MF
[19:43:29] <Reedy>	 well, ish
[19:43:59] <hasharAway>	 finishing a 1/1 with greg
[19:44:34] <James_F>	 Reedy: 
[19:44:36] <James_F>	 Meh.
[19:44:42] <James_F>	 Reedy: Is Special:Version just stale then?
[19:44:43] <Reedy>	 James_F: it works now? :P
[19:45:05] <hasharAway>	 ostriches: kill zuul 
[19:45:08] <Reedy>	 Mebbe.. isn't that a known issue?
[19:45:21] <hasharAway>	 ostriches: cant find out a good fix for the current situation
[19:45:36] <James_F>	 Reedy: Now you're making me doubt myself. But eh, if that code's in Beta Cluster MF I'm able to test now.
[19:46:33] <ostriches>	 hasharAway: zuul-merger restart?
[19:47:43] <hasharAway>	 na zuul on gallium
[19:47:47] <hasharAway>	 and drop the queue of jobs
[19:48:06] <ostriches>	 I can't do service zuul restart.
[19:48:09] <ostriches>	 Lack enough sudos
[19:48:21] <ostriches>	 Just zuul-merger
[19:49:59] <thcipriani>	 https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Restart ?
[19:50:25] <wikibugs>	 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Setup a Swift cluster on beta-cluster to match production - https://phabricator.wikimedia.org/T64835#2459420 (10AlexMonk-WMF)
[19:50:27] <wikibugs>	 10Beta-Cluster-Infrastructure, 13Patch-For-Review: "UnderflowException from ... FancyCaptcha.class.php: Ran out of captcha images" at Special:CreateAccount - https://phabricator.wikimedia.org/T140220#2459419 (10AlexMonk-WMF) 05Open>03Resolved
[19:50:29] <ostriches>	 Ah duh
[19:50:32] <hasharAway>	 back
[19:50:32] <ostriches>	 sudo to zuul
[19:50:48] <hasharAway>	 there are two way to stop/restart it
[19:50:51] <hasharAway>	 one keeps the queue of changes
[19:50:54] <hasharAway>	 the other discard it
[19:51:10] <hasharAway>	  /etc/init.d/zuul restart  is the friendly one
[19:51:18] <hasharAway>	 it waits for running jobs to complete, save the queue
[19:51:22] <hasharAway>	 and on start restore the queue
[19:51:24] <ostriches>	 I did a full stop.
[19:51:27] <ostriches>	 Then start.
[19:51:29] <hasharAway>	 but if you want to flush the queue:  stop  then start
[19:51:37] <hasharAway>	 whoever wrote that doc tyler pointed to is a champion
[19:53:11] <hasharAway>	 that is Timo https://www.mediawiki.org/w/index.php?title=Continuous_integration/Zuul&diff=1149774&oldid=1109717 :)
[19:53:53] <hasharAway>	 ostriches: 
[19:53:54] <hasharAway>	 @gallium:~$ zuul show running-jobs
[19:53:54] <hasharAway>	 No jobs currently running
[19:53:56] <hasharAway>	 :)
[19:54:27] <hasharAway>	 and if you look at the gearman queue graph at bottom of https://integration.wikimedia.org/zuul/
[19:54:44] <hasharAway>	 you can see the huge spike of waiting gearman functions (1700 of them)
[19:55:09] <Krinkle>	 Thanks :)
[19:55:20] <hasharAway>	 and the slow process https://graphite.wikimedia.org/render/?from=-2hours&height=600&width=900&target=alias(color(zuul.geard.queue.running,%27blue%27),%27Running%27)&target=alias(color(zuul.geard.queue.waiting,%27red%27),%27Waiting%27)&target=alias(color(zuul.geard.queue.total,%27888888%27),%27Total%27)&title=Gearman%20job%20queue%20(8%20hours)
[19:55:32] <hasharAway>	 Krinkle: yeah that restart doc is really great thank you a ton :)
[19:55:34] * Krinkle couldn't have done it without hasharAway - I think I based that off a similar page you wrote on wikitech
[20:06:09] <hasharAway>	 "Don't modify the Visible to and Editable by fields unless you know what you are doing."
[20:06:12] <hasharAway>	 where is da doc
[20:06:13] <hasharAway>	 !!! :)
[20:06:47] <paladox>	 hasharAway there's the new gerrit skin
[20:07:03] <paladox>	 that is made for gerrit 3.0 but is being trialled in gerrit
[20:07:11] <paladox>	 has to be enabled by a special thing
[20:07:45] <hasharAway>	 ostriches: thcipriani I am filling a bug for the issue Zuul had
[20:17:57] <wikibugs>	 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Setup a Swift cluster on beta-cluster to match production - https://phabricator.wikimedia.org/T64835#2459517 (10AlexMonk-WMF) * `root@deployment-ms-fe01:/data/project/upload7/private/gwtoolset/wikipedia/commons/commonswiki-gwtoolset-metadata# swift upload wiki...
[20:19:26] <Krenair>	 you want to use those fields hasharAway?
[20:28:45] <Krenair>	 So...
[20:28:50] <Krenair>	 I think I'm done with that.
[20:29:04] <Krenair>	 Do I need to send an email somewhere?
[20:29:13] <Krenair>	 I guess we should delete deployment-upload at some stage
[20:31:17] <hasharAway>	 Krenair: yeah definitely mail QA list and celebrate!
[20:32:12] <hasharAway>	 Krenair: have you managed to solve the captchas issue ?
[20:32:13] <Krenair>	 yes
[20:32:42] <hasharAway>	 magic
[20:32:43] <hasharAway>	 for math
[20:32:49] <hasharAway>	 we can loose them all since they are regenerated
[20:33:03] <hasharAway>	 the challenge is figuring out whether math (or is that mathoid nowadays???) manage to upload to swift
[20:33:12] <hasharAway>	 and that, honestly I have no clue
[20:33:43] <Krenair>	 I don't know, I went to the trouble of copying the math files to swift only to find that it seems to send users to either RB or prod's upload instead?
[20:33:50] <paladox>	 hasharAway they added preferences to polygerrit
[20:34:15] <paladox>	 and it has a nice ui but currently broken in 2.12 might be fixed in 2.13 or which ever version they go with next
[20:34:16] <paladox>	 :)
[20:37:07] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Labs: Completely remove Beta Cluster dependency on NFS - https://phabricator.wikimedia.org/T102953#2459656 (10AlexMonk-WMF)
[20:37:12] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Commons, 10MediaWiki-File-management, 06Multimedia, 07Tracking: Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking) - https://phabricator.wikimedia.org/T84950#2459657 (10AlexMonk-WMF)
[20:37:14] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 07Tracking: Use Beta cluster as a true canary for code deployments (tracking) - https://phabricator.wikimedia.org/T53494#2459658 (10AlexMonk-WMF)
[20:37:16] <wikibugs>	 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Setup a Swift cluster on beta-cluster to match production - https://phabricator.wikimedia.org/T64835#2459653 (10AlexMonk-WMF) 05Open>03Resolved Thanks to @fgiunchedi for setting up the instances and to @arielglenn for getting a few last pieces of informati...
[20:38:37] <andrewbogott>	 So… Jenkins is super slow and/or broken, right?
[20:38:58] <andrewbogott>	 And… is that my fault?
[20:39:14] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Labs: Completely remove Beta Cluster dependency on NFS - https://phabricator.wikimedia.org/T102953#2459672 (10AlexMonk-WMF) 05stalled>03Resolved Given that the only place NFS is mounted in deployment-prep is deployment-upload, which is shutoff and to be deleted at some poin...
[20:39:17] <paladox>	 hasharAway ^^
[20:40:22] <hasharAway>	 andrewbogott: it can be your fault :)
[20:40:25] <hasharAway>	 ERRR
[20:40:33] <hashar>	 andrewbogott: it can NOT be your fault :)
[20:41:28] <hashar>	 andrewbogott: looks like it is processing properly as far as I can tell
[20:41:40] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Commons, 10MediaWiki-File-management, 06Multimedia, 07Tracking: Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking) - https://phabricator.wikimedia.org/T84950#2459681 (10AlexMonk-WMF)
[20:42:23] <paladox>	 hasharAway ostriches https://groups.google.com/group/repo-discuss/attach/127c2bd2eb0799/Screen%20Shot%202016-01-29%20at%2012.03.07.png?part=0.1.1&authuser=0
[20:42:25] <paladox>	 :)
[20:42:36] <andrewbogott>	 hashar: it's definitely working a little bit — I just keep missing the tip and having to resubmit (after which there's a shockingly long delay until I get the +2)
[20:42:37] <paladox>	 https://groups.google.com/group/repo-discuss/attach/127c2bd2eb0799/Screen%20Shot%202016-01-29%20at%2012.02.52.png?part=0.1.2&authuser=0
[20:45:02] <hashar>	 andrewbogott: any change specially I can look at ?
[20:45:26] <Krenair>	 https://lists.wikimedia.org/pipermail/qa/2016-July/002516.html
[20:45:34] <andrewbogott>	 https://gerrit.wikimedia.org/r/#/c/298538/
[20:45:37] <andrewbogott>	 hashar: ^
[20:46:07] <andrewbogott>	 Krenair: that's super fantastic!
[20:47:13] <hashar>	 Krenair: pick up stuff from https://store.wikimedia.org/ you are my guest :)
[20:47:30] <hashar>	 Krenair: that is like super impressive. Been pending for like 3 years maybe
[20:47:57] <Krenair>	 4
[20:48:48] <Krenair>	 Not sure what we want to do about T84950 - maybe just a single imagescaler host that swift contacts directly?
[20:48:48] <stashbot>	 T84950: Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking) - https://phabricator.wikimedia.org/T84950
[20:48:54] <Krenair>	 Don't think we can have load balancers in labs
[20:49:17] <Krenair>	 at the moment I dodge this problem by sending requests to the text varnish
[20:49:46] <Krenair>	 which will send it to one of the apaches (which aren't dedicated imagescalers and don't have all the right packages etc.)
[20:58:22] <hashar>	 andrewbogott: I guess that is when we had to kill zuul
[20:59:01] <andrewbogott>	 Should I resubmit?
[20:59:48] <hashar>	 andrewbogott: I commented 'recheck'
[21:00:36] <hashar>	 andrewbogott: that is a shortcut to reenqueue the change as if the patchset had just been sent
[21:01:45] <bd808>	 Krenair: yeah I think there is something technical in the network layer that keeps us from bringing up a real LB, but a single image-scaler host should be doable I think.
[21:01:53] <hashar>	 andrewbogott: oh also, could we raise the nodepool # of instances back to 20 ?  Since there the RAM overcommit is now 1.0 and there is a couple more labvirt servers?
[21:02:07] <hashar>	 unless the new servers are still warming up :)
[21:10:19] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2459777 (10hashar) **summary**  During a chat with @thcipriani he conducted a bunch of proof checks. Among others:  * Confirm host finger print wi...
[21:11:40] <wikibugs>	 06Release-Engineering-Team (Deployment-Blockers), 05Release, 05WMF-deploy-2016-07-05_(1.28.0-wmf.9): MW-1.28.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T138555#2459781 (10greg) 05Open>03Resolved We're skipping wmf.9. group0 and group1 are now on wmf.10 (group2 is on wmf.8).
[21:18:54] <wikibugs>	 10Beta-Cluster-Infrastructure: Setup poolcounter daemon in Beta Cluster - https://phabricator.wikimedia.org/T38891#2459811 (10greg)
[21:18:57] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Operations, 07Puppet, 07Tracking: Minimize differences between beta and production (Tracking) - https://phabricator.wikimedia.org/T87220#2459810 (10greg)
[21:19:46] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 07Tracking: Use Beta cluster as a true canary for code deployments (tracking) - https://phabricator.wikimedia.org/T53494#2459812 (10greg)
[21:19:52] <hashar>	 thcipriani: and the zuul/ssh bad interaction seems to nail down to that lock at https://phabricator.wikimedia.org/P3415$223-241  :(
[21:20:14] <hashar>	 suggesting paramiko unable to initialize some random number generator
[21:20:41] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Operations, 07Puppet, 07Technical-Debt, 07Tracking: Minimize differences between beta and production (Tracking) - https://phabricator.wikimedia.org/T87220#986422 (10greg)
[21:21:24] <paladox>	 hashar could that be because we doint support all of ssh hard keys
[21:22:05] <hashar>	 paladox: that is an eventuality
[21:22:13] <paladox>	 Oh
[21:23:32] <hashar>	 there is an apparently blocked read() in the thread below  https://phabricator.wikimedia.org/P3415$266-269
[21:23:48] <hashar>	 which is reading 32*32 bytes from /dev/urandom
[21:24:11] <paladox>	 Oh
[21:24:40] <paladox>	 Could that mean it was updated in the new zuul version meaning the one we tryed but failed
[21:24:57] <paladox>	 but would be a different version in the current one were using
[21:25:09] <thcipriani>	 hmm that seems weird. were you able to recreate?
[21:26:05] <paladox>	 Ive been testing on a test instance
[21:26:11] <paladox>	 and seems to be stable with ssh using jessie
[21:26:29] <paladox>	 but havent tested stats
[21:26:38] <paladox>	 hashar ^^
[21:37:07] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 10Monitoring, 07Jenkins, 07Technical-Debt: Create metrics of Beta Cluster stability using a Jenkins job - https://phabricator.wikimedia.org/T106421#2459869 (10greg)
[21:40:30] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2459875 (10hashar) I have filtered out a bunch of threads from P3415 and ends up with three threads P3423  {P3423}  Namely:  * zuul.connection.ger...
[21:50:18] <hashar>	 thcipriani: havent managed to recreate :(
[21:50:48] <hashar>	 I am digging in zuul debug logs :D
[21:54:41] <paladox>	 hashar :)
[21:56:36] <hashar>	 nothing of interest :(
[21:57:24] <etonkovidova>	 something is wrong with fluorine?   ssh deployment-fluorine.eqiad.wmflabs  -- does not do anything 
[21:58:44] <wmf-insecte>	 Yippee, build fixed!
[21:58:44] <wmf-insecte>	 Project selenium-Core » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #86: 09FIXED in 6 min 43 sec: https://integration.wikimedia.org/ci/job/selenium-Core/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/86/
[22:06:48] <etonkovidova>	 greg-g: any idea why ssh deployment-fluorine.eqiad.wmflabs does not work for me? and for matt_flaschen too
[22:08:23] <matt_flaschen>	 Yeah, deployment-tin is fine, but that just hangs.
[22:08:23] <wikibugs>	 10Beta-Cluster-Infrastructure: Can't ssh to deployment-fluorine - https://phabricator.wikimedia.org/T140313#2459953 (10greg)
[22:08:28] <greg-g>	 etonkovidova: not sure ^
[22:08:42] <greg-g>	 I can't get to it from deployment-tin either
[22:09:58] <etonkovidova>	 greg-g: not too urgent, but I'd like to take a look at betalabs logs whether centralauth errors are there
[22:10:09] <greg-g>	 yeah
[22:10:39] <wikibugs>	 10Beta-Cluster-Infrastructure: Can't ssh to deployment-fluorine - https://phabricator.wikimedia.org/T140313#2459968 (10greg) Last mention in RelEng SAL is from May: https://tools.wmflabs.org/sal/releng?p=0&q=fluorine&d=
[22:13:15] <paladox>	 hashar i guess we can make the task a high priorty since it could cause us issues with the current zuul version we have now later
[22:13:31] <paladox>	 Unless the issue is only because of week ssh
[22:13:51] <paladox>	 seems ssh works very good for zuul in jessie and gerrit 2.12
[22:16:54] <hashar>	 thcipriani: seems openstack had the same issue in December 2014 :)
[22:17:28] <thcipriani>	 Oh good.
[22:17:44] <hashar>	 Oh god.
[22:17:46] <hashar>	 http://paste.openstack.org/show/148067/
[22:17:49] <hashar>	 might be a repro
[22:18:05] <hashar>	 I am copy pasting snippet from the irc log
[22:18:15] <hashar>	 2014-12-09T18:32:28  <jeblair> clarkb, fungi: that script takes 3 seconds on our precise nodepool config, and 1 minute 12 seconds on trusty
[22:18:17] <hashar>	 heeek
[22:18:35] <paladox>	 hashar how did they fix it?
[22:19:40] <hashar>	 I have no idea
[22:19:53] <paladox>	 oh
[22:19:53] <hashar>	 seems it is reproduced when upgrading pycrypto from 2.4.1 to 2.6.1
[22:20:02] <paladox>	 oh
[22:20:09] <paladox>	 Maybe downgrading to 2.4.1
[22:20:13] <paladox>	 may fix it for us
[22:20:40] <paladox>	 But it hasent even been updated since 2014
[22:20:45] <paladox>	 https://github.com/dlitz/pycrypto
[22:20:49] <paladox>	 hashar ^^
[22:21:07] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2460027 (10hashar) HINT HINT. OpenStack infra had a discussion about a similar thing on http://eavesdrop.openstack.org/irclogs/%23openstack-infra/...
[22:27:21] <paladox>	 hashar we should defintly go with 2.4.1 or fix it ou self since upstream havent been active since 2014
[22:29:33] <hashar>	 paladox: does 2.4.1 fix it ?
[22:29:43] <hashar>	 and really it works all fine from the cli
[22:29:48] <paladox>	 Im not sure
[22:29:53] <paladox>	 but it woulden hurt to try
[22:30:02] <paladox>	 if it does then we deffintly can say
[22:30:06] <paladox>	 upstream need to fix it
[22:30:29] <paladox>	 hashar how do we pin pycrypto  to 2.4.1
[22:30:51] <paladox>	 Oh wait we can do pycrypto=>2.4.1<2.4.2
[22:31:23] <paladox>	 How did that get installed
[22:31:26] <paladox>	 through which package
[22:31:29] <paladox>	 ?
[22:31:35] <paladox>	 hashar ^^
[22:32:48] <hashar>	 thcipriani: tldr :  might just be thread starved waiting for paramiko to generate some random numbers
[22:32:59] <hashar>	 thcipriani: and I guess we want some new paramiko :)
[22:33:30] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2460051 (10hashar) Post lunch: ``` lang=irc 2014-12-09T21:20:28  <jeblair> clarkb, fungi: warning: GMP or MPIR library not found; Not building Cry...
[22:33:35] <hashar>	 pasting IRC logs on the task was again a terrible idea :(
[22:33:46] <thcipriani>	 :D
[22:34:40] <paladox>	 hashar so upgrading paramiko may resolve this problem
[22:34:56] <paladox>	 ?
[22:34:59] <hashar>	 switching to Jessie might
[22:35:00] <hashar>	 but then
[22:35:16] <hashar>	 zuul-server running in foreground worked just fine for 45 or so minutes
[22:35:28] <hashar>	 while being started as a daemon it failed completely
[22:35:45] <thcipriani>	 ^ that's what I don't understand
[22:36:11] <thcipriani>	 Why would it be starved for randomness only as a daemon?
[22:36:33] <hashar>	 yeah that does not make sense
[22:36:39] <hashar>	 their investigation was related to Nodepool
[22:36:43] <hashar>	 which has a ton of threads
[22:36:55] <hashar>	 (I think a thread per instance in the pool +  other threads)
[22:37:04] <hashar>	 so they might have contention
[22:37:14] <paladox>	 hashar it seems upgrading to jessie resolves the issue
[22:37:38] <paladox>	 Would installing pycrypto 2.4.1 work 
[22:38:37] <paladox>	 hashar it is set like paramiko>=1.8.0,<2.0.0
[22:39:23] <paladox>	 hashar http://www.paramiko.org/changelog.html
[22:39:38] <paladox>	 •[Bug] #537: Fix a bug in BufferedPipe.set_event which could cause deadlocks/hangs when one uses select.select against Channel objects (or otherwise calls Channel.fileno after the channel has closed). Thanks to Przemysław Strzelczak for the report & reproduction case, and to Krzysztof Rusek for the fix.
[22:39:44] <paladox>	 •[Bug] #520: (Partial fix) Fix at least one instance of race condition driven threading hangs at end of the Python interpreter session. (Includes a docs update as well - always make sure to .close() your clients!)
[22:40:26] <paladox>	 Maybe all we have to do is update to 1.17.1 and it might fix our problem
[22:41:06] <paladox>	 hashar https://github.com/paramiko/paramiko/issues/520
[22:42:07] <hashar>	 paladox: there might thousands of bug reports and commit related to that
[22:42:16] <paladox>	 oh
[22:43:04] <paladox>	 hashar should we try just 1.8.0
[22:43:18] <hashar>	 thcipriani: in the repro script they try to generate 2048 bits RSA keys
[22:43:57] <hashar>	 thcipriani: so maybe with a pool of 500 threads  quickly spawning/despawning that might exhaust the paramiko built in urandom generator 
[22:44:08] <wikibugs>	 06Release-Engineering-Team: Define broad criteria for what blocks (and what doesn't block) the train - https://phabricator.wikimedia.org/T140319#2460093 (10greg)
[22:44:08] <hashar>	 thcipriani: anyway seems unrelated 
[22:44:22] <hashar>	 thcipriani: will try to repro on labs and if I manage to do,  use strace :]
[22:44:25] <hashar>	 the nuke option
[22:44:37] <paladox>	 hashar
[22:44:38] <paladox>	 •[Bug] #577: (via #578; should also fix #718, #560) Fix stalled/hung SFTP downloads by cleaning up some threading lock issues. Thanks to Stephen C. Pope for the patch.
[22:44:40] <hashar>	 meanwhile it is 1am here so .. sleep(now)
[22:45:31] <thcipriani>	 heh, yeah I figured it was getting late for you :)
[22:46:43] <hashar>	 thcipriani: and with Jessie systemd  zuul will be running in the foreground :D
[22:46:48] <hashar>	 but might well hit the same issue bah
[22:46:55] <hashar>	 whatever. it is to late
[22:46:59] <hashar>	 too late
[22:47:07] <wikibugs>	 06Release-Engineering-Team: Define post-blocked train resumption strategies - https://phabricator.wikimedia.org/T140320#2460109 (10greg)
[22:48:08] <greg-g>	 hashar: go sleep!
[22:48:14] <hashar>	 sure 
[22:48:24] <hashar>	 greg-g: can you fill a task? ;]
[22:48:33] <hashar>	 good night folks!
[22:58:48] <wikibugs>	 06Release-Engineering-Team, 06ArchCom, 06Developer-Relations, 10Phabricator, 06Team-Practices: Consider alternative processes for Unbreak Now bugs, especially those which cross-cut components - https://phabricator.wikimedia.org/T140207#2456573 (10Jdforrester-WMF) >>! In T140207#2459178, @greg wrote: > Th...
[23:03:46] <wikibugs>	 10Beta-Cluster-Infrastructure: Can't ssh to deployment-fluorine - https://phabricator.wikimedia.org/T140313#2459953 (10AlexMonk-WMF) I pressed buttons and it seems to work now. Try again?
[23:24:02] <wikibugs>	 10Beta-Cluster-Infrastructure: Can't ssh to deployment-fluorine - https://phabricator.wikimedia.org/T140313#2460205 (10Mattflaschen-WMF) 05Open>03Resolved a:03AlexMonk-WMF
[23:28:35] <wikibugs>	 10Beta-Cluster-Infrastructure: Can't ssh to deployment-fluorine - https://phabricator.wikimedia.org/T140313#2460214 (10Etonkovidova) @AlexMonk-WMF and @greg  - it miraculously works now. Thx!
[23:31:56] <wikibugs>	 10Beta-Cluster-Infrastructure: Can't ssh to deployment-fluorine - https://phabricator.wikimedia.org/T140313#2460232 (10AlexMonk-WMF) So I first tried a 'hard reboot' in horizon but didn't immediately notice anything, I then logged into silver and `OS_TENANT_NAME=deployment-prep nova reboot deployment-fluorine` a...