[00:01:46] <grrrit-wm>	 (03CR) 10Dduvall: [C: 032] Fix MEDIAWIKI_URL and enable screenshots [integration/jenkins] - 10https://gerrit.wikimedia.org/r/222219 (owner: 10Dduvall)
[00:02:22] <grrrit-wm>	 (03Merged) 10jenkins-bot: Fix MEDIAWIKI_URL and enable screenshots [integration/jenkins] - 10https://gerrit.wikimedia.org/r/222219 (owner: 10Dduvall)
[00:16:29] <shinken-wm>	 RECOVERY - Puppet failure on deployment-stream is OK Less than 1.00% above the threshold [0.0]
[00:20:43] <grrrit-wm>	 (03PS1) 10Dduvall: Fix tests directory and "missing skin" errors in MW-Selenium jobs [integration/config] - 10https://gerrit.wikimedia.org/r/222223 (https://phabricator.wikimedia.org/T103039) 
[02:08:34] <wikibugs>	 10Beta-Cluster, 10MediaWiki-API: mw: interwiki prefix missing on beta cluster, so API's "complete documentation" is a 404. - https://phabricator.wikimedia.org/T104504#1419533 (10Legoktm) Interwikis are a local site customization, so messages should not depend on it and should use full urls instead.
[03:41:58] <shinken-wm>	 PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL 100.00% of data above the critical threshold [0.0]
[04:59:12] <hashar>	 favorite day of the year is today
[04:59:38] <hashar>	 being wake up by the storm, having breakfast while looking at thunder strikes and listening to the rain
[05:36:18] <wmf-insecte>	 Yippee, build fixed!
[05:36:19] <wmf-insecte>	 Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #467: FIXED in 34 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/467/
[06:40:17] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-bastion is OK All targets OK
[06:59:02] <shinken-wm>	 PROBLEM - Puppet failure on integration-zuul-server is CRITICAL 100.00% of data above the critical threshold [0.0]
[07:12:08] <wikibugs>	 10Beta-Cluster, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 5ContentTranslation-Release6: Setup new wikis in Beta Cluster for Content Translation - https://phabricator.wikimedia.org/T90683#1419882 (10Arrbee)
[07:40:49] <polybuildr>	 git.wikimedia.org is down again?
[07:50:35] <hashar>	 polybuildr: yeah recurring issue
[07:50:44] <hashar>	 been crashing for a few days now :-(
[07:50:46] <polybuildr>	 okay :)
[07:50:48] <polybuildr>	 oh :/
[07:50:53] <hashar>	 gotta poke ops in #wikimedia-operations
[07:51:01] <hashar>	 apparently we got to restart the service
[07:51:04] <hashar>	 and that needs root access
[07:51:19] <polybuildr>	 are you poking there now?
[07:51:24] <polybuildr>	 okay :P
[07:51:27] <hashar>	 don't have access
[07:51:27] <polybuildr>	 just did.
[07:51:30] <hashar>	 well I have basic shell 
[07:55:19] <hashar>	 polybuildr: from a quick glance, it seems to be overloaded by some requests
[07:55:27] <polybuildr>	 possibly a robot
[07:55:31] <hashar>	 yeah
[07:55:32] <hashar>	 sounds likely
[07:55:53] <hashar>	 I can't access any log though :-/
[08:12:37] <polybuildr>	 okay, I've been wondering for about a month now. what's a "callsign" with reference to gerrit repos?
[08:13:14] <polybuildr>	 never got around to asking
[08:15:57] <polybuildr>	 hmm
[08:15:58] <polybuildr>	 https://www.mediawiki.org/wiki/Phabricator/Diffusion/Callsign_naming_conventions/Existing_repositories
[08:16:00] <polybuildr>	 gives me some hint
[08:29:49] <polybuildr>	 hashar: what do you think should be done about the if/elses?
[08:29:56] <polybuildr>	 An exception in the extension's phpcs.xml?
[08:30:40] <polybuildr>	 hashar: https://www.mediawiki.org/wiki/Manual:Coding_conventions/PHP#Alternative_syntax_for_control_structures only says "should be avoided", so this may be a valid case.
[08:36:38] <grrrit-wm>	 (03CR) 10Paladox: [C: 031] Use upstream codesniffer 2.3.3 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/220399 (owner: 10Legoktm)
[08:38:47] <hashar>	 polybuildr: I think we should one-liners 
[08:38:54] <hashar>	 like <?php whatever ?>
[09:00:28] <grrrit-wm>	 (03PS1) 10Mobrovac: Add script which installs the devDependencies for Node.js services [integration/jenkins] - 10https://gerrit.wikimedia.org/r/222259 
[09:01:01] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Add script which installs the devDependencies for Node.js services [integration/jenkins] - 10https://gerrit.wikimedia.org/r/222259 (owner: 10Mobrovac)
[09:03:40] <grrrit-wm>	 (03PS2) 10Mobrovac: Add script which installs the devDependencies for Node.js services [integration/jenkins] - 10https://gerrit.wikimedia.org/r/222259 
[09:04:11] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Add script which installs the devDependencies for Node.js services [integration/jenkins] - 10https://gerrit.wikimedia.org/r/222259 (owner: 10Mobrovac)
[09:04:19] <hashar>	 mobrovac: 
[09:04:19] <hashar>	 " Wikimedia style uses 4 spaces for indentation
[09:04:20] <hashar>	 autocmd BufRead $HOME/projects/operations/puppet/* set sw=4 ts=4 sts=4 et
[09:04:20] <hashar>	 autocmd BufRead $HOME/puppet/* set sw=4 ts=4 sts=4 et
[09:05:24] <hashar>	 mobrovac: https://github.com/hashar/alix/blob/master/vimrc
[09:05:26] <hashar>	 mon vimrc
[09:05:36] <hashar>	 thcipriani must have a rather nice vimrc as well
[09:06:06] <hashar>	 plugins https://github.com/hashar/alix/tree/master/vim
[09:06:14] <hashar>	 as git submodules
[09:06:20] <hashar>	 loaded using some magic other module 
[09:07:39] <grrrit-wm>	 (03PS3) 10Mobrovac: Add script which installs the devDependencies for Node.js services [integration/jenkins] - 10https://gerrit.wikimedia.org/r/222259 
[09:08:09] <hashar>	 mobrovac: https://github.com/scrooloose/syntastic
[09:08:14] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Add script which installs the devDependencies for Node.js services [integration/jenkins] - 10https://gerrit.wikimedia.org/r/222259 (owner: 10Mobrovac)
[09:08:16] <hashar>	 will launch flake8 on save :-}
[09:08:24] <hashar>	 gotta install flake8 first though
[09:09:18] <grrrit-wm>	 (03PS4) 10Mobrovac: Add script which installs the devDependencies for Node.js services [integration/jenkins] - 10https://gerrit.wikimedia.org/r/222259 
[09:11:17] <wikibugs>	 10Beta-Cluster: Make MathML rendering default on BETA wikipedia - https://phabricator.wikimedia.org/T104550#1420076 (10Physikerwelt) 3NEW a:3Physikerwelt
[09:11:25] <mobrovac>	 hashar: | node_modules | package.json -> src/package.json
[09:11:58] <mobrovac>	 ./package.json
[09:12:09] <wikibugs>	 10Beta-Cluster: Make MathML rendering default on BETA wikipedia - https://phabricator.wikimedia.org/T104550#1420085 (10Physikerwelt)
[09:16:56] <hashar>	 mobrovac: I got some network issue on my end :(
[09:23:12] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] "Lets go!" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/222259 (owner: 10Mobrovac)
[09:23:44] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add script which installs the devDependencies for Node.js services [integration/jenkins] - 10https://gerrit.wikimedia.org/r/222259 (owner: 10Mobrovac)
[09:24:59] <grrrit-wm>	 (03CR) 10Hashar: "Force deployed with:" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/222259 (owner: 10Mobrovac)
[09:25:18] <wikibugs>	 6Release-Engineering, 10Gather, 10MobileFrontend, 10Reading Web Planning, 7Epic: [EPIC] Create a formal release process for MobileFrontend/Gather - https://phabricator.wikimedia.org/T100296#1420114 (10Jhernandez)
[09:26:06] <grrrit-wm>	 (03CR) 10Hashar: "Typo:" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/222259 (owner: 10Mobrovac)
[09:29:06] <wikibugs>	 6Release-Engineering, 10Gather, 10MobileFrontend, 10Reading Web Planning, 7Epic: [EPIC] Create a formal release process for MobileFrontend/Gather - https://phabricator.wikimedia.org/T100296#1420135 (10Jhernandez) p:5Triage>3High
[09:38:26] <grrrit-wm>	 (03PS1) 10Hashar: Inject npm devDependencies into deploy.git node_modules [integration/config] - 10https://gerrit.wikimedia.org/r/222261 
[09:38:37] <hashar>	 mobrovac: https://gerrit.wikimedia.org/r/222261  :D
[09:39:08] <grrrit-wm>	 (03CR) 10Mobrovac: [C: 031] Inject npm devDependencies into deploy.git node_modules [integration/config] - 10https://gerrit.wikimedia.org/r/222261 (owner: 10Hashar)
[09:40:44] <hashar>	 mobrovac: https://www.mediawiki.org/wiki/Continuous_integration/Jenkins_job_builder :D
[09:52:35] <mobrovac>	 hashar: 
[09:52:36] <mobrovac>	 WARNING:jenkins_jobs.modules.scm:git:'clean: bool' configuration format is deprecated, please use the extension style format to configure this option.
[09:52:36] <mobrovac>	 WARNING:jenkins_jobs.modules.scm:git:'disable-submodules' is deprecated, please convert to use the 'submodule' section instead as support for this top level option will be removed in a future release.
[10:04:55] <wmf-insecte>	 Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #483: FAILURE in 43 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/483/
[10:06:59] <hashar>	 !log adding mobrovac to the integration project so he can ssh to slaves and sudo as jenkins-deploy user
[10:07:03] <qa-morebots>	 Logged the message, Master
[10:09:03] <hashar>	 mobrovac: ssh integration-slave-trusty-1011.integration.eqiad.wmflabs
[10:09:29] <hashar>	 mobrovac: cd /mnt/jenkins-workspace/workspace/mathoid-deploy-npm
[10:17:47] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] "Kartik, the cxserver-deploy-npm job now install the devDependencies in case they are not provided in the deploy.git repo. I verified the" [integration/config] - 10https://gerrit.wikimedia.org/r/222261 (owner: 10Hashar)
[10:21:03] <grrrit-wm>	 (03Merged) 10jenkins-bot: Inject npm devDependencies into deploy.git node_modules [integration/config] - 10https://gerrit.wikimedia.org/r/222261 (owner: 10Hashar)
[11:01:24] <grrrit-wm>	 (03PS1) 10Addshore: Addshore -> WMDE section of whitelist +2 users [integration/config] - 10https://gerrit.wikimedia.org/r/222275 
[11:03:58] <grrrit-wm>	 (03PS1) 10Addshore: Add Paladox to users to have +2V jobs run! [integration/config] - 10https://gerrit.wikimedia.org/r/222276 
[11:31:09] <grrrit-wm>	 (03CR) 10Paladox: [C: 031] Add Paladox to users to have +2V jobs run! [integration/config] - 10https://gerrit.wikimedia.org/r/222276 (owner: 10Addshore)
[12:04:05] <wikibugs>	 6Release-Engineering, 10Gather, 10MobileFrontend, 10Reading Web Planning, and 3 others: [EPIC] Encourage developers to increase code coverage - https://phabricator.wikimedia.org/T100294#1420462 (10Jhernandez)
[12:04:27] <wikibugs>	 6Release-Engineering, 10Gather, 10MobileFrontend, 10Reading Web Planning, and 3 others: [EPIC] Encourage developers to increase code coverage - https://phabricator.wikimedia.org/T100294#1309866 (10Jhernandez)
[12:07:16] <shinken-wm>	 PROBLEM - Puppet failure on integration-slave-jessie-1001 is CRITICAL 100.00% of data above the critical threshold [0.0]
[12:09:34] <grrrit-wm>	 (03PS1) 10Hashar: Migrate whitespaces jobs to labs (on Trusty) [integration/config] - 10https://gerrit.wikimedia.org/r/222280 (https://phabricator.wikimedia.org/T101966) 
[12:10:11] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] Migrate whitespaces jobs to labs (on Trusty) [integration/config] - 10https://gerrit.wikimedia.org/r/222280 (https://phabricator.wikimedia.org/T101966) (owner: 10Hashar)
[12:12:15] <grrrit-wm>	 (03Merged) 10jenkins-bot: Migrate whitespaces jobs to labs (on Trusty) [integration/config] - 10https://gerrit.wikimedia.org/r/222280 (https://phabricator.wikimedia.org/T101966) (owner: 10Hashar)
[12:14:27] <grrrit-wm>	 (03PS1) 10Hashar: Migrate 'mediawiki-core-jslint to labs (on Trusty) [integration/config] - 10https://gerrit.wikimedia.org/r/222281 (https://phabricator.wikimedia.org/T101966) 
[12:14:44] <grrrit-wm>	 (03PS2) 10Hashar: Migrate mediawiki-core-jslint to labs (on Trusty) [integration/config] - 10https://gerrit.wikimedia.org/r/222281 (https://phabricator.wikimedia.org/T101966) 
[12:15:27] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] Migrate mediawiki-core-jslint to labs (on Trusty) [integration/config] - 10https://gerrit.wikimedia.org/r/222281 (https://phabricator.wikimedia.org/T101966) (owner: 10Hashar)
[12:17:26] <grrrit-wm>	 (03Merged) 10jenkins-bot: Migrate mediawiki-core-jslint to labs (on Trusty) [integration/config] - 10https://gerrit.wikimedia.org/r/222281 (https://phabricator.wikimedia.org/T101966) (owner: 10Hashar)
[12:18:04] <grrrit-wm>	 (03PS1) 10Hashar: Stop generating mediawiki-core-phpcs (ununsed) [integration/config] - 10https://gerrit.wikimedia.org/r/222282 (https://phabricator.wikimedia.org/T101966) 
[12:18:42] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] Stop generating mediawiki-core-phpcs (ununsed) [integration/config] - 10https://gerrit.wikimedia.org/r/222282 (https://phabricator.wikimedia.org/T101966) (owner: 10Hashar)
[12:20:44] <grrrit-wm>	 (03Merged) 10jenkins-bot: Stop generating mediawiki-core-phpcs (ununsed) [integration/config] - 10https://gerrit.wikimedia.org/r/222282 (https://phabricator.wikimedia.org/T101966) (owner: 10Hashar)
[12:24:37] <grrrit-wm>	 (03PS1) 10Hashar: Migrate erblint-HEAD to labs slave (still on Precise) [integration/config] - 10https://gerrit.wikimedia.org/r/222283 (https://phabricator.wikimedia.org/T101966) 
[12:24:54] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] Migrate erblint-HEAD to labs slave (still on Precise) [integration/config] - 10https://gerrit.wikimedia.org/r/222283 (https://phabricator.wikimedia.org/T101966) (owner: 10Hashar)
[12:26:49] <grrrit-wm>	 (03Merged) 10jenkins-bot: Migrate erblint-HEAD to labs slave (still on Precise) [integration/config] - 10https://gerrit.wikimedia.org/r/222283 (https://phabricator.wikimedia.org/T101966) (owner: 10Hashar)
[13:13:55] <grrrit-wm>	 (03PS1) 10Hashar: conftool: no more py34, etcd to be ported on trusty [integration/config] - 10https://gerrit.wikimedia.org/r/222288 
[13:14:05] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] conftool: no more py34, etcd to be ported on trusty [integration/config] - 10https://gerrit.wikimedia.org/r/222288 (owner: 10Hashar)
[13:14:21] <grrrit-wm>	 (03PS2) 10Hashar: conftool: no more py34, etcd to be ported on trusty [integration/config] - 10https://gerrit.wikimedia.org/r/222288 (https://phabricator.wikimedia.org/T103972) 
[13:15:46] <wikibugs>	 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Install etcd on Jessie CI slaves - https://phabricator.wikimedia.org/T103976#1420662 (10hashar)
[13:15:51] <wikibugs>	 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Create CI slaves using Debian Jessie (tracking) - https://phabricator.wikimedia.org/T94836#1420664 (10hashar)
[13:19:57] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] conftool: no more py34, etcd to be ported on trusty [integration/config] - 10https://gerrit.wikimedia.org/r/222288 (https://phabricator.wikimedia.org/T103972) (owner: 10Hashar)
[13:21:44] <grrrit-wm>	 (03Merged) 10jenkins-bot: conftool: no more py34, etcd to be ported on trusty [integration/config] - 10https://gerrit.wikimedia.org/r/222288 (https://phabricator.wikimedia.org/T103972) (owner: 10Hashar)
[13:42:53] <grrrit-wm>	 (03PS1) 10Hashar: conftool tox-flake8 is now voting [integration/config] - 10https://gerrit.wikimedia.org/r/222293 
[13:43:23] <grrrit-wm>	 (03CR) 10Hashar: [C: 04-2] "Depends on changes to be merged in conftool repo. See commit summary." [integration/config] - 10https://gerrit.wikimedia.org/r/222293 (owner: 10Hashar)
[14:02:09] <grrrit-wm>	 (03PS2) 10Addshore: Add Paladox to the whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/222276 
[14:02:29] <grrrit-wm>	 (03PS3) 10Addshore: Add Paladox to the whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/222276 
[14:03:01] <grrrit-wm>	 (03CR) 10Addshore: "It looks like I reopened this @ https://gerrit.wikimedia.org/r/#/c/222276/ before seeing this one" [integration/config] - 10https://gerrit.wikimedia.org/r/215064 (owner: 10Aude)
[14:03:19] <grrrit-wm>	 (03CR) 10Addshore: "People reviewing should not https://gerrit.wikimedia.org/r/#/c/215064/ was abandoned" [integration/config] - 10https://gerrit.wikimedia.org/r/222276 (owner: 10Addshore)
[14:03:26] <grrrit-wm>	 (03CR) 10Addshore: "*note" [integration/config] - 10https://gerrit.wikimedia.org/r/222276 (owner: 10Addshore)
[14:05:11] <jzerebecki>	 hashar: Paladox pointed out that Krenair blocked a past patch to add Paladox to the whitelist https://gerrit.wikimedia.org/r/#/c/215064/ is it supposed to work this way? i.e. that we hold the bar high enough instead of letting people try and if they fail remove them and clean up after them?
[14:05:50] <hashar>	 jzerebecki: I sent a patch as well
[14:05:54] <Krenair>	 jzerebecki, you realise what this whitelist is for, right?
[14:05:56] <hashar>	 krenair pointed me to https://gerrit.wikimedia.org/r/#/c/215064/
[14:06:02] <hashar>	 where Krinkle -1 ed the whitelist
[14:06:24] <Krenair>	 It's to let people run code on our hosts.
[14:07:40] <ebernhardson>	 i dunno how isolated our systems are ... but as an example systems like travis ci let completely random people compile and run C code on their hosts...
[14:08:48] <jzerebecki>	 ebernhardson: we don't have that much isolation yet, there is the ci isolation project that will achive that
[14:12:23] <jzerebecki>	 Krenair: i realise and agree with your assessment. the thing is that 1) people do recheck on his patches without first reading them because they trust him enough and 2) somethimes in wikimedia we give out rights even without being sure that people might break stuff, but just clean up afterwards
[14:12:57] <jzerebecki>	 it seems from h ashars answer that there is agreement that 2 is not the case for ci
[14:14:43] <jzerebecki>	 addshore: ^^
[14:15:48] <addshore>	 *agrees* I mean I love typing recheck and everything... but....
[14:18:45] <addshore>	 On the flip side, if the isolation is really that bad and we can't trust he isn't going to accidently blow the world up then abandon it again
[14:18:53] <addshore>	 as I said I hadn't seen aude's previous patchset
[15:04:00] <grrrit-wm>	 (03CR) 10Paladox: Migrate mediawiki-core-jslint to labs (on Trusty) (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/222281 (https://phabricator.wikimedia.org/T101966) (owner: 10Hashar)
[15:11:58] <wikibugs>	 10Browser-Tests: mediawiki_selenium always use the same default xvfb display 99 - https://phabricator.wikimedia.org/T73602#1421129 (10zeljkofilipin) a:5zeljkofilipin>3None
[15:15:52] <grrrit-wm>	 (03PS1) 10Paladox: Use Trusty for mediawiki-core-jslint [integration/config] - 10https://gerrit.wikimedia.org/r/222318 
[15:16:25] <grrrit-wm>	 (03PS2) 10Paladox: Use Trusty for mediawiki-core-jslint [integration/config] - 10https://gerrit.wikimedia.org/r/222318 
[15:16:37] <grrrit-wm>	 (03PS3) 10Paladox: Use Trusty for mediawiki-core-jslint [integration/config] - 10https://gerrit.wikimedia.org/r/222318 
[15:17:47] <grrrit-wm>	 (03CR) 10Paladox: "Hi I followed it up here https://gerrit.wikimedia.org/r/#/c/222318/ since tests for jslint are failing. Please see https://integration.wik" [integration/config] - 10https://gerrit.wikimedia.org/r/222281 (https://phabricator.wikimedia.org/T101966) (owner: 10Hashar)
[15:18:11] <grrrit-wm>	 (03CR) 10Hashar: "I haven't had time today to review this change. I am worried about the potential side effects on the mwext-*-testextension-* jobs :\" [integration/config] - 10https://gerrit.wikimedia.org/r/222223 (https://phabricator.wikimedia.org/T103039) (owner: 10Dduvall)
[15:19:30] <grrrit-wm>	 (03CR) 10Paladox: "Causing jslint to fail this fixes it by using trusty instead. Please see an example of it failing at https://integration.wikimedia.org/ci/" [integration/config] - 10https://gerrit.wikimedia.org/r/222318 (owner: 10Paladox)
[15:19:52] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] "Thanks!" [integration/config] - 10https://gerrit.wikimedia.org/r/222318 (owner: 10Paladox)
[15:21:03] <grrrit-wm>	 (03CR) 10Hashar: "reconfigured mediawiki-core-jslint" [integration/config] - 10https://gerrit.wikimedia.org/r/222318 (owner: 10Paladox)
[15:22:42] <grrrit-wm>	 (03Merged) 10jenkins-bot: Use Trusty for mediawiki-core-jslint [integration/config] - 10https://gerrit.wikimedia.org/r/222318 (owner: 10Paladox)
[15:22:44] <hashar>	 off to get kids back home *wave*
[15:23:08] <greg-g>	 hah! there's now a dislike (thumbs down) icon in phab for declined tasks
[15:26:59] <wikibugs>	 10Beta-Cluster, 10MediaWiki-API: mw: interwiki prefix missing on beta cluster, so API's "complete documentation" is a 404. - https://phabricator.wikimedia.org/T104504#1421182 (10Anomie) >>! In T104504#1419533, @Legoktm wrote: > Interwikis are a local site customization, so messages should not depend on it and...
[15:27:38] <grrrit-wm>	 (03CR) 10Paladox: "It says this" [integration/config] - 10https://gerrit.wikimedia.org/r/222318 (owner: 10Paladox)
[15:28:50] <wmf-insecte>	 Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #582: FAILURE in 50 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/582/
[15:31:37] <grrrit-wm>	 (03CR) 10Paladox: "Never mind works now." [integration/config] - 10https://gerrit.wikimedia.org/r/222318 (owner: 10Paladox)
[15:31:46] <grrrit-wm>	 (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/222318 (owner: 10Paladox)
[15:36:06] <grrrit-wm>	 (03CR) 10JanZerebecki: [C: 04-1] "Heh, we are now up to 3 of this patch. We should really get https://www.mediawiki.org/wiki/Continuous_integration/Architecture/Isolation ." [integration/config] - 10https://gerrit.wikimedia.org/r/222276 (owner: 10Addshore)
[15:49:42] <wikibugs>	 6Release-Engineering, 10Gather, 10MobileFrontend, 3Reading-Web: Ensure tests are green before they hit production - https://phabricator.wikimedia.org/T101865#1421262 (10phuedx) Alternatively, the tests could be run when we merge from `development` to `master`, which we (#Reading-Web) control when we do.
[16:12:31] <grrrit-wm>	 (03PS5) 10Addshore: Run mw-set-env for WikibaseQuality* tests [integration/config] - 10https://gerrit.wikimedia.org/r/221899 (https://phabricator.wikimedia.org/T103626) 
[16:13:52] <grrrit-wm>	 (03PS1) 10Addshore: Fix comment typo in mw-set-env.sh [integration/jenkins] - 10https://gerrit.wikimedia.org/r/222328 
[16:19:48] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-restbase01 is CRITICAL 100.00% of data above the critical threshold [43200.0]
[16:24:41] <wikibugs>	 10Browser-Tests: Support headless gem's video recording feature for headless Jenkins jobs - https://phabricator.wikimedia.org/T104583#1421370 (10dduvall) 3NEW
[16:25:32] <grrrit-wm>	 (03PS6) 10JanZerebecki: Run mw-set-env for WikibaseQuality* tests [integration/config] - 10https://gerrit.wikimedia.org/r/221899 (https://phabricator.wikimedia.org/T103626) (owner: 10Addshore)
[16:31:43] <grrrit-wm>	 (03CR) 10JanZerebecki: [C: 032] "Is deployed to Jenkins." [integration/config] - 10https://gerrit.wikimedia.org/r/221899 (https://phabricator.wikimedia.org/T103626) (owner: 10Addshore)
[16:31:52] <ostriches>	 twentyafterfour: We need that Phab/Gerrit plugin thingie soonish. I'd like to start redirecting gerrit "viewer" traffic to it soon.
[16:31:58] <ostriches>	 Instead of using Gitblit
[16:37:49] <grrrit-wm>	 (03Merged) 10jenkins-bot: Run mw-set-env for WikibaseQuality* tests [integration/config] - 10https://gerrit.wikimedia.org/r/221899 (https://phabricator.wikimedia.org/T103626) (owner: 10Addshore)
[16:40:37] <twentyafterfour>	 ostriches: the phab/gerrit plugin thingie?
[16:45:13] <twentyafterfour>	 which plugin are you referring to?
[16:52:02] <ostriches>	 twentyafterfour: The mapping of gerrit repo names -> Phab callsigns thingie
[16:52:11] <ostriches>	 + redirection
[17:01:44] <twentyafterfour>	 ah, yeah
[17:01:55] <twentyafterfour>	 that won't be difficult
[17:09:02] <twentyafterfour>	 ostriches: are the repos named properly in phab currently? hmm
[17:09:28] <twentyafterfour>	 we actually know the mapping because they all have the gerrit url stored
[17:22:19] <ostriches>	 twentyafterfour: Yeah, we just need to store that now in a way that persists after gerrit.wm.o doesn't actually exist :)
[17:24:10] <James_F>	 greg-g: Want me to pull the plug?
[17:30:02] <grrrit-wm>	 (03PS1) 10Dduvall: WIP: Attempting video recording with headless [selenium] - 10https://gerrit.wikimedia.org/r/222346 
[17:32:32] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] WIP: Attempting video recording with headless [selenium] - 10https://gerrit.wikimedia.org/r/222346 (owner: 10Dduvall)
[18:01:27] <twentyafterfour>	 ostriches: ok I'll come up with something...
[18:01:48] <ostriches>	 Ok. If you can at least get the skeleton of something working and up, I can help hack on it
[18:01:58] <ostriches>	 I know the full range of urls we'll want to redirect eventually.
[18:02:15] <twentyafterfour>	 ostriches: ok sounds good. how about a plain php array or json with static mappings?
[18:02:48] <ostriches>	 Something like that. It'll need to know a couple of URL schemes though
[18:02:51] <twentyafterfour>	 a static .php include file would probably be most efficient
[18:03:03] <twentyafterfour>	 what url schemes?
[18:03:16] <twentyafterfour>	 it needs to be able to redirect gitblit urls and what else?
[18:05:47] <twentyafterfour>	 ostriches: other than gitblit urls, what else do we need to redirect?
[18:10:34] <ostriches>	 Possibly some gerrit urls in the future as we get closer to decom.
[19:03:44] <wikibugs>	 10Continuous-Integration-Infrastructure, 6Labs, 10Labs-Infrastructure: "puppet-compiler02" Jenkins slave is no more connected - https://phabricator.wikimedia.org/T104428#1422138 (10yuvipanda) p:5Unbreak!>3Normal (Switching out of UBN!, nobody is working on it)
[19:14:37] <grrrit-wm>	 (03CR) 10Krinkle: "Yep. The Release Engineering team is working on it. Bear in mind it is a task of epic proportions. There's a lot of work involved in it. I" [integration/config] - 10https://gerrit.wikimedia.org/r/222276 (owner: 10Addshore)
[19:16:51] <grrrit-wm>	 (03CR) 10Alex Monk: [C: 04-1] "How does this keep getting proposed?" [integration/config] - 10https://gerrit.wikimedia.org/r/222276 (owner: 10Addshore)
[19:17:36] <grrrit-wm>	 (03CR) 10Krinkle: "I'd prefer this wasn't merged. The precompiled legacy node_modules (e.g. not npm jobs but the globally installed jshint) is compiled for P" [integration/config] - 10https://gerrit.wikimedia.org/r/222318 (owner: 10Paladox)
[19:18:58] <grrrit-wm>	 (03PS1) 10Krinkle: Revert "Use Trusty for mediawiki-core-jslint" [integration/config] - 10https://gerrit.wikimedia.org/r/222379 
[19:19:58] <grrrit-wm>	 (03Abandoned) 10Addshore: Add Paladox to the whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/222276 (owner: 10Addshore)
[19:53:19] <grrrit-wm>	 (03CR) 10Paladox: "This should then be recompiled." [integration/config] - 10https://gerrit.wikimedia.org/r/222318 (owner: 10Paladox)
[19:54:18] <grrrit-wm>	 (03CR) 10Paladox: "Will cause it to not run since contintLabsSlave is not on UbuntuPrecise. The patch was a follow-up because he or she forgot to add UbuntuT" [integration/config] - 10https://gerrit.wikimedia.org/r/222379 (owner: 10Krinkle)
[19:54:26] <grrrit-wm>	 (03CR) 10Paladox: [C: 04-1] Revert "Use Trusty for mediawiki-core-jslint" [integration/config] - 10https://gerrit.wikimedia.org/r/222379 (owner: 10Krinkle)
[19:54:34] <grrrit-wm>	 (03CR) 10Krinkle: "Yep. I'll do that later this week. Deploying such recompilation is tricky because it requires all relevant jobs to upgraded at the same ti" [integration/config] - 10https://gerrit.wikimedia.org/r/222318 (owner: 10Paladox)
[19:55:26] <grrrit-wm>	 (03CR) 10Paladox: "Jslint needs to be compiled for trusty." [integration/config] - 10https://gerrit.wikimedia.org/r/222379 (owner: 10Krinkle)
[19:56:14] <grrrit-wm>	 (03CR) 10Paladox: "Ok." [integration/config] - 10https://gerrit.wikimedia.org/r/222318 (owner: 10Paladox)
[19:57:06] <grrrit-wm>	 (03CR) 10Paladox: "If you would like to revert it. Please do this https://gerrit.wikimedia.org/r/#/c/222281/ patch instead or wait until it is recompiled bec" [integration/config] - 10https://gerrit.wikimedia.org/r/222379 (owner: 10Krinkle)
[19:57:53] <grrrit-wm>	 (03CR) 10Krinkle: "No, that is incorrect. Hashar didn't forget anything. And no, contintLabsSlave label is used in labs. We have a dozen Jenkins slaves exact" [integration/config] - 10https://gerrit.wikimedia.org/r/222379 (owner: 10Krinkle)
[19:58:39] <grrrit-wm>	 (03CR) 10Krinkle: [C: 032] Revert "Use Trusty for mediawiki-core-jslint" [integration/config] - 10https://gerrit.wikimedia.org/r/222379 (owner: 10Krinkle)
[20:16:23] <grrrit-wm>	 (03PS2) 10Krinkle: Revert "Use Trusty for mediawiki-core-jslint" [integration/config] - 10https://gerrit.wikimedia.org/r/222379 
[20:16:32] <grrrit-wm>	 (03CR) 10Krinkle: [C: 032] Revert "Use Trusty for mediawiki-core-jslint" [integration/config] - 10https://gerrit.wikimedia.org/r/222379 (owner: 10Krinkle)
[20:18:35] <grrrit-wm>	 (03Merged) 10jenkins-bot: Revert "Use Trusty for mediawiki-core-jslint" [integration/config] - 10https://gerrit.wikimedia.org/r/222379 (owner: 10Krinkle)
[20:23:26] <grrrit-wm>	 (03PS1) 10Krinkle: Remove dead-end symlinks for phplint [integration/jenkins] - 10https://gerrit.wikimedia.org/r/222471 
[20:23:28] <grrrit-wm>	 (03PS1) 10Krinkle: Rebuild tools packages for Ubuntu Trusty [integration/jenkins] - 10https://gerrit.wikimedia.org/r/222472 
[20:24:19] <grrrit-wm>	 (03CR) 10Krinkle: [C: 032] Remove dead-end symlinks for phplint [integration/jenkins] - 10https://gerrit.wikimedia.org/r/222471 (owner: 10Krinkle)
[20:25:26] <grrrit-wm>	 (03Merged) 10jenkins-bot: Remove dead-end symlinks for phplint [integration/jenkins] - 10https://gerrit.wikimedia.org/r/222471 (owner: 10Krinkle)
[20:30:39] <grrrit-wm>	 (03PS2) 10Krinkle: Rebuild tools packages for Ubuntu Trusty [integration/jenkins] - 10https://gerrit.wikimedia.org/r/222472 
[20:39:34] <grrrit-wm>	 (03CR) 10Paladox: "But one of my patches failed because of this change." [integration/config] - 10https://gerrit.wikimedia.org/r/222379 (owner: 10Krinkle)
[20:41:14] <grrrit-wm>	 (03CR) 10Paladox: [C: 031] Rebuild tools packages for Ubuntu Trusty [integration/jenkins] - 10https://gerrit.wikimedia.org/r/222472 (owner: 10Krinkle)
[20:41:40] <grrrit-wm>	 (03CR) 10Krinkle: "That has nothing to do with Trusty/Precise. That has to do with labs/production. More specifically, that relates to the 'defaults' propert" [integration/config] - 10https://gerrit.wikimedia.org/r/222379 (owner: 10Krinkle)
[20:43:18] <grrrit-wm>	 (03CR) 10Paladox: "Hi but works if we are on trusty. I mean the problem fixed its self when it was changed to trusty." [integration/config] - 10https://gerrit.wikimedia.org/r/222379 (owner: 10Krinkle)
[20:47:57] <grrrit-wm>	 (03CR) 10Krinkle: "No, the problem was not fixed when this moved to Trusty." [integration/config] - 10https://gerrit.wikimedia.org/r/222379 (owner: 10Krinkle)
[20:48:30] <grrrit-wm>	 (03CR) 10Krinkle: "See for example https://integration.wikimedia.org/ci/job/mediawiki-core-jslint/12952/console – this one is passing, but also says "ERROR: " [integration/config] - 10https://gerrit.wikimedia.org/r/222379 (owner: 10Krinkle)
[20:56:24] <grrrit-wm>	 (03CR) 10Krinkle: [C: 032] Rebuild tools packages for Ubuntu Trusty [integration/jenkins] - 10https://gerrit.wikimedia.org/r/222472 (owner: 10Krinkle)
[20:56:58] <grrrit-wm>	 (03Merged) 10jenkins-bot: Rebuild tools packages for Ubuntu Trusty [integration/jenkins] - 10https://gerrit.wikimedia.org/r/222472 (owner: 10Krinkle)
[20:58:19] <grrrit-wm>	 (03PS1) 10Krinkle: Migrate jshint jobs to Trusty and Labs [integration/config] - 10https://gerrit.wikimedia.org/r/222480 
[21:05:44] <grrrit-wm>	 (03CR) 10Krinkle: [C: 032] "Re-compiled and deployed all affected jobs: mediawiki-core-jslint (1x), mwext-*-jslint (338x), apps-jslint, labs-toollabs-jslint, wikimedi" [integration/config] - 10https://gerrit.wikimedia.org/r/222480 (owner: 10Krinkle)
[21:11:00] <grrrit-wm>	 (03Merged) 10jenkins-bot: Migrate jshint jobs to Trusty and Labs [integration/config] - 10https://gerrit.wikimedia.org/r/222480 (owner: 10Krinkle)
[21:13:21] <legoktm>	 zuul stuck?
[21:13:44] <legoktm>	 precise jobs stalled?
[21:14:35] <legoktm>	 slowly moving...
[21:18:34] <wikibugs>	 10Deployment-Systems, 7Graphite: Record sync wikiversions event in graphite - https://phabricator.wikimedia.org/T104635#1422908 (10Krinkle) 3NEW
[21:18:51] <Krinkle>	 legoktm: pushing 300 jobs one by one
[21:19:04] <legoktm>	 oh wonderful :)
[21:19:14] <Krinkle>	 To fix jshint
[21:19:16] <Krinkle>	 check job
[21:19:26] <Krinkle>	 Dunno why I'm fixing that but I feel obligated since I created that mess
[21:19:35] <Krinkle>	 and even hashar didn't realise it.
[21:24:01] <legoktm>	 :/
[21:27:36] <Krinkle>	 It's at mwext-P now
[21:27:49] <Krinkle>	 Can't wait for open-stack to finish Zuul v3
[21:27:59] <Krinkle>	 legoktm: Have you been following?
[21:28:20] <legoktm>	 a little bit
[21:28:47] <Krinkle>	 Basically what they're gonna do is build on top of the ability to inject information into a build dependent on the project/branch etc. Similar to our DOC_SUBPATH for example.
[21:28:53] <Krinkle>	 But instead for the entire job.
[21:29:10] <Krinkle>	 So there'd only be one jenkins job "Job" or whatever. Except not even that, because Zuul can just use gearman directly.
[21:29:42] <polybuildr>	 "Your connection to gerrit.wikimedia.org is encrypted with obsolete cryptography." is that Chromium tells me... Because just plain RSA, afaics.
[21:29:42] <legoktm>	 yeah, AIUI they want to get rid of jenkins?
[21:29:52] <Krinkle>	 The only catch is that retriggering builds can then only happen via zuul-cli. And we'll need to follow suit to store build output statically ourselves.
[21:30:07] <Krinkle>	 Other than that, pure win :)
[21:30:27] <Krinkle>	 Yeah.
[21:31:16] <polybuildr>	 And... I suspect the gate-and-submit jobs just ran twice for a patch set of mine. o.O
[21:31:25] <polybuildr>	 Any idea why?
[21:32:02] <legoktm>	 polybuildr: link?
[21:32:43] <polybuildr>	 legoktm: well, no. not sure I can give a link. I was staring at https://integration.wikimedia.org/zuul/ and saw all the jobs complete successfully. I come back 2 minutes later, and they're all queued again.
[21:33:26] <legoktm>	 if a MW before yours fails, yours will be re-tried
[21:33:48] <polybuildr>	 oh, that makes sense.
[21:33:58] <polybuildr>	 there was a core gate-and-submit running
[21:34:21] <polybuildr>	 a patch of yours, I think. :P
[21:34:29] <polybuildr>	 okay, no, not sure.
[21:36:27] <polybuildr>	 legoktm: any comments about the "obsolete crypto" ^
[21:36:37] <legoktm>	 uhh, no idea
[21:36:56] <legoktm>	 firefox doesn't coplain ;P
[21:36:59] <legoktm>	 complain*
[21:37:16] <polybuildr>	 Chromium doesn't complain either. :P It's only if you inquisitively click the padlock icon.
[21:37:35] <polybuildr>	 csteipp: ^ Your connection to gerrit.wikimedia.org is encrypted with obsolete cryptography.
[21:42:13] <Krenair>	 I'd file a ticket against #HTTPS and #Wikimedia-Git-or-gerrit
[21:42:22] <Krenair>	 and #operations
[21:43:40] <polybuildr>	 Krenair: alright :) will do!
[21:44:19] <Krenair>	 My copy of Chrome says "using an obsolete cipher suite"
[21:51:07] <twentyafterfour>	 ostriches: there is a problem with gerrit urls: we can't do a server-side redirect on urls that use the #url-hash
[22:33:53] <aharoni>	 twentyafterfour: hi
[22:34:09] <aharoni>	 you're Phabricator expert, if I'm not mistaken
[22:34:27] <aharoni>	 how can I filter a board to show tasks that don't appear in a certain project?
[22:35:12] <aharoni>	 there is "Not In:", but I cannot write everything after it, only certain projects.
[22:38:49] <aharoni>	 andre__: ^
[22:50:42] <legoktm>	 Krinkle: can redirects like https://phabricator.wikimedia.org/T74454#933518 be implemented in integration-docroot? or does it need to happen somewhere else?
[22:51:39] <Krinkle>	 legoktm: It's tricky, but can be done in htaccess indeed
[22:51:41] <Krinkle>	 legoktm: but!
[22:52:06] <Krinkle>	 legoktm: for the specific case of doc.wikimedia.org, unless it is a top-level directory rename (e.g. mediawiki-core -> mediawiki) you don't need redirects manually
[22:52:21] <Krinkle>	 the docPage class I created for this already redirects to the sole subdirectory
[22:52:30] <Krinkle>	 and if more than one, shows on button for each
[22:52:36] <Krinkle>	 which is all we need 
[22:52:46] <Krinkle>	 like so https://doc.wikimedia.org/mediawiki-core/master/
[22:52:56] <legoktm>	 so the existing link would turn into a page with buttons for "PHP" and "JS"?
[22:53:03] <Krinkle>	 Yep
[22:53:17] <legoktm>	 and that's ok? we'd be breaking all the existing links
[22:53:29] <Krinkle>	 We can't assume they want /js
[22:53:37] <Krinkle>	 Maybe now but not forever
[22:53:42] <Krinkle>	 it's fine to show both
[22:53:57] <legoktm>	 https://www.mediawiki.org/w/index.php?target=https%3A%2F%2Fdoc.wikimedia.org%2Foojs-ui%2F&title=Special%3ALinkSearch
[22:54:26] <legoktm>	 I can run replace.py to fix those
[22:54:39] <Krinkle>	 if php/ doesn't exist, the DocPage handler will redirect to js/
[22:55:13] <legoktm>	 I was going to move jsduck to "js/" and then set up doxygen for php/
[22:56:34] <grrrit-wm>	 (03PS1) 10Legoktm: Move OOUI jsduck documentation to "oojs-ui/js/$branch" [integration/config] - 10https://gerrit.wikimedia.org/r/222502 
[23:01:40] <grrrit-wm>	 (03PS1) 10Legoktm: Add 'oojs-ui-doxygen-publish' job [integration/config] - 10https://gerrit.wikimedia.org/r/222503 (https://phabricator.wikimedia.org/T74454) 
[23:02:06] <grrrit-wm>	 (03PS2) 10Legoktm: Add 'oojs-ui-doxygen-publish' job [integration/config] - 10https://gerrit.wikimedia.org/r/222503 (https://phabricator.wikimedia.org/T74454) 
[23:02:06] <mobrovac>	 ostriches: just a heads up re restbase in staging, we have deployed https://gerrit.wikimedia.org/r/222272 today which adds monitoring of back-ends
[23:02:08] <grrrit-wm>	 (03PS2) 10Legoktm: Move OOUI jsduck documentation to "oojs-ui/js/$branch" [integration/config] - 10https://gerrit.wikimedia.org/r/222502 
[23:02:15] <ostriches>	 mobrovac: mmk
[23:02:24] <mobrovac>	 (just so you know in case rb stops working after a puppet run)
[23:05:02] <James_F>	 legoktm: You're amazing.
[23:06:13] <James_F>	 legoktm: https://gerrit.wikimedia.org/r/#/c/166074/ next? ;-)
[23:06:56] <legoktm>	 :)
[23:07:30] * James_F whistles.
[23:08:51] <legoktm>	 James_F: yeah, I was bugging MatmaRex about that earlier
[23:09:17] <legoktm>	 the demos
[23:10:36] <wmf-insecte>	 Project browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #180: FAILURE in 13 min: https://integration.wikimedia.org/ci/job/browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/180/
[23:10:59] <James_F>	 legoktm: The demos are hard.
[23:11:22] <James_F>	 legoktm: They need building (the repo isn't in a built state), which means running grunt, which means running npm install and composer update…
[23:11:28] <legoktm>	 > Upstream OOjs UI and VisualEditor need to have their demo-directory be standalone (like docs). So must not reference ../ but instead build and copy relevant files.
[23:11:28] <legoktm>	 James_F: ^ is not the case yet for OOUI. Mainly for the PHP side.
[23:11:41] <legoktm>	 my IRC client is lagging
[23:11:43] <James_F>	 That too.
[23:11:57] <legoktm>	 running npm install + composer update is easy
[23:12:05] <James_F>	 On production kit?
[23:12:15] <James_F>	 Hmm.
[23:12:20] <legoktm>	 no, we'd run it in labs and rsync the result to gallium
[23:12:23] <James_F>	 I guess we do it for CI on labs.
[23:12:23] <James_F>	 Yeah.
[23:12:27] <James_F>	 OK, less insane.
[23:30:16] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] Move OOUI jsduck documentation to "oojs-ui/js/$branch" [integration/config] - 10https://gerrit.wikimedia.org/r/222502 (owner: 10Legoktm)
[23:31:45] <shinken-wm>	 PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL 100.00% of data above the critical threshold [0.0]
[23:32:17] <grrrit-wm>	 (03Merged) 10jenkins-bot: Move OOUI jsduck documentation to "oojs-ui/js/$branch" [integration/config] - 10https://gerrit.wikimedia.org/r/222502 (owner: 10Legoktm)
[23:42:44] <grrrit-wm>	 (03PS1) 10Legoktm: Update OOjs UI link [integration/docroot] - 10https://gerrit.wikimedia.org/r/222509 
[23:43:04] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] Update OOjs UI link [integration/docroot] - 10https://gerrit.wikimedia.org/r/222509 (owner: 10Legoktm)
[23:43:16] <grrrit-wm>	 (03Merged) 10jenkins-bot: Update OOjs UI link [integration/docroot] - 10https://gerrit.wikimedia.org/r/222509 (owner: 10Legoktm)
[23:44:00] <wikibugs>	 10Beta-Cluster, 10BetaFeatures, 10Compact-Personal-Bar-(Beta), 7Beta-Feature: Remove Compact Personal Bar from Beta Cluster and Production - https://phabricator.wikimedia.org/T104659#1423453 (10greg) 3NEW