[00:44:42] <grrrit-wm>	 (03CR) 10BryanDavis: "ProxyPassMatch directives should have retry=0 as well shouldn't they?" [operations/apache-config] (betacluster) - 10https://gerrit.wikimedia.org/r/148743 (owner: 10Ori.livneh)
[07:28:12] <wikibugs>	 3Wikimedia / 3Quality Assurance: don't run screenshot tests in normal VE build - 10https://bugzilla.wikimedia.org/68467#c1 (10Željko Filipin) 5NEW>3ASSI a:3Vikas Vikas and I will pair on this, probably today.
[07:57:52] <zeljkof>	 mazza: welcome :)
[07:58:00] <mazza>	 Thanks :)
[08:01:41] <zeljkof>	 mazza: https://github.com/blog/1640-git-internals-pdf-open-sourced
[08:02:26] <zeljkof>	 https://help.github.com/
[08:15:17] <zeljkof>	 mazza: https://github.com/zeljkofilipin/dotfiles/blob/master/.gitconfig
[08:19:45] <grrrit-wm>	 (03PS1) 10Hashar: Make OATHAuth jslint job voting [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/148963 (https://bugzilla.wikimedia.org/61617) 
[08:20:05] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] Make OATHAuth jslint job voting [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/148963 (https://bugzilla.wikimedia.org/61617) (owner: 10Hashar)
[08:20:10] <grrrit-wm>	 (03Merged) 10jenkins-bot: Make OATHAuth jslint job voting [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/148963 (https://bugzilla.wikimedia.org/61617) (owner: 10Hashar)
[08:27:52] <wikibugs>	 3Wikimedia / 3Quality Assurance: Create browser test for a preference tab - 10https://bugzilla.wikimedia.org/67260 (10Željko Filipin) a:3Mazza
[08:28:06] <wikibugs>	 3Wikimedia / 3Quality Assurance: Create browser test for a preference tab - 10https://bugzilla.wikimedia.org/67260 (10Željko Filipin) 5NEW>3ASSI
[08:34:30] <hashar>	 helllo 
[08:34:35] <hashar>	 zeljkof: sorry I was quite busy yesterday
[08:34:38] <hashar>	 anything you need ?
[08:34:47] <zeljkof>	 hashar: in a meeting with mazza
[08:35:02] <zeljkof>	 mazza: https://lastpass.com/
[11:57:08] <vikasyaligar>	 hello zeljkof
[11:57:14] <zeljkof>	 vikasyaligar: hi
[11:57:36] <vikasyaligar>	 zeljkof: when do we pair ? 
[11:57:51] <zeljkof>	 is in an hour or so a good time for you?
[11:58:01] <vikasyaligar>	 zeljkof: yup ! :) 
[11:58:08] <zeljkof>	 vikasyaligar: great
[11:58:24] <zeljkof>	 I will ping you in about an hour
[11:58:40] <vikasyaligar>	 zeljkof: ok thank you :) 
[12:20:37] <grrrit-wm>	 (03CR) 10Zfilipin: "recheck" [ruby/api] - 10https://gerrit.wikimedia.org/r/148083 (owner: 10Zfilipin)
[12:21:07] <grrrit-wm>	 (03CR) 10Zfilipin: "recheck" [selenium] - 10https://gerrit.wikimedia.org/r/148388 (owner: 10Zfilipin)
[12:25:36] <grrrit-wm>	 (03PS1) 10Zfilipin: WIP test [selenium] - 10https://gerrit.wikimedia.org/r/148984 
[12:27:31] <grrrit-wm>	 (03PS2) 10Zfilipin: WIP test [selenium] - 10https://gerrit.wikimedia.org/r/148984 
[12:27:59] <grrrit-wm>	 (03Abandoned) 10Zfilipin: WIP test [selenium] - 10https://gerrit.wikimedia.org/r/148984 (owner: 10Zfilipin)
[12:45:59] <grrrit-wm>	 (03PS1) 10Zfilipin: No longer running Ruby linter for Wikibase [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/148988 
[12:47:42] <grrrit-wm>	 (03PS1) 10Zfilipin: No longer running any jobs for qa/browsertests [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/148989 
[12:55:06] <wikibugs>	 3Wikimedia / 3Continuous integration: Zuul repositories have too many refs causing slow updates - 10https://bugzilla.wikimedia.org/68481#c3 (10Antoine "hashar" Musso) I wrote a quick script which inspect the commit pointed by the Zuul reference and delete the reference whenever it is older than a given numbe...
[13:11:44] <vikasyaligar>	 will be back in few minutes 
[13:27:24] <zeljkof>	 vikasyaligar: sorry to keep you waiting, I will be ready in a few minutes
[13:27:40] <vikasyaligar>	 zeljkof: great !
[13:32:19] <zeljkof>	 vikasyaligar: I have sent you the invite, I will ping you in a few minutes, when I am ready
[13:32:44] <vikasyaligar>	 zeljkof: ok thank you :) 
[13:33:10] <zeljkof>	 vikasyaligar: sorry for the delay, I have visitors in my home office, have to get rid of them first :)
[13:34:29] <vikasyaligar>	 zeljkof: it is not at all a problem as I am free like a bird :) 
[13:34:59] <zeljkof>	 vikasyaligar: I am in the hangout
[13:37:28] <zeljkof>	 vikasyaligar: https://bugzilla.wikimedia.org/show_bug.cgi?id=68467
[13:41:27] <zeljkof>	 hashar: beta is still pretty slow, are you making any changes now?
[13:45:27] <zeljkof>	 hashar, vikasyaligar: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox-sauce/LANGUAGE_SCREENSHOT_CODE=he,label=contintLabsSlave/75/console
[13:45:35] <zeljkof>	 this fails
[13:46:09] <zeljkof>	 after this
[13:46:10] <zeljkof>	 mw-api-siteinfo.py http://en.wikipedia.beta.wmflabs.org/w/api.php git_branch
[13:51:41] <zeljkof>	 vikasyaligar: https://atom.io/
[13:54:02] <zeljkof>	 http://survey.hamptoncatlin.com/stats#question_6
[14:01:08] <hashar>	 zeljkof:  well http://en.wikipedia.beta.wmflabs.org/w/api.php doesn't answer
[14:01:33] <zeljkof>	 hashar:  http://en.wikipedia.beta.wmflabs.org/ does not open too
[14:01:51] <wikibugs>	 3Wikimedia / 3Continuous integration: Zuul repositories have too many refs causing slow updates - 10https://bugzilla.wikimedia.org/68481#c4 (10Antoine "hashar" Musso) zuul@gallium:/srv/ssd/zuul/git/mediawiki/core$ git show-ref|fgrep -c refs/zuul/ 51287   Then ran /home/hashar/zuul_clear_refs.py  --until 360 .
[14:02:10] <zeljkof>	 vikasyaligar: http://ci.openstack.org/jenkins-job-builder/installation.html
[14:07:58] <hashar>	 zeljkof: apparently it is broken since this morning
[14:08:05] <hashar>	 Jul 24 07:53 web.log
[14:09:00] <zeljkof>	 hashar: great, thanks for letting me know
[14:09:03] <zeljkof>	 can you fix it? :)
[14:16:15] <chrismcmahon>	 ori: are you doing more HHVM on beta labs?  http://en.wikipedia.beta.wmflabs.org/w/api.php stopped responding recently
[14:16:20] <chrismcmahon>	 ww
[14:20:56] <hashar>	 !log deployment-prep killed hhvm process on deployment-mediawiki01 and 02.    init script does not work.
[14:21:08] <hashar>	 bah
[14:21:25] <chrismcmahon>	 bd808|BUFFER: you around?
[14:21:33] <hashar>	 just have to restart hhvm on the deployment-mediawiki01 and 02 boxes :D
[14:23:55] <zeljkof>	 vikasyaligar: https://mac.github.com/
[14:40:24] <grrrit-wm>	 (03PS1) 10Vikassy: Run language screenshot job manually [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/149012 
[14:44:46] <grrrit-wm>	 (03PS2) 10Vikassy: Run language screenshot job manually [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/149012 
[15:09:51] <wikibugs>	 3Wikimedia / 3Continuous integration: Zuul repositories have too many refs causing slow updates - 10https://bugzilla.wikimedia.org/68481#c5 (10Antoine "hashar" Musso) And that dropped roughly 21k references:  $ git show-ref |fgrep -c refs/zuul 29639 $  Will process operations/puppet as well.
[15:18:24] <bd808>	 chrismcmahonbrb: I'm here now
[15:18:54] <chrismcmahon>	 hi bd808 HHVM messed up but hashar fixed it
[15:19:01] <hashar>	 oh
[15:19:05] <hashar>	 I just killed hhvm process
[15:19:17] <hashar>	 bd808: /etc/init.d/hhvm stop  doesn't do anything
[15:19:32] <hashar>	 i noticed hhvm had a lot of child process:  [sh]
[15:19:42] <bd808>	 it's an upstart job. service hhvm restart
[15:20:20] * bd808 hasn't looked at the packaging closely
[15:20:20] <hashar>	 bah :-(
[15:20:40] <hashar>	 we should ensure  /etc/init.d/hhvm points to upstart so :]
[15:21:09] <bd808>	 The init.d script may be from the Debian package, not sure
[15:22:52] <bd808>	 chrismcmahon: We have a new build of luasandbox that is hoped will fix the crashes that were happening yesterday. I'm poking at it in my vagrant instance right now, but the real test will be putting beta back to using it.
[15:23:36] <bd808>	 chrismcmahon: Will it cause you great grief if I try changing the beta to use it in a hour or so?
[15:24:47] <chrismcmahon>	 bd808: nope. it would be convenient if things were reasonably stable as of about 11AM PDT
[15:25:17] <bd808>	 Sure. branch deploy and all.
[15:25:49] <bd808>	 Ok, so maybe we should just try it now and then revert if things look sketchy still.
[15:42:59] <grrrit-wm>	 (03PS1) 10Cmcmahon: WIP: duplicate of existing patch [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/149030 
[15:45:02] <bd808>	 chrismcmahon: Ok. Beta is running luasandbox again. This is the commit to revert if it goes wonky -- https://gerrit.wikimedia.org/r/#/c/149029/
[15:45:22] <chrismcmahon>	 thanks bd808
[15:47:06] <grrrit-wm>	 (03Abandoned) 10Cmcmahon: WIP: duplicate of existing patch [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/149030 (owner: 10Cmcmahon)
[15:48:23] <wikibugs>	 3Wikimedia / 3Continuous integration: Zuul repositories have too many refs causing slow updates - 10https://bugzilla.wikimedia.org/68481#c6 (10Antoine "hashar" Musso) p:5Highes>3Normal I have cleaned up a few more repositories  For reference, one can find the top 10 offenders by running:  cd /srv/ssd/zuu...
[15:53:36] <bd808>	 chrismcmahon: It's still crashing. I'll revert the config change.
[16:21:59] <chrismcmahon>	 hashar: are you around? we were working with https://bugzilla.wikimedia.org/show_bug.cgi?id=65486 and figured out that adding db slaves is probably what's causing the 'readonly' lockouts in beta labs
[16:22:22] <hashar>	 we had a db slave for quite a while now
[16:22:56] <hashar>	 and yeah if the slave is lagged up too much, mediawiki complains
[16:23:10] <hashar>	 but I have no idea what would cause the slave to lag out
[16:23:14] <hashar>	 maybe the update.php script from time to time
[16:23:19] <hashar>	 or some nasty queries going on 
[16:23:26] <chrismcmahon>	 hashar: yeah, I reported the bug after seeing it for about 2 weeks, which is when that change happened
[16:24:07] <hashar>	 probably want to start monitoring the database lag
[16:24:31] <chrismcmahon>	 hashar:  could that commit be reverted? do we really need/want db slaves for beta?
[16:24:44] <hashar>	 also I have no idea whether the lagged errors are logged somewhere
[16:25:11] <chrismcmahon>	 hashar:  since we don't run update.php in production it seems
[16:27:48] <hashar>	 well most of update.php calls are boop
[16:27:50] <hashar>	 noop
[16:29:44] <hashar>	 chrismcmahon: DB lag  is far from my area of knowledge though :-(((
[16:29:54] <hashar>	 and we are not entirely sure that is the issue 
[16:31:08] <grrrit-wm>	 (03PS3) 10Hashar: Add jobs for analytics/quarry/web [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/148361 (owner: 10Legoktm)
[16:32:41] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] "Jobs deployed :-]" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/148361 (owner: 10Legoktm)
[16:33:08] <grrrit-wm>	 (03PS2) 10Hashar: Add zuul config for analytics/quarry/web [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/148362 (owner: 10Legoktm)
[16:37:51] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add jobs for analytics/quarry/web [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/148361 (owner: 10Legoktm)
[16:38:42] <chrismcmahon>	 hashar: I am pretty sure :-)
[16:45:29] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] "deploying" [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/148362 (owner: 10Legoktm)
[16:45:34] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add zuul config for analytics/quarry/web [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/148362 (owner: 10Legoktm)
[16:48:34] <hashar>	 I am out !
[17:05:03] <grrrit-wm>	 (03PS1) 10Vikassy: WIP: Language screenshot runs only for language screenshot job [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/149045 (https://bugzilla.wikimedia.org/68467) 
[18:27:38] <grrrit-wm>	 (03PS2) 10Legoktm: Add zuul config for a bunch of extensions I maintain [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/148352 
[18:42:43] <grrrit-wm>	 (03CR) 10Mattflaschen: "This is probably going to keep getting un-mergable. So can we first figure out if it's safe to merge before we have browser tests?" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/144456 (owner: 10Phuedx)
[19:28:17] <grrrit-wm>	 (03PS1) 10Hashar: Switch analytics-quarry-web to tox [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/149094 
[19:30:11] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] Switch analytics-quarry-web to tox [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/149094 (owner: 10Hashar)
[19:31:31] <grrrit-wm>	 (03Merged) 10jenkins-bot: Switch analytics-quarry-web to tox [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/149094 (owner: 10Hashar)
[19:32:39] <grrrit-wm>	 (03PS1) 10Hashar: Switch analytics-quarry-web to tox [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/149095 
[19:33:13] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] "deploying" [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/149095 (owner: 10Hashar)
[19:33:23] <grrrit-wm>	 (03Merged) 10jenkins-bot: Switch analytics-quarry-web to tox [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/149095 (owner: 10Hashar)
[19:49:30] <wikibugs>	 3Wikimedia / 3Continuous integration: Let MediaWiki qunit jobs run on lanthanum - 10https://bugzilla.wikimedia.org/68529 (10Antoine "hashar" Musso) 3NEW p:3Unprio s:3normal a:3None The MediaWiki extensions qunit jobs are bound to gallium.  We have more and more extensions adding qunit support, the jo...
[21:01:21] <wikibugs>	 3Wikimedia / 3Continuous integration: Write job to ensure Parsoid settings on beta cluster is sane - 10https://bugzilla.wikimedia.org/68532 (10Antoine "hashar" Musso) 3NEW p:3Unprio s:3normal a:3None Bug 65939 was filled because the Parsoid daemon on beta cluster ended up fetching content from produc...