[03:08:20] <wmf-insecte>	 Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #775: 04FAILURE in 18 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/775/
[05:23:29] <shinken-wm>	 PROBLEM - Puppet failure on deployment-cache-upload04 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[05:29:51] <wmf-insecte>	 Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #598: 04FAILURE in 27 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/598/
[05:58:26] <shinken-wm>	 RECOVERY - Puppet failure on deployment-cache-upload04 is OK: OK: Less than 1.00% above the threshold [0.0]
[06:14:28] <SMalyshev>	 anybody around who knows about trebuchet deployments?
[07:28:49] <thedj>	 fyi, IE8 javascript is broken currently in production.
[08:57:24] <zeljkof>	 hashar: ping me if you have the time to merge/deploy all ruby ci craziness :)
[09:54:47] <shinken-wm>	 PROBLEM - Puppet staleness on integration-dev is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0]
[10:01:42] <hashar>	 zeljkof: ok done :-}
[10:04:04] <zeljkof>	 hashar: want to merge/deploy now? or later?
[10:04:21] <hashar>	 yeah
[10:04:25] <hashar>	 change # ? 
[10:04:28] <hashar>	 my inbox is filled :(
[10:04:33] <zeljkof>	 hashar: :)
[10:04:36] <zeljkof>	 let me see
[10:04:52] <hashar>	 i think the easiest is to review the CI change
[10:04:54] <hashar>	 get it deployed
[10:05:00] <hashar>	 then recheck  all the extensions changes
[10:05:04] <hashar>	 or maybe just +2 all of them
[10:05:09] <zeljkof>	 hashar: https://gerrit.wikimedia.org/r/#/c/251979/
[10:05:19] <hashar>	 great 
[10:05:19] <hashar>	 on it
[10:05:35] <zeljkof>	 hashar: and then all of these https://gerrit.wikimedia.org/r/#/q/status:open+topic:T117993,n,z
[10:06:03] <zeljkof>	 about 20 of them, some of repos already merged commits with rakefiles
[10:07:28] <kart_>	 hashar: integration.wm.org uses same login/passwed as Gerrit?
[10:07:42] <hashar>	 kart_: that site is public  without any auth
[10:08:06] <kart_>	 I need to use puppet compiler from lab.
[10:08:09] <hashar>	 ohhh
[10:08:12] <hashar>	 JENKINS!!!!!!!!!!!!!!!!
[10:08:17] <hashar>	 yeah your labs account
[10:08:28] <hashar>	 needs to be either in "wmf" or "nda" groups
[10:08:53] <kart_>	 Thanks!
[10:09:56] <hashar>	 curl https://integration.wikimedia.org/ci/job/integration-zuul-layoutdiff/6427/consoleText | vim -
[10:09:59] <hashar>	 pff
[10:11:46] <kart_>	 ok. another question.
[10:11:54] <kart_>	 https://integration.wikimedia.org/ci/job/mwext-qunit/7759/console - why this fails :/
[10:12:56] <hashar>	 zeljkof: that impacts a few mediawiki/skins as well
[10:15:23] <zeljkof>	 hashar: what impacts skins?
[10:17:51] <hashar>	 https://gerrit.wikimedia.org/r/#/c/251979/1/zuul/layout.yaml
[10:17:59] <hashar>	 the extension-rubylint is used on a ton of repos
[10:21:53] <hashar>	 lets do it  ™
[10:22:15] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] "Go go go ! We will have to cherry pick a bunch of patches on REL1_26 / wmf branches :-(" [integration/config] - 10https://gerrit.wikimedia.org/r/251979 (https://phabricator.wikimedia.org/T114262) (owner: 10Zfilipin)
[10:23:07] <grrrit-wm>	 (03Merged) 10jenkins-bot: Run Ruby jobs using Rake [integration/config] - 10https://gerrit.wikimedia.org/r/251979 (https://phabricator.wikimedia.org/T114262) (owner: 10Zfilipin)
[10:23:25] <zeljkof>	 hashar: but none of the skins have any ruby or run ruby jobs, right?
[10:23:54] <zeljkof>	 I think I have covered all repos where extension-rubylint is used
[10:24:14] <zeljkof>	 there are still 5-10 repos that are not extensions (core, ops/puppet...)
[10:26:03] <hashar>	 zeljkof: I think I got confused
[10:26:17] <hashar>	 I have rechecked / rebased all patches
[10:26:53] <hashar>	 zeljkof: PageTriage fails https://gerrit.wikimedia.org/r/251997
[10:26:58] <zeljkof>	 I was confused until yesterday, now I think what needs to be done :)
[10:27:03] <zeljkof>	 hashar: lemme see...
[10:27:09] <hashar>	 I rebased it
[10:27:22] <shinken-wm>	 RECOVERY - Host deployment-parsoidcache02 is UP: PING OK - Packet loss = 0%, RTA = 0.75 ms
[10:27:24] <hashar>	 and the Rakefile does not pass rubocop :D
[10:27:53] <zeljkof>	 hashar: oops, looks like that repo likes double quotes
[10:27:54] <zeljkof>	 on it
[10:28:00] <hashar>	 :-}
[10:28:37] <hashar>	 ahhh
[10:28:52] <hashar>	 I need to get rid of the Nodepool 1 minute delay before deleting an instance
[10:31:59] <hashar>	 zeljkof: a few other fails https://gerrit.wikimedia.org/r/#/q/is:open+owner:%22Zfilipin+%253Czeljko.filipin%2540gmail.com%253E%22+label:verified%253D-1,n,z :D
[10:37:59] <hashar>	 zeljkof: wanna share the load to fix the four falling jobs ?
[10:40:25] <zeljkof>	 hashar: sure, go ahead :)
[10:40:41] <zeljkof>	 I mean, I can manage, but if you have nothing else to do... ;)
[10:40:49] <hashar>	 zeljkof: fix up PageTriage and ULS
[10:40:49] <zeljkof>	 just let me know which one you take
[10:40:50] <hashar>	 i am doing TwnMainPage and Zeroportal
[10:41:56] <zeljkof>	 hashar: deal
[10:43:19] <hashar>	 bah
[10:43:19] <hashar>	 Prefer double-quoted strings unless you need single quotes to avoid extra backslashes for escaping.
[10:43:21] <hashar>	 so
[10:43:29] <hashar>	 some repos have double quotes and others prefer single quotes right?
[10:43:35] <zeljkof>	 hashar: no
[10:43:46] <zeljkof>	 we did not copy the template to all repos :/
[10:44:00] <zeljkof>	 https://www.mediawiki.org/wiki/Manual:Coding_conventions/Ruby#Base_configuration
[10:44:20] <zeljkof>	 you can fix the problem with this 
[10:44:21] <zeljkof>	 bundle exec rubocop -a -c .rubocop.yml
[10:44:33] <zeljkof>	 I will clean up the repos later
[10:45:28] <hashar>	 yeah sounds wise
[10:46:17] <hashar>	 bundle exec rubocop -a -c .rubocop.yml   <--- lovely
[10:46:49] <hashar>	 {done}
[10:46:59] <zeljkof>	 ULS did not fail because of ruby
[10:46:59] <zeljkof>	 https://gerrit.wikimedia.org/r/#/c/252003/
[10:47:10] <zeljkof>	 mediawiki-extensions-qunit FAILURE in 3m 28s
[10:47:16] <zeljkof>	 did recheck
[10:47:39] <hashar>	 Error: Pending AJAX requests: 0 (active: 1)
[10:47:41] <hashar>	 yeah
[10:47:41] <zeljkof>	 and you need "-c .rubocop.yml" only if the repo is a subfolder of mediawiki/vagrant
[10:47:45] <zeljkof>	 (like I do)
[10:47:49] <hashar>	 that one is reallyannoying
[10:48:01] <zeljkof>	 since rubocop then picks the parent (vagrant's) rubocop config file
[10:48:05] <zeljkof>	 and you want the local one
[10:49:55] <hashar>	 zeljkof: handling the TwnMainPage one
[10:50:03] <zeljkof>	 hashar: great, thanks
[10:50:19] <zeljkof>	 I have to go, will be back in 1-2 hours
[10:50:37] <hashar>	 oh no
[10:50:39] <hashar>	 you already did
[10:50:49] <hashar>	 zeljkof: I will finish it
[10:50:51] <zeljkof>	 if this gets merged without trouble, I will finish the migration with the rest of the repos
[10:50:52] <hashar>	 and do the cherry pick to REL branches
[10:50:58] <zeljkof>	 hashar: thans
[10:50:59] <zeljkof>	 thanks
[10:51:32] <zeljkof>	 hashar: uls all good https://gerrit.wikimedia.org/r/#/c/252003/
[10:51:37] <hashar>	 yeah
[10:51:49] <hashar>	 will mass cherry pick / +2
[11:02:40] <shinken-wm>	 PROBLEM - Host deployment-parsoidcache02 is DOWN: CRITICAL - Host Unreachable (10.68.16.145)
[11:12:22] <shinken-wm>	 RECOVERY - Host deployment-parsoidcache02 is UP: PING OK - Packet loss = 0%, RTA = 1.11 ms
[11:23:14] <kart_>	 zeljkof: do you know why https://gerrit.wikimedia.org/r/#/c/252172/ fails for wmf5 and not for wmf6?
[11:23:41] <kart_>	 (ie can you please check :) )
[11:31:04] <shinken-wm>	 PROBLEM - Host deployment-parsoidcache02 is DOWN: CRITICAL - Host Unreachable (10.68.16.145)
[11:37:37] <kart_>	 hashar: any clue on https://integration.wikimedia.org/ci/job/mwext-qunit/7759/consoleFull ?
[11:38:19] <hashar>	 kart_: yeah the qunit jobs are somehow hit by a race condition
[11:38:23] <hashar>	 or suffers from slaves being overloaded
[11:38:27] <kart_>	 oh.
[11:38:36] <kart_>	 Should I do recheck after some time?
[11:38:37] <hashar>	 hard to investigate :-(
[11:38:49] <hashar>	 yeah just recheck
[11:38:49] <hashar>	 and hope :-(
[11:38:52] <kart_>	 hashar: force merge is fine then?
[11:38:53] <hashar>	 I have nothing better to offer sorry
[11:38:59] <kart_>	 hashar: no problem.
[11:39:01] <hashar>	 force merges kills zuul
[11:39:27] <hashar>	 there are bunch of exceptions https://integration.wikimedia.org/ci/job/mwext-qunit/7757/artifact/log/mw-error.log/*view*/
[11:39:29] <hashar>	 maybe unrelated 
[11:39:41] <hashar>	 PHP Warning: Cannot modify header information - headers ...
[11:40:03] <hashar>	 and a bunch of errors on the server side https://integration.wikimedia.org/ci/job/mwext-qunit/7757/artifact/log/mw-debug-www.log/*view*/
[11:40:12] <hashar>	 [DBPerformance] Expectation (writes <= 0) by MediaWiki::main not met:
[11:40:12] <hashar>	 query-m: DELETE FROM `l10n_cache` WHERE lc_lang = 'X' [TRX#a7cd3555976d]
[11:40:19] <hashar>	 seems the database back end is too slow
[11:40:34] <kart_>	 hashar: has bug for that? :)
[11:40:41] <hashar>	 nop
[11:59:21] <wmf-insecte>	 Project browsertests-CentralAuth-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #290: 04FAILURE in 2 min 20 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralAuth-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/290/
[12:01:22] <grrrit-wm>	 (03PS1) 10Hashar: Ignore rake-jessie on wmf/1.27.0-wmf{4,5} [integration/config] - 10https://gerrit.wikimedia.org/r/252206 
[12:02:51] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] Ignore rake-jessie on wmf/1.27.0-wmf{4,5} [integration/config] - 10https://gerrit.wikimedia.org/r/252206 (owner: 10Hashar)
[12:04:03] <grrrit-wm>	 (03Merged) 10jenkins-bot: Ignore rake-jessie on wmf/1.27.0-wmf{4,5} [integration/config] - 10https://gerrit.wikimedia.org/r/252206 (owner: 10Hashar)
[12:07:15] <shinken-wm>	 RECOVERY - Host deployment-parsoidcache02 is UP: PING OK - Packet loss = 0%, RTA = 0.51 ms
[12:59:50] <wmf-insecte>	 Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #844: 09SUCCESS in 27 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/844/
[13:13:14] <zeljkof>	 kart_: you should ping hashar, his CI fu is black belt :)
[13:13:50] <grrrit-wm>	 (03PS1) 10Hashar: Add experimental debian-glue to integration/zuul [integration/config] - 10https://gerrit.wikimedia.org/r/252213 
[13:14:04] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] Add experimental debian-glue to integration/zuul [integration/config] - 10https://gerrit.wikimedia.org/r/252213 (owner: 10Hashar)
[13:14:07] <hashar>	 zeljkof: all patches merged
[13:14:17] <zeljkof>	 hashar: nice!
[13:14:17] <hashar>	 zeljkof: and I got Zuul to ignore rake-jessie on the current wmf branches
[13:14:24] <hashar>	 been too lazy to handle all the backports
[13:14:43] <zeljkof>	 I will continue moving all ruby jobs today, there should be just a few repos left
[13:15:09] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add experimental debian-glue to integration/zuul [integration/config] - 10https://gerrit.wikimedia.org/r/252213 (owner: 10Hashar)
[13:17:04] <shinken-wm>	 PROBLEM - Host deployment-parsoidcache02 is DOWN: CRITICAL - Host Unreachable (10.68.16.145)
[13:26:17] <dcausse>	 zeljkof: hi! I have build failures when I +2 on Cirrus: https://gerrit.wikimedia.org/r/#/c/251658/, I'm not sure how to fix the issue but looks like it's related to Rakefile
[13:26:35] <zeljkof>	 dcausse: looking...
[13:27:57] <zeljkof>	 dcausse: oops, looks like somehow we managed to merge Rakefile that violates RuboCop configuration
[13:27:58] <zeljkof>	 will push the patch in a few minutes
[13:28:21] <dcausse>	 thanks! :)
[13:29:17] <zeljkof>	 dcausse: I see
[13:29:18] <zeljkof>	 https://gerrit.wikimedia.org/r/#/c/251988/
[13:30:16] <zeljkof>	 looks like bundler was not running for the patch
[13:30:16] <zeljkof>	 will fix in a minute
[13:31:19] <hashar>	 dcausse: sorry looks like we missed that one
[13:32:10] <dcausse>	 np :)
[13:35:32] <Nikerabbit>	 evening
[13:35:36] <wikibugs>	 7Browser-Tests, 10Continuous-Integration-Config, 5Patch-For-Review, 7Ruby, 5WMF-deploy-2015-11-10_(1.27.0-wmf.6): Add Rakefile to repositories with Ruby code - https://phabricator.wikimedia.org/T117993#1795629 (10zeljkofilipin)
[13:39:08] <hashar>	 dcausse: https://gerrit.wikimedia.org/r/#/c/252215/ !
[13:39:33] <hashar>	 dcausse: this morning we added a Rakefile to a bunch of repo
[13:39:34] <zeljkof>	 dcausse: https://gerrit.wikimedia.org/r/#/c/252215/
[13:39:48] <hashar>	 so folks can reproduce  locally what CI runs
[13:39:48] <hashar>	 ie:
[13:39:50] <hashar>	 bundle install
[13:39:55] <hashar>	 bundle exec rake
[13:40:05] <zeljkof>	 as soon as you merge that, CI will be happy
[13:40:18] <hashar>	 +2ed
[13:40:38] <hashar>	 zeljkof: can you cherry pick it to REL1_26 please ?
[13:40:49] <hashar>	 there is a conflict though, so gotta be done locally
[13:42:04] <zeljkof>	 hashar: I have never done that :) (cherry pick to REL...)
[13:42:32] <zeljkof>	 looking at what needs to be done there
[13:43:27] <hashar>	 zeljkof: 
[13:43:36] <hashar>	 git checkout -b REL1_26 -t origin/REL1_26
[13:43:51] <hashar>	 git cherry-pick -x 4d30685d29977e955464150092f525de5e2fed79
[13:43:54] <hashar>	 fix conflicts
[13:43:58] <hashar>	 git add
[13:44:04] <hashar>	 git cherry-pick --continue
[13:44:09] <hashar>	 git-review REL1_26
[13:44:13] <hashar>	 done;}
[13:44:27] <zeljkof>	 hashar: thanks, doing
[13:44:35] <hashar>	 dcausse: should work now. 
[13:44:45] <dcausse>	 hashar, zeljkof: thanks!
[13:44:52] <hashar>	 dcausse: note that you do not even need to rebase.  Zuul/Jenkins always tests patchset on tip of branch
[13:45:18] <hashar>	 dcausse: zuul  first attempt to merge your patch with the tip of the branch, if that works the resulting merge commit is what is being tested out :D
[13:45:38] <dcausse>	 ah ok, do I need to +2 again or will it try again itself?
[13:45:48] <hashar>	 dcausse: the current wmf/ branches should not run the rake-jessie so you should be able to swat just fine
[13:45:58] <dcausse>	 ok
[13:46:07] <zeljkof>	 dcausse: I think you have to remove +2, then +2 again
[13:46:12] <hashar>	 dcausse: you will need to remove yourself as reviewer and +2 again
[13:46:16] <zeljkof>	 hashar: is that correct? ^
[13:46:19] <dcausse>	 ok
[13:46:19] <hashar>	 but you are right
[13:46:31] <hashar>	 a patchset that has a +2 should just merge
[13:54:26] <zeljkof>	 hashar: something is wrong :/
[13:54:27] <zeljkof>	 https://etherpad.wikimedia.org/p/git
[13:54:50] <zeljkof>	 looks like origin/REL1_26 does not exist
[13:55:52] <zeljkof>	 nevermind
[13:55:59] <zeljkof>	 this fixed the problem: git remote update origin --prune
[14:00:31] <zeljkof>	 hashar: https://gerrit.wikimedia.org/r/#/c/252219/
[14:00:57] <hashar>	 :-}
[14:01:10] <hashar>	 did not know you could pass -prune
[14:01:33] <hashar>	 +2ed
[14:01:46] <zeljkof>	 did not know myself :) but google did
[14:22:15] <zeljkof>	 hashar: updated this according to your comment https://gerrit.wikimedia.org/r/#/c/250407/
[14:22:27] <zeljkof>	 (rubocop fix in mediawiki/vagrant)
[14:23:53] <hashar>	 zeljkof: yeah
[14:23:57] <hashar>	 cause 123_456
[14:24:01] <hashar>	 reads like a string to me
[14:24:05] <hashar>	 not integer  123456
[14:24:35] <zeljkof>	 hashar: you are just not yet breathing ruby ;)
[14:24:46] <zeljkof>	 ruby is all about readability 
[14:24:54] <hashar>	 the child one
[14:24:54] <hashar>	 https://gerrit.wikimedia.org/r/#/c/250408/
[14:25:10] <hashar>	 which enforces $1  to become Regexp.last_match(1)   
[14:25:28] <hashar>	 i think it rather dumb as well but don't have any strong opinion about it :D
[14:25:44] <zeljkof>	 hashar: I would stick to rubocop defaults if nobody complains
[14:25:54] <zeljkof>	 we can always change the setting later
[14:26:03] <hashar>	 wasn't dan saying that some of the defaults don't make sense?
[14:26:12] <zeljkof>	 hashar: and we fixed it :)
[14:26:16] <hashar>	 ah
[14:26:20] <zeljkof>	 with our custom .rubocop.yml
[14:26:27] <zeljkof>	 and a few of his patches to upstream
[14:26:36] <zeljkof>	 this one apparently was minor one
[14:26:42] <zeljkof>	 if you do not like it, I can ignore it too
[14:26:53] <zeljkof>	 but If you are :| let's stick to defaults
[14:27:03] <hashar>	 would need to seek other developers
[14:27:12] <hashar>	 maybe whoever added that $1 originally
[14:27:24] <hashar>	 also
[14:27:32] <hashar>	 we should find a way to get rake to run tasks in parallel :-}
[14:27:55] <hashar>	 though most of the time is spend installing gems :-(
[14:28:24] <zeljkof>	 hashar: yeah, noticed the install every time
[14:28:36] <zeljkof>	 caching solution needed asap! :D
[14:29:00] <zeljkof>	 back to mail for a while, have a survey to fill :)
[14:37:18] <shinken-wm>	 RECOVERY - Host deployment-parsoidcache02 is UP: PING OK - Packet loss = 0%, RTA = 0.62 ms
[15:01:11] <wikibugs>	 10Continuous-Integration-Config, 10ArticlePlaceholder, 10Wikidata, 3Wikidata-Sprint-2015-11-03: [Task] add CI to extension ArticlePlaceholder - https://phabricator.wikimedia.org/T113049#1795843 (10Lucie)
[15:18:49] <wikibugs>	 10Continuous-Integration-Infrastructure: mwext-qunit tests fails for wmf5 branch - https://phabricator.wikimedia.org/T118263#1795866 (10KartikMistry) 3NEW
[15:19:04] <kart_>	 hashar: zeljkof ^^
[15:19:08] <shinken-wm>	 PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0]
[15:19:29] <wmf-insecte>	 Yippee, build fixed!
[15:19:30] <wmf-insecte>	 Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce build #236: 09FIXED in 1 min 29 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce/236/
[15:19:36] <hashar>	 kart_: we were talking about it with jan and zeljko :)
[15:19:43] <kart_>	 :)
[15:20:07] <kart_>	 Did it something to do with wikidata by chance?
[15:20:49] <shinken-wm>	 PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[15:20:57] <wikibugs>	 10Continuous-Integration-Infrastructure: mwext-qunit tests fails for wmf5 branch - https://phabricator.wikimedia.org/T118263#1795877 (10hashar) From a discussion with @JanZerebecki today:    Warning: include_once(/mnt/jenkins-workspace/workspace/mwext-qunit/src/extensions/Wikidata/extensions/ExternalValidation/W...
[15:22:33] <hashar>	 kart_: jan should come back to you 
[15:22:34] <kart_>	 sigh.
[15:22:40] <hashar>	 kart_: apparently the fix is rather straightforward
[15:22:44] <kart_>	 okay. great!
[15:42:57] <kart_>	 hashar: https://gerrit.wikimedia.org/r/#/c/252227/
[15:45:18] <hashar>	 kart_: let me try something now
[15:46:42] <hashar>	 kart_: Depends-On: I0312c23628d706deb507b5534b868480945b6163
[15:46:52] <hashar>	 Zuul supports cross repositories dependencies now
[15:46:56] <hashar>	 though untested / unannonced
[15:48:10] <kart_>	 I see.
[15:48:25] <kart_>	 Can you announce this? :D
[15:48:39] <hashar>	 jzerebecki: ^^^  
[15:48:48] <hashar>	 so yeah
[15:49:06] <hashar>	 that trick Zuul so that the ContentTranslation change depends on  Wikidata 
[15:49:52] <kart_>	 hashar: if that works, can you +2 jzerebecki's patch?
[15:50:12] <jzerebecki>	 kart_: it will be done during the SWAT
[15:54:08] <shinken-wm>	 RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0]
[15:55:12] <hashar>	 kart_: some magic is going on :-}
[15:55:47] <shinken-wm>	 RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0]
[15:58:39] <hashar>	 kart_: thank you a ton to have filled the bug and noticed all of that mess
[15:58:53] <hashar>	 that provides excellent material for the cross repositories dependency system :D
[15:59:17] <kart_>	 jzerebecki: oh. sorry, yes! :)
[16:12:53] <andrewbogott>	 Hey, so, I have a dumb question:  Was the tuesday checkin moved to Mondays, permanently?
[16:14:33] <andrewbogott>	 greg-g: ^
[16:14:38] <wikibugs>	 10Continuous-Integration-Infrastructure, 5Patch-For-Review: mwext-qunit tests fails for wmf5 branch - https://phabricator.wikimedia.org/T118263#1796077 (10hashar) a:3JanZerebecki We confirmed the ContentTranslation qunit tests are fixed by editing https://gerrit.wikimedia.org/r/#/c/252172/ and making it depe...
[16:14:43] <wikibugs>	 10Continuous-Integration-Infrastructure, 5Patch-For-Review: mwext-qunit tests fails for wmf5 branch - https://phabricator.wikimedia.org/T118263#1796079 (10hashar) 5Open>3Resolved
[16:15:24] <greg-g>	 andrewbogott: at least for this month. 9am pacific is a hot timeslot and we needed a meeting with all of the "cto group" managers and tuesday was the only day that worked for everyone :/
[16:16:19] <andrewbogott>	 greg-g: ok.  The new time overlaps with the Ops meeting (as I’m sure you’ve noticed) so I’ll be absent for a while :)
[16:17:30] <greg-g>	 yeah, no worries, thanks :)
[16:23:56] <wikibugs>	 10MediaWiki-Releasing, 6Developer-Relations, 10Wikimedia-Blog-Content, 3DevRel-November-2015, 5MW-1.26-release: Write blog post announcing MW 1.26 - https://phabricator.wikimedia.org/T112842#1796136 (10Qgil) According to https://www.mediawiki.org/wiki/MediaWiki_1.26#Release_schedule the expected release...
[16:38:18] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2015-11-03: mwext-qunit tests fails for wmf5 branch - https://phabricator.wikimedia.org/T118263#1796175 (10JanZerebecki)
[16:42:16] <shinken-wm>	 PROBLEM - Host deployment-parsoidcache02 is DOWN: CRITICAL - Host Unreachable (10.68.16.145)
[16:49:01] <grrrit-wm>	 (03PS1) 10JanZerebecki: Wikidata headless browser tests: disable triggering [integration/config] - 10https://gerrit.wikimedia.org/r/252234 (https://phabricator.wikimedia.org/T117561) 
[16:51:21] <grrrit-wm>	 (03CR) 10JanZerebecki: [C: 032] "Deployed to Jenkins jobs: (['browsertests-Wikidata-PerformanceTests-linux-firefox', 'browsertests-Wikidata-SmokeTests-linux-firefox', 'bro" [integration/config] - 10https://gerrit.wikimedia.org/r/252234 (https://phabricator.wikimedia.org/T117561) (owner: 10JanZerebecki)
[16:53:42] <shinken-wm>	 RECOVERY - Host deployment-parsoidcache02 is UP: PING OK - Packet loss = 0%, RTA = 0.75 ms
[17:01:50] <grrrit-wm>	 (03Merged) 10jenkins-bot: Wikidata headless browser tests: disable triggering [integration/config] - 10https://gerrit.wikimedia.org/r/252234 (https://phabricator.wikimedia.org/T117561) (owner: 10JanZerebecki)
[17:07:15] <shinken-wm>	 PROBLEM - Host deployment-parsoidcache02 is DOWN: CRITICAL - Host Unreachable (10.68.16.145)
[17:07:45] <wikibugs>	 10Continuous-Integration-Infrastructure: some tests run from mwext-testextension-hhvm will pick up files from extensions that were not checked out for this job - https://phabricator.wikimedia.org/T117710#1796286 (10JanZerebecki) Needed to clean this up again on a slave.
[17:08:20] <jzerebecki>	 !log https://phabricator.wikimedia.org/T117710 ssh integration-slave-trusty-1012.eqiad.wmflabs 'sudo -u jenkins-deploy rm -rf /mnt/jenkins-workspace/workspace/mwext-testextension-hhvm/src/skins/BlueSky'
[17:08:23] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[17:20:37] <wikibugs>	 7Browser-Tests, 10Wikidata: move testing things from browser tests to qunit or phpunit tests where possible - https://phabricator.wikimedia.org/T118283#1796333 (10JanZerebecki) 3NEW a:3Jonas
[17:21:54] <wikibugs>	 7Browser-Tests, 10Wikidata: move testing things from browser tests to qunit or phpunit tests where possible - https://phabricator.wikimedia.org/T118283#1796333 (10JanZerebecki) a:5Jonas>3None
[17:25:06] <wikibugs>	 7Browser-Tests, 10Continuous-Integration-Config, 10Wikidata, 3Wikidata-Sprint-2015-11-03: create a Wikibase browser test job running against a fresh MediaWiki installation - https://phabricator.wikimedia.org/T118284#1796346 (10JanZerebecki) 3NEW
[17:36:31] <grrrit-wm>	 (03PS1) 10JanZerebecki: add composer version of mwext-mw-selenium [integration/config] - 10https://gerrit.wikimedia.org/r/252240 
[17:37:49] <grrrit-wm>	 (03PS2) 10JanZerebecki: add composer version of mwext-mw-selenium [integration/config] - 10https://gerrit.wikimedia.org/r/252240 (https://phabricator.wikimedia.org/T118284) 
[17:42:44] <grrrit-wm>	 (03CR) 10JanZerebecki: "Deployed to Jenkins: https://integration.wikimedia.org/ci/job/mwext-mw-selenium-composer/" [integration/config] - 10https://gerrit.wikimedia.org/r/252240 (https://phabricator.wikimedia.org/T118284) (owner: 10JanZerebecki)
[17:46:26] <wikibugs>	 10Deployment-Systems, 3Scap3: default lock file for scap3 should be repo-dependent - https://phabricator.wikimedia.org/T116208#1796415 (10mmodell) @thcipriani: I agree, I will add the 'reason' by writing a message to the lock file.
[17:53:47] <wikibugs>	 7Browser-Tests, 10Wikidata: [Task] get Wikidata browser tests running on MediaWiki-Vagrant - https://phabricator.wikimedia.org/T95863#1796468 (10JanZerebecki)
[17:57:25] <jzerebecki>	 zeljkof: does mwext-mw-selenium support browser tests being in a different repo than the extension?
[18:05:57] <grrrit-wm>	 (03CR) 10JanZerebecki: [C: 032] add composer version of mwext-mw-selenium [integration/config] - 10https://gerrit.wikimedia.org/r/252240 (https://phabricator.wikimedia.org/T118284) (owner: 10JanZerebecki)
[18:07:05] <grrrit-wm>	 (03Merged) 10jenkins-bot: add composer version of mwext-mw-selenium [integration/config] - 10https://gerrit.wikimedia.org/r/252240 (https://phabricator.wikimedia.org/T118284) (owner: 10JanZerebecki)
[18:07:19] <wikibugs>	 7Browser-Tests, 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2015-11-03: create a Wikibase browser test job running against a fresh MediaWiki installation - https://phabricator.wikimedia.org/T118284#1796602 (10JanZerebecki)
[18:07:39] <ostriches>	 twentyafterfour: Your code move for Deploy to deploy.py lgtm and works. Land-ahoy!
[18:07:58] <zeljkof>	 jzerebecki: good question, but I do not know the answer
[18:08:06] <zeljkof>	 marxarelli: ^
[18:08:17] <ostriches>	 We should have a general task for "clean the hell up of main.py"
[18:13:57] <jzerebecki>	 zeljkof, marxarelli: thx. after reading the jjb output the answer is no.
[18:15:19] <marxarelli>	 jzerebecki: the "other" repo would have to be a submodule of, or cloned somewhere beneath a repo containing the Gemfile
[18:15:55] <marxarelli>	 that, or mediawiki-selenium would have to be installed and executed outside the context of bundler
[18:17:35] <jzerebecki>	 marxarelli: so that needs to be at src/extensions/$EXT_NAME/tests/browser/Gemfile ?
[18:19:12] <marxarelli>	 jzerebecki: there's some disagreement about that, but the convention for now is to put the Gemfile at the root of the project
[18:19:33] <marxarelli>	 i.e. src/extensions/$EXT_NAME/Gemfile
[18:20:34] <jzerebecki>	 ok thx
[18:42:25] <wikibugs>	 7Browser-Tests, 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2015-11-03: create a Wikibase browser test job running against a fresh MediaWiki installation - https://phabricator.wikimedia.org/T118284#1796814 (10JanZerebecki)
[18:42:38] <wikibugs>	 7Browser-Tests, 10Continuous-Integration-Config, 10Wikidata, 3Wikidata-Sprint-2015-11-03: create a Wikibase browser test job running against a fresh MediaWiki installation - https://phabricator.wikimedia.org/T118284#1796346 (10JanZerebecki)
[18:52:18] <shinken-wm>	 RECOVERY - Host deployment-parsoidcache02 is UP: PING OK - Packet loss = 0%, RTA = 0.86 ms
[19:01:45] <thedj>	 ostriches: do we have rel 1.26 up live somewhere per chance ?
[19:01:58] <ostriches>	 I do not
[19:02:13] <ostriches>	 Not anywhere public, at least. I have a dedicated vagrant running REL1_26 right now
[19:02:46] <thedj>	 ok, i'll retool my local setup then.
[19:03:19] <thedj>	 we might have broken IE8 support a while ago...
[19:03:29] <ostriches>	 Could spin something up in labs quick enough
[19:04:25] <wikibugs>	 10Deployment-Systems, 3Scap3: Scap3 repo-cache should clean unused revs - https://phabricator.wikimedia.org/T118305#1796903 (10thcipriani) 3NEW
[19:05:53] <wikibugs>	 10Deployment-Systems, 3Scap3: Scap3 repo-cache should clean unused revs - https://phabricator.wikimedia.org/T118305#1796912 (10demon) What about having them use the same underlying base with `--shared` or similar?
[19:17:40] <shinken-wm>	 PROBLEM - Host deployment-parsoidcache02 is DOWN: CRITICAL - Host Unreachable (10.68.16.145)
[19:19:37] <ostriches>	 thedj: Got an instance up, installer is CSS-less, yay.
[19:20:41] <ostriches>	 Oh, skinless, gotcha.
[19:21:35] <thedj>	 cool. easier then fixing the routing in my VM apparently...
[19:27:19] <ostriches>	 thedj: http://pending-release.wmflabs.org/w/index.php/Main_Page
[19:30:52] <wikibugs>	 10Gitblit-Deprecate, 10MediaWiki-General-or-Unknown: Remove uses of git.wikimedia.org from MediaWiki - https://phabricator.wikimedia.org/T118307#1796981 (10demon) 3NEW
[19:33:40] <thedj>	 ostriches: nice. 
[19:34:21] <thedj>	 ostriches: it's looking like a core lib, that is only used by VE. so not as broken as I feared for a moment.
[19:36:29] <ostriches>	 Wherps vector is running master
[19:36:31] * ostriches fixes
[19:36:35] <jzerebecki>	 !log reloading zuul for 5009316..e81ee95
[19:36:38] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[19:46:32] <ostriches>	 thedj: What do you think about going ahead and getting an rc0 out?
[19:50:27] <wikibugs>	 10Deployment-Systems, 3Scap3: Scap3 repo-cache should clean unused revs - https://phabricator.wikimedia.org/T118305#1797071 (10thcipriani) Seems like a lot of deploys could still use up a lot of disk space using `--shared`. While getting rid of a ton of duplicate `.git/objects` would probably save a lot of spa...
[19:51:27] <thedj>	 ostriches: the thing is that the library is in core... even though only VE triggers it.. i mean patching it wouldn't be too much work. we can make a 'dirty-fix' real easily by patching the upstream lib locally. but .....
[19:51:41] <ostriches>	 hmm
[19:51:42] <thedj>	 meh. /me not sure
[19:52:40] <thedj>	 for an rc0, i think it's ok. even if we still patch it, it would be really minor change I guess, with a minimal risk footprint, in that in practice you would have to install VE as well...
[19:53:15] <ostriches>	 thcipriani: The space usage for core would end up being (N + M * 95) where N is the size of .git/objects and M is the number of branches. As opposed to ((N+95) * M)
[19:54:45] <thcipriani>	 ostriches: yeah, it still seems like, after a few months of deploying, we'd still want to clean that up.
[19:54:55] <ostriches>	 Yeah, I'm not saying we shouldn't.
[19:55:08] <ostriches>	 I've been beating on the "wasted object space" drum for awhile tho
[19:55:47] <thcipriani>	 heh, yeah, I don't think any of the documented problems with shared repos will really apply in this instance. There's certainly no reason not to use --shared.
[19:58:27] <ostriches>	 --shared was a bad example.
[19:58:43] <ostriches>	 --reference is better.
[19:59:05] <ostriches>	 Only problem, really with both, is that you need to clean up before deleting what's being referenced.
[19:59:13] <ostriches>	 (dereference, if you will)
[20:01:03] <thcipriani>	 ideally this would mostly be a non-issue. These repos shouldn't be manually getting futzed with.
[20:01:57] <thcipriani>	 also, for all practical purposes, the final repository checkedout inside /rev doesn't need git info anymore.
[20:07:09] <ostriches>	 true.
[20:08:51] <ostriches>	 I want git 2.3 everywhere :p
[20:09:14] <ostriches>	 --reference --dissociate is wonderful
[20:12:37] <thcipriani>	 I think that may be outside of scope :)
[20:22:46] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-restbase01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0]
[21:15:31] <grrrit-wm>	 (03PS1) 10Hashar: (WIP) Port trusty version to jessie-wikimedia (WIP) [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/252283 
[21:15:48] <grrrit-wm>	 (03CR) 10Hashar: "check experimental" [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/252283 (owner: 10Hashar)
[21:16:26] <grrrit-wm>	 (03CR) 10Hashar: "come on Jenkins !" [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/252283 (owner: 10Hashar)
[21:20:50] <grrrit-wm>	 (03PS1) 10Legoktm: Add composer-test to UserMerge extension [integration/config] - 10https://gerrit.wikimedia.org/r/252287 
[21:41:06] <twentyafterfour>	 greg-g: fyi, new branch is fubar thanks to someone changing the permissions on /srv/mediawiki-staging
[21:49:00] <grrrit-wm>	 (03PS2) 10Hashar: (WIP) Port trusty version to jessie-wikimedia (WIP) [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/252283 
[21:50:42] <hashar>	 pfff
[21:52:36] <grrrit-wm>	 (03PS3) 10Hashar: (WIP) Port trusty version to jessie-wikimedia (WIP) [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/252283 (https://phabricator.wikimedia.org/T117223) 
[21:53:29] <grrrit-wm>	 (03PS4) 10Hashar: Port trusty version to jessie-wikimedia [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/252283 (https://phabricator.wikimedia.org/T117223) 
[21:53:40] <grrrit-wm>	 (03CR) 10Hashar: [C: 032 V: 032] Port trusty version to jessie-wikimedia [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/252283 (https://phabricator.wikimedia.org/T117223) (owner: 10Hashar)
[21:54:27] <wikibugs>	 5Continuous-Integration-Scaling, 6operations, 5Patch-For-Review: install/deploy scandium as zuul merger (ci) server - https://phabricator.wikimedia.org/T95046#1797612 (10hashar)
[21:54:28] <wikibugs>	 5Continuous-Integration-Scaling, 5Patch-For-Review: Package Zuul for Debian Jessie - https://phabricator.wikimedia.org/T117223#1797610 (10hashar) 5Open>3Resolved Done! The package material is at https://people.wikimedia.org/~hashar/debs/zuul_2.1.0-60-g1cc37f7-wmf1jessie1/
[22:32:41] <wikibugs>	 5Continuous-Integration-Scaling, 6operations: Upload new Zuul packages on apt.wikimedia.org for Precise / Trusty / Jessie  - https://phabricator.wikimedia.org/T118340#1797788 (10hashar) 3NEW a:3hashar
[22:33:43] <wikibugs>	 5Continuous-Integration-Scaling, 5Patch-For-Review: Package Zuul for Debian Jessie - https://phabricator.wikimedia.org/T117223#1797805 (10hashar) Upload to apt.wikimedia.org is tracked by {T118340}
[23:10:35] <wmf-insecte>	 Yippee, build fixed!
[23:10:37] <wmf-insecte>	 Project browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #321: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/321/
[23:28:54] <wikibugs>	 10Deployment-Systems, 6Release-Engineering-Team, 6operations: deployment broken on wdqs1001 - https://phabricator.wikimedia.org/T118148#1798026 (10ori) ``` [wdqs1001:/var/log] $ sudo cat /var/log/salt/minion 2015-11-10 23:16:02,487 [salt.loaded.int.module.cmdmod][ERROR   ] Command u'/usr/bin/git checkout --f...
[23:45:07] <wikibugs>	 10Deployment-Systems, 6Release-Engineering-Team, 6operations: deployment broken on wdqs1001 - https://phabricator.wikimedia.org/T118148#1798058 (10ori) 5Open>3Resolved a:3ori First, I restarted `salt-minion`. This may or may not have been necessary. I also could have done it more concisely with 'restar...