[00:27:04] <shinken-wm>	 PROBLEM - Puppet failure on deployment-tin is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0]
[00:34:38] <grrrit-wm>	 (03PS1) 10Thcipriani: Better usage. No positional args. [tools/release] - 10https://gerrit.wikimedia.org/r/258074 
[00:36:46] <grrrit-wm>	 (03PS2) 10Thcipriani: Better usage. No positional args. [tools/release] - 10https://gerrit.wikimedia.org/r/258074 
[01:07:04] <shinken-wm>	 RECOVERY - Puppet failure on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0]
[01:18:29] <grrrit-wm>	 (03PS1) 10Legoktm: Whitelist two GCI students [integration/config] - 10https://gerrit.wikimedia.org/r/258090 
[01:21:25] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] Whitelist two GCI students [integration/config] - 10https://gerrit.wikimedia.org/r/258090 (owner: 10Legoktm)
[01:24:57] <yurik>	 are the steps to deploy a new ext to betalabs still relevant?  https://www.mediawiki.org/wiki/Writing_an_extension_for_deployment#Deploy_to_beta_cluster_on_Labs
[01:27:35] <grrrit-wm>	 (03Merged) 10jenkins-bot: Whitelist two GCI students [integration/config] - 10https://gerrit.wikimedia.org/r/258090 (owner: 10Legoktm)
[01:27:46] <yurik>	 legoktm, ^ ?
[01:27:57] <legoktm>	 !log deploying https://gerrit.wikimedia.org/r/258090
[01:28:02] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[01:28:10] * legoktm looks
[01:28:34] <legoktm>	 yurik: yes, but you need to have a security review done first
[01:28:46] <yurik>	 legoktm, before beta cluster?
[01:28:49] <legoktm>	 yes
[01:32:43] <yurik>	 thx
[01:32:43] <yurik>	 https://phabricator.wikimedia.org/T121058
[01:33:28] <csteipp>	 yurik: Cool. We're fully scheduled for the rest of the quarter, so I honestly can't guarantee we're going to get to it. But I'll certainly try.
[01:34:56] <yurik>	 csteipp, thanks!  i modeled it almost entirely on the simplified graph ext, and it basically uses the leaflet library + mapbox wrapper. Nothing major on the server.
[01:49:22] <wikibugs>	 10Continuous-Integration-Infrastructure: jenkins-bot is not running for MtDu's patches - https://phabricator.wikimedia.org/T121062#1868295 (10Legoktm) 3NEW
[03:50:23] <wmf-insecte>	 Project beta-scap-eqiad build #81812: 04FAILURE in 2 min 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/81812/
[04:01:38] <shinken-wm>	 PROBLEM - Parsoid on deployment-parsoid05 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[04:01:38] <wmf-insecte>	 Yippee, build fixed!
[04:01:39] <wmf-insecte>	 Project beta-scap-eqiad build #81813: 09FIXED in 6 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/81813/
[04:06:29] <shinken-wm>	 RECOVERY - Parsoid on deployment-parsoid05 is OK: HTTP OK: HTTP/1.1 200 OK - 1514 bytes in 0.042 second response time
[04:11:37] <wmf-insecte>	 Project beta-scap-eqiad build #81814: 04FAILURE in 7 min 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/81814/
[04:24:23] <shinken-wm>	 PROBLEM - Parsoid on deployment-parsoid05 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[04:24:44] <wmf-insecte>	 Yippee, build fixed!
[04:24:45] <wmf-insecte>	 Project beta-scap-eqiad build #81815: 09FIXED in 9 min 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/81815/
[04:26:51] <testing-shinken->	 PROBLEM - Parsoid on deployment-parsoid05 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[04:27:54] <testing-shinken->	 RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0]
[04:33:47] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Staging, 6Collaboration-Team-Backlog, 10DBA: Use External Store on Beta Cluster - https://phabricator.wikimedia.org/T95871#1868484 (10Catrope)
[06:29:26] <wmf-insecte>	 Yippee, build fixed!
[06:29:26] <wmf-insecte>	 Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #831: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/831/
[06:47:58] <testing-shinken->	 RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK
[07:51:27] <testing-shinken->	 PROBLEM - Host deployment-cache-parsoid04 is DOWN: CRITICAL - Host Unreachable (10.68.19.197)
[08:31:54] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Wikidata: Cannot find site jenkins_u3_mw - https://phabricator.wikimedia.org/T121083#1868666 (10Nikerabbit) 3NEW
[09:23:51] <wikibugs>	 6Release-Engineering-Team, 10Browser-Tests-Infrastructure, 7Ruby, 7Tracking: Update repositories that use mediawiki_selenium Ruby gem to version 1.x - https://phabricator.wikimedia.org/T94083#1868791 (10zeljkofilipin)
[09:31:48] <grrrit-wm>	 (03PS2) 10Hashar: nodepool: run apt-get upgrade while snapshoting [integration/config] - 10https://gerrit.wikimedia.org/r/257940 (https://phabricator.wikimedia.org/T120961) 
[09:31:53] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] nodepool: run apt-get upgrade while snapshoting [integration/config] - 10https://gerrit.wikimedia.org/r/257940 (https://phabricator.wikimedia.org/T120961) (owner: 10Hashar)
[09:32:53] <grrrit-wm>	 (03Merged) 10jenkins-bot: nodepool: run apt-get upgrade while snapshoting [integration/config] - 10https://gerrit.wikimedia.org/r/257940 (https://phabricator.wikimedia.org/T120961) (owner: 10Hashar)
[09:34:11] <hashar>	 !log Updating Nodepool snapshot image ; setup_node.sh now runs apt-get upgrade ( https://gerrit.wikimedia.org/r/#/c/257940/ )
[09:34:16] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[09:38:35] <wmf-insecte>	 Yippee, build fixed!
[09:38:35] <wmf-insecte>	 Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #699: 09FIXED in 1 min 33 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/699/
[09:40:22] <hashar>	 !log Image ci-jessie-wikimedia-1449740024 in wmflabs-eqiad is ready   "etcd got upgraded in the snapshot image:  etcd (2.0.10-1 => 2.2.1+dfsg-1)"
[09:40:26] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[09:43:00] <wikibugs>	 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 5Patch-For-Review: Nodepool snapshot refresh should run apt-get upgrade - https://phabricator.wikimedia.org/T120961#1868809 (10hashar) 5Open>3Resolved ``` 2015-12-10 09:36:13,725 INFO nodepool.image.build.wmflabs-eqiad.ci-jessie-wi...
[09:46:46] <wikibugs>	 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 7Nodepool: Nodepool no more refresh snapshot images automatically - https://phabricator.wikimedia.org/T120076#1868812 (10hashar)
[09:50:39] <wikibugs>	 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 7Nodepool: Nodepool no more refresh snapshot images automatically - https://phabricator.wikimedia.org/T120076#1868817 (10hashar) 5Open>3Resolved a:3hashar It created one properly on Dec 0th at 14:00 UTC and deleted the old one (4...
[09:53:38] <wikibugs>	 10Continuous-Integration-Config, 5Continuous-Integration-Scaling, 7WorkType-NewFunctionality: Migrate javascript npm CI jobs to Nodepool - https://phabricator.wikimedia.org/T119143#1868821 (10hashar)
[09:53:40] <wikibugs>	 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 7Tracking: tracking: Cleanup contint puppet manifests so they are easier to reuse / split slaves by functions - https://phabricator.wikimedia.org/T110864#1868822 (10hashar)
[09:55:15] <grrrit-wm>	 (03PS1) 10Hashar: dib: include nodejs/npm in images [integration/config] - 10https://gerrit.wikimedia.org/r/258118 (https://phabricator.wikimedia.org/T119143) 
[10:52:21] <testing-shinken->	 RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0]
[11:43:34] <zeljkof>	 mobrovac: vagrant is making me sad again :(
[11:46:00] <zeljkof>	 https://phabricator.wikimedia.org/P2394
[11:48:51] <zeljkof>	 when I remove remote origin (with the instead-of trick) and add it manually to be ssh, it works fine :/
[12:30:19] <wikibugs>	 7Browser-Tests, 10Browser-Tests-Infrastructure, 10CirrusSearch, 6Discovery, and 2 others: Upgrade CirrusSearch browser tests to use mediawiki_selenium 1.x - https://phabricator.wikimedia.org/T99653#1869022 (10zeljkofilipin)
[12:40:56] <wikibugs>	 7Browser-Tests, 10Browser-Tests-Infrastructure, 10CirrusSearch, 6Discovery, and 2 others: Upgrade CirrusSearch browser tests to use mediawiki_selenium 1.x - https://phabricator.wikimedia.org/T99653#1869036 (10zeljkofilipin)
[12:46:07] <zeljkof>	 hashar: what is the syntax for downloading a previous patch set of a commit?
[12:46:11] <zeljkof>	 for example this one https://gerrit.wikimedia.org/r/#/c/257928/1
[12:46:45] <zeljkof>	 tried `git review -d 257928,1` and `git review -d 257928/1`
[12:47:02] <zeljkof>	 wait, problem between keyboard and monitor
[12:47:30] <zeljkof>	 please ignore, the first one works
[13:02:28] <wmf-insecte>	 Yippee, build fixed!
[13:02:29] <wmf-insecte>	 Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #874: 09FIXED in 30 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/874/
[13:17:58] <mobrovac>	 zeljkof: the remote is not good
[13:18:14] <mobrovac>	 zeljkof: ssh://zfilipin@gerrit.wikimedia.org:29418/mediawiki/extensions/CirrusSearch should be ssh://zfilipin@gerrit.wikimedia.org:29418/r/mediawiki/extensions/CirrusSearch
[13:18:21] <mobrovac>	 (note the extra /r/ in the path)
[13:31:18] <zeljkof>	 mobrovac: oops
[13:31:42] <zeljkof>	 hm, not sure why it is missing, I think I have copy/pasted it
[13:34:33] <mobrovac>	 copy/pasting - making devs miserable since 1983 :)
[13:34:33] <zeljkof>	 mobrovac: just checked gerrit
[13:34:37] <zeljkof>	 https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/CirrusSearch
[13:34:47] <zeljkof>	 git clone ssh://zfilipin@gerrit.wikimedia.org:29418/mediawiki/extensions/CirrusSearch
[13:34:54] <zeljkof>	 there is no "r"
[13:35:10] <zeljkof>	 but let me try that
[13:35:51] <mobrovac>	 it'll work with /r/, i've just tried the URL from your paste
[13:36:20] <zeljkof>	 so gerrit provides url with the r, vagrant needs the r
[13:36:25] <zeljkof>	 story of my life :)
[13:37:31] <mobrovac>	 there are R's everywhere you turn :D
[13:38:23] <mobrovac>	 zeljkof: actually, the trick is that git doesn't need it, the repo is accessible on both urls, but gerrit in vagrant needs it because of the change from ssh to https
[13:39:18] <zeljkof>	 I see
[13:41:26] <wikibugs>	 7Browser-Tests, 10Browser-Tests-Infrastructure, 10CirrusSearch, 6Discovery, and 2 others: Upgrade CirrusSearch browser tests to use mediawiki_selenium 1.x - https://phabricator.wikimedia.org/T99653#1869149 (10zeljkofilipin)
[13:43:06] <grrrit-wm>	 (03PS1) 10JanZerebecki: Work around race condition in TMPDIR creation/cleanup [integration/jenkins] - 10https://gerrit.wikimedia.org/r/258140 (https://phabricator.wikimedia.org/T120824) 
[13:43:23] <grrrit-wm>	 (03PS2) 10Hashar: dib: include nodejs/npm in images [integration/config] - 10https://gerrit.wikimedia.org/r/258118 (https://phabricator.wikimedia.org/T119143) 
[13:43:25] <grrrit-wm>	 (03PS1) 10Hashar: dib: tweak messages in 01-mirror-gerrit-repos [integration/config] - 10https://gerrit.wikimedia.org/r/258141 
[13:43:25] <hashar>	 Error: Realm 'labs' required. at /puppet/modules/contint/manifests/packages/javascript.pp:7 on node integration-dev.integration.eqiad.wmflabs
[13:43:27] <grrrit-wm>	 (03PS1) 10Hashar: dib: move repo mirroring after debootstrap [integration/config] - 10https://gerrit.wikimedia.org/r/258142 
[13:43:29] <grrrit-wm>	 (03PS1) 10Hashar: dib: make sure /etc/puppet/* dir are proper symlink [integration/config] - 10https://gerrit.wikimedia.org/r/258143 
[13:43:32] <hashar>	 that is never ending :D
[13:45:19] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] dib: tweak messages in 01-mirror-gerrit-repos [integration/config] - 10https://gerrit.wikimedia.org/r/258141 (owner: 10Hashar)
[13:45:30] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] dib: move repo mirroring after debootstrap [integration/config] - 10https://gerrit.wikimedia.org/r/258142 (owner: 10Hashar)
[13:47:09] <grrrit-wm>	 (03Merged) 10jenkins-bot: dib: tweak messages in 01-mirror-gerrit-repos [integration/config] - 10https://gerrit.wikimedia.org/r/258141 (owner: 10Hashar)
[13:47:31] <grrrit-wm>	 (03PS2) 10Hashar: dib: make sure /etc/puppet/* dir are proper symlinks [integration/config] - 10https://gerrit.wikimedia.org/r/258143 
[13:47:37] <grrrit-wm>	 (03Merged) 10jenkins-bot: dib: move repo mirroring after debootstrap [integration/config] - 10https://gerrit.wikimedia.org/r/258142 (owner: 10Hashar)
[13:47:52] <grrrit-wm>	 (03PS3) 10Hashar: dib: include nodejs/npm in images [integration/config] - 10https://gerrit.wikimedia.org/r/258118 (https://phabricator.wikimedia.org/T119143) 
[13:48:06] <grrrit-wm>	 (03CR) 10Hashar: [C: 04-1] "Fails with:" [integration/config] - 10https://gerrit.wikimedia.org/r/258118 (https://phabricator.wikimedia.org/T119143) (owner: 10Hashar)
[13:50:57] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] dib: make sure /etc/puppet/* dir are proper symlinks [integration/config] - 10https://gerrit.wikimedia.org/r/258143 (owner: 10Hashar)
[13:52:44] <grrrit-wm>	 (03Merged) 10jenkins-bot: dib: make sure /etc/puppet/* dir are proper symlinks [integration/config] - 10https://gerrit.wikimedia.org/r/258143 (owner: 10Hashar)
[13:56:09] <zeljkof>	 mobrovac: :( https://phabricator.wikimedia.org/P2398
[13:56:35] <zeljkof>	 my .gitconfig https://github.com/zeljkofilipin/dotfiles/blob/master/.gitconfig#L21-L22
[13:57:38] <mobrovac>	 ah!
[13:57:46] <mobrovac>	 zeljkof: try removing /r in https://github.com/zeljkofilipin/dotfiles/blob/master/.gitconfig#L21
[13:58:02] <mobrovac>	 zeljkof: and also restore the remote not to contain /r/
[13:58:13] * mobrovac keeps his fingers crossed
[13:58:28] <zeljkof>	 mobrovac: but I have just added /r to line 21 :)
[13:58:35] <mobrovac>	 hm, so i realised
[13:58:36] <mobrovac>	 damn
[13:58:41] <zeljkof>	 ok, reverting
[13:59:45] <mobrovac>	 zeljkof: duuuh
[14:00:06] <mobrovac>	 zeljkof: https://www.mediawiki.org/wiki/MediaWiki-Vagrant#Pushing_commits -> there's an extra p/ at the end of the insteadOf
[14:00:27] * mobrovac closes this mystery as "resolved"
[14:03:14] <zeljkof>	 mobrovac: I think I have noticed it yesterday but forgot to update the docs, my .gitconfig does not have the extra p/
[14:03:32] <zeljkof>	 https://github.com/zeljkofilipin/dotfiles/blob/master/.gitconfig#L22
[14:04:03] <mobrovac>	 exactly, so try putting it there :)
[14:04:17] <zeljkof>	 mobrovac: oh, it _should_ be there?
[14:04:26] <zeljkof>	 I thought it was a typo :)
[14:04:38] <mobrovac>	 hahaha
[14:05:05] <zeljkof>	 but the project url does not have the p/
[14:05:06] <zeljkof>	 https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/CirrusSearch
[14:07:34] <mobrovac>	 zeljkof: web ui urls and api urls slightly vary for gerrit
[14:07:48] <zeljkof>	 I mean
[14:08:04] <zeljkof>	 Anonymous HTTP: git clone https://gerrit.wikimedia.org/r/mediawiki/extensions/CirrusSearch
[14:08:09] <zeljkof>	 from that page
[14:08:13] <zeljkof>	 there is no /p
[14:08:17] <zeljkof>	 that is what confused me
[14:08:28] <zeljkof>	 let me try to /p 
[14:08:56] <mobrovac>	 zeljkof: from my mw .git/config in mw-vagrant:
[14:08:59] <mobrovac>	 [submodule "vendor"]
[14:08:59] <mobrovac>	 	url = https://gerrit.wikimedia.org/r/p/mediawiki/vendor.git
[14:09:19] <zeljkof>	 strange
[14:09:26] <zeljkof>	 not sure where the /p comes from
[14:10:54] <zeljkof>	 so I should clone everything with gerrit.../r/p...repo
[14:11:27] <zeljkof>	 example: https://gerrit.wikimedia.org/r/p/mediawiki/extensions/CirrusSearch
[14:11:32] <zeljkof>	 I have https://gerrit.wikimedia.org/r/mediawiki/extensions/CirrusSearch now
[14:12:45] <mobrovac>	 yes, you need the /p/
[14:12:59] <zeljkof>	 ok, this works... $ git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/CirrusSearch
[14:13:31] <zeljkof>	 remote is rewritten to ssh://zfilipin@gerrit.wikimedia.org:29418/mediawiki/extensions/CirrusSearch
[14:14:27] <mobrovac>	 and then they say small things don't make a difference :)
[14:14:40] <zeljkof>	 mobrovac: something is still wrong :(
[14:14:51] * mobrovac sighs
[14:14:51] <zeljkof>	 copy/pasting terminal output to phab
[14:14:55] <mobrovac>	 k
[14:19:51] <zeljkof>	 mobrovac: https://phabricator.wikimedia.org/P2399
[14:28:06] <mobrovac>	 zeljkof: hm, did you run git review -s in the repo after cloning?
[14:30:40] <grrrit-wm>	 (03PS4) 10Hashar: dib: include nodejs/npm in images [integration/config] - 10https://gerrit.wikimedia.org/r/258118 (https://phabricator.wikimedia.org/T119143) 
[14:30:42] <grrrit-wm>	 (03PS1) 10Hashar: dib: hardcode $::realm to 'labs' [integration/config] - 10https://gerrit.wikimedia.org/r/258147 
[14:33:04] <grrrit-wm>	 (03CR) 10DCausse: Work around race condition in TMPDIR creation/cleanup (032 comments) [integration/jenkins] - 10https://gerrit.wikimedia.org/r/258140 (https://phabricator.wikimedia.org/T120824) (owner: 10JanZerebecki)
[14:33:04] <zeljkof>	 mobrovac: I thought that was optional :/ I never run it
[14:33:09] <zeljkof>	 but let's check :)
[14:33:29] <zeljkof>	 no, did not help
[14:33:50] <zeljkof>	 can you share your .gitconfig and try cloning CirrusSearch?
[14:34:09] <zeljkof>	 I am probably doing something wrong, but not sure how to find the mistake
[14:37:13] <wmf-insecte>	 Yippee, build fixed!
[14:37:13] <wmf-insecte>	 Project browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce build #349: 09FIXED in 9 min 12 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/349/
[14:37:43] <mobrovac>	 zeljkof: did you clone the repo manually or did you enable the cirrussearch role in vagrant?
[14:38:02] <zeljkof>	 mobrovac: this one cloned manually
[14:38:11] <zeljkof>	 I should enable the role?
[14:38:46] <mobrovac>	 zeljkof: now that we have identified the /p/, try removing the repo, enable the role and provision
[14:38:52] <mobrovac>	 perhaps that'll do the trick
[14:39:12] <zeljkof>	 since sometimes I work on extensions that do not have mw-vagrant roles, I still put them in the same repo
[14:39:27] <zeljkof>	 in the same _folder_ (mediawiki/extensions)
[14:39:42] <zeljkof>	 mobrovac: will do thanks
[14:43:28] <zeljkof>	 mobrovac: :(
[14:43:32] <zeljkof>	 no luck
[14:43:57] <zeljkof>	 removed CirrusSearch folder from mediawiki/vagrant/mediawiki/extensions/
[14:43:58] <mobrovac>	 lemme try
[14:44:05] <zeljkof>	 enabled cirrussearch role
[14:44:08] <zeljkof>	 provisioned
[14:47:05] <zeljkof>	 git pull works just fine, but not git review
[14:52:45] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] dib: hardcode $::realm to 'labs' [integration/config] - 10https://gerrit.wikimedia.org/r/258147 (owner: 10Hashar)
[14:52:56] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] dib: include nodejs/npm in images [integration/config] - 10https://gerrit.wikimedia.org/r/258118 (https://phabricator.wikimedia.org/T119143) (owner: 10Hashar)
[14:53:48] <grrrit-wm>	 (03Merged) 10jenkins-bot: dib: hardcode $::realm to 'labs' [integration/config] - 10https://gerrit.wikimedia.org/r/258147 (owner: 10Hashar)
[14:54:07] <grrrit-wm>	 (03Merged) 10jenkins-bot: dib: include nodejs/npm in images [integration/config] - 10https://gerrit.wikimedia.org/r/258118 (https://phabricator.wikimedia.org/T119143) (owner: 10Hashar)
[14:55:09] <grrrit-wm>	 (03PS1) 10Hashar: nodepool: apt-get clean in snapshots [integration/config] - 10https://gerrit.wikimedia.org/r/258150 
[14:55:25] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] nodepool: apt-get clean in snapshots [integration/config] - 10https://gerrit.wikimedia.org/r/258150 (owner: 10Hashar)
[14:56:24] <grrrit-wm>	 (03Merged) 10jenkins-bot: nodepool: apt-get clean in snapshots [integration/config] - 10https://gerrit.wikimedia.org/r/258150 (owner: 10Hashar)
[14:57:13] <hashar>	 !log created new Nodepool base image and pushing it to labs
[14:57:18] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[14:59:10] <hashar>	 !log New image id is 82a708eb-fd1a-4320-a054-6f1d4a319caa
[14:59:15] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[14:59:52] <hashar>	 !log Refreshing Nodepool snapshot ( doc is https://wikitech.wikimedia.org/wiki/Nodepool#Manually_generate_a_new_snapshot )
[14:59:56] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[15:00:25] <hashar>	 zeljkof: so in short, I spent the day to get nodejs/npm added to the nodepool instances :(
[15:00:46] <hashar>	 though to be honest a good chunk of that was fixing a totally unrelated caching issue ;-}
[15:01:10] <zeljkof>	 hashar: and I have spent some time fighting with vagrant :) with a lot of help from mobrovac 
[15:01:45] <mobrovac>	 zeljkof: in a meeting now, will need to continue l8r
[15:02:14] <zeljkof>	 mobrovac: thanks
[15:02:14] <zeljkof>	 I will change the remotes manually for now
[15:06:17] <hashar>	 !log Image ci-jessie-wikimedia-1449759571 in wmflabs-eqiad is ready
[15:06:23] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[15:06:26] <hashar>	 qa-morebots: come on
[15:06:26] <qa-morebots>	 I am a logbot running on tools-exec-1211.
[15:06:26] <qa-morebots>	 Messages are logged to https://tools.wmflabs.org/sal/releng.
[15:06:26] <qa-morebots>	 To log a message, type !log <msg>.
[15:06:42] <Luke081515>	 :)
[15:07:57] <wikibugs>	 6Release-Engineering-Team, 10Browser-Tests-Infrastructure, 7Ruby, 7Tracking: Update repositories that use mediawiki_selenium Ruby gem to version 1.x - https://phabricator.wikimedia.org/T94083#1869267 (10zeljkofilipin)
[15:10:16] <hashar>	 !log deleted all nodepool snapshot image ci-jessie-wikimedia-1449740024	
[15:10:21] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[15:12:02] <wikibugs>	 10Continuous-Integration-Config, 5Continuous-Integration-Scaling, 5Patch-For-Review, 7WorkType-NewFunctionality: Migrate javascript npm CI jobs to Nodepool - https://phabricator.wikimedia.org/T119143#1869279 (10hashar) We now have nodejs/npm provisioned in Nodepool images.
[15:13:22] <grrrit-wm>	 (03PS2) 10Krinkle: Avoid race condition with EXECUTOR_NUMBER for TMPDIR [integration/jenkins] - 10https://gerrit.wikimedia.org/r/258140 (https://phabricator.wikimedia.org/T120824) (owner: 10JanZerebecki)
[15:19:06] <wikibugs>	 6Release-Engineering-Team, 10Browser-Tests-Infrastructure, 7Ruby, 7Tracking: Update repositories that use mediawiki_selenium Ruby gem to version 1.x - https://phabricator.wikimedia.org/T94083#1869293 (10zeljkofilipin)
[15:20:06] <zeljkof>	 hashar: looks like ZeroPortal has gem "mediawiki_selenium", "~> 1.0.0.pre.1" in Gemfile
[15:20:28] <zeljkof>	 but mediawiki_selenium (1.0.2) in Gemfile.lock
[15:20:41] <zeljkof>	 that I why it was ignored in https://phabricator.wikimedia.org/T94083
[15:22:24] <grrrit-wm>	 (03CR) 10Krinkle: [C: 04-1] Avoid race condition with EXECUTOR_NUMBER for TMPDIR (031 comment) [integration/jenkins] - 10https://gerrit.wikimedia.org/r/258140 (https://phabricator.wikimedia.org/T120824) (owner: 10JanZerebecki)
[15:23:01] <grrrit-wm>	 (03CR) 10Krinkle: Avoid race condition with EXECUTOR_NUMBER for TMPDIR (031 comment) [integration/jenkins] - 10https://gerrit.wikimedia.org/r/258140 (https://phabricator.wikimedia.org/T120824) (owner: 10JanZerebecki)
[15:24:45] <wikibugs>	 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Dozens of jobs failing on integration-slave-trusty-1012 because chmod fails for /tmp/jenkins-2 - https://phabricator.wikimedia.org/T120824#1869302 (10Krinkle) Even if it is true that post-build runs concurrent with the next build (which I can't reprod...
[15:47:10] <grrrit-wm>	 (03CR) 10JanZerebecki: [C: 04-1] Avoid race condition with EXECUTOR_NUMBER for TMPDIR (031 comment) [integration/jenkins] - 10https://gerrit.wikimedia.org/r/258140 (https://phabricator.wikimedia.org/T120824) (owner: 10JanZerebecki)
[15:51:17] <grrrit-wm>	 (03CR) 10JanZerebecki: [C: 04-2] Avoid race condition with EXECUTOR_NUMBER for TMPDIR [integration/jenkins] - 10https://gerrit.wikimedia.org/r/258140 (https://phabricator.wikimedia.org/T120824) (owner: 10JanZerebecki)
[15:52:41] <grrrit-wm>	 (03Abandoned) 10JanZerebecki: Avoid race condition with EXECUTOR_NUMBER for TMPDIR [integration/jenkins] - 10https://gerrit.wikimedia.org/r/258140 (https://phabricator.wikimedia.org/T120824) (owner: 10JanZerebecki)
[15:53:41] <wikibugs>	 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Dozens of jobs failing on integration-slave-trusty-1012 because chmod fails for /tmp/jenkins-2 - https://phabricator.wikimedia.org/T120824#1869539 (10JanZerebecki) I misunderstood your comment T120824#1865989 .
[15:57:00] <testing-shinken->	 PROBLEM - Puppet failure on integration-dev is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[15:57:00] <wikibugs>	 6Release-Engineering-Team, 10Browser-Tests-Infrastructure, 7Ruby, 7Tracking: Update repositories that use mediawiki_selenium Ruby gem to version 1.x - https://phabricator.wikimedia.org/T94083#1869552 (10zeljkofilipin)
[16:02:06] <wikibugs>	 6Release-Engineering-Team, 10Browser-Tests-Infrastructure, 6Security: Update all repositories that use mediawiki_selenium Ruby gem to version 1.6.x - https://phabricator.wikimedia.org/T114241#1869563 (10zeljkofilipin)
[16:04:16] <wikibugs>	 5Release-Engineering-Epics, 10Browser-Tests-Infrastructure, 7Epic, 7Tracking: Fix or delete failing browser tests Jenkins jobs - https://phabricator.wikimedia.org/T94150#1869569 (10zeljkofilipin) a:5zeljkofilipin>3None
[16:07:04] <wikibugs>	 6Release-Engineering-Team, 5Testing-Initiative-2015, 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-Examples, 7Documentation: Improve documentation around running/writing (with lots of examples) browser tests - https://phabricator.wikimedia.org/T108108#1869592 (10zeljkofilipin) a:5zeljkofilipin...
[16:09:20] <mobrovac>	 zeljkof: w4m
[16:09:37] <mobrovac>	 lemme paste
[16:09:48] <zeljkof>	 mobrovac: argh
[16:09:54] <zeljkof>	 what am I doing wrong?
[16:12:05] <testing-shinken->	 RECOVERY - Puppet failure on integration-dev is OK: OK: Less than 1.00% above the threshold [0.0]
[16:13:33] <mobrovac>	 zeljkof: https://phabricator.wikimedia.org/P2401
[16:21:51] <zeljkof>	 mobrovac: but wait, how come `cat .git` works for you?
[16:21:59] <zeljkof>	 I get "cat: .git: Is a directory"
[16:22:10] <zeljkof>	 am I doing something wrong?
[16:22:25] <zeljkof>	 (I was following your steps, you have two typos)
[16:22:34] <zeljkof>	 $ vagrant role enable cirussearch && vagrant provision
[16:22:36] <zeljkof>	 should be
[16:22:42] <zeljkof>	 $ vagrant roles enable cirrussearch && vagrant provision
[16:23:16] <mobrovac>	 yeah, didn't c/p that part, just wrote it
[16:23:36] <zeljkof>	 I was literally copy/pasting (again) :)
[16:23:42] <zeljkof>	 not sure what am I doing wrong
[16:27:52] <mobrovac>	 zeljkof: ok, try removing the cirrussearch dir, disable the role, provision, enable the role, provision and then run git-update
[16:28:09] <zeljkof>	 mobrovac: ok, will do
[16:28:16] <zeljkof>	 have to go to a meeting, will run in the backgrount
[16:28:21] <zeljkof>	 backgrounD
[16:28:31] <mobrovac>	 me too
[16:28:32] <mobrovac>	 :)
[16:29:23] <hashar>	 !jenkins
[16:29:23] <wm-bot>	 https://integration.wikimedia.org/ci/job/
[16:31:04] <zeljkof>	 mobrovac: the usual "shut down all windows..." ;)
[16:31:42] <mobrovac>	 :P
[16:43:05] <grrrit-wm>	 (03PS2) 10Hashar: [ContentTranslation] Move jshint to check: [integration/config] - 10https://gerrit.wikimedia.org/r/257957 (owner: 10Paladox)
[16:43:30] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] [ContentTranslation] Move jshint to check: [integration/config] - 10https://gerrit.wikimedia.org/r/257957 (owner: 10Paladox)
[16:44:25] <grrrit-wm>	 (03Merged) 10jenkins-bot: [ContentTranslation] Move jshint to check: [integration/config] - 10https://gerrit.wikimedia.org/r/257957 (owner: 10Paladox)
[17:20:23] <zeljkof>	 mobrovac: disable role, provision, enable role, provision - still the same error :(
[17:20:36] <mobrovac>	 damn it
[17:21:05] <zeljkof>	 I do not think I have anything custom in mw-vagrant
[17:21:30] <zeljkof>	 I am trying to use it in the most usual way
[17:21:40] <zeljkof>	 not sure what am I doing wrong
[17:22:31] <zeljkof>	 anyway, thanks for the help, I owe you a beer (or six) ;)
[17:22:37] <zeljkof>	 time to get some food
[17:37:04] <mobrovac>	 :)
[17:37:12] <mobrovac>	 zeljkof: dobar tek!
[17:37:41] <zeljkof>	 Hvala :)
[17:48:55] <wikibugs>	 10MediaWiki-Releasing, 6Developer-Relations, 10Wikimedia-Blog-Content, 3DevRel-December-2015, 5MW-1.26-release: Write blog post announcing MW 1.26 - https://phabricator.wikimedia.org/T112842#1869929 (10greg) >>! In T112842#1865220, @Qgil wrote: > @greg could you help assessing what is missing and/or when...
[17:54:15] <wikibugs>	 10MediaWiki-Releasing, 6Developer-Relations, 10Wikimedia-Blog-Content, 3DevRel-December-2015, 5MW-1.26-release: Write blog post announcing MW 1.26 - https://phabricator.wikimedia.org/T112842#1869955 (10greg) Ah, one thing I was looking for was this: https://twitter.com/mediawiki/status/630865572654264320...
[18:25:59] <wikibugs>	 10Beta-Cluster-Infrastructure, 5Patch-For-Review, 7Varnish, 7WorkType-Maintenance: On beta cluster varnish stats process points to production statsd - https://phabricator.wikimedia.org/T116898#1870175 (10Milimetric)
[18:28:26] <wikibugs>	 10MediaWiki-Releasing, 6Developer-Relations, 10Wikimedia-Blog-Content, 3DevRel-December-2015, 5MW-1.26-release: Write blog post announcing MW 1.26 - https://phabricator.wikimedia.org/T112842#1870190 (10Legoktm) >>! In T112842#1863897, @jrbs wrote: >>>! In T112842#1862678, @Legoktm wrote: >> Hold on a bit...
[18:31:13] <twentyafterfour>	 so I'm getting a kinda odd git error in my harbormaster-test jenkins job now: https://integration.wikimedia.org/ci/job/harbormaster-test/224/console
[18:31:32] <twentyafterfour>	 Caused by: hudson.plugins.git.GitException: Command "git -c core.askpass=true fetch --tags --progress https://phabricator.wikimedia.org/diffusion/PHSTAGE/harbormaster-staging.git +refs/tags/*:refs/remotes/staging/*" returned status code 128:
[19:24:33] <legoktm>	 https://integration.wikimedia.org/ci/job/npm/41361/console ??
[19:24:42] <legoktm>	 a lot of 19:09:50 rm: cannot remove ‘/mnt/home/jenkins-deploy/tmpfs/jenkins-3/lessphp_asrimhyzbkg8gk48k84ss0c8ogcs484.lesscache’: Permission denied
[19:26:44] <legoktm>	 https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/42477/console
[19:27:26] <legoktm>	 same host
[19:27:29] <legoktm>	 is it busted?
[19:28:29] <thcipriani>	 hmm, all owned by www-data
[19:28:42] <thcipriani>	 user and group
[19:30:10] <legoktm>	 !log marked integration-slave-trusty-1011 as offline, all jobs failing due to tmpfs/lesscache permission denied errors
[19:30:15] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[19:31:15] <legoktm>	 thcipriani: looks like https://phabricator.wikimedia.org/T120824
[19:31:44] <wikibugs>	 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Dozens of jobs failing on integration-slave-trusty-1012 because chmod fails for /tmp/jenkins-2 - https://phabricator.wikimedia.org/T120824#1870416 (10Legoktm) I just marked integration-slave-trusty-1011 as offline because all jobs on it were failing.
[19:38:55] <thcipriani>	 blerg, would be nice to not get a forbidden for this https://integration.wikimedia.org/ci/computer/integration-slave-trusty-1011/builds :\
[19:41:03] <James_F>	 greg-g: Meh, 'Reduce deploy tools' should be 'Consolidate deploy tools' for clarity. ;-)
[19:44:59] <greg-g>	 James_F: it was "reduce number of" before, i think
[19:45:05] <greg-g>	 but yeah, /me fixes
[19:45:09] <James_F>	 greg-g: :-)
[19:53:03] <wikibugs>	 10MediaWiki-Releasing, 6Developer-Relations, 10Wikimedia-Blog-Content, 3DevRel-December-2015, 5MW-1.26-release: Write blog post announcing MW 1.26 - https://phabricator.wikimedia.org/T112842#1870488 (10greg) doit
[19:54:19] <wikibugs>	 10MediaWiki-Releasing, 6Developer-Relations, 10Wikimedia-Blog-Content, 3DevRel-December-2015, 5MW-1.26-release: Write blog post announcing MW 1.26 - https://phabricator.wikimedia.org/T112842#1870493 (10greg) And thanks a ton, @Legoktm. I'm totally fine with your name replacing mine at the end, fwiw :)
[19:56:55] <testing-shinken->	 PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<12.50%)
[20:05:16] <grrrit-wm>	 (03PS1) 10Thcipriani: Add flag to resume branching from named extension [tools/release] - 10https://gerrit.wikimedia.org/r/258203 
[20:16:56] <testing-shinken->	 RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK
[20:52:06] <James_F>	 Gah. How did the SWAT get full already.
[20:52:45] * James_F considers special pleading to greg-g to let him add more VE things.
[20:54:20] <bd808>	 James_F: I would guess SWAT got full because people have been waiting for weeks to get new code on enwiki :)
[20:54:27] <James_F>	 bd808: Yeah. :-(
[20:54:45] <bd808>	 deploy freezes suck
[20:55:17] <James_F>	 And we've got another one just ahead of us.
[20:56:00] * Reedy signs up to deploy the MediaWiki train on Christmas day
[20:56:06] <James_F>	 Reedy: Hic.
[20:56:10] <James_F>	 It's a Friday, tsk.
[20:56:18] <Reedy>	 Fine fine
[20:56:22] <Reedy>	 New Years Day
[20:56:48] <James_F>	 :-)
[21:22:07] <wmf-insecte>	 Yippee, build fixed!
[21:22:08] <wmf-insecte>	 Project browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #95: 09FIXED in 6 min 6 sec: https://integration.wikimedia.org/ci/job/browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/95/
[21:26:27] <wmf-insecte>	 Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #854: 04STILL FAILING in 1 hr 0 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/854/
[21:35:06] <wikibugs>	 6Release-Engineering-Team, 10DBA, 7user-notice: Requests to globally reset a user's skin preferences - https://phabricator.wikimedia.org/T119206#1870832 (10NickK) >>! In T119206#1838886, @demon wrote: > No that isn't possible I'm afraid. It will only set the preference on wikis in which you have a local acco...
[22:16:05] <wikibugs>	 10Continuous-Integration-Config, 6Community-Tech: mediawiki/extensions/PageAssessments history should be cleaned and reimported + other concerns - https://phabricator.wikimedia.org/T121157#1870926 (10hashar) 3NEW
[22:30:28] <testing-shinken->	 RECOVERY - Puppet failure on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:38:54] <legoktm>	 hashar is awake but not on IRC :(
[22:40:43] <wikibugs>	 10Deployment-Systems: [scap] [l10n] Atomic updates for sync scripts - https://phabricator.wikimedia.org/T22085#1871011 (10Krinkle) >>! In T22085#253069, @Jdforrester-WMF wrote: > This bug not yet being fixed caused T65791 today, the fifth or sixth VE breakage from this that I recall. :-( It'd be really great if...
[22:41:24] <wikibugs>	 10Deployment-Systems: [scap] Local sync script on any individual server should be atomic - https://phabricator.wikimedia.org/T22085#1871015 (10Krinkle)
[22:41:51] <grrrit-wm>	 (03CR) 10Hashar: [C: 031] "+1 but I would suggest to use https://docs.python.org/2.7/library/argparse.html#action instead of php getop :-)" [tools/release] - 10https://gerrit.wikimedia.org/r/258074 (owner: 10Thcipriani)
[22:58:10] <testing-shinken->	 PROBLEM - Puppet failure on integration-dev is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0]
[23:25:33] <wikibugs>	 10Deployment-Systems: [scap] Local sync script on any individual server should be atomic - https://phabricator.wikimedia.org/T22085#1871562 (10bd808) >>! In T22085#1871011, @Krinkle wrote: > Because for a short time an individual server had the newer version of CommonSettings.php but not yet the updated Initiali...
[23:25:49] <wikibugs>	 10Deployment-Systems: [scap] Local sync script on any individual server should be atomic - https://phabricator.wikimedia.org/T22085#1871566 (10bd808)
[23:25:50] <wikibugs>	 6Release-Engineering-Team, 10Wikimedia-Apache-configuration, 6operations: Make it possible to quickly and programmatically pool and depool application servers - https://phabricator.wikimedia.org/T73212#760100 (10bd808)
[23:25:53] <wikibugs>	 10Deployment-Systems, 3Scap3, 6Performance-Team, 6operations, 7HHVM: Make scap able to depool/repool servers via the conftool API - https://phabricator.wikimedia.org/T104352#1414314 (10bd808)
[23:28:34] <wmf-insecte>	 Project beta-scap-eqiad build #81922: 04FAILURE in 2 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/81922/
[23:33:08] <testing-shinken->	 RECOVERY - Puppet failure on integration-dev is OK: OK: Less than 1.00% above the threshold [0.0]
[23:38:58] <wmf-insecte>	 Yippee, build fixed!
[23:38:59] <wmf-insecte>	 Project beta-scap-eqiad build #81923: 09FIXED in 7 min 52 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/81923/
[23:39:22] <wikibugs>	 10Deployment-Systems: [scap] Local sync script on any individual server should be atomic - https://phabricator.wikimedia.org/T22085#1871600 (10Krinkle) >>! In T22085#1871562, @bd808 wrote: >>>! In T22085#1871011, @Krinkle wrote: >> Because for a short time an individual server had the newer version of CommonSett...
[23:39:23] <wikibugs>	 10Beta-Cluster-Infrastructure: Review rights removal by user Vogone - https://phabricator.wikimedia.org/T121168#1871601 (10Luke081515) 3NEW
[23:45:39] <Krenair>	 ^ There better be a seriously good reason for this.
[23:45:47] <wikibugs>	 10Deployment-Systems: [scap] Local sync script on any individual server should be atomic - https://phabricator.wikimedia.org/T22085#1871611 (10bd808) >>! In T22085#1871600, @Krinkle wrote: >> {T73212} >> {T104352} >  > Reloading HHVM and/or using RepoAuthoritive mode is also an option.  Both of those options are...
[23:56:09] <wikibugs>	 10Beta-Cluster-Infrastructure: Review rights removal by user Vogone - https://phabricator.wikimedia.org/T121168#1871640 (10Krenair) I'm told you've been evading IRC bans?  >I used my rights too, to find security issues, for users who can see security bugs, [[https://phabricator.wikimedia.org/maniphest/query/_35S...