[17:04:11] <zeljkof>	 greg-g: left a few comments in the job description
[17:04:17] * zeljkof is over and out
[17:04:20] <greg-g>	 cool, thanks!
[17:37:57] <Krenair>	     666 No such file or directory in /srv/mediawiki/php-1.25wmf22/extensions/SpamBlacklist/BaseBlacklist.php on line 269
[17:43:21] <Glaisher>	 wtf.
[17:43:43] <Glaisher>	 Krenair: might be related to the config change earlier
[17:43:56] <Krenair>	 I was wondering the same thing
[17:43:59] <Glaisher>	 https://meta.wikimedia.org/w/index.php?title=User:Glaisher&diff=11721125&oldid=11404504
[17:44:00] <Glaisher>	 crap
[17:44:08] <Glaisher>	 SBL's not working now
[17:44:52] <Glaisher>	 preg_match( '/^http:\/\//', $fileName ) ) {
[17:45:54] <Krenair>	 Looks like they're doing a train deploy right now, so I can't really do the revert at the moment
[17:59:10] <wmf-insecte>	 Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #494: FAILURE in 1 min 10 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/494/
[18:22:11] <legoktm>	 does anyone in here have any objections to https://gerrit.wikimedia.org/r/#/c/200770/ ?
[18:22:53] <grrrit-wm>	 (03PS3) 10Legoktm: Create generic 'mwext-qunit' job [integration/config] - 10https://gerrit.wikimedia.org/r/200327 (https://phabricator.wikimedia.org/T94326) 
[18:29:26] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] Create generic 'mwext-qunit' job [integration/config] - 10https://gerrit.wikimedia.org/r/200327 (https://phabricator.wikimedia.org/T94326) (owner: 10Legoktm)
[18:39:08] <James_F>	 Beta Cluster down?
[18:39:18] <James_F>	 Down == 503ing from api.php.
[18:39:20] <YuviPanda>	 is it?
[18:39:32] <James_F>	 http://en.wikipedia.beta.wmflabs.org/w/api.php?action=visualeditor&page=European_robin&paction=parse&format=jsonfm
[18:39:50] <legoktm>	 http://en.wikipedia.beta.wmflabs.org/w/api.php?action=query
[18:39:54] <James_F>	 Hmm.
[18:39:56] <legoktm>	 so fatal?
[18:40:01] <James_F>	 Eh.
[18:40:07] * James_F blames Aaron.
[18:40:22] <YuviPanda>	 yeah, http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page?debug=True seems ok
[18:40:26] <legoktm>	 Mar 31 18:40:13 deployment-mediawiki01:  #012Fatal error: Class undefined: Utfnormal\Utils in /srv/mediawiki/php-master/includes/libs/normal/UtfNormalUtil.php on line 86
[18:40:28] <thedj>	 was beta up then ?
[18:40:30] <YuviPanda>	 maybe some code that was merged fucked up?
[18:40:30] <YuviPanda>	 hah
[18:40:59] <legoktm>	 oh screw you php
[18:41:03] <James_F>	 legoktm: Is composer update being run on Beta Labs?
[18:41:12] <legoktm>	 no I made a typo
[18:41:40] <James_F>	 Fun.
[18:42:11] <grrrit-wm>	 (03Merged) 10jenkins-bot: Create generic 'mwext-qunit' job [integration/config] - 10https://gerrit.wikimedia.org/r/200327 (https://phabricator.wikimedia.org/T94326) (owner: 10Legoktm)
[18:48:30] <legoktm>	 !LOG DEPLOYING https://gerrit.wikimedia.org/r/200327
[18:48:34] <legoktm>	 err, oops
[18:48:35] <qa-morebots>	 Logged the message, Master
[19:08:59] <grrrit-wm>	 (03PS1) 10Legoktm: Use generic mwext-qunit job for all extensions where possible [integration/config] - 10https://gerrit.wikimedia.org/r/200926 
[19:18:36] <grrrit-wm>	 (03PS2) 10Legoktm: Use generic mwext-qunit job for all extensions where possible [integration/config] - 10https://gerrit.wikimedia.org/r/200926 
[19:44:22] <^demon|away>	 thcipriani: updateBranchPointers assumes sync-common has been run at least once on the deployment host. All works dandy now :)
[19:44:39] <greg-g>	 thcipriani: twentyafterfour ^demon|away marxarelli: duders, I'm currently at a coffee shop near sebastopol (needed to stop on way back from appt, tomake our meeting in time) which means I'm on a mifi, which means who know re hangout. I'll try to join and kick off the meeting and then dip (as previously planned anywho)
[19:45:15] <marxarelli>	 greg-g: aight
[19:45:22] <thcipriani>	 ^demon|away: hooray!
[19:45:52] <greg-g>	 (btw, if ya'll ever want to explore north bay when you're in town, I got some good places to take you)
[19:46:18] <greg-g>	 (same goes to those who are local, of course ;) )
[19:58:22] <^demon|away>	 greg-g: My favorite hiking place in the north bay is the Marin Water District / Mt. Tam watershed
[19:58:30] <^demon|away>	 All these really nice lakes nestled back in the hills
[19:58:41] <grrrit-wm>	 (03PS3) 10Hashar: Disable mediawiki-core-release on tag publishing [integration/config] - 10https://gerrit.wikimedia.org/r/200876 (https://phabricator.wikimedia.org/T94583) 
[19:59:32] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] "because you know. I have to do it :D" [integration/config] - 10https://gerrit.wikimedia.org/r/200876 (https://phabricator.wikimedia.org/T94583) (owner: 10Hashar)
[20:00:37] <aharoni>	 hashar: I LOVE YOUR LANGUAGE
[20:00:41] <grrrit-wm>	 (03Merged) 10jenkins-bot: Disable mediawiki-core-release on tag publishing [integration/config] - 10https://gerrit.wikimedia.org/r/200876 (https://phabricator.wikimedia.org/T94583) (owner: 10Hashar)
[20:00:54] <aharoni>	 hashar: https://fr.wikipedia.org/wiki/Sp%C3%A9cial:ContentTranslationStats
[20:01:03] <aharoni>	 SIX TRANSLATIONS
[20:01:14] <aharoni>	 I don't use all-caps without a good reason.
[20:01:34] <hashar>	 I DO NOT EITHER UNLESS IT IS INTERNATIONAL CAPS LOCK DAY
[20:02:30] <aharoni>	 hashar: Well, my team has been developing ContentTranslation for over a year, and now it's deployed in the French Wikipedia, and in just a few years SIX articles were created
[20:02:37] <aharoni>	 SO I HAVE A DAMN GOOD REASON \o/
[20:02:52] <hashar>	 so
[20:03:00] <YuviPanda>	 years? :)
[20:03:03] <hashar>	 a year of development helped create six articles in a full year?
[20:03:07] <hashar>	 WTF !!!!
[20:03:19] <YuviPanda>	 hours i bet :)
[20:03:57] <hashar>	 aharoni: the thing is that ton of French peoples knows other languages
[20:04:07] <aharoni>	 [ hours, of course ]
[20:04:09] <hashar>	 aharoni: and other countries have a ton of french as a second languages
[20:04:14] <aharoni>	 [ des heures ]
[20:04:27] <hashar>	 aharoni: more over, for some reasonn french people prefer to read stuff in their language, so there are a LOT of translators
[20:04:42] <aharoni>	 hashar: well yes, I very much do hope that African countries will pick this up like crazy
[20:04:57] <hashar>	 aharoni: make sure to announce it on the french bistrot. I am sure that will get a lot of people attracted
[20:05:05] <hashar>	 hopefully! 
[20:05:17] <hashar>	 I love that content translation project
[20:05:42] <hashar>	 to populate the french wiki I was opening an enwiki page on a side of the screen
[20:05:48] <hashar>	 and the empty article to frwiki
[20:06:01] <hashar>	 but keeping them up to date was not very easy
[20:06:05] <aharoni>	 hashar: this community asked for it, actually - https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Sondage/Activation_de_l%27outil_de_traduction
[20:06:30] <aharoni>	 with the percentage of "Support" votes similar to election results in People's Democratic Republic of Korea.
[20:07:05] <hashar>	 aharoni: so how the hell do I start using it ?
[20:07:53] <aharoni>	 hashar: enable in beta features
[20:07:59] <aharoni>	 go to your contributions
[20:08:04] <aharoni>	 press "New contribution"
[20:08:12] <grrrit-wm>	 (03PS3) 10Legoktm: Use generic mwext-qunit job for all extensions where possible [integration/config] - 10https://gerrit.wikimedia.org/r/200926 
[20:08:21] <aharoni>	 select an article to translate, and translate :)
[20:08:30] <hashar>	 30 users enabled it on frwiki :D
[20:08:40] <aharoni>	 (we are going to make this "New contribution" button much more accessible soon)
[20:08:57] <aharoni>	 and btw kart_ - YOU'RE AWESOME
[20:09:01] <hashar>	  ahah
[20:09:23] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] Use generic mwext-qunit job for all extensions where possible [integration/config] - 10https://gerrit.wikimedia.org/r/200926 (owner: 10Legoktm)
[20:09:52] <hashar>	 points to labs oh no https://tools.wmflabs.org/not-in-the-other-language/?lang1=fr&lang2=fr
[20:10:44] <hashar>	 aharoni: terrible :/
[20:10:57] <hashar>	 I was expecting to be asked languages I kno
[20:10:57] <hashar>	 w
[20:11:07] <hashar>	 then be presented a list of candidates to translte
[20:11:46] <hashar>	 aharoni: glad to see the frwiki community strongly support the utility; That is nice to see!
[20:14:02] <grrrit-wm>	 (03Merged) 10jenkins-bot: Use generic mwext-qunit job for all extensions where possible [integration/config] - 10https://gerrit.wikimedia.org/r/200926 (owner: 10Legoktm)
[20:15:01] <legoktm>	 !log deploying https://gerrit.wikimedia.org/r/200926
[20:15:06] <qa-morebots>	 Logged the message, Master
[20:30:34] <grrrit-wm>	 (03PS5) 10JanZerebecki: Switch wikidata qunit jobs from qunit to qunit-karma [integration/config] - 10https://gerrit.wikimedia.org/r/198699 (https://phabricator.wikimedia.org/T94393) (owner: 10Adrian Lang)
[20:30:56] <grrrit-wm>	 (03CR) 10JanZerebecki: [C: 031] "Works. Updated jobs mwext-Wikibase-qunit and mwext-Wikidata-qunit in jenkins. Please merge." [integration/config] - 10https://gerrit.wikimedia.org/r/198699 (https://phabricator.wikimedia.org/T94393) (owner: 10Adrian Lang)
[20:31:15] <grrrit-wm>	 (03CR) 10JanZerebecki: "PS5 was a rebase only." [integration/config] - 10https://gerrit.wikimedia.org/r/198699 (https://phabricator.wikimedia.org/T94393) (owner: 10Adrian Lang)
[20:37:57] <aharoni>	 gilles: as I told your colleague and compatriot hashar, ContentTranslation is enabled in the French Wikipedia!!!
[20:38:22] <gilles>	 woohoo :)
[20:39:25] <aharoni>	 gilles: and it's amazing: six translations in just a few hours
[20:39:28] <aharoni>	 https://fr.wikipedia.org/wiki/Sp%C3%A9cial:ContentTranslationStats
[20:39:33] <aharoni>	 https://fr.wikipedia.org/w/index.php?title=Sp%C3%A9cial:Modifications_r%C3%A9centes&tagfilter=contenttranslation
[20:54:00] <grrrit-wm>	 (03CR) 10Krinkle: [C: 032] Switch wikidata qunit jobs from qunit to qunit-karma [integration/config] - 10https://gerrit.wikimedia.org/r/198699 (https://phabricator.wikimedia.org/T94393) (owner: 10Adrian Lang)
[20:57:18] <greg-g>	 legoktm: do you know if it's safe to re-enable the puppet compiler slave?
[20:57:31] <greg-g>	 not sure if that was disabled yseterdday due to OMGNFS or what?
[20:58:35] <mutante>	 hello, so the puppet compiler is broken
[20:58:39] <greg-g>	 mutante: I just pinged legoktm re puppet compiler, not sure if it was disabled due to nfs stuff or not
[20:58:42] <mutante>	 the one at https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/
[20:59:02] <mutante>	 and it tells me that the instance puppet-compiler02 is down
[20:59:16] <greg-g>	 "This node is offline because Jenkins failed to launch the slave agent on it."
[20:59:20] <greg-g>	 https://integration.wikimedia.org/ci/computer/puppet-compiler02.eqiad.wmflabs/
[20:59:20] <mutante>	 i reported that yesterday and both Andrew and Coren were already on it
[20:59:30] <mutante>	 and fixed that it has a /home mounted and stuff
[20:59:33] <mutante>	 which was due to NFS
[20:59:48] <mutante>	 then Coren said he mounted it back, but also Andrew said he migrated the instance
[20:59:52] <mutante>	 and the IP address changed
[20:59:58] <greg-g>	 !log puppet-compiler02: This node is offline because Jenkins failed to launch the slave agent on it.
[21:00:01] <mutante>	 so i guess now it's just broken because the IP changed
[21:00:02] <qa-morebots>	 Logged the message, Master
[21:00:13] <mutante>	 and we need to tell Jenkins that?
[21:00:23] <greg-g>	 [03/31/15 20:54:04] [SSH] Remote file system root /mnt/jenkins-workspace does not exist. Will try to create it...
[21:00:26] <greg-g>	 hudson.util.IOException2: Could not copy slave.jar into '/mnt/jenkins-workspace' on slave
[21:00:53] <mutante>	 yea, that's basically what i reported yesterday
[21:00:59] <mutante>	 Coren claimed it was fixed :)
[21:02:00] <greg-g>	 I assume clicking on the "launch slave agent" button again won't hurt anything?
[21:02:12] <greg-g>	 (other than probably fail again?)
[21:02:19] <mutante>	 i assumed the same and clicked it before :p
[21:02:24] <mutante>	 yesterday though
[21:02:39] <greg-g>	 same
[21:02:47] <greg-g>	 same result, that is
[21:09:39] <legoktm>	 sorry was having lunch
[21:09:41] <legoktm>	 looking
[21:11:50] <legoktm>	 mutante: what labs project is the puppet-compiler in?
[21:14:10] <mutante>	 ehm, i dunno yet, let me try finding out
[21:14:42] <grrrit-wm>	 (03PS1) 10Hharchani: Update ExtraCharacters Sniff to allow valid shebang [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/201010 (https://phabricator.wikimedia.org/T94141) 
[21:27:17] <grrrit-wm>	 (03PS1) 10Legoktm: Create prepare-mediawiki-zuul-project builder macro [integration/config] - 10https://gerrit.wikimedia.org/r/201022 
[21:30:19] <grrrit-wm>	 (03PS1) 10Legoktm: Create generic phpunit job for extensions [integration/config] - 10https://gerrit.wikimedia.org/r/201032 (https://phabricator.wikimedia.org/T94327) 
[21:33:06] <mutante>	 legoktm: project: puppet3-diffs
[21:34:16] <legoktm>	 mutante: I'm not a part of that project...can you ssh in and check that /mnt is mounted properly?
[21:41:15] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[21:42:55] <mutante>	 legoktm: /mnt/jenkins-workspace ??
[21:43:02] <legoktm>	 yes
[21:43:03] <mutante>	 no such file or directory
[21:43:15] <legoktm>	 does /mnt exist?
[21:43:21] <mutante>	 yes
[21:43:28] <mutante>	 there is "home" in it , that's it
[21:44:19] <legoktm>	 hmm
[21:44:28] <legoktm>	 maybe it has to be manually created? I'm not sure about this
[21:44:51] <YuviPanda>	 hmm
[21:45:01] <mutante>	 root@puppet-compiler02:/mnt/home/jenkins-deploy# ls
[21:45:01] <mutante>	 root@puppet-compiler02:/mnt/home/jenkins-deploy# 
[21:45:40] <YuviPanda>	 not sure why ^ errored, looks ok to me
[21:46:03] <legoktm>	 it should just be /mnt/jenkins-workspace
[21:46:30] <legoktm>	 https://integration.wikimedia.org/ci/computer/puppet-compiler02.eqiad.wmflabs/log
[21:46:35] <mutante>	 YuviPanda: aaww, you didn't say that about the compiler? i was starting to hope
[21:46:37] <legoktm>	 [03/31/15 21:42:04] [SSH] Remote file system root /mnt/jenkins-workspace does not exist. Will try to create it...
[21:46:37] <legoktm>	 hudson.util.IOException2: Could not copy slave.jar into '/mnt/jenkins-workspace' on slave
[21:46:42] <legoktm>	 Caused by: com.trilead.ssh2.SFTPException: Permission denied (SSH_FX_PERMISSION_DENIED: The user does not have sufficient permissions to perform the operation.)
[21:46:44] <YuviPanda>	 mutante: oh, no, I meant the shinken alert
[21:46:47] <mutante>	 yep, been there yesterday
[21:46:50] <mutante>	 sigh
[21:48:46] <mutante>	 i guess the labs puppet changes are tested in another compiler:)
[21:51:05] <shinken-wm>	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 48151 bytes in 0.515 second response time  
[21:56:04] <mutante>	 legoktm: do we know the name of the user that does not have sufficient permissions?
[21:56:15] <mutante>	 id: jenkins: No such user
[21:56:20] <legoktm>	 I think it runs as jenkins-slave or jenkins-deploy
[21:56:37] <mutante>	 ah, both of those exist
[21:56:43] <mutante>	 uid=2947(jenkins-deploy) gid=500(wikidev) groups=50062(project-bastion),50120(project-deployment-prep),50174(project-integration),50876(project-wikidata-build),51904(project-puppet3-diffs),500(wikidev)
[21:56:52] <mutante>	 uid=4098(jenkins-slave) gid=500(wikidev) groups=50062(project-bastion),50174(project-integration),500(wikidev)
[22:00:11] <mutante>	 sudo -u jenkins-slave mkdir /mnt/jenkins-workspace
[22:00:11] <mutante>	 mkdir: cannot create directory `/mnt/jenkins-workspace': Permission denied
[22:06:15] <mutante>	 it's the other user, jenkins-deploy it seems
[22:08:49] <mutante>	 legoktm: i manually created that directory and let the user own it, just to see what it does
[22:09:00] <mutante>	 the error changed to
[22:09:03] <mutante>	 Ping response time is too long or timed out.
[22:10:24] <mutante>	 ah, i think i'm "launching a slave"
[22:10:38] <mutante>	 Slave successfully connected and online
[22:11:47] <mutante>	 yea,uhm, one step forward one step back
[22:11:54] <mutante>	 it still doesn't mean anything appears on http://puppet-compiler.wmflabs.org/
[22:19:03] <csteipp>	 ^demon|away / greg-g, fyi, I'm getting a few emails back disappointed that we have tests in the releases and patches. Is it worth having a -notests tarball next time?
[22:25:39] <wmf-insecte>	 Yippee, build fixed!
[22:25:40] <wmf-insecte>	 Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #549: FIXED in 37 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/549/
[22:56:11] <^demon|away>	 csteipp: Ugh.
[22:56:33] <^demon|away>	 "Why is my download a whole extra 350kb compressed?!?"
[22:56:49] <chrismcmahon>	 we went around on that already a long time ago
[22:57:01] <chrismcmahon>	 nmfp
[22:57:56] <csteipp>	 The biggest issue is the patches. Since people are applying them to previous downloads that didn't have tests/
[23:06:35] <greg-g>	 I guess it's not a horrible idea to have -notests
[23:18:41] <^demon|away>	 Perhaps. But we're only hitting this problem because of our failed idea to exclude tests to begin with :\
[23:29:46] <legoktm>	 https://integration.wikimedia.org/zuul/ stuck?
[23:30:26] <legoktm>	 it looks like anything running on trusty in the test queue
[23:32:23] <wmf-insecte>	 Yippee, build fixed!
[23:32:23] <wmf-insecte>	 Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-firefox-sauce build #6: FIXED in 1 min 20 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-firefox-sauce/6/
[23:32:53] <legoktm>	 and for some reason it's putting all the jobs on slave 1403...
[23:57:17] <wmf-insecte>	 Yippee, build fixed!
[23:57:17] <wmf-insecte>	 Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-chrome-sauce build #6: FIXED in 1 min 5 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-chrome-sauce/6/