[00:03:57] Project beta-code-update-eqiad build #26066: FAILURE in 56 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/26066/ [00:11:09] Project beta-scap-eqiad build #23453: FAILURE in 45 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/23453/ [00:14:01] Yippee, build fixed! [00:14:01] Project beta-code-update-eqiad build #26067: FIXED in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/26067/ [00:14:43] Yippee, build fixed! [00:14:44] Project beta-scap-eqiad build #23454: FIXED in 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/23454/ [00:34:35] !log Updated kibana to latest upstream head 8653aba [00:34:37] Logged the message, Master [03:04:25] Project beta-scap-eqiad build #23472: FAILURE in 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/23472/ [03:30:48] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #193: FAILURE in 2 min 28 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/193/ [03:31:52] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #38: FAILURE in 4 min 27 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/38/ [03:34:37] Yippee, build fixed! [03:34:37] Project beta-scap-eqiad build #23475: FIXED in 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/23475/ [05:11:33] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #194: FAILURE in 4 min 10 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/194/ [05:23:42] Yippee, build fixed! [05:23:42] Project browsertests-Echo-test2.wikipedia.org-linux-firefox-sauce build #73: FIXED in 12 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-firefox-sauce/73/ [05:27:46] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #195: FAILURE in 4 min 3 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/195/ [05:38:52] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #37: FAILURE in 4 min 34 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/37/ [06:12:54] Project beta-mediawiki-config-update-eqiad build #1130: FAILURE in 0.89 sec: https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/1130/ [06:13:53] Project beta-code-update-eqiad build #26103: FAILURE in 52 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/26103/ [06:33:53] Yippee, build fixed! [06:33:54] Project beta-code-update-eqiad build #26105: FIXED in 52 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/26105/ [07:19:02] 3Wikimedia Labs / 3deployment-prep (beta): deployment-rsync01 20GB hard drive is too small - 10https://bugzilla.wikimedia.org/71431#c1 (10Antoine "hashar" Musso) deployment-rsync01.eqiad.wmflabs ( https://wikitech.wikimedia.org/wiki/Nova_Resource:I-000002f4.eqiad.wmflabs ) is a m1 small with 20GB disk alloca... [07:24:17] 3Wikimedia / 3Continuous integration: Jenkins: Re-enable lint checks for Apache config in operations-puppet - 10https://bugzilla.wikimedia.org/70068 (10Antoine "hashar" Musso) a:3Antoine "hashar" Musso [08:44:09] (03PS1) 10Hashar: Reenable Apache lint check [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163812 (https://bugzilla.wikimedia.org/70068) [08:47:39] (03PS1) 10Hashar: Retrigger operations-apache-config-lint (non voting) [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/163813 (https://bugzilla.wikimedia.org/70068) [08:48:40] (03CR) 10Hashar: [C: 04-2] "Still being worked on. The mass amount of sed and configuration tweaks is not ideal." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163812 (https://bugzilla.wikimedia.org/70068) (owner: 10Hashar) [08:49:07] (03CR) 10Hashar: [C: 032] Retrigger operations-apache-config-lint (non voting) [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/163813 (https://bugzilla.wikimedia.org/70068) (owner: 10Hashar) [08:49:15] (03Merged) 10jenkins-bot: Retrigger operations-apache-config-lint (non voting) [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/163813 (https://bugzilla.wikimedia.org/70068) (owner: 10Hashar) [08:53:00] 3Wikimedia / 3Continuous integration: Jenkins: Re-enable lint checks for Apache config in operations-puppet - 10https://bugzilla.wikimedia.org/70068#c7 (10Antoine "hashar" Musso) I have adjusted the job to clone both operations/puppet.git and operations/mediawiki-config.git using Zuul cloner: https://gerrit.... [08:56:28] (03PS2) 10Hashar: Reenable Apache lint check [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163812 (https://bugzilla.wikimedia.org/70068) [08:56:45] (03CR) 10Hashar: "PS2 uses ln -f" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163812 (https://bugzilla.wikimedia.org/70068) (owner: 10Hashar) [09:00:33] 3Wikimedia / 3Continuous integration: Jenkins: Re-enable lint checks for Apache config in operations-puppet - 10https://bugzilla.wikimedia.org/70068#c8 (10Antoine "hashar" Musso) I have proposed two test changes to play with, both have a build passing when doing a noop change: operations/puppet: https://ger... [09:01:03] (03CR) 10Hashar: "From bug 70068:" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163812 (https://bugzilla.wikimedia.org/70068) (owner: 10Hashar) [09:02:03] (03Abandoned) 10Hashar: Example to have a manually defined axis [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163159 (owner: 10Hashar) [09:14:24] Project beta-scap-eqiad build #23507: FAILURE in 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/23507/ [09:35:13] (03CR) 10Zfilipin: [C: 031] Move wikidata-performance browsertests job to WMF Jenkins [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163129 (owner: 10Tobias Gritschacher) [09:37:55] (03PS1) 10Hashar: Remove legacy mediawiki core jobs [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/163817 [09:38:36] (03CR) 10Hashar: [C: 032] "The old jobs do not support mediawiki/vendor :-)" [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/163817 (owner: 10Hashar) [09:38:47] (03Merged) 10jenkins-bot: Remove legacy mediawiki core jobs [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/163817 (owner: 10Hashar) [09:42:06] (03PS1) 10Hashar: Remove legacy mediawiki core jobs [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163819 [09:42:45] (03Abandoned) 10Hashar: Remove legacy mediawiki core jobs [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163819 (owner: 10Hashar) [09:43:44] (03PS1) 10Hashar: Remove legacy mediawiki core jobs [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163820 [09:45:22] (03PS2) 10Hashar: Remove legacy mediawiki core jobs [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163820 [09:46:02] (03CR) 10Hashar: "JJB cleanup with same Change-Id: https://gerrit.wikimedia.org/r/#/c/163820/" [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/163817 (owner: 10Hashar) [09:46:33] (03PS3) 10Hashar: Remove legacy mediawiki core jobs [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163820 [09:47:15] (03CR) 10Hashar: [C: 04-2] "The jobs are no more being triggered since Zuul configuration change https://gerrit.wikimedia.org/r/#/c/163817/ ." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163820 (owner: 10Hashar) [09:47:58] zeljkof: hello :) [09:48:26] hashar: morning [09:48:31] zeljkof: Timo wrote some puppet patch to let us run Chromium tests on the Wikimedia slaves : https://gerrit.wikimedia.org/r/#/c/163791/ [09:48:54] I am not sure what he is willing to do, but he most probably should use SauceLabs instead [09:49:08] hashar: saw that [09:49:18] maybe he wants to run js tests on a local browser [09:49:22] or something [09:49:34] it is apparently for https://github.com/karma-runner/grunt-karma [09:49:37] he could do it on sauce, but running locally is fine too [09:49:41] which is a test runer [09:50:48] https://github.com/karma-runner/karma-sauce-launcher :D [09:54:43] Yippee, build fixed! [09:54:44] Project beta-scap-eqiad build #23511: FIXED in 45 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/23511/ [09:56:38] zeljkof: it might well be some bit of javascript to replace cucumber though which i am more or less worrying about [09:56:44] cause that would mean duplicate effort [09:56:59] anyway, I asked for the use case [10:03:07] hashar: it might be for unit tests [10:03:14] yeah maybe :] [10:03:16] and yes, let's wait for the reply [10:03:17] or to run the qunit jobs [10:03:21] timo is in SF, right? [10:18:46] zeljkof: he was at one point :] [10:18:51] zeljkof: lunch time for me [10:19:01] gotta migrate some more mediawiki core jobs this afternoon :] [11:15:03] re [11:19:16] (03PS4) 10Hashar: Switch extensions qunit jobs to Zuul cloner [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161459 [12:07:16] (03PS5) 10Hashar: Switch extensions qunit jobs to Zuul cloner [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161459 [12:12:23] (03PS1) 10Hashar: Abstract common qunit install steps [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163831 [12:15:40] (03PS2) 10Hashar: Abstract common qunit install steps [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163831 [12:19:28] (03PS6) 10Hashar: Switch extensions qunit jobs to Zuul cloner [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161459 [12:20:12] (03PS3) 10Hashar: Abstract common qunit install steps [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163831 [12:20:35] (03PS7) 10Hashar: Switch extensions qunit jobs to Zuul cloner [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161459 [12:21:04] (03CR) 10jenkins-bot: [V: 04-1] Abstract common qunit install steps [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163831 (owner: 10Hashar) [12:22:09] (03CR) 10jenkins-bot: [V: 04-1] Switch extensions qunit jobs to Zuul cloner [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161459 (owner: 10Hashar) [12:22:40] (03PS4) 10Hashar: Abstract common qunit install steps [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163831 [12:22:57] spam spamspam [12:22:59] damn refactoring [12:25:20] Yippee, build fixed! [12:25:21] Project browsertests-VisualEditor-language-screenshot-linux-firefox » ar,contintLabsSlave && UbuntuPrecise build #10: FIXED in 10 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=ar,label=contintLabsSlave%20&&%20UbuntuPrecise/10/ [12:25:37] (03PS2) 10Zfilipin: WIP: Add languages with translation over 90% [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163241 (owner: 10Amire80) [12:25:57] (03PS8) 10Hashar: Switch extensions qunit jobs to Zuul cloner [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161459 [12:26:39] (03CR) 10Hashar: [C: 032] "Change is a noop." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163831 (owner: 10Hashar) [12:27:50] (03CR) 10jenkins-bot: [V: 04-1] Switch extensions qunit jobs to Zuul cloner [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161459 (owner: 10Hashar) [12:28:10] (03PS1) 10Hashar: Update failure messages: FAILED -> FAILURE [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/163834 [12:28:27] (03PS2) 10Hashar: Update failure messages: FAILED -> FAILURE [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/163834 [12:29:03] (03CR) 10Hashar: [C: 032] Update failure messages: FAILED -> FAILURE [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/163834 (owner: 10Hashar) [12:29:12] (03Merged) 10jenkins-bot: Update failure messages: FAILED -> FAILURE [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/163834 (owner: 10Hashar) [12:30:17] (03PS3) 10Zfilipin: Add languages with translation over 90% to VisualEditor screenshot job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163241 (owner: 10Amire80) [12:30:22] (03CR) 10Zfilipin: [C: 032] Add languages with translation over 90% to VisualEditor screenshot job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163241 (owner: 10Amire80) [12:30:25] (03Merged) 10jenkins-bot: Abstract common qunit install steps [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163831 (owner: 10Hashar) [12:30:27] zeljkof: going to fail [12:30:33] zeljkof: cause I have just merged a change in :] [12:30:44] there is a race condition there :-/ [12:30:48] hashar: what is going to fail= [12:30:50] ? [12:30:55] the job you have just +2ed [12:31:07] I have +2ed a change which merged in just before you C+2ed [12:31:13] (03PS4) 10Zfilipin: Add languages with translation over 90% to VisualEditor screenshot job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163241 (owner: 10Amire80) [12:31:20] so you rchange is going to be rejected because the repository require fast forwards commits :-( [12:31:23] I noticed that yesterday [12:31:31] (03CR) 10Zfilipin: Add languages with translation over 90% to VisualEditor screenshot job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163241 (owner: 10Amire80) [12:31:32] Zuul needs to be smarter hehe [12:31:36] (03CR) 10Zfilipin: [C: 032] Add languages with translation over 90% to VisualEditor screenshot job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163241 (owner: 10Amire80) [12:32:15] hashar: thanks, rebased and +2d again [12:33:16] (03PS9) 10Hashar: Switch extensions qunit jobs to Zuul cloner [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161459 [12:34:03] Project beta-code-update-eqiad build #26140: FAILURE in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/26140/ [12:34:36] 00:01:01.460 error: cannot open .git/FETCH_HEAD: Permission denied [12:34:37] bah [12:35:13] (03Merged) 10jenkins-bot: Add languages with translation over 90% to VisualEditor screenshot job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163241 (owner: 10Amire80) [12:36:51] Yippee, build fixed! [12:36:52] Project beta-code-update-eqiad build #26141: FIXED in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/26141/ [12:37:30] !log Fixed some file permissions under deployment-bastion:/srv/mediawiki-staging/php-master/vendor/.git some files belonged to root instead of mwdeploy [12:37:32] Logged the message, Master [12:37:35] Project browsertests-VisualEditor-language-screenshot-linux-firefox » ko,contintLabsSlave && UbuntuPrecise build #11: SUCCESS in 12 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=ko,label=contintLabsSlave%20&&%20UbuntuPrecise/11/ [12:40:08] (03PS10) 10Hashar: Switch extensions qunit jobs to Zuul cloner [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161459 [12:46:50] (03PS1) 10Hashar: Dedicated VisualEditor qunit job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 [12:48:14] Yippee, build fixed! [12:48:14] Project browsertests-VisualEditor-language-screenshot-linux-firefox » fa,contintLabsSlave && UbuntuPrecise build #12: FIXED in 10 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=fa,label=contintLabsSlave%20&&%20UbuntuPrecise/12/ [12:48:25] (03CR) 10jenkins-bot: [V: 04-1] Dedicated VisualEditor qunit job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 (owner: 10Hashar) [12:49:38] (03PS2) 10Hashar: Dedicated VisualEditor qunit job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 [12:51:02] (03CR) 10jenkins-bot: [V: 04-1] Dedicated VisualEditor qunit job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 (owner: 10Hashar) [12:52:21] (03PS3) 10Hashar: Dedicated VisualEditor qunit job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 [12:53:50] (03CR) 10jenkins-bot: [V: 04-1] Dedicated VisualEditor qunit job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 (owner: 10Hashar) [12:55:55] (03PS4) 10Hashar: Dedicated VisualEditor qunit job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 [13:12:15] 3Wikimedia / 3Quality Assurance: Visual editor screenshot job fails - 10https://bugzilla.wikimedia.org/71298#c5 (10Željko Filipin) 5PATC>3RESO/FIX Looks like the problem is fixed by the above commit. Please reopen the bug if something is still failing. [13:13:15] (03PS5) 10Hashar: Dedicated VisualEditor qunit job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 [13:19:09] (03PS1) 10Hashar: Experimental VisualEditor-qunit job [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/163840 [13:19:17] (03CR) 10jenkins-bot: [V: 04-1] Experimental VisualEditor-qunit job [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/163840 (owner: 10Hashar) [13:27:55] (03CR) 10Hashar: "recheck" [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/163840 (owner: 10Hashar) [13:31:46] (03CR) 10Hashar: [C: 032] Experimental VisualEditor-qunit job [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/163840 (owner: 10Hashar) [13:31:55] (03Merged) 10jenkins-bot: Experimental VisualEditor-qunit job [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/163840 (owner: 10Hashar) [13:34:15] (03PS6) 10Hashar: Dedicated VisualEditor qunit job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 [14:23:23] zeljkof: the browsertests builders in JJB are growing out of control :-] [14:23:35] hashar: what do you mean? [14:23:40] https://gerrit.wikimedia.org/r/#/c/163129/3/browsertests.yaml [14:23:43] refactoring needed asap? [14:23:47] Tobias proposed yet another copy paste :] [14:23:54] either refactoring [14:23:59] or a shell script [14:24:02] hashar: yes, we wanted to move the last of wikidata jobs [14:24:14] and serious refactoring is needed [14:24:15] yeah that is a fair point :] [14:24:27] lets pretend Tobias change is the last to copy paste code ? :] [14:24:29] when would you have time to do some serious refactoring? [14:24:36] hashar: no, it is [14:24:38] period [14:24:38] given my schedule, in 2016 [14:24:42] no more copy-paste [14:24:49] ok, then it is up to me :) [14:25:01] I will add you to cc, you can comment on my ideas [14:25:14] AH cucumber --tags accepts comma separated tags [14:25:15] great [14:26:09] hashar: cucumber tags have different meaning when comma separated [14:26:23] https://github.com/cucumber/cucumber/wiki/Tags [14:26:23] I am looking at both Wikidata jobs [14:26:33] they both have --tags ~@skip [14:26:37] https://github.com/cucumber/cucumber/wiki/Tags#logically-anding-and-oring-tags [14:26:46] and the other tag is either @wikidata.beta.wmflabs.org or @performance_testing [14:26:54] ah or vs and grrr [14:32:17] (03CR) 10Hashar: [C: 04-1] "I am quite happy to know this is the last job that needed to be migrated. Lets say this is the LAST time we copy paste that huge builder ;" (032 comments) [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163129 (owner: 10Tobias Gritschacher) [14:32:49] Tobi_WMDE_SWE: the Wikidata performance browser test job might be missing a --tags @wikidata.beta.wmflabs.org [14:32:57] Tobi_WMDE_SWE: https://gerrit.wikimedia.org/r/#/c/163129/3/browsertests.yaml [14:33:08] zeljkof: should we use the performance publisher on all browsertests jobs ? [14:33:31] hashar: probably not [14:34:12] zeljkof: any reason ? :] [14:34:22] hashar: not sure, what does it do? [14:34:35] it parses the junit XML file [14:34:38] does it hurt to use it everywhere? [14:34:41] and craft trend graphs. ex: https://wiki.jenkins-ci.org/display/JENKINS/Performance+Plugin [14:34:45] if not, go ahead [14:34:57] I would say, lets use it everywhere and see what happens :] [14:35:15] hashar: go ahead [14:35:19] :-] [14:38:52] zeljkof: thanks for the code review about removing the sleep from the Flow tests. I didn't get to review the original code, but it will be nice to clean up that repo some. [14:39:20] that was about 3 minutes per build of just wait time [14:39:28] (03PS1) 10Hashar: browsertests: enable Performance publisher [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163857 [14:40:00] chrismcmahon: in a meeting with manybubbles [14:40:02] (03CR) 10Hashar: Move wikidata-performance browsertests job to WMF Jenkins (031 comment) [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163129 (owner: 10Tobias Gritschacher) [14:40:37] (03CR) 10Hashar: "That might be useful. I don't think it is going to cause too much stress on Jenkins." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163857 (owner: 10Hashar) [14:40:44] zeljkof: done :] [14:40:59] chrismcmahon: sleep() should be banned entirely :D [14:41:37] hashar: I actually needed a sleep the other day because of an obscure race condition in ChromeDriver itself. sometimes it's the tool of last resort. [14:41:50] yeah last resort [14:41:54] that is fine to me :] [14:42:05] but I often see stuff like: while { sleep 5 } [14:42:07] which is cumbersome [14:42:31] anyway, we were talking about enabling Jenkins Performance plugin on all browser tests jobs. That is being used by Wikidata team already [14:42:32] and prone to failure [14:42:37] Link with screenshots: https://wiki.jenkins-ci.org/display/JENKINS/Performance+Plugin [14:42:42] hashar: I saw that, I like it [14:43:55] chrismcmahon: do you know any quick job that I can use to test it out ? [14:43:58] now if ori would just stop breaking beta labs [14:44:35] hashar: this is a fast build: https://integration.wikimedia.org/ci/view/BrowserTests/view/-All/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/ [14:45:06] refreshing it to enable the performance plugin [14:45:09] * hashar crosses fingers [14:51:55] (03CR) 10Hashar: [V: 031] "I have refreshed the configuration of a single job and gave it a try" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163857 (owner: 10Hashar) [14:56:40] Reedy: something bad happened to logstash in beta. No new events logged since 2014-09-30T00:24:30.000Z. [14:57:29] bd808: could that be related to what ori did with redis? we're still unable to save an edit or click Preferences after that change [14:58:15] chrismcmahon: Nah. Reedy upgraded the logstash engine and apparently we are using a filter that has been removed. Crashes on start up. [14:58:24] ha [14:58:35] Also if that hasn't been fixed, why not revert the config change? [14:58:47] You can do that you know :) [14:59:29] bd808: mostly because I was working on test2 yesterday and there is some sort of partial fix in place. [14:59:41] !log Logstash rules need to be adjusted for latest upstream version: "Couldn't find any filter plugin named 'prune'" [14:59:43] Logged the message, Master [15:00:13] and because I lack context :-) [15:01:00] Me too. I saw the crash on Special:Verison once and the it worked for me on the next reload. [15:01:26] And it worked for me right now [15:02:40] !log Logstash doesn't bundle the prune filter by default any more -- http://logstash.net/docs/1.4.2/filters/prune [15:02:42] Logged the message, Master [15:04:19] manybubbles: lost your audio [15:04:26] you froze [15:04:32] manybubbles: no, you froze :) [15:06:16] Yippee, build fixed! [15:06:16] Project beta-mediawiki-config-update-eqiad build #1131: FIXED in 2.5 sec: https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/1131/ [15:13:49] !log Fixed logstash by installing http://packages.elasticsearch.org/logstash/1.4/debian/pool/main/l/logstash-contrib/logstash-contrib_1.4.2-1-efd53ef_all.deb [15:13:50] Logged the message, Master [15:21:09] !sal [15:21:09] https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:23:32] bd808: ahh [15:23:46] I saw the contrib package but we didn't have it installed already [15:23:56] Yeah. It's new I guess. [15:24:04] beta is all better after installing [15:24:10] sweet :) [15:24:12] thanks [15:24:21] documented on the bug I just assigned to you for the prod upgrade. :) [15:24:41] I guess it's optional stuff, hence dpkg not barfing [15:24:51] do we have that .deb in apt.wikimedia.org ? [15:24:59] hashar: Not yet [15:25:00] for Jenkins we reuse an upstream .deb package as well [15:25:12] hashar: We have an older deb [15:25:18] Yeah, I just dpkg -i 'd it for now [15:25:19] 3Wikimedia Labs / 3deployment-prep (beta): deployment-rsync01 20GB hard drive is too small - 10https://bugzilla.wikimedia.org/71431#c2 (10Greg Grossmeier) p:5Unprio>3Normal Let's not make the Jenkins beta-scap-eqiad job very divergent from prod (at all). Let's make the Beta Cluster like prod... [15:25:23] since we're still testing and such [15:25:25] and Faidon crafted some magic reprepro configuration to easily update them from upstream :] [15:25:40] works like a charm, but needs root access :/ [15:26:50] I'm going to maybe break beta by turning puppet back on on the mediawiki0[12] servers. [15:27:05] If we don't then they will break from ldap changes anyway [15:27:13] wee [15:27:49] !log enabling puppet and forcing run on deployment-mediawiki01 [15:27:52] Logged the message, Master [15:28:48] I really wish this SAL and the deployment-prep SAL were the same SAL :) [15:28:53] errr :( [15:29:39] !log puppet showed no changes on mediawiki01‽ [15:29:41] Logged the message, Master [15:30:35] bd808: we can make yet another suite of channels: #wikimedia-beta and #wikimedia-ci to make it clear :/ [15:30:58] Noooooooooo [15:31:04] exactly [15:34:19] !log enabling puppet and forcing run on deployment-mediawiki02 [15:34:21] Logged the message, Master [15:34:26] Project beta-scap-eqiad build #23547: FAILURE in 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/23547/ [15:36:52] !log enabling puppet and forcing run on deployment-mediawiki03 [15:36:54] Logged the message, Master [15:38:57] greg-g: I'm probably not going to be able to attend the rel eng meeting today. Was at parents trying to get decent internet installed. It failed due to lack of strong enough signal for reliable connection [15:39:06] So using a PAYG mifi [15:39:17] yuck [15:39:18] I suspect hangouts will kill the data allowance :) [15:39:29] Part of me does want to try it and see [15:39:53] !log hhvm not starting after puppet run on deployment-mediawiki03. Investigating. [15:39:54] anything to update? just update the etherpad [15:39:55] Logged the message, Master [15:43:09] omg every package on mediawiki03 was out of date :( [15:43:23] s/every/lots and lots and lots/ [15:43:35] (03CR) 10Zfilipin: [C: 031] "Looks good to me! Chris, Tobi, do you have any objections to getting this merged?" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163857 (owner: 10Hashar) [15:44:56] !log scap failing in beta due to "Permission denied (publickey)" talking to deployment-rsync01.eqiad.wmflabs [15:44:58] Logged the message, Master [15:45:03] ugh [15:45:20] !log Updating our Jenkins job builder fork 686265a..ee80dbc (no job changed) [15:45:21] Logged the message, Master [15:45:22] dunno what's up with that all of a sudden [15:45:27] Reedy: can you help bd808 with the above issues? ^ [15:45:56] bd808: I upgraded packages on a handful of beta machines last night [15:45:59] Maybe puppet took the role off that grants access to the shared key? [15:46:55] It worked at 3:29Z and failed at 3:33Z [15:47:04] with puppet run inbetween? [15:47:07] So somethign changed ~15 mintues ago [15:47:14] 3Wikimedia Labs / 3deployment-prep (beta): deployment-rsync01 20GB hard drive is too small - 10https://bugzilla.wikimedia.org/71431#c3 (10Sam Reed (reedy)) How long did it take to break? I deleted a weird tmp dir, killed the whole cache dir, and re-ran sync-common. Which gave ~2G free space. I'm wondering... [15:47:26] Not sure. I'm up to my elbows in mediawiki03 right now [15:47:41] RECOVERY - BetaLabs: Puppet freshness check on labmon1001 is OK: OK: All targets OK [15:48:19] W00t. puppet is happy [15:49:11] !log apt-get dist-upgrade fixed hhvm on deployment-mediawiki03 [15:49:13] Logged the message, Master [15:51:58] So useless that puppet.log has no timestamps [15:52:07] :( [15:52:16] seriously? wow [15:52:27] also, why'd we need the dist-upgrade versus an upgrade? [15:52:27] Notice: /Stage[main]/Beta::Scap::Target/File[/etc/ssh/userkeys/mwdeploy/.ssh/authorized_keys]/group: group changed 'mwdeploy' to 'mwdeploy' [15:52:41] big change, that [15:52:50] greg-g: I type dist-upgrade by default. Upgrade all the things! [15:53:24] Looks like we may have gotten a local group or something on rsync01. Looking deeper [15:53:33] if there's kernel upgrades, probably a good ida to install them :) [15:54:41] !log Local mwdeploy (gid=996) shadowing ldap group gid=603(mwdeploy) on deployment-rsync01 [15:54:42] Logged the message, Master [15:56:07] !log removed local group/user mwdeploy on deployment-rsync01 [15:56:09] Logged the message, Master [15:57:33] Yippee, build fixed! [15:57:34] Project beta-scap-eqiad build #23551: FIXED in 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/23551/ [15:58:10] !log scap happy again after fixing rogue group/user on rsync01 \o/ Not sure why they were created but likely an ldap hiccup during a puppet run [15:58:11] Logged the message, Master [15:59:28] Reedy: are you the pink-ish color in the etherpad? [16:02:30] twentyafterfour: ping! :) [16:04:32] "Memcached error: Error connecting to 127.0.0.1:11211: Connection refused" Why is that using the old non-nutcracker port? [16:08:49] !log Occasional memecached-serious errors in beta for something trying to connect to the default memcached port (11211) rather than the nutcracker port (11212). [16:08:51] Logged the message, Master [16:09:49] Error messages like that without stack traces are close to useless :( [16:12:37] $wgDonationInterfaceMemcachePort = '11211'; (wrong) [16:13:27] extensions/DonationInterface: $wgDonationInterfaceMemcachePort = '11211'; (wrong) [16:13:50] extensions/PHPExcel: 'memcachePort' => 11211, (wrong) [16:20:46] bd808: that port difference keeps breaking :-( [16:21:10] I think mediawiki defaults to 11211 while nutcracker defaults to 11212 [16:21:14] yeah. It was probably not the greatest idea to use a non-standard port [16:21:29] one time, I fixed the errors by regenerating the configuration cache (iirc) [16:21:35] I thought it was a temporary thing as they rolled it out :( [16:21:38] might have filled a bug about it [16:21:47] 3Wikimedia Labs / 3deployment-prep (beta): Unable to connect to redis server - 10https://bugzilla.wikimedia.org/71415#c7 (10Greg Grossmeier) p:5Highes>3Immedi s:5major>3blocke a:3Ori Livneh Ori: can you please take a look at this ASAP? Redis dependency is breaking Beta/it's unasable for testing now... [16:24:20] bd808: ah I remember now. Yesterday I have abandoned two puppet patches which were related to nutcracker port config [16:24:31] 3Wikimedia Labs / 3deployment-prep (beta): Unable to connect to redis server - 10https://bugzilla.wikimedia.org/71415#c8 (10Greg Grossmeier) (Actually, I might just ask out on [Ops] for some (SWAT) deployer to help out.) [16:25:10] bd808: https://gerrit.wikimedia.org/r/#/c/148041/ [16:25:42] bd808: which was to be able to install nutcracker on bastion https://gerrit.wikimedia.org/r/#/c/148042/ . But anyway ori fixed it differently apparently [16:25:56] (sorry that is not very clear) [16:54:40] Project beta-scap-eqiad build #23558: FAILURE in 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/23558/ [16:56:28] Yippee, build fixed! [16:56:29] Project beta-scap-eqiad build #23559: FIXED in 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/23559/ [16:58:51] greg-g: :( [16:58:54] sorry [16:59:23] twentyafterfour: anything to add to the etherpad? https://etherpad.wikimedia.org/p/RelEngWeekly [16:59:42] make the archives look like you were there ;) [16:59:55] hah ok [17:03:39] twentyafterfour: ping me when you're done so I can archive the page [17:04:24] greg-g: just catching up with Chase so I'll have more to add.. [17:05:55] * greg-g nods [17:06:01] 3Wikimedia Labs / 3deployment-prep (beta): Unable to connect to redis server - 10https://bugzilla.wikimedia.org/71415#c9 (10Greg Grossmeier) 5REOP>3RESO/FIX greg-g: are you a wizard? https://www.google.com/search?hl=en&site=imghp&tbm=isch&source=hp&biw=1330&bih=705&q=muley+point&oq=muley+point&gs_l=img.12..0l5j0i24l5.974.2698.0.4128.11.11.0.0.0.0.227.1202.1j7j1.9.0....0...1ac.1.54.img..2.9.1202.ZDN2nQeNmQQ [17:07:31] greg-g: wrong link https://bugzilla.wikimedia.org/show_bug.cgi?id=71415#c9 [17:07:42] sometimes [17:07:50] :) [17:08:56] greg-g: how'd you do that anyway? [17:09:44] chrismcmahon: ori did it [17:09:55] I think just this: https://gerrit.wikimedia.org/r/163874 [17:10:07] greg-g: he was pretty quiet about it [17:10:19] yeah [17:14:39] Project beta-scap-eqiad build #23561: FAILURE in 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/23561/ [17:18:49] Yippee, build fixed! [17:18:49] Project beta-scap-eqiad build #23562: FIXED in 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/23562/ [17:19:15] greg-g: Etherpad updated [17:19:52] ty sir [17:21:35] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #231: ABORTED in 2 min 59 sec: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/231/ [17:25:07] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #258: ABORTED in 1 min 29 sec: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/258/ [17:30:50] (03CR) 10Krinkle: [C: 04-1] "These repositories are not required to be compatible at every commit, and unless I misunderstand this can never be voting (ve/ve is upstre" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 (owner: 10Hashar) [17:34:48] (03CR) 10Krinkle: Dedicated VisualEditor qunit job (031 comment) [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 (owner: 10Hashar) [17:35:25] (03PS3) 10Krinkle: qunit-cleanup: rm -f when deleting file [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163848 (owner: 10Hashar) [17:38:45] 3Wikimedia / 3Continuous integration: wikidata-jenkins* instances needs puppet update - 10https://bugzilla.wikimedia.org/71411#c5 (10Jan Zerebecki) 5NEW>3RESO/FIX All updated now. [17:44:18] !log Updated scap to 064425b (Remove restart-nutcracker and restart-twemproxy scripts) [17:44:22] Logged the message, Master [18:30:06] .deployment-mediawiki04.puppetagent.failed [18:30:16] integration.integration-dev-trusty.puppetagent.failed [18:33:53] ACKNOWLEDGEMENT - BetaLabs: Puppet failure events on labmon1001 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki04.puppetagent.failed_events.value (100.00%) daniel_zahn reported to #wikimedia-qa [18:33:54] ACKNOWLEDGEMENT - CI: Puppet failure events on labmon1001 is CRITICAL: CRITICAL: integration.integration-dev-trusty.puppetagent.failed_events.value (100.00%) daniel_zahn reported to #wikimedia-qa [18:37:22] Project beta-mediawiki-config-update-eqiad build #1139: FAILURE in 0.27 sec: https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/1139/ [18:43:53] Project beta-code-update-eqiad build #26178: FAILURE in 52 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/26178/ [18:44:57] :( [18:48:38] Yippee, build fixed! [18:48:38] Project beta-mediawiki-config-update-eqiad build #1145: FIXED in 1.5 sec: https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/1145/ [18:48:43] Yippee, build fixed! [18:48:43] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #202: FIXED in 11 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/202/ [18:49:08] Project beta-scap-eqiad build #23576: FAILURE in 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/23576/ [18:55:12] chrismcmahonbrb: now that you just left, fyi, I'm going to be afk for about 2 hours starting at 12:15, in case anyone is looking for me :) [19:01:21] (03PS1) 10EBernhardson: The coding conventions for php state: [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/163898 [19:01:47] greg-g: gotcha [19:04:19] Yippee, build fixed! [19:04:20] Project beta-code-update-eqiad build #26180: FIXED in 1 min 18 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/26180/ [19:05:13] (03PS2) 10EBernhardson: The coding conventions for php state: [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/163898 [19:05:45] (03PS3) 10EBernhardson: Introduce sniff to assert whitespace after control structure [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/163898 [19:12:12] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #260: STILL FAILING in 40 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/260/ [19:36:53] whoops, my bad. fixing... [19:59:52] !log /srv partition on deployment-rsync01 full again. We need a new rsync server with more space [19:59:54] Logged the message, Master [20:02:49] !log Started building deployment-rsync02 to replace deployment-rsync01 [20:02:50] Logged the message, Master [20:05:45] chrismcmahon: Beta updates are stopped again due to rsync01's disk issue :( Working to fix now. [20:07:11] thanks bd808 [20:13:24] bd808: can you take some traces ? [20:13:31] i.e. what is actually consuming the disk space [20:13:51] might end up hitting prod as well one day :/ [20:16:29] hashar: Yeah I'll leave it alive for debugging. My guess is that some extensions are being updated with largish change sets and we don't have enough extra disk to handle the new copy that rsync makes while it is pulling the change set over. [20:18:20] The disk is 8.5G. /srv/mediawiki is 4.9G :( [20:19:00] Not sure how big /srv/common-local is because I deleted it in an attempt to get things to limp along again [20:24:15] Yippee, build fixed! [20:24:16] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #233: FIXED in 36 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/233/ [20:24:38] It may even just be l10n updates that are killing it at the current level of disk utilization [20:26:53] bd808: yeah that is what Reedy has seen (l10n filling everything) [20:27:09] I guess a bit more that 8G will help so it is worth creating a new instance [20:27:25] unless ops are able to resize the instance extended disk space, but that is unlikely [20:34:06] Yeah. I think a bigger instance is the easiest. We have used all of the default disk allocation for the node already. Moving from a small instance to a medium instance will basically double our available disk. [20:35:28] !log Initial puppet run with role::beta::rsync_slave applied on rsync02 failed spectacularly in /Stage[main]/Mediawiki::Scap/Exec[fetch_mediawiki] stage [20:35:30] Logged the message, Master [20:36:23] !log lots and lots of "file has vanished" errors from rsync. Not sure why [20:36:26] Logged the message, Master [20:39:06] Project beta-code-update-eqiad build #26183: FAILURE in 1 min 42 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/26183/ [20:40:15] chrismcmahon or anyone, I'm getting "superclass mismatch for class FlowPage (TypeError)" when I run browser tests. I guess I messed up my bundle somehow? [20:40:48] spagewmf: in a Vagrant env I am guessing? [20:42:16] spagewmf: seems that somehow you are trying to define the FlowPage class twice [20:42:48] chrismcmahon: a local mediawiki install. I figured it out, it's leftover cmcmahon_bug.rb where I was trying out your textarea bug :) [20:43:19] awesome, people name bug files after me :-) [20:51:15] 3Wikimedia / 3Quality Assurance: browser tests should assert there are no pink error boxes on the page - 10https://bugzilla.wikimedia.org/61304#c2 (10spage) (In reply to Željko Filipin from comment #1) > S, do you need help implementing this? Always :) A simple test case for the utility of this is any brow... [20:54:02] Yippee, build fixed! [20:54:02] Project beta-code-update-eqiad build #26185: FIXED in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/26185/ [20:54:24] Yippee, build fixed! [20:54:24] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #237: FIXED in 47 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/237/ [20:54:44] bd808: since today I noticed on beta a bunch of "error: cannot open .git/FETCH_HEAD: Permission denied" [20:54:48] example https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/1143/console [20:55:07] something is going wild in /srv/mediawiki-staging :-D [20:55:13] or maybe its LDAP related [20:56:57] anyway sleep time [20:57:13] hashar: ldap-related? [20:57:42] andrewbogott: I have no clue how unix figure whether a user can read/write a file [20:57:52] would it end up looking up in LDAP from time to time? [20:58:09] Not on a production box [20:58:17] As far as I know [20:59:12] hashar: Hmmm.. I wonder if that's related to the local mwdeploy user I found on the rsync01 server earlier today [21:00:11] that one was on deployment-bastion though [21:00:15] I couldn't find evidence in the puppet logs but my guess was that there was a ldap blip during a puppet run and puppet decided to make a local user [21:00:25] ahhh [21:00:47] * hashar blames puppet [21:00:48] I can look on deployment-bastion and see if something happened there too [21:01:21] eployment-bastion:~$ grep mwdeploy /etc/passwd [21:01:21] mwdeploy:x:994:994::/var/lib/mwdeploy:/bin/bash [21:01:32] \O/ [21:01:50] if only we could define something like User { 'mwdeploy': provider => LDAP } [21:01:53] frack. Yeah that's what I saw on rsync01 as well [21:01:57] and have puppet skip the local creation [21:02:21] if the ldap lookup works it won't create a local but if it fails then boom! [21:02:37] Then when ldap comes back that user ghosts the local [21:02:57] That may actually be what's behind rsync01 filling up too [21:03:23] if the ownership was flipping back and forth rsync might go nuts [21:03:48] maybe we can switch production to use LDAP as well? [21:04:51] Looks like deployment-bastion is the only other host with a local right now based on a salt search [21:05:40] I can fix it or you can. delete the local user and group and things should be fine [21:06:06] please do its 11pm and I am busy on some online game :D [21:06:12] heh [21:06:17] on it [21:06:34] !log Local mwdeploy user on deployment-bastion making things sad [21:06:36] Logged the message, Master [21:07:21] ah [21:07:23] puppet is nice [21:07:33] the User type has a forcelocal => [21:07:45] setting it true would cause puppet to create a local user [21:07:55] Yeah, but in prod we want local and in beta we don't :( [21:08:03] and there is also a provider => ldap [21:08:18] puppet "discovers" the appropriate method [21:08:21] We might be able to use hiera to fix this [21:08:23] and probably fallback to add_user [21:08:29] but it would take some refactoring [21:08:45] hiera('user_provider') ? [21:08:54] and update all freaking User uses :-/ [21:09:34] If we had a parameterized class then hiera could be used to set provider => ldap in beta and provider => whatever in prod [21:10:03] I do stuff like that in mw-vagrant/labs-vagrant [21:10:46] * bd808 looks suspiciously at "Notice: /Stage[main]/Mediawiki::Users/File[/home/l10nupdate/.ssh]/owner: owner changed 'l10nupdate' to 'l10nupdate'" [21:11:06] Do we have an l10nupdate user in ldap too? [21:13:23] We do. Damn. Another local ghost [21:14:33] filled a bug https://bugzilla.wikimedia.org/show_bug.cgi?id=71480 [21:15:00] we might need a Shinken monitor now to ensure none of the instances have mwdeploy / l10nupdate as local users [21:15:02] grr [21:15:22] !log local l10nupdate users on bastion, mediawiki01 and rsync01 [21:15:24] Logged the message, Master [21:15:25] and I haven't kept track of the user I have asked to be added in LDAP [21:15:34] but I did fill bug for each of them :] [21:30:22] greg-g: [21:31:01] greg-g: Bryan probably found the cause of the rsync errors. At least we had two mwdeploy user with different UID. One defined locally and one in LDAP :D [21:33:30] the puppet ldap provider can even create users in ldap db for you l\O/| [21:37:58] !log I figured out the disk space problem on rsync01 (just as I was ready to replace it with rsync02). The old /src/common-local directory was still there which doubled the disk utilization. /src/mediawiki is the correct sync dir now following prod changes. [21:38:00] Logged the message, Master [21:38:52] !log /srv on rsync01 now has 3.2G of free space and should be fine fro quite a while again. [21:38:53] Logged the message, Master [21:43:03] kudos on fixing https://bugzilla.wikimedia.org/show_bug.cgi?id=71431 \O/ [21:43:39] * hashar sleeps [21:45:27] !log jobrunner not running. ebernhardson is debugging. [21:45:29] Logged the message, Master [21:55:48] Yippee, build fixed! [21:55:48] Project beta-scap-eqiad build #23584: FIXED in 1 min 45 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/23584/ [22:19:17] (03PS4) 10Krinkle: Introduce sniff to assert whitespace after control structure [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/163898 (owner: 10EBernhardson) [23:35:44] (03PS5) 10EBernhardson: Introduce sniff to assert whitespace after control structure [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/163898 [23:47:17] !log jobrunner using outdated ip address for redis01. Testing patch to use hostname rather than hardcoded ip [23:47:22] Logged the message, Master [23:59:41] Yippee, build fixed! [23:59:41] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #192: FIXED in 20 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/192/