[00:46:28] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #696: SUCCESS in 26 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/696/ [00:53:32] Project UploadWizard-api-commons.wikimedia.org build #557: SUCCESS in 31 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.org/557/ [02:08:13] Project beta-scap-eqiad build #21586: FAILURE in 4 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/21586/ [02:18:26] Yippee, build fixed! [02:18:27] Project beta-scap-eqiad build #21587: FIXED in 4 min 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/21587/ [02:48:45] 3Wikimedia / 3Quality Assurance: Create browser test builds for MMV in IE back to IE8 - 10https://bugzilla.wikimedia.org/70744#c2 (10Gilles Dubuc) Are you just running the existing test for other browsers on IE8 and checking that it fails continuously? I think that wouldn't catch the issue we had. Because w... [03:05:18] Project browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs build #143: SUCCESS in 17 sec: https://integration.wikimedia.org/ci/job/browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs/143/ [03:14:55] Yippee, build fixed! [03:14:55] Project browsertests-TwnMainPage-sandbox.translatewiki.net-linux-firefox-sauce build #127: FIXED in 10 min: https://integration.wikimedia.org/ci/job/browsertests-TwnMainPage-sandbox.translatewiki.net-linux-firefox-sauce/127/ [03:15:38] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #107: SUCCESS in 43 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/107/ [03:17:07] Project browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce build #44: STILL FAILING in 11 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce/44/ [03:20:31] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #163: SUCCESS in 3 min 22 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/163/ [03:22:37] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #8: SUCCESS in 6 min 58 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/8/ [03:24:18] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #8: STILL FAILING in 1 min 40 sec: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/8/ [03:25:44] Project browsertests-UniversalLanguageSelector-language-browsertests.wmflabs.org-linux-firefox-sauce build #149: STILL FAILING in 1 min 25 sec: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-language-browsertests.wmflabs.org-linux-firefox-sauce/149/ [03:36:52] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #169: SUCCESS in 11 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/169/ [03:42:58] Project browsertests-Flow-test2.wikipedia.org-windows_8-internet_explorer-sauce build #149: STILL FAILING in 41 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-windows_8-internet_explorer-sauce/149/ [03:43:19] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce build #174: SUCCESS in 6 min 27 sec: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce/174/ [03:44:00] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #140: SUCCESS in 1 min 1 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/140/ [03:45:26] Project browsertests-UniversalLanguageSelector-sandbox.translatewiki.net-linux-firefox-sauce build #138: SUCCESS in 1 min 25 sec: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-sandbox.translatewiki.net-linux-firefox-sauce/138/ [03:46:41] Project browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce build #80: SUCCESS in 40 sec: https://integration.wikimedia.org/ci/job/browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce/80/ [03:59:54] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #221: STILL FAILING in 39 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/221/ [04:07:34] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #60: STILL FAILING in 7 min 38 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/60/ [04:25:23] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #6: SUCCESS in 39 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/6/ [04:41:00] Project browsertests-VisualEditor-test2.wikipedia.org-linux-chrome-sauce build #186: SUCCESS in 57 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-chrome-sauce/186/ [04:41:58] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #96: SUCCESS in 57 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/96/ [04:43:27] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #199: SUCCESS in 35 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/199/ [04:49:00] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #182: SUCCESS in 7 min 2 sec: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/182/ [04:51:30] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #140: SUCCESS in 2 min 29 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/140/ [04:51:32] Project browsertests-ContentTranslation-language-stage.wmflabs.org-linux-firefox-sauce build #110: STILL FAILING in 1.9 sec: https://integration.wikimedia.org/ci/job/browsertests-ContentTranslation-language-stage.wmflabs.org-linux-firefox-sauce/110/ [05:00:10] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #47: STILL FAILING in 8 min 36 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/47/ [05:01:01] Project browsertests-Translate-meta.wikimedia.org-linux-firefox-sauce build #165: SUCCESS in 49 sec: https://integration.wikimedia.org/ci/job/browsertests-Translate-meta.wikimedia.org-linux-firefox-sauce/165/ [05:02:19] Project browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce build #139: SUCCESS in 1 min 16 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce/139/ [05:03:04] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #107: SUCCESS in 45 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/107/ [05:10:58] Project beta-scap-eqiad build #21605: FAILURE in 7 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/21605/ [05:16:58] Yippee, build fixed! [05:16:59] Project beta-scap-eqiad build #21606: FIXED in 2 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/21606/ [05:20:40] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #274: STILL FAILING in 55 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/274/ [05:27:24] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #204: SUCCESS in 43 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/204/ [05:45:25] (03PS1) 10Yurik: Added graph extension [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160396 [05:45:44] (03CR) 10jenkins-bot: [V: 04-1] Added graph extension [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160396 (owner: 10Yurik) [05:47:42] Project beta-scap-eqiad build #21609: FAILURE in 3 min 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/21609/ [05:51:01] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #241: STILL FAILING in 47 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/241/ [05:56:45] Yippee, build fixed! [05:56:45] Project beta-scap-eqiad build #21610: FIXED in 2 min 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/21610/ [06:02:08] (03PS1) 10Yurik: Removed ZRMA [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160400 [06:03:31] Project browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce build #169: SUCCESS in 42 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce/169/ [06:20:04] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #210: STILL FAILING in 52 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/210/ [06:26:25] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #164: SUCCESS in 6 min 19 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/164/ [06:28:36] (03CR) 10Yurik: "recheck" [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160396 (owner: 10Yurik) [06:30:54] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #154: SUCCESS in 39 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/154/ [06:31:38] Project browsertests-Flow-test2.wikipedia.org-linux-firefox-sauce build #151: STILL FAILING in 28 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-linux-firefox-sauce/151/ [06:36:55] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #165: SUCCESS in 6 min 0 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/165/ [06:37:55] Project browsertests-Echo-test2.wikipedia.org-linux-firefox-sauce build #43: STILL FAILING in 11 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-firefox-sauce/43/ [06:43:49] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #7: SUCCESS in 6 min 54 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/7/ [06:44:34] Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #39: STILL FAILING in 44 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/39/ [06:51:13] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-8-sauce build #7: STILL FAILING in 6 min 37 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-8-sauce/7/ [06:51:14] Project beta-scap-eqiad build #21615: FAILURE in 7 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/21615/ [06:54:18] Project browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce build #166: SUCCESS in 3 min 4 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce/166/ [06:57:26] Yippee, build fixed! [06:57:27] Project beta-scap-eqiad build #21616: FIXED in 3 min 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/21616/ [07:08:15] Project beta-scap-eqiad build #21617: FAILURE in 4 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/21617/ [07:10:39] Project browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce build #152: STILL FAILING in 38 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce/152/ [07:13:13] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #165: SUCCESS in 2 min 33 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/165/ [07:17:57] Yippee, build fixed! [07:17:57] Project beta-scap-eqiad build #21618: FIXED in 3 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/21618/ [07:20:00] PROBLEM - BetaLabs: Puppet failure events on labmon1001 is CRITICAL: CRITICAL: deployment-prep.deployment-videoscaler01.puppetagent.failed_events.value (30.00%) [07:38:39] Project browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce build #183: SUCCESS in 1 hr 0 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce/183/ [07:40:37] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #18: STILL FAILING in 46 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/18/ [07:41:27] Project browsertests-Translate-sandbox.translatewiki.net-linux-firefox-sauce build #120: STILL FAILING in 28 min: https://integration.wikimedia.org/ci/job/browsertests-Translate-sandbox.translatewiki.net-linux-firefox-sauce/120/ [07:41:43] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #108: SUCCESS in 1 min 4 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/108/ [07:41:53] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #697: SUCCESS in 25 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/697/ [07:42:13] Project UploadWizard-api-commons.wikimedia.org build #558: SUCCESS in 30 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.org/558/ [07:57:39] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #161: SUCCESS in 18 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/161/ [08:13:26] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #698: SUCCESS in 24 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/698/ [08:21:26] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #699: SUCCESS in 24 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/699/ [08:27:23] hello [08:40:29] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #700: SUCCESS in 27 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/700/ [08:41:25] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #701: SUCCESS in 23 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/701/ [09:13:43] (03PS2) 10Hashar: Whitelist Florianschmidtwelzow [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/159414 (owner: 10Legoktm) [09:14:27] (03CR) 10Hashar: [C: 032] "Rebased. Deploying!" [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/159414 (owner: 10Legoktm) [09:14:32] yay :D [09:14:35] (03Merged) 10jenkins-bot: Whitelist Florianschmidtwelzow [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/159414 (owner: 10Legoktm) [09:16:59] (03CR) 10Florianschmidtwelzow: "Thanks :)" [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/159414 (owner: 10Legoktm) [09:18:09] (03PS4) 10Legoktm: Add code coverage template for extensions and try to add one for Echo [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/154757 (https://bugzilla.wikimedia.org/69685) [09:19:09] (03CR) 10jenkins-bot: [V: 04-1] Add code coverage template for extensions and try to add one for Echo [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/154757 (https://bugzilla.wikimedia.org/69685) (owner: 10Legoktm) [09:19:20] (03PS5) 10Legoktm: Add code coverage template for extensions and try to add one for GlobalCssJs [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/154757 (https://bugzilla.wikimedia.org/69685) [09:19:37] hashar: do you want to look at ^ ? I basically got it running on labs: https://tools.wmflabs.org/coverage/GlobalCssJs/extensions_GlobalCssJs.html [09:19:56] (but I have no idea on how to do it in jenkins) [09:20:13] (03CR) 10jenkins-bot: [V: 04-1] Add code coverage template for extensions and try to add one for GlobalCssJs [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/154757 (https://bugzilla.wikimedia.org/69685) (owner: 10Legoktm) [09:21:40] legoktm: well I am just back from a 2 weeks leave [09:21:49] so got a bunch of mails to process :-] [09:21:49] whenever you have time :) [09:21:58] Project beta-scap-eqiad build #21630: FAILURE in 7 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/21630/ [09:22:03] 270 mails just for integration/*.git :( [09:22:13] well, 1 down so far :D [09:23:07] (03CR) 10Hashar: "Great! Feel free to add a new Trusty slave in labs. If unsure we can pair it together." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159224 (https://bugzilla.wikimedia.org/66056) (owner: 10Krinkle) [09:24:52] Yippee, build fixed! [09:24:53] Project beta-scap-eqiad build #21631: FIXED in 1 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/21631/ [09:26:33] (03CR) 10Hashar: [C: 04-1] "I defined them explicitly because some editors / linters complains when a variable is not initialized before use. Also that is a good wa" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/156988 (owner: 10Addshore) [09:26:50] (03PS2) 10Hashar: Change spaces to tab [integration/jenkins] - 10https://gerrit.wikimedia.org/r/156993 (owner: 10Addshore) [09:27:01] (03CR) 10Hashar: [C: 032] Change spaces to tab [integration/jenkins] - 10https://gerrit.wikimedia.org/r/156993 (owner: 10Addshore) [09:27:03] (03Merged) 10jenkins-bot: Change spaces to tab [integration/jenkins] - 10https://gerrit.wikimedia.org/r/156993 (owner: 10Addshore) [09:29:57] (03PS3) 10Hashar: Run Ruby syntax check for Echo [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/157806 (https://bugzilla.wikimedia.org/69130) (owner: 10Zfilipin) [09:31:08] (03CR) 10Hashar: [C: 032] "Deployed:" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/157806 (https://bugzilla.wikimedia.org/69130) (owner: 10Zfilipin) [09:32:12] (03PS3) 10Hashar: Run Ruby syntax check for Echo [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/157804 (https://bugzilla.wikimedia.org/69130) (owner: 10Zfilipin) [09:34:18] (03CR) 10Hashar: [C: 032] Run Ruby syntax check for Echo [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/157804 (https://bugzilla.wikimedia.org/69130) (owner: 10Zfilipin) [09:34:26] (03Merged) 10jenkins-bot: Run Ruby syntax check for Echo [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/157804 (https://bugzilla.wikimedia.org/69130) (owner: 10Zfilipin) [09:35:35] (03PS3) 10Hashar: Update benestar's email address (trusted user) [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/157302 (owner: 10Addshore) [09:35:37] (03Merged) 10jenkins-bot: Run Ruby syntax check for Echo [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/157806 (https://bugzilla.wikimedia.org/69130) (owner: 10Zfilipin) [09:36:31] (03PS4) 10Hashar: Update benestar's email address (trusted user) [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/157302 (owner: 10Addshore) [09:37:16] (03CR) 10Hashar: [C: 032] "The change missed an entry in the other email filter. Amended and deploying :)" [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/157302 (owner: 10Addshore) [09:37:25] (03Merged) 10jenkins-bot: Update benestar's email address (trusted user) [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/157302 (owner: 10Addshore) [10:02:04] (03PS2) 10Hashar: Add jenkins jobs for SandboxLink extension [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/158898 (owner: 10Legoktm) [10:02:23] (03CR) 10Hashar: [C: 032] "deploying jobs" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/158898 (owner: 10Legoktm) [10:03:46] (03CR) 10Hashar: "deployed:" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/158898 (owner: 10Legoktm) [10:05:33] (03Merged) 10jenkins-bot: Add jenkins jobs for SandboxLink extension [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/158898 (owner: 10Legoktm) [10:06:21] (03PS2) 10Hashar: Add zuul config for SandboxLink extension [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/158899 (owner: 10Legoktm) [10:06:39] (03CR) 10jenkins-bot: [V: 04-1] Add zuul config for SandboxLink extension [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/158899 (owner: 10Legoktm) [10:16:01] (03CR) 10Hashar: "recheck" [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/158899 (owner: 10Legoktm) [10:18:10] (03PS7) 10Hashar: WIP kill jenkins slaves WIP [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/156455 [10:18:53] (03CR) 10Hashar: [C: 032] Add zuul config for SandboxLink extension [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/158899 (owner: 10Legoktm) [10:19:01] (03Merged) 10jenkins-bot: Add zuul config for SandboxLink extension [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/158899 (owner: 10Legoktm) [10:41:14] Project beta-scap-eqiad build #21638: FAILURE in 7 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/21638/ [10:49:59] Yippee, build fixed! [10:49:59] Project beta-scap-eqiad build #21639: FIXED in 5 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/21639/ [11:02:56] Project beta-scap-eqiad build #21640: FAILURE in 8 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/21640/ [11:16:37] Yippee, build fixed! [11:16:38] Project beta-scap-eqiad build #21642: FIXED in 2 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/21642/ [12:08:49] zeljkof: I am back around :D [12:14:54] (03PS2) 10Hashar: Remove ZeroRatedMobileAccess [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160400 (owner: 10Yurik) [12:14:58] (03PS3) 10Hashar: Remove ZeroRatedMobileAccess [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160400 (owner: 10Yurik) [12:15:15] (03CR) 10Hashar: [C: 032] Remove ZeroRatedMobileAccess [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160400 (owner: 10Yurik) [12:15:23] (03Merged) 10jenkins-bot: Remove ZeroRatedMobileAccess [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160400 (owner: 10Yurik) [12:15:53] hashar: welcome back! :) [12:16:23] (03PS1) 10Hashar: Remove ZeroRatedMobileAccess [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160433 [12:16:38] (03CR) 10Hashar: [C: 032] Remove ZeroRatedMobileAccess [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160433 (owner: 10Hashar) [12:18:41] (03Merged) 10jenkins-bot: Remove ZeroRatedMobileAccess [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160433 (owner: 10Hashar) [12:19:38] (03CR) 10Hashar: [C: 04-1] "We need jobs to be defined in Jenkins Job Builder configuration https://www.mediawiki.org/wiki/Continuous_integration/Tutorials/Adding_a_M" [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160396 (owner: 10Yurik) [12:20:08] (03PS3) 10Hashar: Stop SemanticForms-testextensions from voting [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/157131 (owner: 10Addshore) [12:20:30] (03CR) 10Hashar: [C: 032] Stop SemanticForms-testextensions from voting [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/157131 (owner: 10Addshore) [12:20:38] (03Merged) 10jenkins-bot: Stop SemanticForms-testextensions from voting [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/157131 (owner: 10Addshore) [12:21:09] zeljkof: thx! I noticed there is a lot of changes pending [12:21:16] wanna have a look at them this afternoon? [12:21:36] hashar: sure [12:21:42] I have a few that are waiting for you [12:22:20] lets process them :] [12:22:56] Project beta-scap-eqiad build #21648: FAILURE in 9 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/21648/ [12:23:05] hashar: just ping me when ever you have some time [12:23:21] zeljkof: I do right now :] [12:23:43] I am clearing up my Gerrit dashboard and there is too many changes by Zfilipin [12:23:44] hashar: just a second to find a free conference room [12:24:01] zeljkof: I can't google hangout though [12:24:04] connection is shitty there [12:24:13] hashar: ok, here then? [12:25:30] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #702: SUCCESS in 28 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/702/ [12:25:30] zeljkof: obviously :D [12:25:56] hashar: ok, not sure how to do it here :) [12:26:00] do you have any questions? [12:26:23] lets pick patches and review them :D [12:26:27] https://gerrit.wikimedia.org/r/#/q/status:open+project:integration/jenkins-job-builder-config,n,z [12:26:31] I am not sure which one to start with [12:26:44] hashar: ok, let me find one to start [12:26:53] ah your patches are https://gerrit.wikimedia.org/r/#/q/status:open+project:integration/jenkins-job-builder-config+owner:%22Zfilipin+%253Czfilipin%2540wikimedia.org%253E%22,n,z [12:28:04] (03PS3) 10Zfilipin: The first PdfHandler job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/157696 [12:29:07] hashar: not sure why this one is not merged [12:29:08] https://gerrit.wikimedia.org/r/#/c/157696/ [12:30:18] it is deployed apparently [12:30:34] yes, and all related commits are merged, but this one is not [12:30:36] was apparently depending on a bunch of other changes [12:30:42] feel free to +2 [12:31:05] (03CR) 10Hashar: [C: 032] "Job deployed, all dependencies merged." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/157696 (owner: 10Zfilipin) [12:31:13] next! [12:32:02] hashar: great, good start, let's find another easy one [12:32:19] Added NeilK to UploadWizard mail notifications [12:32:23] https://gerrit.wikimedia.org/r/#/c/159061/ [12:32:30] I am wondering whether we can merge lists in YAML [12:32:30] (03PS2) 10Zfilipin: Added NeilK to UploadWizard mail notifications [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159061 (https://bugzilla.wikimedia.org/70307) [12:33:11] (03Merged) 10jenkins-bot: The first PdfHandler job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/157696 (owner: 10Zfilipin) [12:33:14] hashar: not sure what you have asked [12:35:58] forget me :) [12:37:28] (03CR) 10Hashar: [C: 032] "All fine. I have refreshed the jobs configuration to be sure." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159061 (https://bugzilla.wikimedia.org/70307) (owner: 10Zfilipin) [12:37:32] Yippee, build fixed! [12:37:32] Project beta-scap-eqiad build #21650: FIXED in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/21650/ [12:37:38] so much spam :) [12:37:44] (03PS3) 10Hashar: Added NeilK to UploadWizard mail notifications [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159061 (https://bugzilla.wikimedia.org/70307) (owner: 10Zfilipin) [12:37:54] (03CR) 10Hashar: [C: 032] Added NeilK to UploadWizard mail notifications [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159061 (https://bugzilla.wikimedia.org/70307) (owner: 10Zfilipin) [12:38:24] zeljkof: someone suggested to merge the jjb and zuul configuration repositories [12:39:07] hashar: is that doable? [12:39:27] yup [12:39:32] would it be a lot of work? [12:39:34] just have to create a new repo such as integration/config.git [12:39:40] and use two different directories [12:39:44] ie /jjb/ and /zuul/ [12:40:23] I see [12:43:29] will see :] [12:44:23] hashar: should we continue with the review? [12:44:27] yes [12:44:33] (03Merged) 10jenkins-bot: Added NeilK to UploadWizard mail notifications [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159061 (https://bugzilla.wikimedia.org/70307) (owner: 10Zfilipin) [12:44:51] ok, the next in the queue is... [12:45:10] (03PS2) 10Zfilipin: Internet Explorer (8, 9, 11) jobs for MultimediaViewer [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160018 (https://bugzilla.wikimedia.org/70744) [12:45:28] hashar: see ^ [12:45:49] yeah net is super slow :( [12:49:38] hashar: no problem, ping me when you are ready for the next commit :) [12:50:20] hi, any tips why this might happen? https://integration.wikimedia.org/ci/job/mwext-MultimediaViewer-qunit/2421/console [12:50:52] I am trying to backport a patch to wmf21, and it looks like it doesn't pick up the qunit tests [12:55:47] zeljkof: back [12:57:45] hashar: https://gerrit.wikimedia.org/r/#/c/160018/ [12:57:54] I think that is the next easy one [12:58:45] zeljkof: yeah looking at it [12:59:56] zeljkof: the VERSION is hardcoded in macro.yaml [13:00:00] based on the value of BROWSER [13:00:13] hashar: oh, that is a long story :( [13:00:18] ohh wait [13:00:21] we actually changed the implementation [13:00:23] code has been duplicated [13:00:26] copy/pasting [13:00:26] there is a second macro [13:00:31] yeah :( [13:00:36] that is a ugly [13:00:40] it was a WIP and chris merged it by mistake, I think [13:01:08] hashar: see https://gerrit.wikimedia.org/r/#/c/160008/1/macro.yaml,cm [13:01:15] sorry https://gerrit.wikimedia.org/r/#/c/160008/ [13:01:39] yeah [13:01:44] hashar: I would like to clean that up today [13:01:46] we talked about it back in June [13:01:53] the idea was to set the version in the 'default' section [13:02:04] and override it at project / job level [13:02:04] yes, I was never able to figure out how to do that [13:02:17] and the patch I made for it in JJB got changed a few weeks ago [13:02:20] jjb was always reporting some problems [13:02:21] so it is doable now :] [13:02:45] should we work on that right now, or can we merge what is in the queue and the clean up? [13:03:19] lets merge pending changes :) [13:03:40] hashar: ok [13:03:48] I think none of them touch that code [13:03:53] so it will be easy to clean up later [13:05:27] ok so https://gerrit.wikimedia.org/r/#/c/159323/3 [13:05:31] Add browsertests to file names that are related to browsertests [13:05:38] seems to just be some file renaming [13:05:44] hashar: yes [13:05:59] to make it more consistent with the naming convention of the repo [13:06:01] (03PS4) 10Hashar: Rename browsertests files [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159323 (owner: 10Zfilipin) [13:06:03] (03CR) 10jenkins-bot: [V: 04-1] Rename browsertests files [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159323 (owner: 10Zfilipin) [13:06:50] hashar: ok, merge conflict, will rebase locally [13:07:06] :( [13:07:30] I can't work [13:07:36] moving to another place [13:07:39] the net is way too slow [13:07:51] be back in a few [13:08:46] (03PS5) 10Zfilipin: Rename browsertests files [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159323 [13:15:10] zeljkof: back [13:15:14] moved to a local bar with A+ grade internet [13:15:19] and cheap prices! [13:15:25] hashar: good beer? :) [13:15:33] na nothing special [13:15:50] having coffee and sparkling water [13:16:17] (03CR) 10Hashar: [C: 032] Rename browsertests files [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159323 (owner: 10Zfilipin) [13:16:40] zeljkof: next sort emails and need a rebase https://gerrit.wikimedia.org/r/#/c/159334/ [13:16:50] hashar: will do [13:17:11] and then you can rebase the change that sort the browertest jobs https://gerrit.wikimedia.org/r/#/c/159474/ [13:18:14] (03PS6) 10Zfilipin: Add browsertests to file names that are related to browsertests [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159323 [13:18:16] (03PS4) 10Zfilipin: Sort notification-emails in browsertests.yaml alphabetically [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159334 [13:19:02] hashar: both commits actually did not need any human intervention [13:19:09] git did all the work [13:19:18] not sure why gerrit did not do it itself [13:19:40] "Falling back to patching base and 3-way merge..." [13:19:45] ^ that was all [13:19:50] but no action needed from me [13:19:54] yeah [13:20:05] Gerrit only handles very trivial conflicts [13:20:15] Krinkle|detached: [13:20:42] Krinkle|detached: can't do any hangout today, net is flaky at my usual working place. We can do the checkin over IRC though :] [13:21:10] (03CR) 10Hashar: [C: 032] Add browsertests to file names that are related to browsertests [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159323 (owner: 10Zfilipin) [13:22:26] (03CR) 10Hashar: [C: 032] Sort notification-emails in browsertests.yaml alphabetically [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159334 (owner: 10Zfilipin) [13:22:44] zeljkof: next is https://gerrit.wikimedia.org/r/#/c/159474/ ( sort browsertests jobs) [13:23:12] hashar: rebasing [13:23:14] (03Merged) 10jenkins-bot: Add browsertests to file names that are related to browsertests [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159323 (owner: 10Zfilipin) [13:24:30] (03Merged) 10jenkins-bot: Sort notification-emails in browsertests.yaml alphabetically [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159334 (owner: 10Zfilipin) [13:31:56] hashar: argh, "remote rejected" [13:32:16] hehe [13:32:50] hashar: ok, will try again, stand by... [13:34:26] hashar: I see [13:34:29] [remote rejected] HEAD -> refs/publish/master/browsertests (change 159334 closed) [13:34:51] why is it trying to push there...? [13:35:29] hashar: http://etherpad.wikimedia.org/p/jjb [13:36:18] the previous one got merged [13:36:21] ie c937c0c Sort notification-emails in browsertests.yaml alphabetically [13:36:25] so you want to rebase [13:36:32] hashar: did that [13:36:44] ok, let me paste the entire bash output [13:36:47] the change is https://gerrit.wikimedia.org/r/#/c/159334/3 [13:36:57] and none of the path set have c937c0c [13:37:05] so Gerrit consider it a new patchset and attempt to update the change [13:37:11] but it has been merged, and the change is closed [13:37:12] so it whine [13:37:16] if you rebase, that should work :-] [13:37:44] hashar: that is what I thought too [13:37:54] not sure what am I doing wrong [13:38:01] checking if I have the latest master [13:46:29] 3Wikimedia / 3Continuous integration: Have Jenkins run a voting PHPUnit job for GettingStarted - 10https://bugzilla.wikimedia.org/70631#c1 (10Antoine "hashar" Musso) The step by step guide at https://www.mediawiki.org/wiki/Continuous_integration/Tutorials/Adding_a_MediaWiki_extension should let you craft all... [13:48:31] (03PS1) 10Hashar: CodeMirror now pass jshint [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160441 (https://bugzilla.wikimedia.org/69869) [13:50:01] (03CR) 10Hashar: [C: 032] CodeMirror now pass jshint [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160441 (https://bugzilla.wikimedia.org/69869) (owner: 10Hashar) [13:51:28] (03Merged) 10jenkins-bot: CodeMirror now pass jshint [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/160441 (https://bugzilla.wikimedia.org/69869) (owner: 10Hashar) [13:54:46] hashar: ok, not sure what am I doing wrong [13:54:49] will start over [13:55:48] :D [13:56:02] 3Wikimedia Labs / 3deployment-prep (beta): hhvm fill up /var/log/upstart/hhvm.log - 10https://bugzilla.wikimedia.org/69976#c3 (10Antoine "hashar" Musso) I think puppet is now passing on the hhvm instances. I am not sure where the log are written to though. [13:56:07] back in 5 minutes, need a break [13:57:23] (03PS2) 10Hashar: wikilove passes jshint after I0278575 [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/158777 (https://bugzilla.wikimedia.org/61645) (owner: 10TheDJ) [13:57:41] (03CR) 10Hashar: [C: 032] "Thank you!" [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/158777 (https://bugzilla.wikimedia.org/61645) (owner: 10TheDJ) [13:58:11] (03Merged) 10jenkins-bot: wikilove passes jshint after I0278575 [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/158777 (https://bugzilla.wikimedia.org/61645) (owner: 10TheDJ) [14:09:37] (03PS1) 10Hashar: Revert "Workaround hphpize injecting some bad include path" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160447 [14:09:45] (03PS2) 10Hashar: Revert "Workaround hphpize injecting some bad include path" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160447 [14:11:07] (03PS2) 10Zfilipin: Sort browsertests jobs [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159474 [14:12:22] hashar: finally :( https://gerrit.wikimedia.org/r/#/c/159474/ [14:13:29] 3Wikimedia / 3Continuous integration: hphpize inject $PWD used at time of building to HHVM_INCLUDE_DIRS - 10https://bugzilla.wikimedia.org/68944#c4 (10Antoine "hashar" Musso) 5RESO/FIX>3REOP I have updated the Trusty Jenkins slave: Unpacking hhvm-dev (3.3-dev+20140728+wmf7) over (3.3-dev+20140728+wmf5)... [14:13:38] (03PS3) 10Hashar: Revert "Workaround hphpize injecting some bad include path" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160447 (https://bugzilla.wikimedia.org/68944) [14:14:09] (03CR) 10Hashar: [C: 04-2] "hphize isn't fixed yet see comment on https://bugzilla.wikimedia.org/show_bug.cgi?id=68944#c4 which I reopened." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160447 (https://bugzilla.wikimedia.org/68944) (owner: 10Hashar) [14:14:24] zeljkof: great! [14:14:51] (03CR) 10Hashar: [C: 032] "Noop!" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159474 (owner: 10Zfilipin) [14:15:00] as usual, the problem was between keyboard and computer [14:16:59] (03Merged) 10jenkins-bot: Sort browsertests jobs [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159474 (owner: 10Zfilipin) [14:17:27] zeljkof: next! https://gerrit.wikimedia.org/r/#/c/158873/ Half of Flow jobs are on test2, it does not make sense to have beta as default [14:17:40] which needs rebasing hehe [14:18:00] that one is straightforward, if it is a noop (it should) just self merge! [14:20:53] hashar: rebasing [14:21:56] (03PS5) 10Zfilipin: Half of Flow jobs are on test2, it does not make sense to have beta as default [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/158873 [14:22:37] hashar: rebased ^ [14:23:31] gooo [14:23:32] d [14:24:10] that is a good cleanup [14:24:23] now gotta think about a way to refactor the browser version code duplication [14:24:42] hashar: it should be easy, merging copied code was a mistake [14:24:51] (03CR) 10Hashar: [C: 032] "noop change \O/" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/158873 (owner: 10Zfilipin) [14:25:03] yeah we probably can find something nicer [14:25:12] though we would still need two job templates [14:25:23] one having -{version}- and another without [14:25:39] hashar: that was what was confusing me [14:26:14] so, if I want to pass "version" as a variable to a template, it has to be in the job name? [14:26:26] no [14:26:43] but if you need a job per version, they better have to be named differently [14:26:50] ie test-ie-7 and test-ie-8 [14:26:51] (03Merged) 10jenkins-bot: Half of Flow jobs are on test2, it does not make sense to have beta as default [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/158873 (owner: 10Zfilipin) [14:27:01] which would be realized by a template named test-{browser}-{version} [14:27:05] ok, so all jobs should have version in the name then? [14:27:17] not necessarly [14:27:21] or just ie jobs? [14:27:34] since only IE will have different versions, yeah I would only do it for ie [14:28:01] (03PS3) 10Zfilipin: Internet Explorer (8, 9, 11) jobs for MultimediaViewer [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160018 (https://bugzilla.wikimedia.org/70744) [14:28:02] which is more or less what we did for the -sauce jobs [14:28:11] hashar: yes [14:28:50] looking at job-templates-browsertests.yaml [14:28:56] hashar: should we merge https://gerrit.wikimedia.org/r/#/c/160018/ ? [14:29:08] the version template (lines 17-31) is a overriding the builders section [14:29:11] since the refactoring will not change that commit anyway [14:29:16] which is just to be able to pass "version" [14:29:32] what we could do is defined a version: 'undefined' in the default section [14:29:38] hashar: yes, that was a temp workaround to see if it works [14:29:42] this way version would always be defined [14:29:51] hashar: good idea [14:29:58] and in 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-{version}-sauce' [14:30:03] remove the builders section [14:30:14] then have it invoked as usual with something like: [14:30:16] jobs: [14:30:19] - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-{version}-sauce': [14:30:34] browser: IE [14:30:34] version: 8 [14:30:42] that might work [14:31:11] nature's call / cig [14:44:51] hashar: https://gerrit.wikimedia.org/r/#/c/160018/ [14:44:57] should we merge this? [14:44:58] you are fast [14:45:43] zeljkof: yeah deploy the job and +2 :-D [14:46:06] though we could shorten internet_explorer to simply "IE" [14:46:21] hashar: let me document that the jobs are already deployed :) [14:46:24] but that is another issue hehe [14:46:30] +2 it! [14:46:38] hashar: i _hate_ to self merge :) [14:47:59] (03PS4) 10Zfilipin: Internet Explorer (8, 9, 11) jobs for MultimediaViewer [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160018 (https://bugzilla.wikimedia.org/70744) [14:49:02] hashar: should I +2 this, or would you do it? https://gerrit.wikimedia.org/r/#/c/160018/ [14:49:08] do it :] [14:49:15] {{be bold}} hehe [14:49:18] (03CR) 10Zfilipin: [C: 032] Internet Explorer (8, 9, 11) jobs for MultimediaViewer [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160018 (https://bugzilla.wikimedia.org/70744) (owner: 10Zfilipin) [14:49:24] hashar: :) done [14:50:07] hashar: I need help with this one [14:50:07] https://gerrit.wikimedia.org/r/#/c/160100/ [14:50:16] rebasing... [14:50:28] zeljkof: is there a way to disable mail warnings from the IE8 test? [14:50:37] tgr: sure [14:50:40] MediaViewer is known to not work on IE8 [14:50:57] tgr: we can disable the job, delete it, remove you from mail notifications... [14:51:01] what ever you prefere [14:51:03] prefer [14:51:19] (03PS4) 10Zfilipin: WIP Disable a few jobs that are known to be broken [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160100 [14:51:36] (03Merged) 10jenkins-bot: Internet Explorer (8, 9, 11) jobs for MultimediaViewer [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160018 (https://bugzilla.wikimedia.org/70744) (owner: 10Zfilipin) [14:52:40] hashar: I have noticed that jobs that are manually disabled (via jenkins web site, not via jjb) tend to get enabled at random times [14:52:49] disabling would be best I think [14:52:53] my theory is that they get enable when jenkins is restarted [14:52:57] zeljkof: ah I have no idea [14:53:02] should I write a mail about that? [14:53:13] zeljkof: but that is probably because whenever we run JJB, that removes the disabled: true that has been set manually [14:53:14] tgr: no, just disable the job :) [14:53:19] tgr: do you know how to do it? [14:53:47] hashar: no, it is not documented in the job change log [14:53:56] and jjb would make the entry there, right? [14:54:04] zeljkof: on the Jenkins interface or in the config? [14:54:15] tgr: both :) [14:54:31] tgr: I am working on a patch to make it in jjb https://gerrit.wikimedia.org/r/#/c/160100/ [14:54:45] tgr: but for now you can do it from the jenkins web interface [14:55:03] ok, thanks [14:55:05] zeljkof: for disabled: I have to look it up [14:55:13] zeljkof: seems there is some annoying thing in JJB :] [14:56:49] zeljkof: does Jenkins use the labs login? [14:57:34] apparently yes [14:59:31] someone going to play the piano in the bar :] [15:00:51] tgr: I call it gerrit credentials :) [15:01:08] tgr: but if you can log in, go to job and click "disable" button [15:01:22] yeah, found it, thanks [15:02:14] 3Wikimedia / 3Quality Assurance: Create browser test builds for MMV in IE back to IE8 - 10https://bugzilla.wikimedia.org/70744 (10Željko Filipin) 5PATC>3RESO/FIX [15:02:24] (03PS5) 10Zfilipin: WIP Disable a few jobs that are known to be broken [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160100 [15:04:55] (03CR) 10Hashar: [C: 04-1] WIP Disable a few jobs that are known to be broken [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160100 (owner: 10Zfilipin) [15:07:37] (03PS6) 10Zfilipin: WIP Disable a few jobs that are known to be broken [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160100 [15:09:14] 3Wikimedia / 3Continuous integration: Jenkins: Set up PHPUnit testing on PostgreSQL backend - 10https://bugzilla.wikimedia.org/37602#c15 (10Antoine "hashar" Musso) Someone need to figure out a way to: * create a unique username / database when a job start * inject that in LocalSettings.php or have the dbname... [15:09:23] hashar: as far as I remember, ops people said we should run rubocop on a labs machine [15:09:31] ah yeah [15:09:43] so _joe_ desperately attempted to package rubocop and all its dependencies [15:09:53] and eventually gave up because it was a looong list of deps [15:09:58] so we need to use ruby gems on labs instance indeed [15:10:07] hashar: I am fine with that [15:10:07] with a per instance cache of gems to speed things up [15:10:23] hashar: is that something you can work on these days? [15:10:57] I am sure you can figure it out :] [15:11:07] hashar: I have no clue where to start [15:11:23] for browsertests we are using gem [15:11:25] I remember playing with labs a while ago, but it was not easy to set up [15:11:30] with something like GEM_HOME=../gemcache or something [15:11:35] then just gem install rubocop [15:11:39] and: rubocop . [15:11:39] :-D [15:11:42] hashar: I see [15:11:53] you will need the job to be tied on labs salaves [15:12:19] node: contintLabsSlave && UbuntuPrecise [15:12:21] that should do [15:13:25] hashar: where is that set up? in zuul? [15:14:07] (03PS5) 10Zfilipin: WIP Added job template and builder that runs rubocop, Ruby linter [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/152918 (https://bugzilla.wikimedia.org/69245) [15:15:29] zeljkof: na in JJB -config :D [15:15:38] hashar: ok, so all that needs to be done is to tie this job template to specific machines? https://gerrit.wikimedia.org/r/#/c/152918/5/job-templates.yaml,cm [15:15:42] remember Jenkins defines the tasks / steps to run [15:15:50] Zuul only react to events and trigger jobs [15:16:15] zeljkof: yeah add a node: there [15:16:19] node: contintLabsSlave && UbuntuPrecise [15:18:23] (03PS6) 10Zfilipin: WIP Added job template and builder that runs rubocop, Ruby linter [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/152918 (https://bugzilla.wikimedia.org/69245) [15:19:35] (03CR) 10Hashar: [C: 04-1] "The rubocop macro need to install the utility via gem and using a per instance cache :D See inline!" (031 comment) [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/152918 (https://bugzilla.wikimedia.org/69245) (owner: 10Zfilipin) [15:19:36] zeljkof: commented :] [15:21:02] (03PS6) 10Zfilipin: WIP Run rubocop, Ruby linter, for all repositories that have Ruby code [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/152926 (https://bugzilla.wikimedia.org/69245) [15:21:43] (03CR) 10jenkins-bot: [V: 04-1] WIP Run rubocop, Ruby linter, for all repositories that have Ruby code [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/152926 (https://bugzilla.wikimedia.org/69245) (owner: 10Zfilipin) [15:24:26] hashar: ok, this is complaining that some rubocop jobs are not yet created, right? https://integration.wikimedia.org/ci/job/integration-zuul-layoutvalidation/1735/consoleFull [15:24:39] yup [15:24:51] the job query Jenkins for a list of jobs configured [15:24:52] hashar: is that new, I do not remember seeing it before? [15:25:09] I added it just before my paternity leave [15:25:12] heya hashar :) welcome back [15:25:16] to make sure people were not going to screw it up hehe [15:25:21] greg-g: !!! [15:25:46] zeljkof: so whenever the rubocop jobs are deployed in jenkins, the zuul change will pass [15:26:02] hashar: ok, that is easy [15:27:14] 3Wikimedia / 3Continuous integration: CI: mw-api-siteinfo.py should more clearly report site failure - 10https://bugzilla.wikimedia.org/70695#c5 (10Antoine "hashar" Musso) - https://bugzilla.wikimedia.org/show_bug.cgi?id=51497 which is dealing with monitoring. [15:27:15] 3Wikimedia Labs / 3deployment-prep (beta): Setup monitoring for Beta cluster - 10https://bugzilla.wikimedia.org/51497 (10Antoine "hashar" Musso) [15:29:38] * bd808S waves to hashar [15:29:56] good to see you back. Hope the family is doing well. [15:31:07] greg-g: bd808S family is all fine :] [15:31:14] my latest kid is already sleeping 8+ hours straight [15:31:33] bd808S: will definitely work on completing the zuul cloner / vendor stuff [15:31:38] and have it run for extensions [15:31:44] I am close to it! ™ © [15:32:36] hashar: That would be pretty cool! I'm glad you have made so much time to work on my "simple change" [15:32:54] bd808: well I started as far back as in Febuary [15:32:55] bd808: is there any use for the scap statsd data at all now? [15:33:00] (it's on graphite.wmflabs.org) [15:33:05] your vendor repo was a good occasion to prioritize a bit more :] [15:33:20] ended up refactoring a bunch of things in the process [15:33:44] YuviPanda: If it's actually working then it would be useful to fix the "deployments" checkbox in gdash. [15:33:54] (03PS7) 10Zfilipin: Added job template and builder that runs rubocop, Ruby linter [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/152918 (https://bugzilla.wikimedia.org/69245) [15:34:02] bd808: I see data under scap.* [15:34:09] YuviPanda: https://bugzilla.wikimedia.org/show_bug.cgi?id=62667 [15:34:26] (03CR) 10Zfilipin: Added job template and builder that runs rubocop, Ruby linter (031 comment) [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/152918 (https://bugzilla.wikimedia.org/69245) (owner: 10Zfilipin) [15:35:30] YuviPanda: and scap.scap.count is showing a constant 1/sample period :( [15:35:38] oh [15:35:51] I suppose that's txstatsd filling in? [15:36:08] I actually may be that the client built into scap is sending the wrong messages [15:36:18] at least wrong for txstatsd [15:36:41] They *look* right to me, but have never worked with txstatsd [15:37:21] sigh [15:37:38] we should look into replacing txstatsd. I see no upstream changes in quite a while... [15:37:49] * YuviPanda is currently distracted by implementing shinken [15:37:57] I don't have time to mess with it this week (or even next) but the scap client code is at -- https://github.com/wikimedia/mediawiki-tools-scap/blob/master/scap/log.py#L248-L274 [15:38:10] yeah, I took a look at it before [15:38:47] bd808: the poor beta-scap-eqiad job keep failing due to some DNS resolution failure [15:39:05] bd808: I think the labs dnsmasq is overwhelmed hehe [15:39:20] hashar: again? Sadness. That was fixed for a while by Coren adding longer dsn cache config on all the labs hosts. [15:39:32] The labs dns server is crap. [15:39:53] last time it was due to the security audit generating hundreds of queries per seconds, each doing a bunch of dns queries [15:41:28] I remember in some conf i used the IP to avoid relying on DNS [15:41:36] hashar: whoa! 8+ hours before a month old?! crazy [15:41:46] greg-g: yeah big baby :] [15:42:07] she was 4,1 kg (blame US for still using obsolete imperial system) [15:42:35] heading out! [15:42:44] zeljkof: I am out, will resume tomorrow morning :] [15:42:52] meeting at school hehe [15:43:08] greg-g: good news: I have read all my emails spam! [15:43:16] hashar: no way [15:43:22] [delete] [15:43:23] works fine [15:43:26] productive first day [15:43:27] haha [15:43:51] will spend the rest of the week pleasing bd808 :] He has been waiting for too long [15:44:15] see you tomorrow! [16:04:35] (03PS7) 10Zfilipin: Run rubocop, Ruby linter, for all repositories that have Ruby code [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/152926 (https://bugzilla.wikimedia.org/69245) [16:08:13] (03PS8) 10Zfilipin: Run rubocop, Ruby linter, for all repositories that have Ruby code [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/152926 (https://bugzilla.wikimedia.org/69245) [16:10:51] anyone around who can unstick jenkins? [16:10:55] I never remember how [16:12:51] (03PS1) 10Cmcmahon: Add Flow builds for monobook user in beta labs [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160469 [16:13:36] (03CR) 10jenkins-bot: [V: 04-1] Run rubocop, Ruby linter, for all repositories that have Ruby code [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/152926 (https://bugzilla.wikimedia.org/69245) (owner: 10Zfilipin) [16:13:37] aude: you can try now [16:18:41] bd808: btw, I see more usable scap data in the prod graphite, so I presume that's not a txstatsd problem [16:18:45] * YuviPanda puts it on backlog to investigate [18:32:57] beta scap has been broken for an hour or so :/ [18:33:04] rsync can't connect to deployment-bastion [18:33:48] oh, those salt changes.... [18:33:52] or not [18:33:59] * greg-g is reaching [18:34:00] 00:08:20.909 rsync: failed to connect to deployment-bastion.eqiad.wmflabs (10.68.16.58): Connection timed out (110) [18:34:00] :D [18:34:39] !log [18:34:41] !sal [18:35:00] 3Wikimedia / 3Continuous integration: Jenkins: Set up integration slaves with Ubuntu Trusty - 10https://bugzilla.wikimedia.org/68256 (10Krinkle) 5PATC>3ASSI a:3Krinkle [18:35:39] wm-bot: help [18:35:44] hrm [18:35:55] !help [18:35:58] qa-morebots: help [18:35:58] I am a logbot running on tools-exec-08. [18:35:58] Messages are logged to https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL. [18:35:58] To log a message, type !log . [18:36:01] there we go [18:36:19] !sal is https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [18:36:19] Key was added [18:36:22] !sal [18:36:22] https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [18:37:18] !log beta-scap-eqiad job is broken since ~17:20 UTC https://integration.wikimedia.org/ci/job/beta-scap-eqiad/21680/console || rsync: failed to connect to deployment-bastion.eqiad.wmflabs (10.68.16.58): Connection timed out (110) [18:37:21] Logged the message, Master [18:37:43] well I am not sure where qa-morebots logs though hehe [18:37:58] !del sal [18:38:06] !delete sal [18:38:07] !sal is https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:38:08] This key already exist - remove it, if you want to change it [18:38:16] @delete sal [18:38:17] @remove sal [18:38:19] !remove sal [18:38:37] !**** *** [18:38:55] @del sal [18:38:58] wm-bot: del sal [18:39:00] wm-bot: help [18:40:57] !sal del [18:40:57] Successfully removed sal [18:41:01] !sal is https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:41:01] Key was added [18:41:03] pff [18:47:19] so that is ferm [18:48:23] !log https://gerrit.wikimedia.org/r/#/c/160485/ tweaked a default ferm configuration file which caused puppet to reload ferm. It ends up having rules that prevent ssh from other host thus breaking rsync \O/ [18:48:26] Logged the message, Master [18:57:57] !log scap breakage due to ferm is logged as https://bugzilla.wikimedia.org/show_bug.cgi?id=70858 [18:57:59] Logged the message, Master [18:58:01] 3Wikimedia Labs / 3deployment-prep (beta): ferm policy on deployment-bastion prevents scap rsync from mw hosts - 10https://bugzilla.wikimedia.org/70858 (10Antoine "hashar" Musso) 3NEW p:3Unprio s:3normal a:3None The deployment-bastion.eqiad.wmflabs has ferm enabled. A changed occurred at 17:20 UTC w... [19:00:16] 3Wikimedia Labs / 3deployment-prep (beta): ferm policy on deployment-bastion prevents scap rsync from mw hosts - 10https://bugzilla.wikimedia.org/70858 (10Greg Grossmeier) p:5Unprio>3Highes [19:50:19] !log killed on deployment-bastion a bunch of python /usr/local/sbin/grain-ensure contains ... and /usr/bin/python /usr/bin/salt-call --out=json grains.append deployment_target scap commands [19:50:21] Logged the message, Master [20:01:44] 3Wikimedia Labs / 3deployment-prep (beta): Search and page loads extremely slow on beta cluster (cause being investigated) - 10https://bugzilla.wikimedia.org/70103#c9 (10Rummana Yasmeen) 5RESO/FIX>3REOP This is again happening. [20:03:44] 3Wikimedia Labs / 3deployment-prep (beta): Search and page loads extremely slow on beta cluster (cause being investigated) - 10https://bugzilla.wikimedia.org/70103 (10Rummana Yasmeen) 5REOP>3ASSI [20:09:29] 3Wikimedia Labs / 3deployment-prep (beta): Setup monitoring for Beta cluster - 10https://bugzilla.wikimedia.org/51497 (10Antoine "hashar" Musso) [20:09:33] 3Wikimedia Labs / 3deployment-prep (beta): monitor unsigned salt keys - 10https://bugzilla.wikimedia.org/70862 (10Antoine "hashar" Musso) 3NEW p:3Unprio s:3normal a:3None Whenever an instance is added to the beta cluster and switched to the local salt master, we might forget to sign the key on the sa... [20:16:46] 3Wikimedia Labs / 3deployment-prep (beta): salt commands exceed timeout - 10https://bugzilla.wikimedia.org/70863 (10Antoine "hashar" Musso) [20:16:46] 3Wikimedia Labs / 3deployment-prep (beta): ferm policy on deployment-bastion prevents scap rsync from mw hosts - 10https://bugzilla.wikimedia.org/70858 (10Antoine "hashar" Musso) [20:16:47] 3Wikimedia Labs / 3deployment-prep (beta): salt commands exceed timeout - 10https://bugzilla.wikimedia.org/70863 (10Antoine "hashar" Musso) 3NEW p:3Unprio s:3normal a:3None deployment-bastion has a bunch of idling python commands. When running puppet I noticed some timeout being exceeded: Error: /St... [20:18:16] !log restarted salt-master [20:18:19] Logged the message, Master [21:12:46] 3Wikimedia Labs / 3deployment-prep (beta): sudo rights for matanya - 10https://bugzilla.wikimedia.org/70864 (10matanya) 3NEW p:3Unprio s:3normal a:3None Hello I would like to get sudo rights on beta project in order to help debug puppet/firewall and other issues. I have signed an NDA, can verify wit... [21:15:29] 3Wikimedia Labs / 3deployment-prep (beta): sudo rights for matanya - 10https://bugzilla.wikimedia.org/70864#c1 (10jeremyb) no need for ops: $ groups matanya | perl -lpe 's/\s+/\n/g;' | fgrep -x -e project-deployment-prep -e nda project-deployment-prep nda [21:19:46] 3Wikimedia Labs / 3deployment-prep (beta): sudo rights for matanya - 10https://bugzilla.wikimedia.org/70864#c2 (10Bryan Davis) 5NEW>3RESO/FIX p:5Unprio>3Normal a:3Bryan Davis $ ldapsearch -x cn="Matanya" \* + |grep nda isMemberOf: cn=nda,ou=groups,dc=wikimedia,dc=org Easy decision. Welcome to the... [21:25:15] 3Wikimedia Labs / 3deployment-prep (beta): Search and page loads extremely slow on beta cluster (cause being investigated) - 10https://bugzilla.wikimedia.org/70103#c10 (10Rummana Yasmeen) Its slow while searching for pages, link target names, media files etc but page loading is fine at my end. [21:27:21] whoa https://code.google.com/p/selenium/issues/detail?id=7445 got fixed. welcome back Firefox :-) [21:29:32] !log salt calls failing in beta with errors like "This master address: 'salt' was previously resolvable but now fails to resolve!" [21:29:34] Logged the message, Master [21:31:57] DNS broken in beta? [21:32:01] !log only hosts responding to salt in beta are deployment-mathoid, deployment-pdf01 and deployment-stream [21:32:06] Logged the message, Master [21:32:47] mutante: I think it's something weirder than that. Maybe the salt-minons all died? [21:35:55] chrismcmahon: browsertest puppet ins't applying cleanly [21:36:01] do you know anything about those manifests? [21:36:08] https://gist.githubusercontent.com/Krinkle/fd6cf70688d110809440/raw/ [21:36:09] !log Trying to fix salt with `salt '*' service.restart salt-minion` [21:36:12] Logged the message, Master [21:37:03] For some reason it's creating a new directory and then failing to remove it. Smells like something that was written to apply to an existing node but not accounting for a clean state [21:37:53] Krinkle: manybubbles and/or marxarelli would know more than me I think. Is it all elasticsearch stuff? [21:38:10] I'm here? [21:38:14] Well, it's cascading from Role::Ci::Slave::Browsertests [21:38:29] manybubbles: Providing a new labs instance with CI slave is failing repeatedly due to elasticsearch [21:38:35] manybubbles: https://gist.githubusercontent.com/Krinkle/fd6cf70688d110809440/raw/ [21:38:59] Will it grind anyone's buzz if I reboot deployment-sentry2 and deployment-videoscaler01? There'll be a few minute downtime while I relocate the instances. [21:39:06] I don't want to dig into elasticsearch so I'm going to ignore it for now and have these new slaves be for npm tests, but FYI. [21:39:25] Krinkle: k - I think that is something that hashar put together but I can look [21:40:12] andrewbogott: bd808 might have an opinion, he's futzing with salt right now... (but also currently speaking on a team meeting right now) [21:40:35] andrewbogott: Have at them. No worries from me. [21:40:40] ok, thanks [21:40:53] andrewbogott: should be OK, I'll say something in -qa [21:41:05] chrismcmahon: you just did ;) [21:41:17] er, so many channels... [21:41:20] :) [21:43:59] greg-g: I'm stumped on salt in beta. Restarting the salt-master process and then the salt-minin on deployment-bastion didn't fix. I thought I saw salt woes in prod earlier today but I didn [21:44:12] but I didn't follow if it was fax and how. [21:44:19] s/fax/fixed/ [21:44:44] bd808: I don't remember.. wasn't able to pay attention :/ [21:44:54] andrewbogott: ^^ do you remember anything about that? [21:45:18] yep, there were some stuck processes. [21:45:26] Try 'sudo killall salt-call' [21:45:29] …everywhere :) [21:45:34] That's what we did in prod [21:45:46] oh right... [21:45:47] Well, bring your puppetmaster up to date first [21:46:32] 3Wikimedia Labs / 3deployment-prep (beta): deployment-salt can't talk to itself, git deploy hangs - 10https://bugzilla.wikimedia.org/70868 (10C. Scott Ananian) 3NEW p:3Unprio s:3normal a:3None cscott-free: git deploy sync is hanging (no console output at all) on beta. bd808: lame. I wonder if salt is... [21:48:59] 3Wikimedia Labs / 3deployment-prep (beta): deployment-salt can't talk to itself, git deploy hangs - 10https://bugzilla.wikimedia.org/70868#c1 (10Bryan Davis) On deployment-salt: $ salt '*' cmd.run hostname i-00000396.eqiad.wmflabs: deployment-pdf01 i-00000504.eqiad.wmflabs: deployment-math... [21:50:08] matanya: hey [21:50:29] 3Wikimedia Labs / 3deployment-prep (beta): deployment-salt can't talk to itself, git deploy hangs - 10https://bugzilla.wikimedia.org/70868#c2 (10Bryan Davis) I tried restarting the salt-master process on deployment-salt and the salt-minion process on deployment-bastion and this didn't seem to help anything. [21:50:49] or bd808 [21:51:02] hi jeremyb [21:51:06] don't want to step on toes [21:51:13] i'm going offline in 15-20 mins [21:51:20] not doing anything atm [21:51:22] anything i can take a quick look at before i go? [21:51:23] ok [21:51:27] PROBLEM - BetaLabs: Puppet failure events on labmon1001 is CRITICAL: CRITICAL: deployment-prep.deployment-memc05.puppetagent.failed_events.value (100.00%) deployment-prep.deployment-memc04.puppetagent.failed_events.value (100.00%) deployment-prep.deployment-cxserver01.puppetagent.failed_events.value (100.00%) deployment-prep.deployment-mediawiki01.puppetagent.failed_events.value (100.00%) deployment-prep.deployment-cache-mobile03.puppetagent. [21:51:36] hah [21:51:40] * jeremyb hugs icinga-wm [21:52:45] really icinga-wm ? [21:52:51] jeremyb, matanya: all yours. I shouldn't be here anyway :) [22:06:30] Hm, was deployment-videoscaler01 broken before, or did I break it? [22:09:10] yay @ icinga-wm [22:09:28] matanya: yes:) beta->qa and toollabs->labs [22:09:56] yeah, saw your mail, thanks for this [22:09:57] there wasn't even a puppetmaster running at all [22:10:07] matanya: role/echoirc.pp [22:10:11] now doing puppet run on -salt [22:10:23] jeremyb: beacuse it broke [22:10:29] but of course grain-ensure is going to fail [22:10:38] so first need to fix salt [22:19:49] false alarm, deployment-videoscaler01 should be fine [22:20:27] jeremyb: Did the puppetmaster on deployment-salt die again? If so maybe we need an icinga check for that. [22:20:38] sure we should have a check [22:21:56] yes, 9405 was killed by oom killer, was a master [22:22:13] Sep 15 20:12:09 deployment-salt kernel: [8739588.577549] Out of memory: Kill process 9405 (puppet) score 83 or sacrifice child [22:22:13] Sep 15 20:12:09 deployment-salt kernel: [8739588.578328] Killed process 9405 (puppet) total-vm:482828kB, anon-rss:319876kB, file-rss:2820kB [22:22:40] so, maybe puppet would have fixed it for me. but i manually fixed /etc/salt/minion on deployment-salt [22:22:49] then i saw puppet fix the same file on -bastion [22:24:44] 3Wikimedia Labs / 3deployment-prep (beta): Security test load caused search and page loads extremely slow on beta cluster - 10https://bugzilla.wikimedia.org/70103#c11 (10Greg Grossmeier) 5ASSI>3RESO/FIX I just sat next to Rummana to see the symptoms. There a bit sporatic but noticable. I'll open a new b... [22:26:32] 3Wikimedia Labs / 3deployment-prep (beta): Search is sometimes slow - 10https://bugzilla.wikimedia.org/70869 (10Greg Grossmeier) 3NEW p:3Unprio s:3normal a:3None Rummana saw the issue described in bug 70103 again. The search requests (either in the drop down on the top right, or within VE) are somet... [22:26:59] 3Wikimedia Labs / 3deployment-prep (beta): Search is sometimes slow - 10https://bugzilla.wikimedia.org/70869#c1 (10Greg Grossmeier) p:5Unprio>3Normal (setting normal for now, but if it starts causing browser test failures or otherwise, we'll bump it up) [22:28:59] 3Wikimedia Labs / 3deployment-prep (beta): Search is sometimes slow - 10https://bugzilla.wikimedia.org/70869#c2 (10Greg Grossmeier) Better graph: http://graphite.wmflabs.org/render/?width=586&height=308&_salt=1410820101.127&from=-6hours&target=deployment-prep.deployment-elastic01.loadavg.01.value&target=depl... [22:31:15] 3Wikimedia Labs / 3deployment-prep (beta): Search is sometimes slow - 10https://bugzilla.wikimedia.org/70869#c3 (10Greg Grossmeier) Bah, those graphs are relative time based (last 6 hours) and will change. Here's a static one for today from 17:30 - 23:30 UTC: http://graphite.wmflabs.org/render/?width=586&hei... [22:35:28] jeremyb: status ? [22:38:28] basically just watching puppet runs to make sure they're all working. puppet is fixing salt. i think [22:39:44] thanks [22:41:34] what's deployment-pdf01.eqiad.wmflabs ? is that ocg? [22:41:37] cscott: ? [22:41:53] should be, yes [22:42:38] this ssh banner crap is sooooo annoying [22:43:14] "If you are having access problems, please see" [22:43:19] (in a for loop...) [22:44:26] Sep 15 22:43:30 deployment-pdf01 puppet-agent[5239]: Caching catalog for i-00000396.eqiad.wmflabs [22:44:26] Sep 15 22:43:33 deployment-pdf01 puppet-agent[5239]: Could not run command from prerun_command: Cannot allocate memory - fork(2) [22:44:26] Sep 15 22:43:33 deployment-pdf01 puppet-agent[5239]: Could not run command from postrun_command: Cannot allocate memory - fork(2) [22:44:54] whoa [22:45:04] lots of grain-ensure hanging around [22:46:34] jeremyb: would reverting https://gerrit.wikimedia.org/r/#/c/160485/ still help? [22:47:18] looking [22:51:12] greg-g: helps if i look on the right host... [22:53:34] btw... [22:53:45] !log deployment-pdf01: pkill -f grain-ensure [22:53:48] Logged the message, Master [22:58:32] greg-g: so, what's the current status? still can't connect? [22:58:41] i haven't looked at jenkins [23:02:44] jeremyb: beta loads [23:02:58] ? [23:04:14] who thought it was a good idea for puppet logs to not have timestamps????????? [23:20:43] (03PS4) 10Krinkle: [WIP] job-templates: Move {name}-npm from Precise to Trusty [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/159224 (https://bugzilla.wikimedia.org/66056)