[01:35:03] (03CR) 10Krinkle: "Maybe use (or extend) the built-in PSR2 sniff for this? It has ControlStructureSpacingSniff. We do this for others as well." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/163898 (owner: 10EBernhardson) [01:36:58] (03PS2) 10Krinkle: [WIP] mwext-{ext-name}-npm: Move from Precise to Trusty [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/162691 [01:37:21] (03PS3) 10Krinkle: mwext-{ext-name}-npm: Move from Precise to Trusty [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/162691 [02:57:52] (03CR) 10Krinkle: [C: 032] "Deployed mwext-DonationInterface-npm, mwext-TemplateData-npm, mwext-UploadWizard-npm, mwext-VisualEditor-npm" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/162691 (owner: 10Krinkle) [03:01:04] (03Merged) 10jenkins-bot: mwext-{ext-name}-npm: Move from Precise to Trusty [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/162691 (owner: 10Krinkle) [03:24:46] Project beta-scap-eqiad build #23618: FAILURE in 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/23618/ [03:34:49] Yippee, build fixed! [03:34:50] Project beta-scap-eqiad build #23619: FIXED in 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/23619/ [04:11:39] Yippee, build fixed! [04:11:39] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #261: FIXED in 40 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/261/ [04:34:34] Yippee, build fixed! [04:34:35] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #26: FIXED in 37 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/26/ [04:43:47] Yippee, build fixed! [04:43:48] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #93: FIXED in 9 min 13 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/93/ [04:54:50] Yippee, build fixed! [04:54:50] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #38: FIXED in 41 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/38/ [05:15:55] kart_: now you're making me re-think the whole schedule :) [05:16:07] but for now, I must sleep. (still getting over a cold) [05:24:22] greg-g: good night! [05:24:41] greg-g: feel free. [05:30:49] Yippee, build fixed! [05:30:49] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #79: FIXED in 8 min 8 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/79/ [05:48:18] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #238: FAILURE in 46 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/238/ [07:11:02] Yippee, build fixed! [07:11:03] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #186: FIXED in 37 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/186/ [07:29:27] aharoni: could you please merge this? https://gerrit.wikimedia.org/r/#/c/163175/ [07:30:06] loogin [07:30:08] looking [07:30:13] aharoni: oops, looks like there is a merge conflict there, will fix [07:30:45] Probably because of the languages I added yesterday. [07:32:31] aharoni: I am updating all jobs so I can not mess with the code now, it should be done in a few minutes, will rebase and fix the merge conflict [07:32:48] redis problem should be solved, so we can finally run all jobs [07:44:45] (03PS8) 10Zfilipin: Make it possible to run VisualEditor screenshot job for arbitrary number of languages [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163175 [07:44:50] aharoni: ^ [07:45:02] could you please take a look if the rebase looks good? [07:45:05] and merge if it does [07:45:46] this is the change https://gerrit.wikimedia.org/r/#/c/163175/7..8/job-templates-browsertests.yaml,unified [07:47:55] I have deployed the change and started the job https://integration.wikimedia.org/ci/view/BrowserTests/view/-All/job/browsertests-VisualEditor-language-screenshot-linux-firefox/ [07:55:21] Project browsertests-VisualEditor-language-screenshot-linux-firefox » es,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 8 min 1 sec: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=es,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [07:57:47] (03CR) 10Amire80: [C: 032] Make it possible to run VisualEditor screenshot job for arbitrary number of languages [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163175 (owner: 10Zfilipin) [08:00:47] (03Merged) 10jenkins-bot: Make it possible to run VisualEditor screenshot job for arbitrary number of languages [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163175 (owner: 10Zfilipin) [08:03:03] Project browsertests-VisualEditor-language-screenshot-linux-firefox » yue,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 15 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=yue,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [08:12:27] Project browsertests-VisualEditor-language-screenshot-linux-firefox » gl,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 25 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=gl,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [08:20:10] Project browsertests-VisualEditor-language-screenshot-linux-firefox » he,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 32 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=he,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [08:28:43] Project browsertests-VisualEditor-language-screenshot-linux-firefox » nb,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 41 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=nb,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [08:32:12] (03CR) 10Tobias Gritschacher: "@Hashar: no, the tags are correct. This job should only execute the performance tests which are tagged with @performance_testing" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163129 (owner: 10Tobias Gritschacher) [08:35:31] Project browsertests-VisualEditor-language-screenshot-linux-firefox » pt,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 48 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=pt,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [08:43:06] Project browsertests-VisualEditor-language-screenshot-linux-firefox » zh-hans,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 55 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=zh-hans,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [08:58:07] Project browsertests-VisualEditor-language-screenshot-linux-firefox » cs,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 1 hr 10 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=cs,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [09:06:51] Project browsertests-VisualEditor-language-screenshot-linux-firefox » zh-hant,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 1 hr 19 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=zh-hant,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [09:14:36] Project browsertests-VisualEditor-language-screenshot-linux-firefox » sl,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 1 hr 27 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=sl,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [09:22:05] Project browsertests-VisualEditor-language-screenshot-linux-firefox » sv,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 1 hr 34 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=sv,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [09:37:31] Project browsertests-VisualEditor-language-screenshot-linux-firefox » fr,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 1 hr 50 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=fr,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [09:44:51] Project browsertests-VisualEditor-language-screenshot-linux-firefox » fi,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 1 hr 57 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=fi,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [09:52:55] Project browsertests-VisualEditor-language-screenshot-linux-firefox » ar,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 2 hr 5 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=ar,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [09:58:41] (03CR) 10Tobias Gritschacher: [C: 031] "Fine." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163857 (owner: 10Hashar) [10:00:25] Project browsertests-VisualEditor-language-screenshot-linux-firefox » uk,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 2 hr 13 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=uk,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [10:09:20] Project browsertests-VisualEditor-language-screenshot-linux-firefox » mk,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 2 hr 22 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=mk,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [10:25:50] Project browsertests-VisualEditor-language-screenshot-linux-firefox » et,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 2 hr 38 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=et,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [10:28:15] 3Wikimedia / 3Quality Assurance: browser tests should assert there are no pink error boxes on the page - 10https://bugzilla.wikimedia.org/61304#c3 (10Željko Filipin) Chris, Dan and me should be able to help you. Feel free to ping me via e-mail if you would like to pair with me in your morning (my afternoon). [10:33:32] Project browsertests-VisualEditor-language-screenshot-linux-firefox » ilo,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 2 hr 46 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=ilo,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [10:40:49] Project browsertests-VisualEditor-language-screenshot-linux-firefox » ast,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 2 hr 53 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=ast,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [10:50:18] Project browsertests-VisualEditor-language-screenshot-linux-firefox » el,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 3 hr 2 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=el,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [11:00:20] Project browsertests-VisualEditor-language-screenshot-linux-firefox » ms,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 3 hr 13 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=ms,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [11:07:58] Project browsertests-VisualEditor-language-screenshot-linux-firefox » ru,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 3 hr 20 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=ru,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [11:15:30] Project browsertests-VisualEditor-language-screenshot-linux-firefox » fa,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 3 hr 28 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=fa,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [11:22:48] Project browsertests-VisualEditor-language-screenshot-linux-firefox » de,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 3 hr 35 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=de,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [11:31:37] Project browsertests-VisualEditor-language-screenshot-linux-firefox » vi,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 3 hr 44 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=vi,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [11:40:00] Project browsertests-VisualEditor-language-screenshot-linux-firefox » nl,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 3 hr 52 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=nl,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [11:44:44] Project beta-scap-eqiad build #23668: FAILURE in 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/23668/ [11:47:05] Project browsertests-VisualEditor-language-screenshot-linux-firefox » ko,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 3 hr 59 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=ko,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [11:54:24] Project browsertests-VisualEditor-language-screenshot-linux-firefox » en,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 4 hr 7 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=en,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [11:54:43] Yippee, build fixed! [11:54:44] Project beta-scap-eqiad build #23669: FIXED in 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/23669/ [12:03:13] Project browsertests-VisualEditor-language-screenshot-linux-firefox » pl,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 4 hr 15 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=pl,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [12:11:04] Project browsertests-VisualEditor-language-screenshot-linux-firefox » it,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 4 hr 23 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=it,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [12:20:22] Project browsertests-VisualEditor-language-screenshot-linux-firefox » lb,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 4 hr 33 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=lb,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [12:27:53] Project browsertests-VisualEditor-language-screenshot-linux-firefox » ja,contintLabsSlave && UbuntuPrecise build #13: FAILURE in 4 hr 40 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-linux-firefox/LANGUAGE_SCREENSHOT_CODE=ja,label=contintLabsSlave%20&&%20UbuntuPrecise/13/ [13:40:00] 3Wikimedia / 3Continuous integration: wikidata-jenkins* instances needs puppet update - 10https://bugzilla.wikimedia.org/71411#c6 (10Antoine "hashar" Musso) And https://wikitech.wikimedia.org/wiki/Ldap_rename#Instances_in_Danger has been updated. Thank you Jan! [13:43:16] 3Wikimedia Labs / 3deployment-prep (beta): deployment-rsync01 20GB hard drive is too small - 10https://bugzilla.wikimedia.org/71431#c4 (10Antoine "hashar" Musso) It seems the root cause of the issue was the LDAP being upgraded/unreacheable intermittently over the past few days. As a result, when puppet run... [13:56:04] (03PS1) 10Hashar: Introduces {name}-tox-{toxenv}-trusty [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164065 [13:57:09] (03CR) 10Hashar: "Lets be lame and copy paste code. https://gerrit.wikimedia.org/r/164065 propose a new template suffixed -trusty and tied to Ubuntu" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/162636 (owner: 10XZise) [14:04:34] Project beta-scap-eqiad build #23682: FAILURE in 37 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/23682/ [14:06:47] (03PS2) 10Hashar: Introduces {name}-tox-{toxenv}-trusty [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164065 [14:06:57] (03PS3) 10Hashar: Introduces {name}-tox-{toxenv}-trusty [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164065 [14:11:24] (03CR) 10XZise: [C: 031] Introduces {name}-tox-{toxenv}-trusty [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164065 (owner: 10Hashar) [14:12:15] (03CR) 10Hashar: [C: 032] "python3.4 is installed on Jenkins Trusty slaves with the puppet change https://gerrit.wikimedia.org/r/#/c/164071/ ." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164065 (owner: 10Hashar) [14:14:22] !log rebased contint puppetmaster [14:14:25] Logged the message, Master [14:14:46] Yippee, build fixed! [14:14:46] Project beta-scap-eqiad build #23683: FIXED in 46 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/23683/ [14:15:28] (03Merged) 10jenkins-bot: Introduces {name}-tox-{toxenv}-trusty [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164065 (owner: 10Hashar) [14:16:30] 3Wikimedia / 3Continuous integration: hhvm 3.3.0-20140925+wmf2 has some annoying build dependency - 10https://bugzilla.wikimedia.org/71413#c1 (10Antoine "hashar" Musso) As a side effect, that breaks puppet runs on the contint Trusty instances (integration-slave1006 integration-slave1007 and integration-slave... [14:17:42] (03PS2) 10Hashar: [FEAT] Add nose34 test for pywikibot and Python 3 [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/162636 (owner: 10XZise) [14:19:47] (03PS3) 10Hashar: Add nose34 test for pywikibot and Python 3 [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/162636 (owner: 10XZise) [14:23:40] (03CR) 10Hashar: "Thank you!" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/162636 (owner: 10XZise) [14:23:46] (03CR) 10Hashar: [C: 032] Add nose34 test for pywikibot and Python 3 [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/162636 (owner: 10XZise) [14:26:54] (03Merged) 10jenkins-bot: Add nose34 test for pywikibot and Python 3 [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/162636 (owner: 10XZise) [14:39:34] (03PS2) 10XZise: [FEAT] Add nose34 test for pywikibot and Python 3 [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/162635 [14:52:07] stupid jenkins [15:10:30] 3Wikimedia Labs / 3deployment-prep (beta): deployment-rsync01 20GB hard drive is too small - 10https://bugzilla.wikimedia.org/71431#c5 (10Greg Grossmeier) Created attachment 16640 --> https://bugzilla.wikimedia.org/attachment.cgi?id=16640&action=edit disk space percent free graph So it appears that things... [15:22:32] 3Wikimedia Labs / 3deployment-prep (beta): deployment-rsync01 20GB hard drive is too small - 10https://bugzilla.wikimedia.org/71431#c6 (10Bryan Davis) 5NEW>3RESO/FIX a:3Bryan Davis (In reply to Antoine "hashar" Musso from comment #4) > It seems the root cause of the issue was the LDAP being > upgraded/... [16:03:15] Yippee, build fixed! [16:03:15] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #198: FIXED in 2 min 53 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/198/ [16:06:00] Yippee, build fixed! [16:06:01] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #198: FIXED in 5 min 49 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/198/ [16:21:31] !log Cherry-picked https://gerrit.wikimedia.org/r/#/c/163078/ into scap for beta. hhvm will be restarted on each scap. Keep your eyes open for weird problems like 503 responses that this may cause. [16:21:33] Logged the message, Master [17:29:14] !log Stopped service udp2log and started service udp2log-mw on deployment-bastion [17:29:16] Logged the message, Master [17:48:49] anyone else seeing raw html/no js/no css on beta labs right now? [17:49:26] i.e. a problem with bits? [17:55:17] chrismcmahon: working for me before and after a hard reload but labs dns has been worse than usual today for intermittent failures [17:55:53] bd808: I happen to be working on a no-javascript test for Flow, so it is not inconvenient right now. it just looks awful [18:07:27] hi. i just happened to see this line [18:07:29] modules/beta/files/monitor_fatals.rb: Net::SMTP.start('mchenry.wikimedia.org') do |smtp| [18:07:37] because we are going to decom mchenry [18:07:52] we should replace with other smtp [18:36:50] Yippee, build fixed! [18:36:51] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #41: FIXED in 6 min 51 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/41/ [18:44:28] Yippee, build fixed! [18:44:29] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #196: FIXED in 12 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/196/ [19:19:31] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #262: FAILURE in 42 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/262/ [19:33:38] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #27: FAILURE in 24 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/27/ [19:34:04] hi, y'all know that npm job in VE is pooped? [19:34:08] https://gerrit.wikimedia.org/r/#/c/164151/ [19:34:13] https://integration.wikimedia.org/ci/job/mwext-VisualEditor-npm/4690/console [19:35:10] seems to be the same error as yesterday? [19:35:12] or the day before [19:43:27] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #94: FAILURE in 9 min 48 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/94/ [19:56:52] (03CR) 10EBernhardson: "I may be misreading the ControlStructureSpacingSniff, but my reading of it is that it asserts a different thing, it asserts the spaces in" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/163898 (owner: 10EBernhardson) [20:00:40] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #39: FAILURE in 39 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/39/ [20:20:38] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #235: FAILURE in 37 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/235/ [20:26:31] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #80: FAILURE in 2 min 21 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/80/ [20:29:07] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #141: FAILURE in 1.1 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/141/ [20:33:17] 3Wikimedia / 3Quality Assurance: Move user agent assignment to Ruby gem - 10https://bugzilla.wikimedia.org/71531 (10Chris McMahon) 3NEW p:3Unprio s:3normal a:3None Three repos (Echo, Flow, MobileFrontend) are using this step: Given(/^I am using user agent "(.+)"$/) do |user_agent| @user_agent = u... [20:39:26] MatmaRex: blerg. I thought hashar fixed that job. It needs to be changed to use a different method for git cloning. [20:39:32] * bd808 looks for example patch [20:40:27] bd808: hey :) [20:40:43] bd808: which job? [20:40:50] hashar: https://integration.wikimedia.org/ci/job/mwext-VisualEditor-npm/4690/console [20:41:07] bd808: Krinkl.e said he's looking into it / will look into it, btw [20:41:24] hm [20:41:25] [22:37] I'm working on the CI breakage. Jenkins GIT layer is completely unresponsive. [20:41:25] ah thanks I was looking for that job earlier today [20:41:25] [22:37] Unknown cause, sort of out of ideas. [20:41:37] (that's four minutes ago) [20:41:38] I spent the afternoon figuring out that error: git reset: not found [20:41:38] See server admin log [20:41:40] I also e-mailed hashar [20:41:52] Krinkle: fill bugs instead please so others are aware :-] [20:42:07] so the issue is that the job got moved to Trusty which has a more recent git version [20:42:13] hmm, something going on with beta? logstash-beta is filled with connection refused messages [20:42:14] Is that not the same as the qunit changes? -- https://gerrit.wikimedia.org/r/#/c/161459/ https://gerrit.wikimedia.org/r/#/c/163837/ [20:42:16] hashar: No [20:42:21] and git 1.9.0 changed the way it handles args passed to git submodule foreach [20:42:26] hashar: I ran the command on the server directly. It works fine [20:42:39] on the same server, same directory, same git checkout hash [20:42:41] same user [20:42:48] so internally the git plugin does: git submodule foreach '"git reset"' and obviously gitreset does not exist :D [20:42:51] ebernhardson: hhvm dead? [20:42:55] * bd808 looks [20:43:06] hashar: Right [20:43:15] ebernhardson: I deployed a scap patch in beta that could be causing that [20:43:26] bd808: well, its "alive" on 01 and 03, not on 02. But nothing responding on port 9000 for either [20:43:31] Krinkle: and that is a bug in the Jenkins git plugin I am not willing to upgrade without properly testing it. Cause last time it screwed everything :-/ [20:43:32] by alive i mean in the process list [20:43:51] $ git submodule foreach --recursive "git reset --hard" [20:43:51] oh, actually i bet i have to check the 10.* addr, sec [20:43:52] works fine [20:44:06] nope, also closed [20:44:14] bd808: looks dead [20:44:20] jenkins-deploy@integration-slave1007:/mnt/jenkins-workspace/workspace/mwext-VisualEditor-npm$ git submodule foreach --recursive "git reset" --hard [20:44:27] ; /usr/lib/git-core/git-submodule: 552: /usr/lib/git-core/git-submodule: git reset: not found [20:44:31] OK, that's the one [20:44:39] ebernhardson: looking now at mw02 [20:45:15] bd808: oh, bsimmmers is logged into 02 and has hhvm.conf open [20:45:20] Krinkle: and that one should pass on Precise slaves [20:45:21] bd808: so thats probably "intentionally down" [20:45:51] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #187: FAILURE in 9.5 sec: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/187/ [20:46:22] hashar: Yeah, works in slave1003 [20:46:27] hashar: So how come this only failed now/ [20:46:36] hashar: I've been using npm on trusty for several days/weeks [20:46:40] Did we upgrade jenkins or something? [20:46:49] hashar: I noticed you enabled jgit [20:46:54] I tried to disable it, but couldn't [20:47:12] did the VE-npm job ever passed on Trusty? [20:47:14] Yes [20:47:31] https://integration.wikimedia.org/ci/job/mwext-VisualEditor-npm/4682/console [20:47:35] That's the last one passing [20:47:41] 4683 + fails [20:48:03] ebernhardson: You see Brett logged in on 02? I don't [20:48:09] so somewhere between 18 and 4 hours ago [20:48:22] bd808: my bad, he must just have a screen session left open [20:48:38] bd808: he has an `emacs hhvm.conf` process open, which i saw grepping for hhvm processes [20:48:45] but not on who, so just an old session [20:48:48] ah. old crap [20:49:18] hhvm isn't starting and it's not logging anything for me :( [20:49:27] Krinkle: ah I remember now [20:49:58] Krinkle: so on an empty workspace, git just clone and everything works fine since "git submodule foreach 'git reset'" is not invoked [20:50:19] Krinkle: but the next build occurring on a Trusty node which already has a workspace would trigger the reset and cause the bug [20:50:28] yes, when I try to do a rebuild, it passes [20:50:35] on a different node maybe? [20:50:38] no [20:51:50] ebernhardson: "syntax error, unexpected '=', expecting $end or TC_SECTION or TC_LABEL or END_OF_LINE in /etc/hhvm/fcgi/php.ini on line 5" and looks like some new patches from ori that may have caused it [20:52:55] bd808: ahha, dynamic extensions via the .ini must be new, maybe it hasn't updated the hhvm package? [20:53:14] well, i havn't been following too close it could have been in for a little while, not really sure [20:53:14] hashar: Can we disable jgit? What is it even for? [20:53:16] I tried to remove it from /configure, but that didn't do anything [20:53:18] Krinkle: I tried jgit which is a different implementation. But it does not even manage to clone a repo [20:53:34] hashar: It was still in the /configure when I opened it earlier today [20:53:39] Krinkle: it is not being used. the default is the cli one [20:53:39] I removed it. [20:53:41] OK [20:53:49] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #782: FAILURE in 23 min: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/782/ [20:53:57] So that brings us back to the first question. What changed today? [20:54:08] Did we change something abtu whether it re-clones or not? [20:54:18] like switching from wipe workspace to git clean? [20:54:37] the job uses submodule, has been moved to Trusty which uses git 1.9.0+ which changed how it handles args passed to submodule foreach [20:54:42] quick fix: migrate back to Precise [20:54:48] 3Wikimedia Labs / 3deployment-prep (beta): login busted - 10https://bugzilla.wikimedia.org/71532 (10Chris McMahon) 3NEW p:3Unprio s:3critic a:3None http://en.wikipedia.beta.wmflabs.org/wiki/Special:UserLogin is 503 [20:54:54] hashar: no, that's not a fix. The code requires node 0.10 [20:55:03] hashar: the jobs has been on trusty for over a week [20:55:05] then you are screwed till the bug is figure dout [20:55:37] it only got changed around 2am https://integration.wikimedia.org/ci/job/mwext-VisualEditor-npm/jobConfigHistory/showDiffFiles?timestamp1=2014-09-16_00-14-39×tamp2=2014-10-01_01-55-34 [20:56:12] you might be confusing with another ve job Iguess [20:57:00] 3Wikimedia Labs / 3deployment-prep (beta): login busted - 10https://bugzilla.wikimedia.org/71532#c1 (10Erik Bernhardson) Currently all pages on beta are down, the hhvm instances are in a restart loop. It might be related to https://gerrit.wikimedia.org/r/#/c/164160/ , still investigating. [20:57:40] !log hhvm servers broken because of I5f9b5c4e452e914b33313d0774fb648c1cdfe7ad [20:57:44] Logged the message, Master [20:59:28] ebernhardson: Ori did it with https://gerrit.wikimedia.org/r/#/c/164160/ I pinged him in another chanel. I'm not waisting my afternoon to clean it up. [20:59:37] bd808: ok, thanks for investigating [20:59:54] we have a separate labs instance can continue our testing on, just means all the browser tests are broken :) livable [21:00:12] Find ori and make him fix it :) [21:00:53] self merge 13 minutes after posting + beta broken == caning offense [21:02:53] it's not just the browser tests, login page is 503 right now [21:03:32] * chrismcmahon is a few minutes behind [21:04:30] hashar: are there / what are; the main problems with a newer GIT Plugin version? [21:04:33] chrismcmahon: hhvm is hosed in beta due to bugs with https://gerrit.wikimedia.org/r/#/c/164160/ [21:04:33] We're quite far behind [21:04:50] bd808: yep, I'm caught up now [21:06:41] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #199: FAILURE in 3 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/199/ [21:08:07] bd808: does ori know he trashed beta labs again? [21:08:27] I told him in #mediawiki-core; no response [21:09:01] Feel free to track him down and tell him how it makes you feel ;) [21:09:13] Krinkle: I might have filled a bug at the time, but it basically screwed up the whole setup [21:10:57] hashar: https://github.com/jenkinsci/git-client-plugin/blob/4362cf66b19519a558984d33f7e4272131087b6a/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L773-L781 [21:12:24] https://github.com/jenkinsci/git-client-plugin/blob/git-client-1.0.6/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L401-L412 [21:12:48] git plugin 1.5.0 (which we use) uses git-client 1.0.6 [21:13:07] yeah [21:13:26] and apparently add an args "git reset" which is interpreted differently by the git version on Trusty [21:13:36] at that point I did my audio and came back home [21:14:08] A safe bet would be to attempt: args.add("git") args.add("reset") [21:14:36] and patch 1.5.0 with that, rebuild the plugin and give it a try on some labs instance [21:14:42] (or in prod if feeling brave :D ) [21:15:17] the submodule foreach argument handling change in git 1.9.0 is described as the first paragraph in https://raw.githubusercontent.com/git/git/master/Documentation/RelNotes/1.9.0.txt [21:15:36] hashar: indeed, the code is just wrong and happens to work on the old git version [21:16:02] which is my .plan for tomorrow morning unless you wanna handle that scary thing :D [21:16:25] note, I even have not found the time to fill a bug nor to write something to actually reproduce the issue with some shell scirpt [21:16:28] I want to fix this know. I feel brave enough to do a documented live hack [21:16:34] Just trying to find where [21:16:45] the code you pointed above seems relevant [21:16:45] I've reproduced it. [21:16:48] that was my assumption [21:16:56] * Krinkle files bug [21:17:16] i found it / debugged it when migrating a pywikibot/core job to Trusty (they need python 3.4) [21:18:31] Krinkle: the VE-qunit job probably suffers from the same issue . Which caused me to craft a dedicated job based on Zuul cloner https://gerrit.wikimedia.org/r/#/c/163837/ [21:19:44] hashar: It does suffer, and as of 6 hours ago, we'v enot been able ot merge or test anything [21:21:32] 3Wikimedia / 3Continuous integration: Jenkins: The GIT Plugin is broken for submodules ("git-submodule: git reset: not found") - 10https://bugzilla.wikimedia.org/71533 (10Krinkle) 3NEW p:3Unprio s:3critic a:3None Jobs running on Ubuntu Trusty for repositories with submodules are currently broken due... [21:21:57] Krinkle: lengthy explanation in git commit https://github.com/git/git/commit/1c4fb136dbad762c9c4350ee79c3474ae8037587 :D [21:22:15] hashar: That's fine. I just need to know how to edit it [21:22:22] it's all precompiled .jpi files, right? [21:22:27] Krinkle: well at worth the job can be reverted back to Precise and the nodejs 10 code reverted till CI fix the bug [21:22:32] Or does jenkins download plugins from source and compile locally? [21:22:46] hashar: not gonna happen, too much baggage. Waste of time. [21:22:48] it downloads .jpi from jenkins upstream [21:22:52] Right [21:22:58] so I need to compile on my local machine.. [21:23:08] you need to clone jenkinsci/git-client-plugin or something [21:23:20] checkout the proper tag, patch and run maven [21:23:25] which *might* work [21:23:56] probably better to do it on a Precise instance instead of your laptop [21:25:09] Krinkle: anyway been up for 18 hours straight and will wake up in less than 7 hours. So gotta sleep for real. Poke the bug with whatever achievement you did and I will follow up [21:27:17] hashar: intersting, we actually run git client pligin 1.4.6 [21:27:19] not the one the Git plugin usually provides [21:27:43] https://github.com/jenkinsci/git-client-plugin/blob/git-client-1.4.6/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L556-L567 [21:27:49] accoding to https://integration.wikimedia.org/ci/pluginManager/ [21:28:08] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #199: FAILURE in 1.6 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/199/ [21:28:32] hashar: before you leave, what does "disabling puppet" mean? I mean technically. [21:29:29] (03CR) 10Hashar: "We have a bug in Jenkins git plugin which cause it to choke when cleaning submodules on Trusty (which has git version 1.9+). https://bugzi" [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/162635 (owner: 10XZise) [21:31:45] 3Wikimedia / 3Continuous integration: Jenkins: The GIT Plugin is broken for submodules ("git-submodule: git reset: not found") - 10https://bugzilla.wikimedia.org/71533#c1 (10Krinkle) Release notes of git 1.9 detailing this change in git-submodule semantics: https://raw.githubusercontent.com/git/git/master/D... [21:36:00] 3Wikimedia / 3Continuous integration: Jenkins: The GIT Plugin is broken for submodules ("git-submodule: git reset: not found") - 10https://bugzilla.wikimedia.org/71533#c2 (10Antoine "hashar" Musso) Created attachment 16642 --> https://bugzilla.wikimedia.org/attachment.cgi?id=16642&action=edit git submodule... [21:43:18] 3Wikimedia / 3Continuous integration: Jenkins: The GIT Plugin is broken for submodules ("git-submodule: git reset: not found") - 10https://bugzilla.wikimedia.org/71533#c3 (10Antoine "hashar" Musso) Created attachment 16643 --> https://bugzilla.wikimedia.org/attachment.cgi?id=16643&action=edit IRC log of a... [21:43:35] Krinkle: ask ops [21:43:42] hashar: https://github.com/wikimedia/git-client-plugin/commit/5d4c94051 [21:43:47] git-client-1.4.6+wmf1 [21:44:07] if args.add() supports multiple arguments that can do :] [21:44:18] hashar: It's used that way in other places already [21:44:25] args.add("submodule", "foreach"); [21:44:25] \O/ [21:44:39] hashar: who in ops do I ask (and what exactly?) [21:44:59] sorry my brain is well past its burnout time [21:45:18] Krinkle: puppet as an agent running locally on each instance/server [21:45:23] it runs every 30 minutes [21:45:31] which means fetching the latest conf from the puppet master [21:45:40] puppet master which is updated from time to time / continuously [21:45:44] puppet agent --disable [21:46:00] sometime you want to prevent puppet from running updates so you can disable it [21:46:05] which is evil :] [21:46:17] 3Wikimedia / 3Continuous integration: Jenkins: The GIT Plugin is broken for submodules ("git-submodule: git reset: not found") - 10https://bugzilla.wikimedia.org/71533#c4 (10Krinkle) (In reply to Krinkle from comment #0) > Jobs running on Ubuntu Trusty for repositories with submodules are currently > broken... [21:46:32] hashar: So you currently disabled puppet on all slaves? Can you say why next time? Or was that only temporary? [21:46:51] mutante: we should wrap that command to force a message to be given in comment and log it somewhere together with $HOST and $USER :] [21:47:26] Krinkle: I might have done it with Andrew on friday while he was testing the LDAP migration [21:47:26] hashar: sounds good, because there was once a bug where puppet disables itself and no admin actually did it :P [21:47:35] Krinkle: I am pretty sure we reenabled it after [21:47:54] mutante: puppet might have such a reporting build in [21:48:28] hashar: we already have monitoring for puppet being disabled [21:48:38] mutante: yeah puppet agent --logdest syslog [21:48:39] :D [21:49:06] hashar: you logged that you disabled it [21:49:10] I assume it was enabled again? [21:50:38] hopefully :] [21:50:47] I ran it today on all slaves so it must be running [21:51:39] or yuvi monitoring would probably complain here [21:51:43] anyway sleep for real; [21:51:55] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #194: FAILURE in 2 sec: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/194/ [21:51:56] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #143: FAILURE in 1 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/143/ [22:02:49] Yippee, build fixed! [22:02:50] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #783: FIXED in 10 min: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/783/ [22:04:00] beta fixed; the puppet master needed to be restarted [23:10:00] 3Wikimedia / 3Continuous integration: Jenkins: The GIT Plugin is broken for submodules ("git-submodule: git reset: not found") - 10https://bugzilla.wikimedia.org/71533#c5 (10Krinkle) 5NEW>3RESO/FIX https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/Jenkins_Plugin Deployed the .jpi file to Jen... [23:10:17] 3Wikimedia / 3Continuous integration: Jenkins: The GIT Plugin is broken for submodules ("git-submodule: git reset: not found") - 10https://bugzilla.wikimedia.org/71533 (10Krinkle) p:5Unprio>3Highes a:3Krinkle