[00:00:10] Krinkle: "fixed" by me manually restarting mysql with "sudo service mysql start" via salt [00:00:36] REL1_23 still dirty https://gerrit.wikimedia.org/r/#/c/274757/ [00:00:44] missing composer.json [00:00:44] right.. [00:00:47] phpunit [00:01:05] uh yeah [00:01:08] legoktm: would anything bad happen if we add a simple require-dev section only composer.json to rel1-23? [00:01:20] you'd break SMW [00:01:41] is that bad? [00:01:44] OK [00:01:49] I'm curious how you know that [00:01:59] Uhhh, let me find the giant bz ticket [00:02:02] (and in what way it'd break) [00:02:20] Is there a jenkins job for SMW that copies stuff into composer.json pre-1.25 because it could? [00:04:52] legoktm: If it only applies to release, we can still exclude composer.json from REL1_23 that way [00:04:58] and we should because of existing installs [00:05:01] that may've taken the spot [00:05:37] Krinkle: break might be a little too strong. But basically in 1.23, installation of extensions via composer was done via creating a composer.json that required your extensions (SMW), so there was no core composer.json because you'd end up with a dirty git repo. Except that meant composer couldn't be used for mangaging deps, which is why we made the composer-merge-plugin in 1.25 so composer.local.json could be used to extend the deps without [00:05:37] ending up with a dirty git repo since we don't commit that [00:05:51] https://gerrit.wikimedia.org/r/#/c/132788/ and https://phabricator.wikimedia.org/T67188 [00:06:02] Right [00:06:36] so adding a composer.json to git in 1.23 would break the no dirty git stuff, which makes upgrading hard, and that's already hard enough without fighting git.... [00:08:22] (03PS1) 10Krinkle: mw-fetch-composer-dev: Add fallback for REL1_23 [integration/jenkins] - 10https://gerrit.wikimedia.org/r/274849 [00:08:30] nvm, fuck it [00:08:32] there's an easier way [00:08:43] We used to magically provide it by CI [00:08:50] so whether one magic way or another doesn't really matter [00:09:00] lol good enough [00:09:22] (03CR) 10Krinkle: [C: 032] mw-fetch-composer-dev: Add fallback for REL1_23 [integration/jenkins] - 10https://gerrit.wikimedia.org/r/274849 (owner: 10Krinkle) [00:10:11] (03Merged) 10jenkins-bot: mw-fetch-composer-dev: Add fallback for REL1_23 [integration/jenkins] - 10https://gerrit.wikimedia.org/r/274849 (owner: 10Krinkle) [00:11:09] !log salt -v --show-timeout '*slave*' cmd.run "bash -c 'cd /srv/deployment/integration/slave-scripts; git pull'" [00:11:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:15:13] legoktm: OK. Everything back to "normal" now? [00:15:24] (aside from the mysterious REL1_26 js failure) [00:15:28] close enough? :P [00:15:47] there's T128802 but I didn't want to deal with that today [00:15:47] T128802: integration-slave-precise-1001 needs to be rebuilt (doesn't have DNS) - https://phabricator.wikimedia.org/T128802 [00:15:55] I already depooled it from jenkins [00:15:56] just depooled? [00:15:58] ok [00:18:02] * legoktm goes afk for a bit [00:25:16] ugh, it still doesn't work because we're needlessly checking out mediawiki/vendor which doesn't have REL1_23 so it gets master which, even if you just add phpunit 3.7, will still fail because vendor master composer.json says we require php 5.5 [01:05:43] (03PS1) 10Arlolra: Set $wgShowExceptionDetails = true; in LocalSettings.php [integration/config] - 10https://gerrit.wikimedia.org/r/274863 [01:08:35] (03CR) 10Arlolra: "See https://gerrit.wikimedia.org/r/#/c/274861/" [integration/config] - 10https://gerrit.wikimedia.org/r/274863 (owner: 10Arlolra) [01:14:39] (03CR) 10Krinkle: [C: 04-1] "This is for web-based testing (qunit and browsertests). If you want to change the general Jenkins settings, go to integration/jenkins.git," [integration/config] - 10https://gerrit.wikimedia.org/r/274863 (owner: 10Arlolra) [01:15:26] (03Abandoned) 10Arlolra: Set $wgShowExceptionDetails = true; in LocalSettings.php [integration/config] - 10https://gerrit.wikimedia.org/r/274863 (owner: 10Arlolra) [01:17:25] (03Restored) 10Arlolra: Set $wgShowExceptionDetails = true; in LocalSettings.php [integration/config] - 10https://gerrit.wikimedia.org/r/274863 (owner: 10Arlolra) [01:18:42] (03CR) 10Arlolra: "And these settings already exist in integration-jenkins, no?" [integration/config] - 10https://gerrit.wikimedia.org/r/274863 (owner: 10Arlolra) [01:22:08] (03CR) 10Krinkle: "The macro being modified here (expose-localhost) is run when constructing qunit or browsertest builds." [integration/config] - 10https://gerrit.wikimedia.org/r/274863 (owner: 10Arlolra) [01:22:13] (03CR) 10Krinkle: "Are you on IRC?" [integration/config] - 10https://gerrit.wikimedia.org/r/274863 (owner: 10Arlolra) [01:23:09] (03CR) 10Krinkle: "Indeed. wikimedia/integration-jenkins – 00_dev_settings.php already sets $wgShowExceptionDetails = true; So this job must be bypassing o" [integration/config] - 10https://gerrit.wikimedia.org/r/274863 (owner: 10Arlolra) [01:25:45] (03CR) 10Krinkle: "parsoidsvc-php-parsertests is defined in jjb/parsoidsvc.yaml. Looking closely it seems this job, contrary to all others everywhere is doin" [integration/config] - 10https://gerrit.wikimedia.org/r/274863 (owner: 10Arlolra) [01:31:35] Krinkle: that job definitely clones mediawiki/vendor, and it uses a custom clonemap thing [01:34:41] legoktm: Which one? [01:34:47] legoktm: parsoid or REL1_23 [01:34:58] parsoid [01:35:34] it uses composer-update afaik [01:35:36] not vendor [01:35:37] which is fine [01:38:41] (03CR) 10Arlolra: "Sorry, not on IRC today but thanks for looking into it." [integration/config] - 10https://gerrit.wikimedia.org/r/274863 (owner: 10Arlolra) [02:47:25] (03PS1) 10Krinkle: Don't run rake-jessie on REL1_25 or REL1_26 of VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/274889 [02:47:35] (03CR) 10Krinkle: [C: 032] Don't run rake-jessie on REL1_25 or REL1_26 of VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/274889 (owner: 10Krinkle) [02:48:31] (03Merged) 10jenkins-bot: Don't run rake-jessie on REL1_25 or REL1_26 of VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/274889 (owner: 10Krinkle) [02:49:05] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/274889 [02:49:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [03:13:35] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #895: 04FAILURE in 23 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/895/ [03:34:29] 10Continuous-Integration-Infrastructure: Jenkins: Assert no PHP errors (notices, warnings) were raised or exceptions were thrown - https://phabricator.wikimedia.org/T50002#2087404 (10Catrope) [04:24:47] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #737: 04FAILURE in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/737/ [06:29:43] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: puppet fail [06:33:23] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:34:01] Yippee, build fixed! [06:34:01] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #34: 09FIXED in 7 min 0 sec: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/34/ [06:54:23] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #746: 04FAILURE in 29 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/746/ [06:57:18] 10Beta-Cluster-Infrastructure, 10Staging, 10DBA, 3Collaboration-Team-Current, and 2 others: Use External Store on Beta Cluster - https://phabricator.wikimedia.org/T95871#2087608 (10jcrespo) I can fix production, but I need the original script for that (even if it is non-working). I will not be able to find... [08:27:01] 10Beta-Cluster-Infrastructure, 10Staging, 10DBA, 3Collaboration-Team-Current, and 2 others: Use External Store on Beta Cluster - https://phabricator.wikimedia.org/T95871#2087771 (10hashar) The `wikiadmin_pass` was provisioned via a puppet erb template: ``` #! /bin/bash echo -n "<%= @wikiadmin_pass %>" ```... [08:32:03] !sal [08:32:03] https://tools.wmflabs.org/sal/releng [08:38:41] 10Continuous-Integration-Infrastructure: integration-slave-precise-1001 needs to be rebuilt (doesn't have DNS) - https://phabricator.wikimedia.org/T128802#2087786 (10hashar) a:3hashar ``` lang=irc [22:37:03] hm... [22:37:22] it doesn’t have dns. Which... [22:37:50] 10Continuous-Integration-Infrastructure: integration-slave-precise-1001 needs to be rebuilt (doesn't have DNS) - https://phabricator.wikimedia.org/T128802#2087806 (10hashar) 5Open>3Resolved Actually I am going to just remove the instance. The reason is we barely have any jobs still tied to Precise beside th... [08:42:54] !log CI deleting integration-slave-precise-1001 (2 executors). It is not in labs DNS which causes bunch of issues, no need for the capacity anymore. T128802 [08:42:55] T128802: integration-slave-precise-1001 needs to be rebuilt (doesn't have DNS) - https://phabricator.wikimedia.org/T128802 [08:42:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [08:47:02] PROBLEM - Host integration-slave-precise-1001 is DOWN: CRITICAL - Host Unreachable (10.68.17.80) [09:02:08] ^^^ I have deleted it [09:09:58] 10MediaWiki-Codesniffer, 10Possible-Tech-Projects: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#2087867 (10Aklapper) @lucky: That is off-topic for this task and not handled here. Please also see [[ https://www.mediawiki.org/wiki/Outreach_programs/Life_of_a_successful... [09:19:22] 10MediaWiki-Codesniffer, 10Possible-Tech-Projects: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#2087884 (10Legoktm) [09:24:33] 10MediaWiki-Codesniffer, 10Possible-Tech-Projects: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#2087895 (10Legoktm) I updated the task description. @Addshore, @EBernhardson, are you both still up for mentoring this? (I have another project or two I'd like to mentor i... [10:19:32] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling: DIB images have /puppet/hieradata files that are empty - https://phabricator.wikimedia.org/T128846#2087975 (10hashar) [10:24:32] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling: DIB images have /puppet/hieradata files that are empty - https://phabricator.wikimedia.org/T128846#2087993 (10hashar) That is definitely a fault in the disk image provisioning. The way I have set it up is the build host as a copy of git... [10:33:36] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling: DIB images have /puppet/hieradata files that are empty - https://phabricator.wikimedia.org/T128846#2088008 (10hashar) On an instance: ``` $ find / -inum 6059 2> /dev/null /puppet/hieradata/eqiad.yaml /sys/module/libata/parameters/acpi_gt... [11:52:14] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer: browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce fails with "757: unexpected token" - https://phabricator.wikimedia.org/T128079#2088213 (10zeljkofilipin) a:3zeljkofilipin [11:52:58] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer: browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce fails with "757: unexpected token" - https://phabricator.wikimedia.org/T128079#2063388 (10zeljkofilipin) Looks like ssl causes this: http://stackoverflow.com/questions/207101... [12:04:04] 10Browser-Tests-Infrastructure, 5Release-Engineering-Epics, 7Epic, 13Patch-For-Review, 7Tracking: Fix scenarios that fail at en.wikipedia.beta.wmflabs.org or do not run them daily - https://phabricator.wikimedia.org/T94150#2088221 (10zeljkofilipin) [12:06:27] 7Browser-Tests, 10VisualEditor, 13Patch-For-Review: Fix language screenshots browser test on VisualEditor - https://phabricator.wikimedia.org/T119482#2088224 (10zeljkofilipin) a:5Vikassy>3zeljkofilipin [12:14:24] Yippee, build fixed! [12:14:24] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #747: 09FIXED in 28 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/747/ [12:18:48] Yippee, build fixed! [12:18:48] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #896: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/896/ [12:38:00] jzerebecki: is there a task already for wikidata browser tests failing with "Build ​timed ​out ​(after ​240 ​minutes). ​Marking ​the ​build ​as ​aborted."? [12:38:11] I have looked in phab, did not find it [12:38:59] I will create one now, so feel free to merge it with the original, if it exists [12:40:40] 10Browser-Tests-Infrastructure, 10Wikidata: Wikidata browser test jobs fail with "Build ​timed ​out ​(after ​240 ​minutes). ​Marking ​the ​build ​as ​aborted." - https://phabricator.wikimedia.org/T128857#2088274 (10zeljkofilipin) [12:42:23] 10Browser-Tests-Infrastructure, 10Wikidata: Wikidata browser test jobs fail with "Build ​timed ​out ​(after ​240 ​minutes). ​Marking ​the ​build ​as ​aborted." - https://phabricator.wikimedia.org/T128857#2088287 (10zeljkofilipin) [12:43:13] 10Browser-Tests-Infrastructure, 10Wikidata: Wikidata browser test jobs fail with "Build ​timed ​out ​(after ​240 ​minutes). ​Marking ​the ​build ​as ​aborted." - https://phabricator.wikimedia.org/T128857#2088274 (10zeljkofilipin) [12:43:16] 10Browser-Tests-Infrastructure, 5Release-Engineering-Epics, 7Epic, 13Patch-For-Review, 7Tracking: Fix scenarios that fail at en.wikipedia.beta.wmflabs.org or do not run them daily - https://phabricator.wikimedia.org/T94150#2088288 (10zeljkofilipin) [12:44:43] 10Browser-Tests-Infrastructure, 5Release-Engineering-Epics, 7Epic, 13Patch-For-Review, 7Tracking: Fix scenarios that fail at en.wikipedia.beta.wmflabs.org or do not run them daily - https://phabricator.wikimedia.org/T94150#1674592 (10zeljkofilipin) [12:45:21] 10Browser-Tests-Infrastructure, 10Wikidata: Wikidata browser test jobs fail with "Build ​timed ​out ​(after ​240 ​minutes). ​Marking ​the ​build ​as ​aborted." - https://phabricator.wikimedia.org/T128857#2088274 (10zeljkofilipin) [12:45:23] 7Browser-Tests, 10Wikidata, 7Tracking: [tracking] make Wikidata browsertests non-flaky - https://phabricator.wikimedia.org/T92619#2088295 (10zeljkofilipin) [12:47:44] (03PS1) 10Zfilipin: WIP Increase job timeout for Wikidata browser tests from 240 to 300 minutes [integration/config] - 10https://gerrit.wikimedia.org/r/274932 (https://phabricator.wikimedia.org/T128857) [12:50:05] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review: Wikidata browser test jobs fail with "Build ​timed ​out ​(after ​240 ​minutes). ​Marking ​the ​build ​as ​aborted." - https://phabricator.wikimedia.org/T128857#2088274 (10zeljkofilipin) p:5Triage>3Normal [12:51:49] (03PS2) 10Zfilipin: WIP Increase job timeout for Wikidata browser tests from 240 to 300 minutes [integration/config] - 10https://gerrit.wikimedia.org/r/274932 (https://phabricator.wikimedia.org/T128857) [12:56:52] (03CR) 10Zfilipin: "I have updated the jobs and started them. We will know if the new timeout (300 minutes) is enough in 5 hours. :)" [integration/config] - 10https://gerrit.wikimedia.org/r/274932 (https://phabricator.wikimedia.org/T128857) (owner: 10Zfilipin) [12:57:13] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review: Wikidata browser test jobs fail with "Build ​timed ​out ​(after ​240 ​minutes). ​Marking ​the ​build ​as ​aborted." - https://phabricator.wikimedia.org/T128857#2088331 (10zeljkofilipin) I have updated the jobs and started them. We will know if t... [13:06:59] Yippee, build fixed! [13:06:59] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #739: 09FIXED in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/739/ [13:09:40] zeljkof: that is a new failure [13:10:05] jzerebecki: really? I thought it was happening for a while [13:10:25] did not look back in job history, looks like I have assumed incorrectly [13:12:48] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling: DIB images have /puppet/hieradata files that are empty - https://phabricator.wikimedia.org/T128846#2088351 (10hashar) The reference image created by dib is `ci-jessie-wikimedia`. Nodepool boot it up, run a setup script to refresh what wa... [13:18:25] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review: Wikidata browser test jobs fail with "Build ​timed ​out ​(after ​240 ​minutes). ​Marking ​the ​build ​as ​aborted." - https://phabricator.wikimedia.org/T128857#2088357 (10JanZerebecki) I think they never took that long. Maybe a bug is causing th... [13:20:43] zeljkof: i'd have to relativate my new: it wasn't the case a few month ago when I last looked [13:21:02] jzerebecki: I will investigate myself [13:21:12] thx [13:22:42] 10Browser-Tests-Infrastructure: mediawiki_selenium raises MediawikiSelenium::ConfigurationError if a configuration in environments.yml is empty string - https://phabricator.wikimedia.org/T128860#2088358 (10zeljkofilipin) [13:29:05] (03PS1) 10Zfilipin: WIP MediawikiSelenium::ConfigurationError is raised if a configuration is empty string [selenium] - 10https://gerrit.wikimedia.org/r/274938 (https://phabricator.wikimedia.org/T128860) [13:29:57] !log Nodepool Image ci-jessie-wikimedia-1457097785 in wmflabs-eqiad is ready [13:30:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:31:36] (03CR) 10jenkins-bot: [V: 04-1] WIP MediawikiSelenium::ConfigurationError is raised if a configuration is empty string [selenium] - 10https://gerrit.wikimedia.org/r/274938 (https://phabricator.wikimedia.org/T128860) (owner: 10Zfilipin) [13:32:36] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling: DIB images have /puppet/hieradata files that are empty - https://phabricator.wikimedia.org/T128846#2088377 (10hashar) On labnodepool I have hacked the setup script to count empty files before and after the `git -C /puppet pull` and the c... [13:58:05] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review: Wikidata browser test jobs fail with "Build ​timed ​out ​(after ​240 ​minutes). ​Marking ​the ​build ​as ​aborted." - https://phabricator.wikimedia.org/T128857#2088405 (10zeljkofilipin) Debugging is harder since cucumber pretty formatter is disa... [14:02:22] (03PS1) 10Hashar: nodepool: setup_node now announce start and end [integration/config] - 10https://gerrit.wikimedia.org/r/274943 [14:02:23] 10Browser-Tests-Infrastructure, 13Patch-For-Review: mediawiki_selenium raises MediawikiSelenium::ConfigurationError if a configuration in environments.yml is empty string - https://phabricator.wikimedia.org/T128860#2088413 (10zeljkofilipin) [14:02:56] 10Browser-Tests-Infrastructure, 13Patch-For-Review: mediawiki_selenium raises MediawikiSelenium::ConfigurationError if a configuration in environments.yml is empty string - https://phabricator.wikimedia.org/T128860#2088358 (10zeljkofilipin) [14:03:42] 10Browser-Tests-Infrastructure, 13Patch-For-Review: mediawiki_selenium raises MediawikiSelenium::ConfigurationError if a configuration in environments.yml is empty string - https://phabricator.wikimedia.org/T128860#2088358 (10zeljkofilipin) p:5Triage>3Normal [14:04:27] (03CR) 10Hashar: [C: 032] nodepool: setup_node now announce start and end [integration/config] - 10https://gerrit.wikimedia.org/r/274943 (owner: 10Hashar) [14:05:36] (03Merged) 10jenkins-bot: nodepool: setup_node now announce start and end [integration/config] - 10https://gerrit.wikimedia.org/r/274943 (owner: 10Hashar) [14:14:59] (03PS2) 10Zfilipin: WIP Fix browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce [integration/config] - 10https://gerrit.wikimedia.org/r/273234 (https://phabricator.wikimedia.org/T128079) [14:15:11] (03PS3) 10Zfilipin: WIP Fix browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce [integration/config] - 10https://gerrit.wikimedia.org/r/273234 (https://phabricator.wikimedia.org/T128079) [14:20:40] (03Abandoned) 10Zfilipin: WIP Fix browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce [integration/config] - 10https://gerrit.wikimedia.org/r/273234 (https://phabricator.wikimedia.org/T128079) (owner: 10Zfilipin) [14:34:21] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce-T128731 build #12: 04FAILURE in 7 min 38 sec: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce-T128731/12/ [14:36:29] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling: DIB images have /puppet/hieradata files that are empty - https://phabricator.wikimedia.org/T128846#2088453 (10hashar) I have downloaded a snapshot image and looked at some random files, they are empty: ``` $ virt-ls --extra-stats -l -R... [14:49:50] (03PS1) 10Hashar: dib: fully clone /puppet [integration/config] - 10https://gerrit.wikimedia.org/r/274954 (https://phabricator.wikimedia.org/T128846) [14:57:17] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox-T119482 » en,contintLabsSlave && UbuntuTrusty build #1: 04FAILURE in 43 sec: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox-T119482/LANGUAGE_SCREENSHOT_CODE=en,label=contintLabsSlave%20&&%20UbuntuTrusty/1/ [15:03:20] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 13Patch-For-Review: DIB images have /puppet/hieradata files that are empty - https://phabricator.wikimedia.org/T128846#2088514 (10hashar) ``` 927M Mar 1 11:15 image-jessie-20160301T100721Z.qcow2 1020M Mar 4 16:02 image-jessie-2016030... [15:12:19] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 13Patch-For-Review: DIB images have /puppet/hieradata files that are empty - https://phabricator.wikimedia.org/T128846#2088520 (10hashar) I have rebuild an image with the patch above `image-t128846`. Will see what happens over the week... [15:12:34] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 13Patch-For-Review: DIB images have /puppet/hieradata files that are empty - https://phabricator.wikimedia.org/T128846#2088521 (10hashar) p:5Triage>3Unbreak! [15:16:35] 10Continuous-Integration-Config, 10Mathoid, 13Patch-For-Review, 15User-mobrovac: Enable librsvg-dev on jenkins - https://phabricator.wikimedia.org/T128280#2088540 (10hashar) Kind of blocked on {T128846} . Random discussion is happening on https://gerrit.wikimedia.org/r/#/c/274675/ [15:22:13] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer, 13Patch-For-Review: browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce fails with "757: unexpected token" - https://phabricator.wikimedia.org/T128079#2088557 (10zeljkofilipin) Reproduced on my machine: ``` $ MEDIAWIKI_ENV... [15:51:17] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer, 13Patch-For-Review: browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce fails with "757: unexpected token" - https://phabricator.wikimedia.org/T128079#2088596 (10zeljkofilipin) Working fine when targeting beta: ``` $ MEDIA... [15:53:04] RECOVERY - Puppet failure on integration-slave-trusty-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [15:56:15] (03CR) 10Paladox: [C: 031] Extension APC was discontinued [integration/config] - 10https://gerrit.wikimedia.org/r/274911 (owner: 10Siebrand) [16:11:40] legoktm: Or Krinkle would you be able to review https://integration.wikimedia.org/zuul/ please. [16:11:58] Woops wrong link this one https://gerrit.wikimedia.org/r/#/c/264333/ please. [16:12:14] hello, could I get access to the deployment project in labs? I'm trying to access deployment-eventlogging03.eqiad.wmflabs as https://wikitech.wikimedia.org/wiki/User:Nschaaf [16:18:55] schana: I don't think we've met? [16:19:32] greg-g: we may have been in a meeting together, but I don't remember. [16:20:14] Luca or? [16:20:29] Nathaniel [16:21:01] ahhhhh [16:21:09] research, not analytics :) [16:21:17] yep [16:22:17] * greg-g logs in [16:23:31] schana: now try [16:23:57] greg-g: it works. thanks! [16:24:03] np [16:25:33] (03PS1) 10Zfilipin: WIP T128079 [ruby/api] - 10https://gerrit.wikimedia.org/r/274973 [16:25:55] (03PS2) 10Zfilipin: WIP T128079 [ruby/api] - 10https://gerrit.wikimedia.org/r/274973 (https://phabricator.wikimedia.org/T128079) [16:28:16] (03PS1) 10Zfilipin: WIP T128079 [selenium] - 10https://gerrit.wikimedia.org/r/274975 (https://phabricator.wikimedia.org/T128079) [16:38:53] (03CR) 10Krinkle: [C: 04-1] "We usually set 'archived' on these to inform any future patch creator." [integration/config] - 10https://gerrit.wikimedia.org/r/274911 (owner: 10Siebrand) [16:40:17] paladox: You're familiar with 'archived' template we used before? [16:40:27] Krinkle yes [16:40:33] paladox: I'm asking because you +1'ed [16:40:46] Just making sure you actually look at the commit before doing that. +1 does not mean "mark as read". [16:41:00] Krinkle yes because the repo is being archived so i agree witht he patch. [16:41:11] Except the patch doesn't do that [16:41:56] 10Beta-Cluster-Infrastructure: Recent deploys to beta failed: - https://phabricator.wikimedia.org/T128492#2088720 (10Luke081515) [16:41:58] 10Beta-Cluster-Infrastructure: Recent deploys to beta failed - https://phabricator.wikimedia.org/T128493#2088721 (10Luke081515) [16:42:10] 10Beta-Cluster-Infrastructure: Recent deploys to beta failed: - https://phabricator.wikimedia.org/T128492#2076674 (10Luke081515) I don't know, why this task was created two times. [16:42:17] paladox: Do you still agree with the patch? [16:42:50] Krinkle i need to have a look at it again. Im going to use chrome since internet explorer keeps crashing when reviewing changes. [16:43:42] Krinkle ive looked again no i doint aggree with that. [16:43:48] okay :) [16:44:05] (03CR) 10Paladox: [C: 04-1] "Please use template archived instead." [integration/config] - 10https://gerrit.wikimedia.org/r/274911 (owner: 10Siebrand) [16:44:58] Krinkle sorry i didnt look clearly. Just it kept crashing the browser and was very slow. Chrome loaded it fast so ive changed my review to -1. [16:46:43] Krinkle could you merge https://gerrit.wikimedia.org/r/#/c/274296/ and https://gerrit.wikimedia.org/r/#/c/274302/ please. I want to see if both of those patches will fix an issue where flow and thanks fail the generic tests with flow and thanks failing with an echo error and thanks erroring with a flow error too. [16:53:47] (03PS2) 10Paladox: Extension APC was discontinued [integration/config] - 10https://gerrit.wikimedia.org/r/274911 (owner: 10Siebrand) [16:53:52] Krinkle ^^ [16:54:18] (03CR) 10Paladox: [C: 031] "Ive added the archived template." [integration/config] - 10https://gerrit.wikimedia.org/r/274911 (owner: 10Siebrand) [16:57:41] I’d like to migrate integration-slave-trusty-1015; is anyone around who can depool it? [17:05:03] 10Browser-Tests-Infrastructure, 13Patch-For-Review: Simplify creating of Jenkins jobs for running browser tests daily - https://phabricator.wikimedia.org/T128190#2088803 (10zeljkofilipin) CI part is done. To minimize code duplication across repositories, we should create a rake task and put it in mediawiki_sel... [17:12:53] (03PS1) 10Paladox: [Citoid] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/274987 [17:14:50] (03PS2) 10Paladox: [Citoid] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/274987 [17:17:25] (03PS1) 10Paladox: [CirrusSearch] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/274990 [17:23:26] Krinkle: do you know how to depool intergration slaves? Or know who does? [17:23:34] andrewbogott: Yes [17:23:44] andrewbogott: from Jenkins web panel [17:23:52] Log in with LDAP [17:24:04] From https://integration.wikimedia.org/ci/computer/ [17:24:08] I want to depool integration-slave-trusty-1015 and migrate it to another host… that’s fairly safe, right? [17:24:12] Will just slow down CI tests for a bit? [17:24:16] mark as temp offline [17:24:21] and (if needed) wait for jobs to finish in the sidebar [17:24:33] andrewbogott: Yeah, should be fine [17:24:40] great, thanks [17:24:44] as long as at least two of each OS type are online at any time [17:24:46] to avoid deadlocks [17:25:25] trusty-1015 currently runs a slow browsertest, so might take a while [17:25:47] I’m patient :) [17:32:36] 6Release-Engineering-Team: MW-1.27-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T128872#2088867 (10greg) [17:32:38] 6Release-Engineering-Team: MW-1.27-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T128872#2088880 (10greg) p:5Triage>3Normal [17:34:24] (03PS1) 10Paladox: [CleanChanges] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/275000 [17:40:17] (03PS1) 10Paladox: [ClientSide] Add npm test [integration/config] - 10https://gerrit.wikimedia.org/r/275005 [18:01:53] PROBLEM - Host integration-slave-trusty-1015 is DOWN: CRITICAL - Host Unreachable (10.68.16.232) [18:09:47] 10Deployment-Systems, 3Scap3, 7WorkType-Maintenance: /srv/deployment/scap is no longer needed - https://phabricator.wikimedia.org/T128386#2089112 (10dduvall) p:5Triage>3High [18:19:27] 10Continuous-Integration-Config, 10Fundraising-Backlog: CI issues on fundraising/REL1_25 - https://phabricator.wikimedia.org/T128883#2089135 (10cwdent) [18:20:31] 10Continuous-Integration-Config, 10Fundraising-Backlog, 3Fundraising Sprint Elevator Maintenance 2016: CI issues on fundraising/REL1_25 - https://phabricator.wikimedia.org/T128883#2089154 (10cwdent) [18:29:27] 10Deployment-Systems, 3Scap3: scap3 upstream/debian versioning - https://phabricator.wikimedia.org/T127828#2089198 (10dduvall) p:5Triage>3Normal [18:29:48] 10Beta-Cluster-Infrastructure, 10Staging, 10DBA, 3Collaboration-Team-Current, and 2 others: Use External Store on Beta Cluster - https://phabricator.wikimedia.org/T95871#2089205 (10Mattflaschen) >>! In T95871#2087608, @jcrespo wrote: > I can fix production, but I need the original script for that (even if... [18:33:40] 10Continuous-Integration-Config, 10Fundraising-Backlog, 3Fundraising Sprint Elevator Maintenance 2016: CI issues on fundraising/REL1_25 - https://phabricator.wikimedia.org/T128883#2089135 (10Legoktm) You need to cherry-pick the patch from {T124574} [18:33:53] RECOVERY - Host integration-slave-trusty-1015 is UP: PING OK - Packet loss = 0%, RTA = 0.42 ms [18:37:33] awesome sprint name [18:37:39] 10Continuous-Integration-Infrastructure, 3Scap3, 10scap, 10Packaging: Develop a CI Testing and Release pipeline for the SCAP package (and potentially other debian packages that we maintain) - https://phabricator.wikimedia.org/T127741#2089255 (10thcipriani) [18:44:56] 6Release-Engineering-Team, 6Developer-Relations, 6Team-Practices, 15User-greg: Set up Code Review office hours - https://phabricator.wikimedia.org/T128371#2089287 (10greg) So, my thinking here: A bi-weekly (every two weeks) event scheduled for the -dev(?) IRC channel that is advertised as a "come work on... [18:45:09] 10Continuous-Integration-Config: Move list of extensions in shared job into a zuul function, and filter by branch - https://phabricator.wikimedia.org/T128886#2089289 (10Legoktm) [18:48:58] 3Scap3, 10scap: Bring co-master / fanout capabilities to `deploy` and friends - https://phabricator.wikimedia.org/T121276#2089327 (10thcipriani) p:5High>3Normal [18:49:33] 3Scap3, 10scap: Scap3 needs a way to handle large binary file transport - https://phabricator.wikimedia.org/T119443#2089329 (10thcipriani) p:5Normal>3High [18:54:54] 6Release-Engineering-Team, 10scap, 6Operations, 10Scap3 (Scap3-MediaWiki-MVP): Depool proxies temporarily while scap is ongoing to avoid taxing those nodes - https://phabricator.wikimedia.org/T125629#2089377 (10mmodell) [18:54:59] 10scap, 10Scap3 (Scap3-MediaWiki-MVP): Bring co-master / fanout capabilities to `deploy` and friends - https://phabricator.wikimedia.org/T121276#2089379 (10mmodell) [18:55:01] 10scap, 10Scap3 (Scap3-MediaWiki-MVP): Implement MediaWiki pre-promote checks - https://phabricator.wikimedia.org/T121597#2089381 (10mmodell) [18:55:03] 10scap, 10Scap3 (Scap3-MediaWiki-MVP): Scap should touch symlinks when originals are touched - https://phabricator.wikimedia.org/T126306#2089383 (10mmodell) [18:55:06] 6Release-Engineering-Team, 10scap, 10Scap3 (Scap3-MediaWiki-MVP), 7Security-General: Scap should apply security patches - https://phabricator.wikimedia.org/T118478#2089385 (10mmodell) [18:55:09] 10scap, 13Patch-For-Review, 10Scap3 (Scap3-MediaWiki-MVP): Parameterize git_server in ops/puppet for /etc/scap.cfg - https://phabricator.wikimedia.org/T126259#2089387 (10mmodell) [18:55:12] 10scap, 10Scap3 (Scap3-MediaWiki-MVP): Scap3 needs a way to handle large binary file transport - https://phabricator.wikimedia.org/T119443#2089389 (10mmodell) [18:55:14] 10scap, 10Scap3 (Scap3-MediaWiki-MVP): refreshCdbJsonFiles should be rewritten in python - https://phabricator.wikimedia.org/T125685#2089391 (10mmodell) [18:55:18] 10scap, 6Operations, 10Scap3 (Scap3-MediaWiki-MVP): Move scap target configuration to etcd - https://phabricator.wikimedia.org/T115899#2089394 (10mmodell) [18:55:20] 10scap, 10Scap3 (Scap3-MediaWiki-MVP): scap3 configuration selection is confusing - https://phabricator.wikimedia.org/T120410#2089396 (10mmodell) [18:55:27] 10scap, 6Operations, 6Performance-Team, 7HHVM, 10Scap3 (Scap3-MediaWiki-MVP): Make scap able to depool/repool servers via the conftool API - https://phabricator.wikimedia.org/T104352#2089398 (10mmodell) [18:55:29] 10scap, 10Scap3 (Scap3-MediaWiki-MVP): Allow scap3 to read target host list from stdin - https://phabricator.wikimedia.org/T122913#2089400 (10mmodell) [18:55:31] 10scap, 10Scap3 (Scap3-MediaWiki-MVP), 7WorkType-NewFunctionality: Scap3 check to monitor logstash and detect changes in error frequency - https://phabricator.wikimedia.org/T110068#2089402 (10mmodell) [18:55:35] 10scap, 10Scap3 (Scap3-MediaWiki-MVP), 7WorkType-NewFunctionality: Remove apache dependency from scap3 deployment host - https://phabricator.wikimedia.org/T116630#2089404 (10mmodell) [18:55:40] 6Release-Engineering-Team, 10scap, 10Scap3 (Scap3-MediaWiki-MVP): updateWikiversions: Don't assume that all versions being operated on +/- of each other - https://phabricator.wikimedia.org/T125702#2089418 (10mmodell) [18:55:59] ooo, a milestone! [19:01:26] 3releng-201516-q2, 3releng-201516-q3, 10scap, 10Scap3 (scap3-adoption): [keyresult] Migrate all Service team owned services and MW deploys to scap3 - https://phabricator.wikimedia.org/T109926#2089446 (10mmodell) [19:01:30] 10scap, 10ContentTranslation-Deployments, 10ContentTranslation-cxserver, 10MediaWiki-extensions-ContentTranslation, 10Scap3 (scap3-adoption): Deploy CXServer with scap3 - https://phabricator.wikimedia.org/T120104#2089448 (10mmodell) [19:01:33] 10scap, 10RESTBase, 10Scap3 (scap3-adoption): Deploy RESTBase with scap3 - https://phabricator.wikimedia.org/T116335#2089450 (10mmodell) [19:01:35] 10scap, 10Citoid, 10VisualEditor, 10Scap3 (scap3-adoption): Deploy Citoid with scap3 - https://phabricator.wikimedia.org/T116337#2089452 (10mmodell) [19:01:37] 10scap, 10Mathoid, 10Scap3 (scap3-adoption): Deploy Mathoid with scap3 - https://phabricator.wikimedia.org/T116338#2089455 (10mmodell) [19:01:41] 10scap, 10RESTBase-Cassandra, 10Scap3 (scap3-adoption): Deploy Cassandra with scap3 - https://phabricator.wikimedia.org/T116340#2089457 (10mmodell) [19:01:43] 3releng-201516-q3, 10scap, 10Scap3 (scap3-adoption), 7WorkType-NewFunctionality: [keyresult] Migrate the MW weekly train deploy to scap3 - https://phabricator.wikimedia.org/T114313#2089459 (10mmodell) [19:01:46] 10Beta-Cluster-Infrastructure, 10scap, 10Analytics, 6Services, and 3 others: Set up AQS in Beta - https://phabricator.wikimedia.org/T116206#2089461 (10mmodell) [19:01:49] 10scap, 10Graphoid, 10Scap3 (scap3-adoption): Deploy Graphoid with scap3 - https://phabricator.wikimedia.org/T120102#2089463 (10mmodell) [19:01:52] 10scap, 10Analytics-EventLogging, 10Scap3 (scap3-adoption): Move EventLogging service to scap3 - https://phabricator.wikimedia.org/T118772#2089465 (10mmodell) [19:01:56] 10scap, 10Parsoid, 10Scap3 (scap3-adoption): Deploy Parsoid with scap3 - https://phabricator.wikimedia.org/T120103#2089467 (10mmodell) [19:01:58] 10scap, 6Discovery, 10Scap3 (scap3-adoption): Create scripts for automatic deployment for wikimedia/portals - https://phabricator.wikimedia.org/T114694#2089469 (10mmodell) [19:02:00] 10scap, 13Patch-For-Review, 10Scap3 (scap3-adoption): Scap3 needs to be deployed on RESTBase boxes and needs a group on tin - https://phabricator.wikimedia.org/T109862#2089471 (10mmodell) [19:04:07] 10scap, 10Analytics-EventLogging, 10Scap3 (scap3-adoption): Move EventLogging service to scap3 - https://phabricator.wikimedia.org/T118772#2089473 (10Ottomata) It would be best to let us fix the upstart vs systemd stuff first. Currently eventlogging deployment takes a manually global sudo pip install step,... [19:22:43] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:34] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 40326 bytes in 2.342 second response time [19:31:51] !log Nodepool Image ci-jessie-wikimedia-1457119603 in wmflabs-eqiad is ready - T128846 [19:31:52] T128846: DIB images have /puppet/hieradata files that are empty - https://phabricator.wikimedia.org/T128846 [19:31:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [19:39:38] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer: browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce fails with "757: unexpected token" - https://phabricator.wikimedia.org/T128079#2089573 (10Jdlrobson) [19:43:56] bd808: stashbot expanding "T\d+" to URL is very useful thank you [19:45:29] hashar or legoktm or Krinkle or jzerebecki can i have help to fix this error please [19:45:34] https://integration.wikimedia.org/ci/job/mediawiki-extensions-php55/1431/console [19:45:46] Per https://phabricator.wikimedia.org/T128883 [19:46:13] 10Continuous-Integration-Config, 10Fundraising-Backlog, 3Fundraising Sprint Elevator Maintenance 2016: CI issues on fundraising/REL1_25 - https://phabricator.wikimedia.org/T128883#2089623 (10Paladox) @Legoktm theres a wikibase error. [19:46:19] yes, I know [19:47:33] legoktm: Ok thanks. [19:48:57] legoktm: Looks like there is no fundraising/REL1_25 branch so it then fallbacks to master. [19:49:08] Yes. [19:50:25] legoktm: Oh, ok. Should we create that branch or would you know an easy fix so instead of failing that branch it can try REL1_25 and if that does not work then master. [19:50:49] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 13Patch-For-Review: DIB images have /puppet/hieradata files that are empty - https://phabricator.wikimedia.org/T128846#2089629 (10hashar) The snapshot `ci-jessie-wikimedia-1457119603` has been made out of the image `image-t128846` whi... [19:52:42] legoktm: Maybe we can create fundraising/mediawiki/core [19:52:48] legoktm: Maybe we can create fundraising/mediawiki-core * [19:53:01] then it wouldn't need to prefix branches [19:53:15] Krinkle: Yes please. [19:53:52] Patching Zuul will be non-trivial [19:53:57] Krinkle: er, what do you mean? [19:54:08] legoktm: Have fundraising maintain a literal fork instead of now like a branch [19:54:15] ohhh [19:54:19] wikimedia/fundraising/mediawik-core [19:54:23] or something like that [19:54:34] idk what they'd think of that [19:54:40] Would solve a bunch of problems I imagine [19:54:53] Or alternatively, we could convince them to drop it and use plain REL1_25 [19:55:05] I also proposed https://phabricator.wikimedia.org/T128886 [19:55:15] T128886 [19:55:15] T128886: Move list of extensions in shared job into a zuul function, and filter by branch - https://phabricator.wikimedia.org/T128886 [19:55:26] so adding new extensions to the shared job doesn't break old branches [19:55:31] Yeah [19:55:37] I tihnk that happened with Wikibase [19:56:51] Krinkle: Maybe we should ask would it be easisar for there development to use it from plain REL1_25 since mediawiki/core is not updated but the extension is. [19:56:55] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 13Patch-For-Review: DIB images have /puppet/hieradata files that are empty - https://phabricator.wikimedia.org/T128846#2089662 (10hashar) Image ci-jessie-wikimedia-1457121113 in wmflabs-eqiad is ready Created with `sync` called in the... [19:57:14] paladox: That requires changing logic to hardcode 'fundraising' in many many places [19:57:28] Zuul's interface is: clone a number of repos with branch X and fallback to master [19:57:34] Krinkle: Oh then i think your first suggestion would be good. [19:57:36] we should not want to change that [19:58:00] Ok. [19:58:30] Krinkle: we could also stop running the shared jobs on fundraising branches, idk which extensions they actually use [19:59:46] legoktm: Should we migrate DonationInterface to generic test. Since it is still using non generic. [20:01:07] No, that one needs to be non-voting for now [20:01:22] legoktm: Oh ok, we could use the non voting generic test. [20:01:42] Right! We could do that yeah [20:02:06] (03PS1) 10Hashar: dib: sync filesystem before snapshotting [integration/config] - 10https://gerrit.wikimedia.org/r/275037 (https://phabricator.wikimedia.org/T128846) [20:02:17] legoktm: Ok. [20:04:16] legoktm: Im running the test here https://integration.wikimedia.org/ci/job/mwext-testextension-hhvm/5032/ to make sure it passes on master then i will do a check on REL1_25 branch. [20:05:00] legoktm: It fails because i need to add the dependacies which i will do now, then if the test passes after that i will switch tests. [20:05:05] 10Continuous-Integration-Config, 10Fundraising-Backlog, 3Fundraising Sprint Elevator Maintenance 2016: CI issues on fundraising/REL1_25 - https://phabricator.wikimedia.org/T128883#2089135 (10Krinkle) The current branching system used by Fundraising is incompatible with how Release Engineering has designed th... [20:06:03] legoktm: Sure [20:06:11] legoktm: That'll be a hotpatch [20:06:26] and we'd ideally create a separate job with a list of extensions they want to test with their core version [20:06:44] (03PS1) 10Paladox: [DonationInterface] Add dependacie on ContributionTracking in parameter_functions.py [integration/config] - 10https://gerrit.wikimedia.org/r/275041 [20:06:58] * legoktm nods [20:07:13] Krinkle or legoktm ^^ please merge. [20:10:13] paladox: "dependency" (its been corrected a few times before) [20:10:37] Krinkle: Sorry i keep misspelling it. [20:10:58] (03PS2) 10Paladox: [DonationInterface] Add dependency on ContributionTracking in parameter_functions.py [integration/config] - 10https://gerrit.wikimedia.org/r/275041 [20:13:07] Krinkle if the fundraising decide todo what you suggested here https://phabricator.wikimedia.org/T128883 would we be able to add a note somewhere so that REL1_25 changes can be backported to them. [20:13:30] paladox: What do you mean? [20:14:43] Krinkle well sometimes we may fix an issue in REL1_25 but if the https://phabricator.wikimedia.org/T128883fundraising team decide to split the branch of into its own project to improve maintenance, could we add a note so those backports which the fundraising teams project. [20:15:10] paladox: Backporting to REL1_25 doesn't go to fundraising either [20:15:21] they can continue to do as they do now which is to backport what they want, themselves. [20:15:33] Krinkle: Oh yes sorry, maybe we should ask them todo that. [20:15:36] Yes. [20:15:43] They alreayd do that [20:15:47] for several years [20:15:48] nothing changes that [20:16:13] Krinkle oh ok, thanks for explaning [20:17:50] (03CR) 10Paladox: "We can switch this to a non voting generic test. Please merge." [integration/config] - 10https://gerrit.wikimedia.org/r/275041 (owner: 10Paladox) [20:18:02] 10Continuous-Integration-Config, 10Fundraising-Backlog, 3Fundraising Sprint Elevator Maintenance 2016: CI issues on fundraising/REL1_25 - https://phabricator.wikimedia.org/T128883#2089716 (10hashar) @Krinkle T124105 was specially to get tests to accommodate CI with `fundraising/REL1_25` and I have suggested... [20:19:55] 10Continuous-Integration-Config, 10Fundraising-Backlog, 3Fundraising Sprint Elevator Maintenance 2016: CI issues on fundraising/REL1_25 - https://phabricator.wikimedia.org/T128883#2089723 (10Paladox) @hashar T124574 is now fixed for them. @Legoktm suggested they backport it, which I did and it was merged. ht... [20:20:30] 10Continuous-Integration-Config, 10Fundraising-Backlog, 3Fundraising Sprint Elevator Maintenance 2016: CI issues on fundraising/REL1_25 - https://phabricator.wikimedia.org/T128883#2089724 (10Paladox) @hashar also wikibase or wikidata or any of the extension dosent have that branch so that where the problem c... [20:22:10] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 13Patch-For-Review: DIB images have /puppet/hieradata files that are empty - https://phabricator.wikimedia.org/T128846#2089725 (10hashar) Image ci-jessie-wikimedia-1457122001 in wmflabs-eqiad is ready Went back to use a clone with har... [20:26:30] 10Continuous-Integration-Config, 10Fundraising-Backlog, 3Fundraising Sprint Elevator Maintenance 2016: CI issues on fundraising/REL1_25 - https://phabricator.wikimedia.org/T128883#2089734 (10hashar) Why does Wikibase/Wikidata repos are included in that job? That does not make sense. It is a specific/narrow j... [20:26:59] paladox: Krinkle: That DonationInterface / Fundraising job is very narrow in scope [20:27:27] that went from a discussion between cwdent and thcipriani and we kind of though it would be a nice feat to have for them [20:28:10] 10Gitblit-Deprecate, 10Diffusion: Update mediawiki.org templates to link to Diffusion, not gitblit - https://phabricator.wikimedia.org/T108864#2089737 (10demon) [20:28:12] 10Gitblit-Deprecate, 6Release-Engineering-Team, 10Diffusion, 7WorkType-NewFunctionality: Use Diffusion as canonical location for browsing code repos (not gitblit) - https://phabricator.wikimedia.org/T752#2089739 (10demon) [20:28:14] 5Gerrit-Migration, 10Gitblit-Deprecate, 6Release-Engineering-Team, 10Diffusion, and 2 others: Import all gerrit.wikimedia.org repositories with Diffusion - https://phabricator.wikimedia.org/T616#2089735 (10demon) 5Open>3Resolved They're all imported. [20:28:20] 5Gerrit-Migration, 10Gitblit-Deprecate, 6Release-Engineering-Team, 10Diffusion, and 2 others: Import all gerrit.wikimedia.org repositories with Diffusion - https://phabricator.wikimedia.org/T616#2089742 (10demon) [20:31:38] (03Abandoned) 10Hashar: dib: fully clone /puppet [integration/config] - 10https://gerrit.wikimedia.org/r/274954 (https://phabricator.wikimedia.org/T128846) (owner: 10Hashar) [20:31:56] (03PS2) 10Hashar: dib: sync filesystem before snapshotting [integration/config] - 10https://gerrit.wikimedia.org/r/275037 (https://phabricator.wikimedia.org/T128846) [20:32:49] hashar: Ok. [20:33:16] 5Gerrit-Migration, 10Gitblit-Deprecate, 6Release-Engineering-Team, 10Diffusion, and 2 others: Import all gerrit.wikimedia.org repositories with Diffusion - https://phabricator.wikimedia.org/T616#2089763 (10hashar) If we create new repositories in Gerrit are they going to be automatically imported in Diffus... [20:34:00] (03CR) 10Hashar: [C: 032] dib: sync filesystem before snapshotting [integration/config] - 10https://gerrit.wikimedia.org/r/275037 (https://phabricator.wikimedia.org/T128846) (owner: 10Hashar) [20:34:20] 10Continuous-Integration-Config, 10Fundraising-Backlog, 3Fundraising Sprint Elevator Maintenance 2016: CI issues on fundraising/REL1_25 - https://phabricator.wikimedia.org/T128883#2089767 (10Paladox) @hashar it dosent run the on the extension specific job but ones on the gate extension mediawiki-extensions-p... [20:34:41] 5Gerrit-Migration, 10Gitblit-Deprecate, 6Release-Engineering-Team, 10Diffusion, and 2 others: Import all gerrit.wikimedia.org repositories with Diffusion - https://phabricator.wikimedia.org/T616#2089771 (10Krenair) no, see {T128757} [20:34:52] 5Gerrit-Migration, 10Gitblit-Deprecate, 6Release-Engineering-Team, 10Diffusion, and 2 others: Import all gerrit.wikimedia.org repositories with Diffusion - https://phabricator.wikimedia.org/T616#2089773 (10Paladox) @hashar the admin who creates the repo has been asked to also create it in diffusion. [20:35:07] (03Merged) 10jenkins-bot: dib: sync filesystem before snapshotting [integration/config] - 10https://gerrit.wikimedia.org/r/275037 (https://phabricator.wikimedia.org/T128846) (owner: 10Hashar) [20:36:08] hashar: Wikidata is run on mediawiki-extensions-php55 which mediawiki/core fundraising/REL1_25 runs [20:44:43] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 13Patch-For-Review: DIB images have /puppet/hieradata files that are empty - https://phabricator.wikimedia.org/T128846#2089789 (10hashar) p:5Unbreak!>3High **Summary** An instance is booted from the reference image. It has a git w... [20:45:28] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling: DIB images have /puppet/hieradata files that are empty - https://phabricator.wikimedia.org/T128846#2087975 (10hashar) [20:46:39] paladox: hashar knows that. He is saying, that was not intended. Fundraising does not use Wikidata, the test should not enable Wikidata. [20:46:52] So the error doesn't matter, in this case, we need to just remove wikidata from that test, there is no resolution [20:47:10] and wikidata is known to be broken for REL1_25 [20:47:18] the tests got removed / skipped [20:47:23] Krinkle: Oh. [20:47:40] hashar Krinkle how would we remove it. [20:47:51] and mediawiki-extensions-php55 should probably not be run on DonationInterface , and definitely not on fundraising/REL* branches [20:48:28] the wikimedia fundraising mediawiki install is quite unique / narrow case [20:48:29] hashar: We should blacklist it then. [20:48:41] Using the skip-if which is really useful. [20:48:45] yup [20:48:52] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #944: 04FAILURE in 22 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/944/ [20:49:03] but then they have a bunch of repos in the specific job [20:49:16] so it is probably easier to just filter out with branch: fundraising/.* [20:50:35] hashar: Yes. [20:50:40] hashar would this do it branch: (?!REL1_23|REL1_24|fundraising/REL.*) [20:50:49] - name: ^mediawiki-extensions-(qunit|hhvm|php53)$ [20:50:49] branch: (?!REL1_23|REL1_24|fundraising/REL.*) [20:50:49] queue-name: mediawiki [20:52:21] (03PS1) 10Paladox: Blacklist mediawiki-extensions-php55 test on branch fundraising/REL [integration/config] - 10https://gerrit.wikimedia.org/r/275050 (https://phabricator.wikimedia.org/T128883) [20:53:37] hashar: Krinkle legoktm ^^^ [20:53:43] (03PS2) 10Paladox: Blacklist mediawiki-extensions-php55 test on branch fundraising/REL [integration/config] - 10https://gerrit.wikimedia.org/r/275050 (https://phabricator.wikimedia.org/T128883) [20:55:51] hashar also mediawiki-extensions-php55 is not run on DonationInterface but is run on mediawiki/core on branch fundraising. [21:04:47] # Extensions tested together, since 1.25alpha ... [21:04:47] - name: ^mediawiki-extensions-(qunit|hhvm|php53)$ [21:04:47] branch: (?!REL1_23|REL1_24|fundraising/REL.*) [21:04:57] that is supposed to get rid of it [21:05:24] note how that one only has 'php53' [21:05:30] adding php55 there might be sufficient [21:06:11] hashar: I thought we are ment to be getting rid of that test for that branch. [21:07:09] oh [21:07:16] that is actually what you have done [21:07:44] https://integration.wikimedia.org/ci/job/integration-zuul-layoutdiff/8305/console is weird [21:08:14] hashar sorry about that, laptop shut down suddaly so it's restarting. [21:10:24] (03CR) 10Hashar: "Looking at the Zuul layout diff, that change the regex which was:" [integration/config] - 10https://gerrit.wikimedia.org/r/275050 (https://phabricator.wikimedia.org/T128883) (owner: 10Paladox) [21:10:45] paladox_: eek [21:10:51] paladox: anyway I have replied ;-} [21:11:09] I am off. But all that mess will have to be refactored eventually [21:19:11] hashar: Hi could you rewrite what you wrote before since laptop just restarted now. [21:19:14] pleasew. [21:19:16] please. [21:21:08] (03PS3) 10Paladox: Blacklist mediawiki-extensions-php55 test on branch fundraising/REL [integration/config] - 10https://gerrit.wikimedia.org/r/275050 (https://phabricator.wikimedia.org/T128883) [21:21:11] hashar: ^^^ [21:21:28] :p [21:21:57] paladox: wait for zuul diff output [21:22:08] hashar: Ok. [21:22:25] hashar do i need to include queue-name: mediawiki [21:23:43] nope [21:23:54] hashar: Ok. [21:24:04] most repos / jobs are in it already [21:24:09] hashar also i think the layout is look like that in the diff because of [21:24:09] - name: ^.*php55.*$ [21:24:10] branch: (?!REL1_2[3-6]$) [21:24:27] (03CR) 10Hashar: [C: 032] Blacklist mediawiki-extensions-php55 test on branch fundraising/REL [integration/config] - 10https://gerrit.wikimedia.org/r/275050 (https://phabricator.wikimedia.org/T128883) (owner: 10Paladox) [21:24:36] hashar: Thanks. [21:25:22] hii [21:25:24] thcipriani: yt? [21:25:33] ottomata: what's up? [21:25:45] looking at doing this [21:25:45] https://phabricator.wikimedia.org/T127099 [21:25:51] (03Merged) 10jenkins-bot: Blacklist mediawiki-extensions-php55 test on branch fundraising/REL [integration/config] - 10https://gerrit.wikimedia.org/r/275050 (https://phabricator.wikimedia.org/T128883) (owner: 10Paladox) [21:26:02] currently, this repo is just a submodule of eventlogging [21:26:04] which is not right [21:26:10] and, it will be used for other things than eventbus [21:26:14] (which is part of eventlogging) [21:26:20] so, i want to deploy it separately [21:26:28] makes sense [21:26:42] also, deploying separately will let me deploy to eventbus hosts and sighup them [21:26:54] instead of restarting them on deploy , which is what eventbus deploy does (because code changes) [21:27:10] so, i'm thinking of puppetizing an eventschemas module that does scap::target [21:27:22] i don't want to put the scap/ dir in the event-schemas repo [21:27:33] i'm thinking of puppetizing the whole scap dir [21:27:36] for it [21:27:46] and i could have an environment for each deploy purpose [21:27:58] that shouldn't be a problem. [21:27:59] eventbus/ [21:27:59] analytics_cluster/ [21:27:59] etc. [21:28:04] does that make sense? [21:28:29] does i make sense to use the same deploy source on tin to deploy to different environment for diffferent purpose? [21:29:01] if it's always going to be the same code then maybe. [21:29:23] I suppose only if you can't think of a reason that it would ever diverge. [21:29:47] its a schema repository [21:29:54] that is always backwards compatible [21:29:59] each change is actually a new file [21:30:00] (03PS1) 10Paladox: Add fundraising/REL.* to ^.*php53.*$ instead [integration/config] - 10https://gerrit.wikimedia.org/r/275060 [21:30:05] (unless you are deleting stuff) [21:31:21] can i override the uhhhh service restart command for each environemnt? [21:31:23] paladox: deployed the other change [21:31:28] uhhh, looking for that property in docs.. [21:31:31] hashar: Thanks. [21:31:51] ottomata: the restart command? It just uses the service_name. [21:32:27] heading to bed, happy hacking folks! [21:32:28] naw but i want to do somethign other than restart [21:32:35] and it will be sighupping a different service [21:32:39] so [21:32:42] event-schemas is a dependency of eventbus [21:32:42] ah, right. [21:32:46] which is deployed differently [21:32:54] i want to sighup eventbus on event-schema deploy [21:33:06] doesn't have to restart on schema change, just reload [21:33:18] service eventlogging-service-eventbus reload [21:33:46] so, if you don't specify a service_name and instead have it run as a post promote command. [21:33:58] in checks.yaml? [21:34:04] then you could run: kill -HUP in checks.yaml, yup. [21:34:17] ok got it, and can each environment have its own checks.yaml? [21:34:41] yeah, that seems to be what you'd need. [21:34:52] cool, ok [21:35:20] so i need a custom dsh hosts file and checks.yaml per env, and one main scap.cfg for the whole thing [21:35:21] ok... [21:35:49] (03PS1) 10Paladox: Allow us to recheck gate and submit [integration/config] - 10https://gerrit.wikimedia.org/r/275062 [21:35:55] hashar: ^^^ :) [21:38:13] paladox: potentially a dupe of the very old https://gerrit.wikimedia.org/r/#/c/227223/ :D [21:38:27] oh man [21:38:29] stop sending so many patchsets really ;-} [21:38:38] hashar: Oh sorry. [21:38:44] (03CR) 10Krinkle: "Yes, but we already have a canonical way. It's just that the parsoid job accidentally left out a very important macro in hits job configur" [integration/config] - 10https://gerrit.wikimedia.org/r/274863 (owner: 10Arlolra) [21:39:06] paladox: that patchset does not work though. The syntax is invalid [21:39:20] but that was the same idea. Will have to dig into it again [21:39:39] hashar: Oh, would mine work. As long as someone has already +2 and is whitelisted. [21:39:49] I can't remember honestly [21:39:57] but there is an integration test on https://gerrit.wikimedia.org/r/#/c/227223/16/tests/test_zuul_scheduler.py,cm [21:40:02] will hae to revisit it [21:40:49] Ok. [21:41:01] I see you use the new gerrit diff view. [21:41:13] yeah [21:41:28] Ok. :) [21:41:35] you should switch to the new Gerrit UI, whenever we upgrade Gerrit that will be the only one available i think [21:41:39] so better adapt now [21:42:06] anyway, we will want to polish up and finish the old patch https://gerrit.wikimedia.org/r/#/c/227223/ [21:42:29] hashar: you said you were sleeping a while ago :) :) [21:42:32] hashar: yep, But i belive the upgrade will make it more polished then it was in 2.8 since that was the begining and 2.9 was the finish. [21:42:56] hashar: Yep we would want to polish your patch up. [21:43:18] thcipriani: should I set get_server in wmnet to deployment.eqiad.wmnet? [21:43:22] instead of hardcoding tin? [21:43:23] greg-g: can't sleep! I have a mission to fulfill ! [21:43:58] paladox: I think we will aim for Gerrit 2.11 or 2.12 can't remember which [21:44:07] anyway as greg-g said. I should sleep for real now [21:44:07] hashar: its 2.12. [21:44:23] hashar: you will get your signed pushes too. [21:44:27] :) :) [21:44:31] ottomata: https://gerrit.wikimedia.org/r/#/c/272947/7 needs to merge at some point. [21:44:38] Meaning it will be supported now. [21:44:56] git_server* :p [21:45:19] Also we will be able to edit in browser on gerrit. :) gerrit is being migrated soon on wikimedia to the new version the server has the hardware now just needs someone to install. [21:45:21] oh nice, then it will be configured globally [21:45:27] instead of in each scap.cfg [21:45:37] ottomata: yeah, should be easy to override for beta, et al. [21:45:41] paladox: awesome! [21:46:13] hashar: I belive it will be soon since the reason is because gerrits current server is out of warrenty soon. [21:48:15] hashar: https://phabricator.wikimedia.org/T123132 and https://phabricator.wikimedia.org/T125018 and https://phabricator.wikimedia.org/T126794 [21:48:26] thcipriani: hm. will what i'm doing be funky? [21:48:36] maybe i'm trying to make environments do too much [21:48:44] we've got eventbus running in beta and in production [21:49:06] would hope to use the same scapification to deploy in both [21:49:14] but, if i'm using 'eventbus' as an environment already [21:49:17] then what? [21:49:26] i guess a make a special 'eventbus-beta' environment? [21:49:35] actually, the other problem is the puppet provider. [21:49:53] relies on .git/DEPLOY_HEAD which would be...weird in this instance. [21:49:57] paladox: thanks! sleeping now [21:50:03] hashar: Ok. [21:52:30] thcipriani: ? [21:54:13] ottomata: so if you use the same repo on tin and rely only on environment, everytime you deploy you'd write a yaml file to .git/DEPLOY_HEAD , but they'd be for different environments and not globally applicable. [21:54:58] then when you spin up a new server and use: package { [blah]: provider => 'scap' } it'd try to use the file at .git/DEPLOY_HEAD to figure out the most recent revision to deploy. [21:55:14] which might not be a big deal, but it may not be what you expected. [21:55:48] Oo, hm, in my case, it would be ok, but i see what you are saying, that's not ideal [21:55:53] .git/DEPLOY_HEAD also stores the configuration of the last deployment, which might change based on env. [21:55:57] that means that you can only deploy one environemnt from one deploy server? [21:56:14] yeah, likely, depending on the use-case. [21:56:34] sounds like a bug! should probably move DEPLOY_HEAD of .git and put it into scap/ somewhere based on env [21:56:51] out of* [21:57:48] thcipriani: alternatively i could just do multiple deployment sources on tin, and make each one use the same repo but deploy to different targets for different purposes [21:57:52] PROBLEM - Host integration-slave-trusty-1016 is DOWN: CRITICAL - Host Unreachable (10.68.18.34) [21:57:59] not really ideal, since i want to make a puppet eventschemas module that makes this easy [21:58:03] rather than having to do it differently each time [21:58:13] hmmm...but how would puppet find the correct settings for the target? [21:58:26] puppet? [21:58:38] I guess you could specify environment in the puppet provider. [21:58:43] does that matter? [21:59:10] hm [21:59:16] PROBLEM - Host cache-rsync is DOWN: CRITICAL - Host Unreachable (10.68.23.165) [21:59:40] if host A and B are for different environments, but each is configured using the same scap::target setting... [21:59:52] then what, the initialization of the target will pull [21:59:55] from DEPLOY_HEAD [21:59:59] buuut [22:00:07] from then on, everything will be hanlded by manual scap deploys, right? [22:00:09] yeah, different sources on tin should be fine. [22:00:19] oh, i'm still talking about a single source [22:00:26] yeah, that's true. just provisioning that uses .git/DEPLOY_HEAD [22:00:27] i'd prefer to do a single source [22:00:39] er, well, it all uses .git/DEPLOY_HEAD, but it's overwritten each deploy [22:00:43] HMMMM [22:00:54] i guess i could make the eventschemas/scap.pp a define or something [22:01:01] and have it easily provision a new deploy source on tin [22:01:03] for each instance [22:01:05] :? [22:01:21] * bd808 really should ditch the watch on "scap" [22:01:25] haha [22:01:35] bd808: it's comforting to know you're watching :P [22:02:03] yeah, different deploy sources on tin would work fine, I think. [22:02:30] yeah, but its messier from the configution /puppet standpoint [22:02:31] hmmm [22:02:38] do you think that's better thcipriani? [22:03:38] I do. Messier from the configuration standpoint, but I think that buys flexibility. Plus that's what the puppet provider expects. [22:03:57] ok [22:04:05] will figure that out then [22:04:06] thanks [22:04:19] thanks for talking it through with me :) [22:10:03] Quit: Updating windows 10 to build 14279 for PC :) https://twitter.com/GabeAul/status/705816926006579200 [22:11:38] QUIT Updating windows 10 to build 14279 for PC :) https://twitter.com/GabeAul/status/705816926006579200 [22:12:21] QUIT: Updating windows 10 to build 14279 for PC :) https://twitter.com/GabeAul/status/705816926006579200 [22:12:46] paladox: you've already posted that a couple times in a row :) [22:12:59] greg-g: Sorry it wont quit. [22:14:45] /quit ctrl+v ;) [22:16:11] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling: DIB images have /puppet/hieradata files that are empty - https://phabricator.wikimedia.org/T128846#2090116 (10mobrovac) Fingers crossed :) Thanks for the effort, @hashar [22:28:13] (03PS2) 10Arlolra: Run mw-apply-settings for parsoidsvc-php-parsertests [integration/config] - 10https://gerrit.wikimedia.org/r/274863 [22:29:38] (03CR) 10Arlolra: "Should it do mw-fetch-composer-dev and mw-run-update-script as well?" [integration/config] - 10https://gerrit.wikimedia.org/r/274863 (owner: 10Arlolra) [22:30:36] RECOVERY - Host integration-slave-trusty-1016 is UP: PING OK - Packet loss = 0%, RTA = 0.99 ms [22:40:33] Yippee, build fixed! [22:40:34] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #976: 09FIXED in 29 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/976/ [23:18:43] Can I shut down deployment-tin for a few minutes? [23:19:10] thcipriani: ^ ? [23:19:28] andrewbogott: sure. [23:22:31] Yippee, build fixed! [23:22:32] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce-T128731 build #13: 09FIXED in 7 min 31 sec: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce-T128731/13/ [23:23:32] PROBLEM - Host deployment-tin is DOWN: CRITICAL - Host Unreachable (10.68.17.240) [23:25:05] 10Continuous-Integration-Config, 10Tool-Labs-tools-stewardbots: Implement jenkins tests on labs/tools/stewardbots - https://phabricator.wikimedia.org/T128503#2090346 (10MarcoAurelio) p:5Triage>3Normal Not sure I'll be able to get this done. [23:36:18] RECOVERY - Host deployment-tin is UP: PING OK - Packet loss = 0%, RTA = 0.92 ms [23:36:43] thcipriani: all done [23:36:57] andrewbogott: didn't feel a thing :) [23:37:06] thanks [23:37:21] Project beta-code-update-eqiad build #94528: 04FAILURE in 3.3 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/94528/ [23:39:59] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10Unplanned-Sprint-Work, 3Fundraising Sprint Elevator Maintenance 2016, 13Patch-For-Review: CI issues on fundraising/REL1_25 - https://phabricator.wikimedia.org/T128883#2089135 (10DStrine) [23:44:42] Yippee, build fixed! [23:44:43] Project beta-code-update-eqiad build #94529: 09FIXED in 1 min 42 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/94529/ [23:46:26] Project beta-scap-eqiad build #92317: 04FAILURE in 1 min 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/92317/ [23:49:28] darn keyholder. [23:53:34] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10Unplanned-Sprint-Work, 3Fundraising Sprint Elevator Maintenance 2016, 13Patch-For-Review: CI issues on fundraising/REL1_25 - https://phabricator.wikimedia.org/T128883#2090478 (10cwdent) Really appreciate everyone's input on this. FWIW having that... [23:56:02] Yippee, build fixed! [23:56:02] Project beta-scap-eqiad build #92318: 09FIXED in 1 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/92318/