[01:12:37] 10Continuous-Integration-Infrastructure, 06Operations, 13Patch-For-Review, 07WorkType-Maintenance: Jenkins master / client ssh connection fails due to missing ssh algorithm - https://phabricator.wikimedia.org/T100509#1314411 (10Dzahn) @hashar do you think this will change on contint1001 with openjdk-7-jdk... [01:17:33] RECOVERY - Puppet staleness on deployment-prometheus01 is OK: OK: Less than 1.00% above the threshold [3600.0] [06:11:43] 10Continuous-Integration-Infrastructure, 06Operations, 13Patch-For-Review, 07WorkType-Maintenance: Jenkins master / client ssh connection fails due to missing ssh algorithm - https://phabricator.wikimedia.org/T100509#2747576 (10hashar) @Dzahn the issue is in Jenkins itself which uses an old / apparently un... [08:54:09] PROBLEM - SSH on deployment-sca02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:14:32] PROBLEM - Puppet run on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [09:20:12] PROBLEM - Puppet run on integration-slave-trusty-1016 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:20:30] PROBLEM - Puppet run on integration-slave-trusty-1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:23:49] PROBLEM - Puppet run on integration-slave-trusty-1004 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:25:13] PROBLEM - Puppet run on integration-slave-trusty-1012 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:30:21] PROBLEM - Puppet run on integration-slave-trusty-1006 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:35:10] PROBLEM - Puppet run on integration-slave-trusty-1011 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:52:16] bah [10:05:29] Error 400 on SERVER: Duplicate declaration: Exec[jenkins-deploy kvm membership] is already declared in file /etc/puppet/modules/contint/manifests/packages/androidsdk.pp:19; cannot redeclare at /etc/puppet/modules/contint/manifests/packages/labs.pp:86 [10:05:43] due to rebase of puppet.git I made earlier [10:17:38] 10Continuous-Integration-Infrastructure, 06Wikipedia-Android-App-Backlog: Puppet on CI Trusty slaves: Duplicate declaration: Exec[jenkins-deploy kvm membership] - https://phabricator.wikimedia.org/T149294#2747883 (10hashar) [10:25:04] !log integration: purge Android packages from Trusty slaves for T149294 : salt -v '*slave-trusty*' cmd.run 'apt-get --yes remove --purge gcc-multilib lib32z1 lib32stdc++6 qemu' [10:25:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:29:08] !log integration: on Trusty slaves, remove jenkins-deploy from KVM which is only needed for Android testing: salt -v '*slave-trusty*' cmd.run 'groupdeluser jenkins-deploy kvm' [10:29:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:29:56] 10Continuous-Integration-Infrastructure, 06Wikipedia-Android-App-Backlog, 13Patch-For-Review: Puppet on CI Trusty slaves: Duplicate declaration: Exec[jenkins-deploy kvm membership] - https://phabricator.wikimedia.org/T149294#2747922 (10hashar) p:05Triage>03High [10:30:26] !log integration: on Trusty slaves, remove jenkins-deploy from KVM which is only needed for Android testing for T149294: salt -v '*slave-trusty*' cmd.run 'deluser jenkins-deploy kvm' [10:30:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:31:14] 10Continuous-Integration-Infrastructure, 06Wikipedia-Android-App-Backlog, 13Patch-For-Review: Puppet on CI Trusty slaves: Duplicate declaration: Exec[jenkins-deploy kvm membership] - https://phabricator.wikimedia.org/T149294#2747925 (10hashar) Solved. I have cherry picked https://gerrit.wikimedia.org/r/31827... [10:34:31] RECOVERY - Puppet run on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [10:40:19] RECOVERY - Puppet run on integration-slave-trusty-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [10:45:12] RECOVERY - Puppet run on integration-slave-trusty-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [10:45:32] PROBLEM - Puppet run on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [11:00:12] RECOVERY - Puppet run on integration-slave-trusty-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [11:00:12] RECOVERY - Puppet run on integration-slave-trusty-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [11:00:32] RECOVERY - Puppet run on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [11:03:50] RECOVERY - Puppet run on integration-slave-trusty-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [11:05:32] RECOVERY - Puppet run on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [11:24:04] RECOVERY - SSH on deployment-sca02 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u3 (protocol 2.0) [12:14:08] (03PS1) 10Hashar: dib: install Doxygen on Jessie images [integration/config] - 10https://gerrit.wikimedia.org/r/318279 [12:14:21] (03CR) 10jenkins-bot: [V: 04-1] dib: install Doxygen on Jessie images [integration/config] - 10https://gerrit.wikimedia.org/r/318279 (owner: 10Hashar) [12:25:06] 10Continuous-Integration-Config, 05Goal, 07I18n: Configure banana checker for i18n files to run on all MediaWiki extensions and skins - https://phabricator.wikimedia.org/T94547#2748087 (10hashar) There are still 373 extensions that lack a `package.json` :( cd mediawiki/extensions find . -mindepth 1 -max... [12:34:05] zeljkof: hashar: any idea why https://integration.wikimedia.org/ci/job/mwext-mw-selenium/11130/console is failing with "stack level too deep (SystemStackError)"? [12:34:18] it's working fine when I run it locally.. [12:34:52] did you see something similar in past and can give me a hint perhaps? [12:39:09] Tobi_WMDE_SW: eeek [12:40:02] Tobi_WMDE_SW: sorry, at the phone, at doctor, hashar should be able to help [12:40:11] vendor/bin/bundle exec cucumber --color --tags @integration --tags '~@skip' --format pretty --format junit --out /srv/jenkins-workspace/workspace/mwext-mw-selenium/log/junit [12:40:12] by looking at the code, I could not find any suspicious piece that would cause an endless recursion or something.. [12:40:17] stack level too deep (SystemStackError) [12:40:20] /mnt/home/jenkins-deploy/.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/core_ext/instance_exec.rb:73 [12:40:34] I have no idea :( [12:41:58] Cucumber expodes [12:42:02] Strange [12:42:31] zeljkof: hashar: could it be "Given I am on the Main Page" which could already exist in some other form..? [12:42:36] Tobi_WMDE_SW: please create a task in phab and assign to me [12:43:07] will try something out first and if I'm not able to fix it, I'll create one zeljkof . thx! [12:45:53] AHHHH [12:45:56] I can reproduce locally :] [12:46:01] bundle install [12:46:03] cd tests/browser [12:46:09] bundle exec cucumber --color --tags @integration --tags '~@skip' --format pretty --format junit --out junit.xml [12:46:09] stack level too deep (SystemStackError) [12:48:24] hashar: ha! found the issue [12:48:51] hashar: forgot to check in environments.yml [12:48:59] :) [12:49:00] ohhhh [12:49:09] that is one of those very annoying message errors so [12:49:15] hashar: but strange that this is failing in such a way.. [12:56:02] (03PS1) 10Tobias Gritschacher: Turn on browsertests for ElectronPdfService extension [integration/config] - 10https://gerrit.wikimedia.org/r/318284 (https://phabricator.wikimedia.org/T149189) [13:34:05] (03CR) 10WMDE-Fisch: [C: 031] Turn on browsertests for ElectronPdfService extension [integration/config] - 10https://gerrit.wikimedia.org/r/318284 (https://phabricator.wikimedia.org/T149189) (owner: 10Tobias Gritschacher) [13:35:17] Tobi_WMDE_SW: so I guess the selenium browser tests should be made voting ? :] [13:36:29] hashar: Yes! [13:50:40] (03CR) 10Hashar: [C: 032] "Well done :]" [integration/config] - 10https://gerrit.wikimedia.org/r/318284 (https://phabricator.wikimedia.org/T149189) (owner: 10Tobias Gritschacher) [13:51:13] (03Merged) 10jenkins-bot: Turn on browsertests for ElectronPdfService extension [integration/config] - 10https://gerrit.wikimedia.org/r/318284 (https://phabricator.wikimedia.org/T149189) (owner: 10Tobias Gritschacher) [13:52:27] Tobi_WMDE_SW: congratulations! [13:52:46] hashar: thx! :D [15:00:08] 10Browser-Tests-Infrastructure: Absence of environments.yml causing endless recursion - https://phabricator.wikimedia.org/T149311#2748445 (10Tobi_WMDE_SW) [15:02:08] (03PS1) 10Tobias Gritschacher: Avoid endless recursion when environments.yml is missing [selenium] - 10https://gerrit.wikimedia.org/r/318305 (https://phabricator.wikimedia.org/T149311) [15:04:31] (03CR) 10jenkins-bot: [V: 04-1] Avoid endless recursion when environments.yml is missing [selenium] - 10https://gerrit.wikimedia.org/r/318305 (https://phabricator.wikimedia.org/T149311) (owner: 10Tobias Gritschacher) [15:11:27] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T147517#2748483 (10mmodell) [15:25:14] (03CR) 10Hashar: "recheck" [selenium] - 10https://gerrit.wikimedia.org/r/302850 (owner: 10Dduvall) [15:25:43] 10Continuous-Integration-Infrastructure, 06Wikipedia-Android-App-Backlog, 13Patch-For-Review: Puppet on CI Trusty slaves: Duplicate declaration: Exec[jenkins-deploy kvm membership] - https://phabricator.wikimedia.org/T149294#2748525 (10Niedzielski) @hashar, sorry for any trouble. I feel a bit out of the loop... [15:28:19] (03CR) 10Hashar: "Looks like I broke the mediawiki-selenium-integration job on October 4th https://integration.wikimedia.org/ci/job/mediawiki-selenium-integ" [selenium] - 10https://gerrit.wikimedia.org/r/318305 (https://phabricator.wikimedia.org/T149311) (owner: 10Tobias Gritschacher) [15:40:47] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T147517#2748574 (10mmodell) [15:42:48] Project selenium-MobileFrontend ยป chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #206: 04FAILURE in 20 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/206/ [15:48:55] (03CR) 10Tobias Gritschacher: "It's looking for Geckodriver.. seems unrelated to my change. :/" [selenium] - 10https://gerrit.wikimedia.org/r/318305 (https://phabricator.wikimedia.org/T149311) (owner: 10Tobias Gritschacher) [15:50:13] 10Browser-Tests-Infrastructure: mediawiki_selenium integration test fails: Unable to find Mozilla geckodriver. - https://phabricator.wikimedia.org/T149319#2748687 (10hashar) [15:51:14] Hey team! Will you release 1.28.0-wmf.24 next week? [15:51:24] https://www.mediawiki.org/wiki/MediaWiki_1.28/Roadmap#Schedule_for_the_deployments has no information about it. [15:52:56] Trizek: I think the plan is for next week to be 1.29.0-wmf.1. I may have misread Chad's email though [15:53:02] Trizek: https://lists.wikimedia.org/pipermail/wikitech-l/2016-October/086859.html [15:53:03] Trizek hi, it's now 1.29 [15:53:13] yeah, 1.29.0-wmf.1 is next week [15:53:45] * bd808 pours one out for the plans to deprecate some things in 1.28 [15:54:24] Thanks! [15:54:39] Idea: add a not on the 1.28 page ;) [15:54:51] (03PS1) 10Hashar: Pin selenium-webdriver < 3 [selenium] - 10https://gerrit.wikimedia.org/r/318311 (https://phabricator.wikimedia.org/T149319) [15:55:02] Trizek: it's a wiki ;) [15:55:08] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:55:44] Good point, bd808. [15:56:03] That was just a suggestion for the next branch. [15:56:05] (03CR) 10Hashar: "Yeah unrelated. Looks like that is due to selenium-webdriver 3.0.0. I filled T149319 about it, and have some patch at https://gerrit.wikim" [selenium] - 10https://gerrit.wikimedia.org/r/318305 (https://phabricator.wikimedia.org/T149311) (owner: 10Tobias Gritschacher) [15:56:45] (03CR) 10Hashar: [C: 031] "The Jenkins job installed selenium-webdriver (2.53.4) which is all happy :]" [selenium] - 10https://gerrit.wikimedia.org/r/318311 (https://phabricator.wikimedia.org/T149319) (owner: 10Hashar) [15:57:11] (03PS2) 10Hashar: Avoid endless recursion when environments.yml is missing [selenium] - 10https://gerrit.wikimedia.org/r/318305 (https://phabricator.wikimedia.org/T149311) (owner: 10Tobias Gritschacher) [16:03:29] Tobi_WMDE_SW_NA: fixed :] [16:04:02] (03CR) 10Hashar: "That probably deserves a new minor version. Will want to review and land https://gerrit.wikimedia.org/r/#/c/318305/ as well :)" [selenium] - 10https://gerrit.wikimedia.org/r/318311 (https://phabricator.wikimedia.org/T149319) (owner: 10Hashar) [16:04:36] 10Browser-Tests-Infrastructure, 13Patch-For-Review: mediawiki_selenium integration test fails: Unable to find Mozilla geckodriver. - https://phabricator.wikimedia.org/T149319#2748820 (10hashar) p:05Triage>03High https://gerrit.wikimedia.org/r/#/c/318311/ fix it by pinning `selenium-webdriver < 3` [16:04:57] I still don't see the benefit of the list of wmf.XX deployments on that page, but maybe I'm missing a use case [16:19:42] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T147517#2748897 (10greg) [16:22:28] Project beta-scap-eqiad build #126244: 04FAILURE in 9.9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/126244/ [16:24:44] hashar: cool! thx! [16:26:58] Yippee, build fixed! [16:26:58] Project beta-scap-eqiad build #126245: 09FIXED in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/126245/ [17:18:24] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T149338#2749149 (10greg) [17:18:33] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T149059#2740552 (10greg) [17:49:54] PROBLEM - Puppet run on deployment-pdfrender is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:24:21] PROBLEM - Host integration-puppetmaster is DOWN: CRITICAL - Host Unreachable (10.68.16.42) [18:50:07] !log stashbot has replaced qa-morebots in this channel as the sole bot handling !log messages [18:50:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:55:03] neat [18:57:09] (03PS1) 10Ejegg: Add no-op for SmashPig gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/318347 [18:57:28] Anyone mind taking a look at that trivial thing ^^^ ? [18:57:57] just trying to get fundraising's SmashPig deployment branch to auto-submit on C+2 [19:14:04] 10Continuous-Integration-Infrastructure, 06Wikipedia-Android-App-Backlog, 13Patch-For-Review: Puppet on CI Trusty slaves: Duplicate declaration: Exec[jenkins-deploy kvm membership] - https://phabricator.wikimedia.org/T149294#2749588 (10hashar) 05Open>03Resolved Unrelated to your Xdummy patch :] The root... [19:17:55] (03CR) 10Hashar: [C: 032] Add no-op for SmashPig gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/318347 (owner: 10Ejegg) [19:18:14] thanks hasharAway ! [19:18:16] ejegg: we should just give you CR+2 on integration/config and give you the ability to deploy :D [19:18:33] (03Merged) 10jenkins-bot: Add no-op for SmashPig gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/318347 (owner: 10Ejegg) [19:18:37] ooh, fun! [19:19:02] those directions sounded complicated, but I've managed to get jjb working locally just for testing stuff [19:19:02] deployed! [19:19:09] ty hashar [19:19:13] ah that is nice [19:20:06] while you are around ejegg, I have added a job for DonationInterface that targets mw 1.28 [19:20:15] I have made it non voting [19:20:26] since I am pretty sure stuff will break soonish eventually [19:29:17] greg-g: one of the cute new stashbot features: !log deployment-prep in #wikimedia-labs will write to both the project SAL and the releng SAL. Same for contintcloud [19:37:05] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T147517#2749655 (10mmodell) [19:39:37] (03PS1) 10Hashar: Remove obsolete scripts [integration/jenkins] - 10https://gerrit.wikimedia.org/r/318357 [19:39:48] hashar: ah thanks, I'll take a look in a bit [19:40:45] bd808: nice thanks :] [19:41:28] (03CR) 10Hashar: [C: 032] "It is gone from JJB and I double checked on gallium by grepping all job configs." [integration/jenkins] - 10https://gerrit.wikimedia.org/r/318357 (owner: 10Hashar) [19:42:01] (03Merged) 10jenkins-bot: Remove obsolete scripts [integration/jenkins] - 10https://gerrit.wikimedia.org/r/318357 (owner: 10Hashar) [19:43:40] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T147517#2749688 (10mmodell) [20:03:31] (03PS1) 10Hashar: JJB macro 'prepare-mediawiki' does not exist anymore [integration/config] - 10https://gerrit.wikimedia.org/r/318362 [20:08:23] (03CR) 10Hashar: [C: 032] "Noop" [integration/config] - 10https://gerrit.wikimedia.org/r/318362 (owner: 10Hashar) [20:09:14] (03Merged) 10jenkins-bot: JJB macro 'prepare-mediawiki' does not exist anymore [integration/config] - 10https://gerrit.wikimedia.org/r/318362 (owner: 10Hashar) [20:16:32] PROBLEM - Puppet run on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [20:20:21] PROBLEM - Puppet run on castor is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:21:15] PROBLEM - Puppet run on integration-slave-trusty-1016 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:21:31] PROBLEM - Puppet run on integration-slave-trusty-1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:22:58] bd808: neat! we were just talking in here (or somewhere) about how it's annoying that "our" !log messages get split between prod and releng sals (things like CI, gallium-related go in prod while "other" go in releng). [20:24:49] PROBLEM - Puppet run on integration-slave-trusty-1004 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:25:15] PROBLEM - Puppet run on integration-puppetmaster01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [20:26:11] PROBLEM - Puppet run on integration-slave-trusty-1012 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [20:26:45] PROBLEM - Puppet run on integration-slave-jessie-android is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:30:34] PROBLEM - Puppet run on integration-publisher is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:31:22] PROBLEM - Puppet run on integration-slave-trusty-1006 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:31:46] PROBLEM - Puppet run on integration-slave-jessie-1003 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [20:33:42] greg-g: hmmm... that could be a solvable problem. [20:34:22] PROBLEM - Puppet run on integration-saltmaster is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:36:11] PROBLEM - Puppet run on integration-slave-trusty-1011 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:40:44] PROBLEM - Puppet run on integration-slave-precise-1011 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:42:00] PROBLEM - Puppet run on repository is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:43:00] PROBLEM - Puppet run on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:43:40] PROBLEM - Puppet run on integration-slave-precise-1012 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [20:44:24] PROBLEM - Puppet run on integration-slave-precise-1002 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:45:54] PROBLEM - Puppet run on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:48:49] PROBLEM - Puppet run on deployment-fluorine02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:49:11] so puppet exploded [20:49:30] really [20:49:44] I am gonna take a couple week of vacations and just move to a master less setup [20:51:05] !log reboot integration-puppetmaster01 [20:51:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:52:37] Could not find class role::ci::slave::labs [20:52:41] that does not make sense [20:55:54] solved [21:04:09] greg-g, hashar: what if I make messages that contain "#releng" copy to your SAL? [21:04:44] so you could "!log whatever #releng" in -operations and get the message on both SAL pages? [21:05:12] RECOVERY - Puppet run on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:05:56] hashar: ugh, thanks [21:06:35] bd808: that'd be neat, yeah. I know we're a special snowflake, but maybe Discovery/whomever would also like that? maybe not [21:06:44] RECOVERY - Puppet run on integration-slave-jessie-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [21:06:44] RECOVERY - Puppet run on integration-slave-jessie-android is OK: OK: Less than 1.00% above the threshold [0.0] [21:07:37] greg-g: they just have to ask :) [21:08:01] bd808: yeah, more I'm trying to justify the time spent on our special snowflakeness :) [21:08:35] meh. let me worry about time investment [21:10:32] RECOVERY - Puppet run on integration-publisher is OK: OK: Less than 1.00% above the threshold [0.0] [21:10:43] * hashar buys bd808 stocks [21:11:20] RECOVERY - Puppet run on integration-slave-trusty-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [21:14:25] RECOVERY - Puppet run on integration-saltmaster is OK: OK: Less than 1.00% above the threshold [0.0] [21:16:09] RECOVERY - Puppet run on integration-slave-trusty-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [21:20:35] deployment-sca02 is not happy btw, loadavg > 200 [21:20:41] RECOVERY - Puppet run on integration-slave-precise-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [21:21:33] RECOVERY - Puppet run on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [21:21:57] RECOVERY - Puppet run on repository is OK: OK: Less than 1.00% above the threshold [0.0] [21:23:01] RECOVERY - Puppet run on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [21:23:38] RECOVERY - Puppet run on integration-slave-precise-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [21:24:24] RECOVERY - Puppet run on integration-slave-precise-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [21:25:52] RECOVERY - Puppet run on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [21:30:20] RECOVERY - Puppet run on castor is OK: OK: Less than 1.00% above the threshold [0.0] [21:31:12] RECOVERY - Puppet run on integration-slave-trusty-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [21:31:12] RECOVERY - Puppet run on integration-slave-trusty-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [21:31:30] RECOVERY - Puppet run on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [21:34:49] RECOVERY - Puppet run on integration-slave-trusty-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [22:59:55] 10Continuous-Integration-Infrastructure, 06Operations, 07Zuul: Upgrade Zuul on gallium to 2.5.0-8-gcbc7f62-wmf2precise1 - https://phabricator.wikimedia.org/T144088#2750516 (10Paladox) @hashar is this task resolved? [23:03:58] 10Continuous-Integration-Infrastructure, 07Zuul: Large number of android app changes slowing down the processing of changes in unrelated repositories - https://phabricator.wikimedia.org/T149382#2750531 (10greg) [23:07:00] 10Continuous-Integration-Infrastructure, 07Upstream, 07WorkType-Maintenance, 07Zuul: Zuul deadlocks if unknown repo has activity in Gerrit - https://phabricator.wikimedia.org/T128569#2750546 (10hashar) [23:07:02] 10Continuous-Integration-Infrastructure, 06Operations, 07Zuul: Upgrade Zuul on gallium to 2.5.0-8-gcbc7f62-wmf2precise1 - https://phabricator.wikimedia.org/T144088#2750543 (10hashar) 05Open>03Resolved Yeah they have all been uploaded / polished with @elukey via a different task. Thanks @Paladox ! Precis... [23:07:18] 10Continuous-Integration-Infrastructure, 06Operations, 07Zuul: Upgrade Zuul on gallium to 2.5.0-8-gcbc7f62-wmf2precise1 - https://phabricator.wikimedia.org/T144088#2750547 (10hashar) [23:11:55] 10Continuous-Integration-Infrastructure, 06Operations, 07Zuul: Upgrade Zuul on gallium to 2.5.0-8-gcbc7f62-wmf2precise1 - https://phabricator.wikimedia.org/T144088#2750548 (10Paladox) your welcome :) [23:13:00] 10Continuous-Integration-Infrastructure, 07Zuul: Large number of android app changes slowing down the processing of changes in unrelated repositories - https://phabricator.wikimedia.org/T149382#2750553 (10hashar) [23:13:24] 10Continuous-Integration-Infrastructure, 07Zuul: Large number of android app changes slowing down the processing of changes in unrelated repositories - https://phabricator.wikimedia.org/T149382#2750531 (10hashar) That is exactly the private task T140297 [23:13:54] 10Continuous-Integration-Infrastructure, 07Zuul: Large number of android app changes slowing down the processing of changes in unrelated repositories - https://phabricator.wikimedia.org/T149382#2750531 (10Paladox) @hashar it seems T140297 is a private task. [23:14:37] 10Continuous-Integration-Infrastructure, 07Zuul: Large number of android app changes slowing down the processing of changes in unrelated repositories - https://phabricator.wikimedia.org/T149382#2750573 (10Paladox) Also this can happen to any repo just that the android repo likes to depend on a lot of open chan... [23:16:35] 10Continuous-Integration-Infrastructure, 07Zuul: Large number of android app changes slowing down the processing of changes in unrelated repositories - https://phabricator.wikimedia.org/T149382#2750579 (10Paladox) @hashar Could we have one task for public viewing please?