[00:12:23] PROBLEM - Puppet staleness on deployment-apertium01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [01:15:01] Project beta-scap-eqiad build #34791: FAILURE in 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34791/ [01:25:03] Project beta-scap-eqiad build #34792: STILL FAILING in 1 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34792/ [01:35:14] Yippee, build fixed! [01:35:15] Project beta-scap-eqiad build #34793: FIXED in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34793/ [03:11:18] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [03:53:41] Yippee, build fixed! [03:53:42] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce build #368: FIXED in 10 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce/368/ [04:16:54] 3Continuous-Integration: Jenkins: point TMP/TEMP to workspace and delete it after build completion - https://phabricator.wikimedia.org/T70563#938175 (10Krinkle) I believe `$TMPDIR` is the most standard (on POSIX systems anyway, which includes Linux, OSX and most Unix variants) . We shouldn't have to set any `$TM... [04:17:35] 3Continuous-Integration: Jenkins: point TMPDIR to tmp dir in workspace and delete after build - https://phabricator.wikimedia.org/T70563#938183 (10Krinkle) [04:21:12] 3operations, Continuous-Integration: Acquire old production API servers for use in CI - https://phabricator.wikimedia.org/T84940#938192 (10Krinkle) Yeah. CI isn't currently in need of more bare-metal nodes as Jenkins slaves (like gallium and lanthanum are at the moment). These resources would be better spent on... [04:23:31] Yippee, build fixed! [04:23:31] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #428: FIXED in 1 hr 2 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/428/ [04:34:46] bd808|BUFFER: Looks like hashar deployed it for you already. [04:44:36] 3Continuous-Integration: QUnit test errors are missing in Jenkins short console log - https://phabricator.wikimedia.org/T63843#938208 (10Krinkle) a:3Krinkle [04:48:29] 3Continuous-Integration: QUnit test errors are missing in Jenkins short console log - https://phabricator.wikimedia.org/T63843#938213 (10Krinkle) The qunit Grunt task is run by Jenkins itself, not part of the test suite in MediaWiki core itself. It's currently this verbose because the Jenkins job invokes grunt... [04:52:51] (03PS1) 10Krinkle: qunit macro: Invoke grunt without --verbose [integration/config] - 10https://gerrit.wikimedia.org/r/181356 [04:54:31] 3Continuous-Integration: QUnit test failures are missing from Jenkins short console log - https://phabricator.wikimedia.org/T63843#938272 (10Krinkle) [04:58:48] 3Continuous-Integration: jenkins - operations-puppet-puppetlint-lenient - --no-autoloader_layout-check - https://phabricator.wikimedia.org/T75117#768414 (10Krinkle) [05:12:37] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #395: FAILURE in 54 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/395/ [05:27:55] (03CR) 10Krinkle: [C: 032] "Deployed:" [integration/config] - 10https://gerrit.wikimedia.org/r/181356 (owner: 10Krinkle) [05:31:54] (03Merged) 10jenkins-bot: qunit macro: Invoke grunt without --verbose [integration/config] - 10https://gerrit.wikimedia.org/r/181356 (owner: 10Krinkle) [06:03:18] 3Continuous-Integration: QUnit test failures are missing from Jenkins short console log - https://phabricator.wikimedia.org/T63843#938332 (10Krinkle) 5Open>3Resolved [06:26:23] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<22.22%) [06:36:25] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [06:56:41] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:09:56] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #202: FAILURE in 27 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/202/ [07:21:42] RECOVERY - Puppet failure on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:25:33] 3Continuous-Integration: Augment jsonlint test with a test for duplicate keys - https://phabricator.wikimedia.org/T73284#938359 (10Legoktm) Moved the repo to https://github.com/legoktm/jsonchecker. >>! In T73284#768151, @hashar wrote: > I like the checker since it has tox and tests :-D I am not too sure how to... [07:35:02] Project beta-scap-eqiad build #34829: FAILURE in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34829/ [07:45:15] Project beta-scap-eqiad build #34830: STILL FAILING in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34830/ [07:55:33] Yippee, build fixed! [07:55:33] Project beta-scap-eqiad build #34831: FIXED in 1 min 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34831/ [08:07:56] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #357: FAILURE in 47 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/357/ [08:22:09] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #351: FAILURE in 34 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/351/ [08:26:55] 3Continuous-Integration: Warn/alert on too many jobs queued - https://phabricator.wikimedia.org/T85034#938399 (10Krinkle) A deadlock results in a growing gueue, but a large queue doesn't always mean a deadlock. There'd be many false positives. And it takes a while for the queue to grow large enough. It should e... [09:45:30] Project beta-scap-eqiad build #34842: FAILURE in 1 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34842/ [09:55:11] Yippee, build fixed! [09:55:12] Project beta-scap-eqiad build #34843: FIXED in 1 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34843/ [13:54:52] Project beta-scap-eqiad build #34867: FAILURE in 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34867/ [14:05:00] Project beta-scap-eqiad build #34868: STILL FAILING in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34868/ [14:15:20] Yippee, build fixed! [14:15:20] Project beta-scap-eqiad build #34869: FIXED in 1 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34869/ [14:22:09] 3Continuous-Integration: Puppet broken on integration slaves: install_zuul - https://phabricator.wikimedia.org/T84917#938593 (10hashar) I have cleaned up the mess: created a new tag `wmf-deploy-20141221-1` and pushed master and labs to point to it. It is based on upstream 685ca22 with: * requirements.txt tweak... [14:40:42] (03PS1) 10Hashar: Fix hack to update VisualEditor submodule [integration/config] - 10https://gerrit.wikimedia.org/r/181365 [14:41:49] (03CR) 10Hashar: MobileFrontend depends on Echo,VisualEditor (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/180740 (owner: 10Hashar) [14:45:24] (03CR) 10jenkins-bot: [V: 04-1] Fix hack to update VisualEditor submodule [integration/config] - 10https://gerrit.wikimedia.org/r/181365 (owner: 10Hashar) [14:52:29] !log Fixed a hack to update VisualEditor submodule https://gerrit.wikimedia.org/r/#/c/181365/ [14:53:35] PROBLEM - Puppet failure on deployment-sentry2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:49:04] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:50:51] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:52:15] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:54:07] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:54:21] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:55:53] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:56:28] Project beta-scap-eqiad build #34879: FAILURE in 2 min 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34879/ [15:57:28] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:57:50] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:57:58] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:01:09] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:01:09] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [16:02:07] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:02:25] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:02:31] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:06:01] PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:06:27] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:06:59] Project beta-scap-eqiad build #34880: STILL FAILING in 3 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34880/ [16:12:43] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:13:10] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:18:23] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:18:39] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:18:53] Project beta-scap-eqiad build #34881: STILL FAILING in 4 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34881/ [16:22:14] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:22:28] RECOVERY - Puppet failure on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:22:58] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [16:23:34] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0] [16:24:19] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [16:24:23] PROBLEM - Puppet failure on deployment-restbase03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:26:49] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:26:49] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:27:34] Project beta-scap-eqiad build #34882: STILL FAILING in 3 min 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34882/ [16:28:35] PROBLEM - Puppet failure on deployment-redis01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:28:39] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:30:13] PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:30:42] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:30:42] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:31:02] RECOVERY - Puppet failure on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [16:32:53] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:35:18] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:37:45] Project beta-scap-eqiad build #34883: STILL FAILING in 3 min 45 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34883/ [16:38:56] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:38:56] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:39:04] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [16:39:34] PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:43:23] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [16:43:37] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:46:57] Project beta-scap-eqiad build #34884: STILL FAILING in 2 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34884/ [16:47:04] PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [16:49:13] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:50:07] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:50:12] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:51:56] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:52:30] RECOVERY - Puppet failure on deployment-cxserver03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:53:28] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:53:36] RECOVERY - Puppet failure on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:55:12] RECOVERY - Puppet failure on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [16:55:42] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [16:57:08] Project beta-scap-eqiad build #34885: STILL FAILING in 3 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34885/ [16:59:03] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:59:25] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:59:35] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:02:46] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [17:03:06] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:03:58] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [17:04:32] RECOVERY - Puppet failure on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [17:05:18] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [17:07:26] Project beta-scap-eqiad build #34886: STILL FAILING in 3 min 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34886/ [17:10:04] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:11:42] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:13:39] RECOVERY - Puppet failure on deployment-parsoidcache02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:14:14] RECOVERY - Puppet failure on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:15:40] RECOVERY - Puppet failure on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [17:15:58] Holy christ that's loud. [17:16:25] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:16:27] Did hashar do this? [17:16:30] And then DISAPPEAR. [17:17:33] Project beta-scap-eqiad build #34887: STILL FAILING in 3 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34887/ [17:20:00] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:20:50] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:26] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:10] PROBLEM - Puppet failure on deployment-redis02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:23:26] RECOVERY - Puppet failure on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:24] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:46] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:11] Project beta-scap-eqiad build #34888: STILL FAILING in 3 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34888/ [17:27:29] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:27:39] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:31:51] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:32:49] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:34:05] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [17:37:15] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [17:37:36] Project beta-scap-eqiad build #34889: STILL FAILING in 3 min 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34889/ [17:38:25] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:40:09] RECOVERY - Puppet failure on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:41:21] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:42:45] RECOVERY - Puppet failure on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:42:49] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:43:45] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:44:59] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [17:47:38] Project beta-scap-eqiad build #34890: STILL FAILING in 3 min 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34890/ [17:48:12] RECOVERY - Puppet failure on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:48:32] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [17:50:16] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [17:51:12] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:52:00] RECOVERY - Puppet failure on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [17:53:20] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:57:13] Project beta-scap-eqiad build #34891: STILL FAILING in 3 min 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34891/ [18:00:57] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:01:49] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:03:29] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:03:31] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:04:38] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:06:04] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #302: FAILURE in 4.3 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/302/ [18:06:56] Project beta-scap-eqiad build #34892: STILL FAILING in 3 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34892/ [18:08:50] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [18:13:42] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:14:27] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:14:29] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:16:11] RECOVERY - Puppet failure on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:17:11] Project beta-scap-eqiad build #34893: STILL FAILING in 3 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34893/ [18:18:15] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [18:19:33] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:20:21] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:20:31] PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:21:42] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [18:23:59] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:24:41] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:24:46] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:26:08] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:26:18] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [18:27:10] Project beta-scap-eqiad build #34894: STILL FAILING in 3 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34894/ [18:27:15] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:27:44] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:28:01] PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:31:15] PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:32:18] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:32:48] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [18:34:24] RECOVERY - Puppet failure on deployment-restbase03 is OK: OK: Less than 1.00% above the threshold [0.0] [18:34:57] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:35:39] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:36:29] Project beta-scap-eqiad build #34895: STILL FAILING in 2 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34895/ [18:36:54] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [18:37:00] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:37:52] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:38:07] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #150: FAILURE in 36 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/150/ [18:39:26] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:42:11] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [18:44:26] RECOVERY - Puppet failure on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:44:32] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [18:45:09] Yippee, build fixed! [18:45:10] Project beta-scap-eqiad build #34896: FIXED in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34896/ [18:45:22] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [18:45:22] PROBLEM - Puppet failure on deployment-restbase03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:45:34] RECOVERY - Puppet failure on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [18:47:05] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [18:48:33] RECOVERY - Puppet failure on deployment-cxserver03 is OK: OK: Less than 1.00% above the threshold [0.0] [18:49:46] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [18:52:42] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [18:53:03] RECOVERY - Puppet failure on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [18:55:04] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [18:55:06] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [18:55:16] RECOVERY - Puppet failure on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:56:13] RECOVERY - Puppet failure on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [18:56:54] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:57:25] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:57:39] RECOVERY - Puppet failure on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [18:58:45] RECOVERY - Puppet failure on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:59:07] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:59:57] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:00:37] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:01:57] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [19:02:10] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:02:18] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:02:50] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [19:06:07] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [19:10:21] RECOVERY - Puppet failure on deployment-restbase03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:14:11] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #429: FAILURE in 49 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/429/ [19:14:39] RECOVERY - Puppet failure on deployment-parsoidcache02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:09:12] Yippee, build fixed! [20:09:13] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #396: FIXED in 55 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/396/ [20:14:55] Project beta-scap-eqiad build #34905: FAILURE in 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34905/ [20:25:05] Project beta-scap-eqiad build #34906: STILL FAILING in 1 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34906/ [20:35:34] Yippee, build fixed! [20:35:35] Project beta-scap-eqiad build #34907: FIXED in 1 min 37 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34907/ [20:58:36] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #403: FAILURE in 1 hr 10 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/403/ [21:13:29] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #407: FAILURE in 31 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/407/ [22:19:34] (03CR) 10Hashar: [C: 032] "deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/181365 (owner: 10Hashar) [22:24:09] 3Continuous-Integration: Puppet broken on integration slaves: install_zuul - https://phabricator.wikimedia.org/T84917#939107 (10hashar) Listing tarballs in zuul requirements.txt caused the integration-config-tox-py27 Jenkins job to fail. That is because pip does not recognize the tarballs being listed there :-/... [22:24:11] (03Merged) 10jenkins-bot: Fix hack to update VisualEditor submodule [integration/config] - 10https://gerrit.wikimedia.org/r/181365 (owner: 10Hashar) [23:46:31] PROBLEM - Free space - all mounts on deployment-cache-upload02 is CRITICAL: CRITICAL: deployment-prep.deployment-cache-upload02.diskspace._srv_vdb.byte_percentfree.value (<100.00%)