[00:12:24] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<11.11%) [00:22:15] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #326: FAILURE in 32 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/326/ [00:31:29] (03PS1) 10EBernhardson: Add ConfirmEdit dependency to Flow phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/179024 [00:37:33] (03CR) 10EBernhardson: [C: 031] "Pushed update to jenkins, tested against https://gerrit.wikimedia.org/r/#/c/178988/ and https://gerrit.wikimedia.org/r/#/c/179007/ , works" [integration/config] - 10https://gerrit.wikimedia.org/r/179024 (owner: 10EBernhardson) [00:44:57] Project browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce build #340: FAILURE in 1 hr 29 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce/340/ [00:52:27] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<11.11%) [01:09:46] 3Parsoid, Continuous-Integration: Move parsoidsvc-php-parsertests to node 0.10 - https://phabricator.wikimedia.org/T76881#839822 (10Arlolra) p:5Triage>3High [01:12:24] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<11.11%) [02:05:59] (03PS1) 10Dduvall: Support for default environment configurations [selenium] (env-abstraction-layer) - 10https://gerrit.wikimedia.org/r/179048 [02:06:01] (03PS1) 10Dduvall: Remote browser factory fixes [selenium] (env-abstraction-layer) - 10https://gerrit.wikimedia.org/r/179049 [02:47:25] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<33.33%) [02:54:01] (03CR) 10Krinkle: "This repo is deployed via git-deploy from tin to gallium/lanthanum. And for labs nodes, it's a puppet-enforced git-clone on latest master." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/163898 (owner: 10EBernhardson) [02:59:14] (03CR) 10Krinkle: "Deployed." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/163898 (owner: 10EBernhardson) [03:00:23] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #337: FAILURE in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/337/ [03:04:11] PROBLEM - Puppet failure on deployment-cache-mobile03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [03:11:18] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [03:15:21] Project beta-scap-eqiad build #33369: FAILURE in 1 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33369/ [03:23:23] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #331: FAILURE in 22 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/331/ [03:24:07] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #129: FAILURE in 22 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/129/ [03:35:26] Yippee, build fixed! [03:35:27] Project beta-scap-eqiad build #33371: FIXED in 1 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33371/ [03:38:03] Project browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce build #355: FAILURE in 1 hr 30 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce/355/ [03:46:52] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [03:58:20] Yippee, build fixed! [03:58:21] Project browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce build #218: FIXED in 19 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce/218/ [04:16:54] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [05:03:45] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #408: FAILURE in 43 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/408/ [05:46:33] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [05:47:23] PROBLEM - Puppet failure on deployment-restbase03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [05:48:20] PROBLEM - Puppet failure on deployment-parsoid04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [05:49:17] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [05:49:39] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [05:49:43] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [05:50:12] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [05:50:36] PROBLEM - Puppet failure on deployment-redis01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [05:50:48] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [05:51:05] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [05:52:38] PROBLEM - Puppet failure on deployment-mx is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [05:52:52] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #237: FAILURE in 13 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/237/ [05:52:52] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [05:52:53] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [05:54:02] PROBLEM - Puppet failure on deployment-sca-cache01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [05:54:06] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [05:54:27] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [05:55:07] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0] [05:55:11] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [05:55:13] PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [05:55:36] PROBLEM - Puppet failure on deployment-sentry2 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [05:56:44] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [05:56:59] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [05:57:23] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [05:58:00] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [05:58:28] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [05:59:12] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [05:59:35] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [06:00:48] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:01:04] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [06:01:04] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [06:03:10] PROBLEM - Puppet failure on deployment-redis02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:04:48] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:07:26] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [06:07:33] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #354: FAILURE in 11 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/354/ [06:07:39] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [06:09:04] PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:12:24] RECOVERY - Puppet failure on deployment-restbase03 is OK: OK: Less than 1.00% above the threshold [0.0] [06:13:20] RECOVERY - Puppet failure on deployment-parsoid04 is OK: OK: Less than 1.00% above the threshold [0.0] [06:14:09] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [06:14:19] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [06:14:38] RECOVERY - Puppet failure on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [06:14:40] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [06:15:15] RECOVERY - Puppet failure on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:15:15] RECOVERY - Puppet failure on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [06:15:35] RECOVERY - Puppet failure on deployment-sentry2 is OK: OK: Less than 1.00% above the threshold [0.0] [06:16:03] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [06:16:58] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:17:38] RECOVERY - Puppet failure on deployment-mx is OK: OK: Less than 1.00% above the threshold [0.0] [06:19:02] RECOVERY - Puppet failure on deployment-sca-cache01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:20:48] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [06:21:43] RECOVERY - Puppet failure on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [06:22:25] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [06:23:27] RECOVERY - Puppet failure on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:24:27] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [06:24:37] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:25:48] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [06:26:05] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [06:26:05] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [06:26:37] RECOVERY - Puppet failure on deployment-cxserver03 is OK: OK: Less than 1.00% above the threshold [0.0] [06:28:12] RECOVERY - Puppet failure on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [06:29:03] RECOVERY - Puppet failure on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [06:29:49] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [06:32:26] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [06:32:39] RECOVERY - Puppet failure on deployment-parsoidcache02 is OK: OK: Less than 1.00% above the threshold [0.0] [06:35:08] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [06:35:14] RECOVERY - Puppet failure on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [06:35:35] RECOVERY - Puppet failure on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:37:24] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [06:37:52] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:37:56] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:39:26] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [06:42:58] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [06:44:13] PROBLEM - Puppet failure on deployment-redis02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:48:24] PROBLEM - Puppet failure on deployment-restbase03 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [06:48:28] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:48:38] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:50:39] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:51:17] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [06:51:17] PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [06:51:35] PROBLEM - Puppet failure on deployment-redis01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:51:50] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:52:56] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:53:40] PROBLEM - Puppet failure on deployment-mx is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:53:52] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [06:54:58] PROBLEM - Puppet failure on deployment-sca-cache01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:55:08] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [06:55:16] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0] [06:57:42] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [06:58:24] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:59:02] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [06:59:28] RECOVERY - Puppet failure on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:01:46] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:04:30] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #461: FAILURE in 1 hr 15 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/461/ [07:07:32] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [07:09:19] PROBLEM - Puppet failure on deployment-parsoid04 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [07:10:05] PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [07:11:12] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [07:13:38] RECOVERY - Puppet failure on deployment-parsoidcache02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:15:08] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:15:26] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [07:17:44] RECOVERY - Puppet failure on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:18:38] RECOVERY - Puppet failure on deployment-mx is OK: OK: Less than 1.00% above the threshold [0.0] [07:18:54] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:20:00] RECOVERY - Puppet failure on deployment-sca-cache01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:20:36] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:22:07] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:23:25] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [07:23:59] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [07:24:11] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:26:50] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [07:27:30] RECOVERY - Puppet failure on deployment-cxserver03 is OK: OK: Less than 1.00% above the threshold [0.0] [07:29:09] RECOVERY - Puppet failure on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:30:40] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [07:33:21] RECOVERY - Puppet failure on deployment-restbase03 is OK: OK: Less than 1.00% above the threshold [0.0] [07:33:27] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:34:20] RECOVERY - Puppet failure on deployment-parsoid04 is OK: OK: Less than 1.00% above the threshold [0.0] [07:34:58] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:35:10] Project beta-scap-eqiad build #33395: FAILURE in 1 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33395/ [07:35:16] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [07:36:12] RECOVERY - Puppet failure on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [07:36:12] RECOVERY - Puppet failure on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:36:12] RECOVERY - Puppet failure on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:36:34] RECOVERY - Puppet failure on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:36:50] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:37:56] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:40:13] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:40:28] RECOVERY - Puppet failure on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:40:36] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:45:33] !log shut up shinken-wm [07:45:36] Logged the message, Master [08:07:50] Yippee, build fixed! [08:07:50] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #417: FIXED in 1 hr 18 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/417/ [08:15:31] Yippee, build fixed! [08:15:31] Project beta-scap-eqiad build #33399: FIXED in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33399/ [08:35:53] Project beta-scap-eqiad build #33401: FAILURE in 1 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33401/ [08:55:28] Yippee, build fixed! [08:55:28] Project beta-scap-eqiad build #33403: FIXED in 1 min 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33403/ [08:59:30] Yippee, build fixed! [08:59:30] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #327: FIXED in 51 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/327/ [09:05:15] Project beta-scap-eqiad build #33404: FAILURE in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33404/ [09:25:40] Yippee, build fixed! [09:25:40] Project beta-scap-eqiad build #33406: FIXED in 1 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33406/ [09:35:47] Project beta-scap-eqiad build #33407: FAILURE in 1 min 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33407/ [10:24:07] Project beta-code-update-eqiad build #35817: FAILURE in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/35817/ [10:34:10] Yippee, build fixed! [10:34:11] Project beta-code-update-eqiad build #35818: FIXED in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/35818/ [10:59:40] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #183: FAILURE in 49 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/183/ [11:20:07] 3Multimedia, MediaWiki-extensions-GWToolset, Beta-Cluster: Creating directory with special characters - https://phabricator.wikimedia.org/T75725#841284 (10JeanFred) [12:25:22] Yippee, build fixed! [12:25:22] Project beta-scap-eqiad build #33423: FIXED in 1 分 15 秒: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33423/ [12:35:33] Project beta-scap-eqiad build #33424: FAILURE in 1 分 25 秒: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33424/ [12:55:20] Yippee, build fixed! [12:55:20] Project beta-scap-eqiad build #33426: FIXED in 1 分 17 秒: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33426/ [13:05:19] Project beta-scap-eqiad build #33427: FAILURE in 1 分 18 秒: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33427/ [13:09:04] bah jenkins using chinese [13:15:23] Yippee, build fixed! [13:15:23] Project beta-scap-eqiad build #33428: FIXED in 1 分 20 秒: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33428/ [13:25:12] Project beta-scap-eqiad build #33429: FAILURE in 1 分 13 秒: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33429/ [14:25:12] Yippee, build fixed! [14:25:12] Project beta-scap-eqiad build #33436: FIXED in 1 分 14 秒: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33436/ [14:35:16] Project beta-scap-eqiad build #33437: FAILURE in 1 分 14 秒: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33437/ [14:37:42] hashar: is jenkins in chinese now? [14:37:45] :) [14:37:56] looks weird [15:05:29] Yippee, build fixed! [15:05:30] Project beta-scap-eqiad build #33440: FIXED in 1 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33440/ [15:15:37] Project beta-scap-eqiad build #33441: FAILURE in 1 min 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33441/ [15:25:13] Yippee, build fixed! [15:25:13] Project beta-scap-eqiad build #33442: FIXED in 1 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33442/ [15:29:07] (03CR) 10Hashar: [C: 032] "Thanks for the job reconfiguration!" [integration/config] - 10https://gerrit.wikimedia.org/r/179024 (owner: 10EBernhardson) [15:32:59] (03Merged) 10jenkins-bot: Add ConfirmEdit dependency to Flow phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/179024 (owner: 10EBernhardson) [15:45:19] Project beta-scap-eqiad build #33444: FAILURE in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33444/ [16:24:19] (03CR) 10Hashar: "I have defined the env variable in Jenkins global configuration so they will be available to all jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/178862 (owner: 10Hashar) [16:50:30] Yippee, build fixed! [16:50:30] Project beta-scap-eqiad build #33454: FIXED in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33454/ [16:52:39] hi hashar_ if you have a moment could you look this over and merge it if it's OK? https://gerrit.wikimedia.org/r/#/c/178714/ [16:55:00] Project beta-scap-eqiad build #33455: FAILURE in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33455/ [16:55:23] chrismcmahon: I guess you can deploy the job with JJB can't you ? ;) [16:56:21] hashar_: I loved your "!log @damons we love you!" :) [16:57:11] greg-g: someone pointed me the !log are twitted :D [16:57:27] 3Release-Engineering: Can "gem build" fail if there are _any_ warnings? - https://phabricator.wikimedia.org/T1333#842040 (10zeljkofilipin) The upstream support said I should report a bug, done: https://github.com/rubygems/rubygems/issues/1098 [16:58:23] hashar_: yeah, I actually find that (the tweets) useful [16:59:08] hashar_: I can, but I can't merge that patch in that repo (and I'd like some review from an expert) [17:01:15] zeljkof: can I be 5 minutes late? I need to wish my son happy birthday real quick [17:01:22] will be right there [17:01:25] greg-g: sure :) [17:01:28] take your time [17:19:18] I give up for today *wave* [17:25:18] Yippee, build fixed! [17:25:18] Project beta-scap-eqiad build #33458: FIXED in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33458/ [17:28:47] hashar_: g'night! [17:29:02] danke [17:35:14] Project beta-scap-eqiad build #33459: FAILURE in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33459/ [17:55:28] Yippee, build fixed! [17:55:29] Project beta-scap-eqiad build #33461: FIXED in 1 min 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33461/ [18:25:38] Project beta-scap-eqiad build #33464: FAILURE in 1 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33464/ [18:26:15] Project browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce build #219: FAILURE in 20 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce/219/ [18:35:27] Yippee, build fixed! [18:35:28] Project beta-scap-eqiad build #33465: FIXED in 1 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33465/ [18:55:15] Project beta-scap-eqiad build #33467: FAILURE in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33467/ [19:07:35] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #183: FAILURE in 37 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/183/ [19:15:25] Yippee, build fixed! [19:15:25] Project beta-scap-eqiad build #33469: FIXED in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33469/ [19:15:57] bd808|LUNCH: am I just noticing it more now, or is beta scap more flakey lately? [19:25:21] Project beta-scap-eqiad build #33470: FAILURE in 1 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33470/ [19:25:51] 3Release-Engineering: [scap] Suppress/de-emphasize errors from hosts marked for maintenance in icinga - https://phabricator.wikimedia.org/T78319#842441 (10bd808) 3NEW [19:33:21] 3Release-Engineering: [scap] Suppress/de-emphasize errors from hosts marked for maintenance in icinga - https://phabricator.wikimedia.org/T78319#842462 (10greg) p:5Triage>3High [19:33:31] greg-g: It's been really up and down for a few days. Not sure what the root cause is [19:34:08] I get a ping on every fail/recover and it's definately been more noisy [19:34:16] bd808|LUNCH: /me nods, but you also think it's worse lately than normal? [19:34:21] * greg-g nods [19:35:14] Yippee, build fixed! [19:35:14] Project beta-scap-eqiad build #33471: FIXED in 1 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33471/ [19:35:16] greg-g: https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/buildTimeTrend -- graphs agree [19:35:26] yuck [19:36:37] rsync: rename "/srv/mediawiki/.wikiversions-labs.cdb.T8kvej" -> ".~tmp~/wikiversions-labs.cdb": Permission denied (13) [19:37:03] not sure what would cause ownership conflicts... [19:38:57] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #338: FAILURE in 43 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/338/ [19:39:31] !log lots of nslcd errors in syslog on deployment-rsync01 which may be causing scap failures [19:39:35] Logged the message, Master [19:43:46] !log nslcd log noise is probably a red herring -- https://access.redhat.com/solutions/58684 [19:43:48] Logged the message, Master [19:45:11] !log I got nerd snipped into looking at beta. Major personal productivity failure. [19:45:13] Logged the message, Master [19:45:23] Project beta-scap-eqiad build #33472: FAILURE in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33472/ [19:45:27] bd808|LUNCH: my apologies [19:45:43] didn't mean to nerd snipe you, was just wanting to confirm my frequency of failures suspicion [19:45:55] greg-g: :) [20:04:34] Yippee, build fixed! [20:04:34] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #409: FIXED in 1 hr 4 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/409/ [20:35:22] Yippee, build fixed! [20:35:22] Project beta-scap-eqiad build #33477: FIXED in 1 min 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33477/ [20:38:49] Yippee, build fixed! [20:38:50] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #238: FIXED in 15 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/238/ [20:41:47] Yippee, build fixed! [20:41:47] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #168: FIXED in 59 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/168/ [21:05:24] Project beta-scap-eqiad build #33480: FAILURE in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33480/ [21:15:19] Yippee, build fixed! [21:15:20] Project beta-scap-eqiad build #33481: FIXED in 1 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33481/ [21:26:13] the UploadWizard UI is a piece of work. [21:34:52] Yippee, build fixed! [21:34:52] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #377: FIXED in 56 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/377/ [21:45:22] Yippee, build fixed! [21:45:23] Project beta-scap-eqiad build #33484: FIXED in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33484/ [21:47:08] !log Jenkins re adding [https://integration.wikimedia.org/ci/computer/integration-slave1009/ integration-slave1009] to the pool of slaves [21:47:11] Logged the message, Master [21:48:02] chrismcmahon: sorry about ignoring whatever you asked earlier [21:48:14] chrismcmahon: was about to leave and trying to fix up some hhvm related patch :-d [21:49:19] hashar: no problem. low priority, add some builds in jjb: https://gerrit.wikimedia.org/r/#/c/178714/ I still need to push those out, but I don't have the privileges to merge the change [21:53:40] Yippee, build fixed! [21:53:40] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #220: FIXED in 13 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/220/ [21:57:24] 3Quality-Assurance, Release-Engineering: Create a basic RSpec unit test for operations/puppet - https://phabricator.wikimedia.org/T78342#842870 (10hashar) 3NEW [22:05:45] Project beta-scap-eqiad build #33486: FAILURE in 1 min 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33486/ [22:11:27] 3Quality-Assurance, Release-Engineering: role/phabricator.pp include a password class in the global puppet scope - https://phabricator.wikimedia.org/T78344#842919 (10hashar) 3NEW [22:12:30] chrismcmahon: yeah just push the jobs :-) this way they will start running [22:12:39] chrismcmahon: I am adding you to the Gerrit 'integration' group so you can +2 them [22:12:45] but be extremely careful :-] [22:13:04] namely, changes affecting Zuul require a manual deploy on the Zuul server [22:13:23] !log Adding chrismcmahon to the 'integration' Gerrit group so he can +2 changes made to integration/config.git [22:13:26] Logged the message, Master [22:17:15] thanks hashar [22:17:38] I'll probably push those new builds out tomorrow with Zeljko. I should have done it today but I forgot. [22:18:59] 3Release-Engineering, Quality-Assurance: role/phabricator.pp include a password class in the global puppet scope - https://phabricator.wikimedia.org/T78344#842953 (10mmodell) I thought that was kinda odd myself. But how does moving the include resolve the issue? Wouldn't the stub still be needed in either case? [22:19:50] chrismcmahon: and we will really need to make the jobs runnable in parallel again [22:20:05] chrismcmahon: the whole run takes too long [22:36:40] (03PS1) 10Hashar: Job operations-puppet-bundle-rspec [integration/config] - 10https://gerrit.wikimedia.org/r/179244 [22:49:39] (03CR) 10Hashar: [C: 032] Job operations-puppet-bundle-rspec [integration/config] - 10https://gerrit.wikimedia.org/r/179244 (owner: 10Hashar) [22:50:35] 3Quality-Assurance, Release-Engineering: Create a basic RSpec unit test for operations/puppet - https://phabricator.wikimedia.org/T78342#843011 (10hashar) p:5Triage>3Low [22:51:22] greg-g: I went wild this week and started a single unit test for operations/puppet based on RSpec. Paired with someone at my coworking place :] [22:53:45] (03Merged) 10jenkins-bot: Job operations-puppet-bundle-rspec [integration/config] - 10https://gerrit.wikimedia.org/r/179244 (owner: 10Hashar) [22:54:20] 6 minutes till friday, lets deploy! [22:59:33] hashar: oh cool [22:59:45] greg-g: yeah marxarelli is in love of it apparently [22:59:59] I am suspecting him to RSpec the world [23:01:21] subject { World }; it { is_expected.to be_conquered_by(marxarelli) } [23:02:09] hashar: i'm going to steal your rspec setup for mw-vagrant :) [23:02:22] suuuure [23:02:49] hashar: btw, should we be targeting Ruby >= 1.9 with those specs, or do we need to maintain 1.8 compatibility? [23:02:56] drop 1.8 [23:03:03] the Jenkins jobs run on Trusty [23:03:15] most of the puppet tests I've ever seen were fragile and useless. I'd love to see some that aren't. [23:04:34] up to us to write them! [23:04:35] bd808: rspec-puppet seems a little strange and undermaintained, but based on hashar has come up with i think we can implement some useful tests [23:04:52] there might be some better gem to test puppet [23:05:23] there is https://github.com/puppetlabs/beaker/wiki/How-to-Write-a-Beaker-Test-for-a-Module [23:05:34] apparently boots vagrant machine to do the tests [23:07:00] hashar: interesting. fwict, it's geared more towards integration/acceptance testing [23:07:32] marxarelli: I can confirm the *-bundle-* jobs are using gem2.0 . I think that is set the shell shebang to point to ruby2 [23:07:51] marxarelli: yup. So for unit tests, I am not aware of anything else but rspec-puppet [23:09:12] hashar: no worries. i think what you have so far is good [23:09:38] we would need some default entry point at the root of the repo to easily run all tests [23:09:56] right now, bundle exec rspec needs to be passed a path (ex: modules/wmflib ) [23:10:07] and I would love to be able to generate a code coverage report :D [23:10:55] example: https://travis-ci.org/garethr/garethr-nginx/jobs/17597307#L105 [23:11:10] it is in rspec-puppet 2.0.0 that we use ( http://www.morethanseven.net/2014/01/25/code-coverage-for-puppet-modules/ ) [23:12:48] hashar: you could create a Rakefile (i think rspec-puppet-init does this by default) and specify a default path for rspec [23:13:08] hashar: http://rspec-puppet.com/setup/ (under Manual Setup) [23:13:19] yeah potentially [23:13:39] or use a Jenkins job that spawn a job per /spec/ directory found [23:13:52] this way they run in parallel [23:14:22] !bug hashar.clone() should provide more copies [23:29:18] I am off [23:29:24] marxarelli: thanks for your enthusiasm :] [23:30:00] hashar: thanks for Getting Shit Done(tm)! [23:41:12] late to the conversation, but we should never have to deal with Ruby 1.8 marxarelli. that way lies madness. and many many dead ends. [23:41:38] yeah with Zeljkof we decided to use gem2.0 / ruby2.0 on Trusty instance [23:41:45] I guess because 1.9 is legacy nowadays [23:41:54] chrismcmahon: that's good to hear. :) i wasn't sure if ops was maintaining 1.8 compatibility for puppet or not [23:42:35] native ruby libs for puppet modules that is [23:42:43] marxarelli: since we've updated Ubuntu/Debian everywhere, Ruby 1.8 anywhere should be considered a bug. [23:43:46] you can ask #wikimedia-operations :] [23:43:59] they most probably use Trusty for the puppet master, and hopefully ruby2.0 as well [23:44:38] * marxarelli summons the all-knowing bd808 for answers [23:44:41] crashing to bed! have a good evening! [23:44:48] hashar: night! [23:44:57] I think we should support 1.9(.3) on general principles, but prefer 2.x [23:45:04] thanks hashar [23:45:42] I think there are still a couple of trusty hosts in prod [23:46:19] err I mean 10.04 [23:46:22] bd808: do you mean precise? [23:46:23] got it [23:46:35] There are lots of 12.04 boxes in prod [23:46:46] lots and lots [23:46:54] but now a large number of 14.04 too [23:47:29] The elasticsearch boxes are trusty as are the app servers and the api servers [23:47:35] are y'all running puppet with 1.9 on the boxes running >= 12.04? [23:47:47] with *ruby* 1.9? [23:48:25] not sure... [23:48:43] I know they had to backport things to get puppet3 on 12.04 [23:48:58] I don't know if that included rubies or not [23:51:40] bd808: rgr that. i'll bug ori about it [23:57:01] most are on 1.9, but there are some that aren't [23:59:24] (03PS1) 10MaxSem: Disable npm checks for MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/179345