[00:12:46] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [00:14:39] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [00:15:14] Project beta-scap-eqiad build #36756: FAILURE in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36756/ [00:20:41] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [00:25:13] Project beta-scap-eqiad build #36757: STILL FAILING in 1 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36757/ [00:32:20] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [00:35:43] Yippee, build fixed! [00:35:44] Project beta-scap-eqiad build #36758: FIXED in 1 min 46 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36758/ [00:42:46] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:44:04] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [00:44:40] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [00:45:42] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [00:46:32] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [00:47:32] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [00:52:43] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [00:55:31] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<10.00%) [00:57:23] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:09:40] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [01:12:32] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [01:16:34] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [01:22:46] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:27:46] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [01:30:31] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<20.00%) [01:39:40] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [01:39:45] RECOVERY - Puppet failure on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:50:31] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<11.11%) [01:57:45] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [02:04:42] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [03:16:51] Yippee, build fixed! [03:16:52] Project browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce build #266: FIXED in 10 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce/266/ [03:17:55] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [03:33:55] Yippee, build fixed! [03:33:55] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #177: FIXED in 31 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/177/ [03:42:57] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [03:46:19] Yippee, build fixed! [03:46:20] Project browsertests-Flow-test2.wikipedia.org-windows_8-internet_explorer-sauce build #371: FIXED in 45 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-windows_8-internet_explorer-sauce/371/ [04:04:30] Yippee, build fixed! [04:04:31] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #285: FIXED in 9 min 56 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/285/ [04:34:03] Yippee, build fixed! [04:34:03] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #215: FIXED in 42 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/215/ [04:37:05] Yippee, build fixed! [04:37:06] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #358: FIXED in 3 min 2 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/358/ [04:45:27] Yippee, build fixed! [04:45:28] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #266: FIXED in 8 min 21 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/266/ [04:45:41] Yippee, build fixed! [04:45:42] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #423: FIXED in 41 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/423/ [05:04:11] Yippee, build fixed! [05:04:11] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #430: FIXED in 40 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/430/ [05:16:58] Yippee, build fixed! [05:16:58] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #434: FIXED in 20 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/434/ [05:32:32] Yippee, build fixed! [05:32:32] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #466: FIXED in 47 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/466/ [06:27:21] Yippee, build fixed! [06:27:22] Project browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce build #373: FIXED in 41 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce/373/ [06:35:01] Project beta-scap-eqiad build #36794: FAILURE in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36794/ [06:40:35] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [06:45:17] Project beta-scap-eqiad build #36795: STILL FAILING in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36795/ [06:51:17] Yippee, build fixed! [06:51:18] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #385: FIXED in 23 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/385/ [06:55:43] Yippee, build fixed! [06:55:43] Project beta-scap-eqiad build #36796: FIXED in 1 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36796/ [08:47:31] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [09:17:29] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [10:55:04] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:07:27] Project beta-scap-eqiad build #36821: FAILURE in 3 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36821/ [11:16:27] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:17:57] Yippee, build fixed! [11:17:57] Project beta-scap-eqiad build #36822: FIXED in 3 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36822/ [11:20:03] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [11:41:25] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:45:37] Project beta-scap-eqiad build #36825: FAILURE in 1 min 35 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36825/ [11:48:45] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:51:42] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:52:25] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:55:24] Project beta-scap-eqiad build #36826: STILL FAILING in 1 min 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36826/ [12:05:41] Yippee, build fixed! [12:05:42] Project beta-scap-eqiad build #36827: FIXED in 1 min 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36827/ [12:16:44] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:18:46] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:22:24] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:54:51] Project beta-scap-eqiad build #36832: FAILURE in 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36832/ [13:05:12] Project beta-scap-eqiad build #36833: STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36833/ [13:15:19] Yippee, build fixed! [13:15:19] Project beta-scap-eqiad build #36834: FIXED in 1 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36834/ [14:45:39] Project beta-scap-eqiad build #36843: FAILURE in 1 min 46 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36843/ [14:47:25] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:49:44] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:53:35] PROBLEM - Puppet failure on deployment-sentry2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:55:25] Yippee, build fixed! [14:55:25] Project beta-scap-eqiad build #36844: FIXED in 1 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36844/ [15:13:43] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:17:26] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:19:46] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:25:20] Project beta-scap-eqiad build #36847: FAILURE in 1 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36847/ [15:35:25] Yippee, build fixed! [15:35:25] Project beta-scap-eqiad build #36848: FIXED in 1 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36848/ [15:38:46] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:42:43] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:07:44] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:49:42] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:15:39] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:19:45] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:40:39] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [17:41:43] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:48:39] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:52:47] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:06:45] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:17:46] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:18:38] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [18:50:59] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:07:58] Yippee, build fixed! [19:07:58] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #457: FIXED in 50 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/457/ [19:14:55] Project beta-scap-eqiad build #36869: FAILURE in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36869/ [19:15:59] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [19:17:43] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [19:24:53] Project beta-scap-eqiad build #36870: STILL FAILING in 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36870/ [19:33:26] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [19:35:30] Yippee, build fixed! [19:35:30] Project beta-scap-eqiad build #36871: FIXED in 1 min 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36871/ [19:47:42] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:48:28] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [19:50:45] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [19:55:40] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [19:58:25] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:06:10] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #431: FAILURE in 41 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/431/ [20:13:33] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [20:20:43] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:25:44] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:48:43] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:58:41] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:18:46] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:28:41] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:39:07] Project beta-scap-eqiad build #36880: FAILURE in 30 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36880/ [21:43:18] Yippee, build fixed! [21:43:19] Project beta-scap-eqiad build #36881: FIXED in 2 min 49 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36881/ [21:45:42] Yippee, build fixed! [21:45:43] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #231: FIXED in 39 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/231/ [21:54:28] 3Continuous-Integration: Upgrade tox on integration slaves - https://phabricator.wikimedia.org/T85662#954110 (10hashar) Thanks to have tracked the issue. I remember now I have been hit by that tox regression locally when 1.7 got released but never bothered with it. I will apply `pip install --upgrade tox` on al... [23:34:14] 3MediaWiki-Documentation, Continuous-Integration: Doxygen 1.8.x is too slow on Trusty labs instance - https://phabricator.wikimedia.org/T75311#954214 (10Aklapper) [23:45:05] (03PS4) 10BryanDavis: Add job template for running composer scripts [integration/config] - 10https://gerrit.wikimedia.org/r/174410 (owner: 10Hashar) [23:45:12] (03CR) 10BryanDavis: [C: 031] Add job template for running composer scripts [integration/config] - 10https://gerrit.wikimedia.org/r/174410 (owner: 10Hashar) [23:46:29] PROBLEM - Free space - all mounts on deployment-cache-upload02 is CRITICAL: CRITICAL: deployment-prep.deployment-cache-upload02.diskspace._srv_vdb.byte_percentfree.value (<100.00%) [23:55:46] 3Librarization, Continuous-Integration: Set up composer validate job for operations/mediawiki-config - https://phabricator.wikimedia.org/T76621#954260 (10bd808) a:3Legoktm [23:56:10] 3Librarization, Continuous-Integration, MediaWiki-Core-Team: Set up composer validate job for operations/mediawiki-config - https://phabricator.wikimedia.org/T76621#807514 (10bd808)