[02:21:15] Yippee, build fixed! [02:21:16] Project beta-update-databases-eqiad build #7042: FIXED in 1 min 15 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/7042/ [03:41:31] Yippee, build fixed! [03:41:31] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #267: FIXED in 34 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/267/ [03:52:54] Yippee, build fixed! [03:52:55] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #431: FIXED in 11 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/431/ [03:57:10] Yippee, build fixed! [03:57:10] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #493: FIXED in 46 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/493/ [04:05:29] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<11.11%) [04:27:37] Yippee, build fixed! [04:27:38] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #251: FIXED in 35 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/251/ [04:30:35] Yippee, build fixed! [04:30:35] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #394: FIXED in 2 min 57 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/394/ [04:35:22] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #459: FAILURE in 34 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/459/ [04:37:32] Yippee, build fixed! [04:37:32] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #267: FIXED in 44 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/267/ [04:54:38] Project beta-scap-eqiad build #39014: FAILURE in 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39014/ [05:04:44] Project beta-scap-eqiad build #39015: STILL FAILING in 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39015/ [05:15:04] Yippee, build fixed! [05:15:04] Project beta-scap-eqiad build #39016: FIXED in 1 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39016/ [05:17:04] 3Beta-Cluster: deployment-mx is its own puppetmaster - https://phabricator.wikimedia.org/T86575#989579 (10yuvipanda) /me pokes @Jgreen [05:20:29] Yippee, build fixed! [05:20:30] Project browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce build #424: FIXED in 39 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce/424/ [05:23:07] Yippee, build fixed! [05:23:08] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #470: FIXED in 19 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/470/ [06:13:18] Project browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce build #420: FAILURE in 2 min 57 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce/420/ [06:35:33] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [06:38:20] Yippee, build fixed! [06:38:20] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #266: FIXED in 42 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/266/ [06:40:49] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [06:50:27] Yippee, build fixed! [06:50:27] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #415: FIXED in 15 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/415/ [06:52:42] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [07:17:43] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:29:29] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:59:28] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:34:28] Project beta-scap-eqiad build #39045: FAILURE in 30 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39045/ [10:38:41] Yippee, build fixed! [10:38:41] Project beta-scap-eqiad build #39046: FIXED in 2 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39046/ [11:04:54] PROBLEM - Puppet failure on deployment-cache-mobile03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [11:09:14] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:12:00] 3Continuous-Integration: Remove dependency on git.wikimedia.org - https://phabricator.wikimedia.org/T74001#989766 (10JanZerebecki) [11:14:12] 3Wikidata, Continuous-Integration: Remove dependency on git.wikimedia.org - https://phabricator.wikimedia.org/T74001#989768 (10JanZerebecki) [11:14:55] Project beta-scap-eqiad build #39050: FAILURE in 47 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39050/ [11:24:56] Project beta-scap-eqiad build #39051: STILL FAILING in 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39051/ [11:29:54] RECOVERY - Puppet failure on deployment-cache-mobile03 is OK: OK: Less than 1.00% above the threshold [0.0] [11:34:16] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [11:37:04] Yippee, build fixed! [11:37:05] Project beta-scap-eqiad build #39052: FIXED in 2 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39052/ [12:09:40] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [12:34:40] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [12:37:36] PROBLEM - SSH on deployment-lucid-salt is CRITICAL: Connection refused [12:43:36] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [12:53:41] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [12:58:57] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [13:08:34] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [13:10:42] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:16:05] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:17:56] 3Phabricator, Phabricator.org: Get rid of "Export to Excel" - https://phabricator.wikimedia.org/T257#989882 (10Qgil) >>! In T257#984606, @valhallasw wrote: > * While PHPExcel is not installed, we can use "Export (disabled)" to clarify it's not working yet: {F29549} > * Afterwards, we can simply use "Export": > {... [13:18:25] 3Phabricator: Get rid of "Export to Excel" - https://phabricator.wikimedia.org/T257#3247 (10Qgil) [13:18:43] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:23:56] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [13:26:11] 3Phabricator: Phabricator mails Message-ID has localhost.localdomain - https://phabricator.wikimedia.org/T75713#989904 (10Qgil) For what is worth, notifications from secure.phabricator.org don't have the Message-ID you would expect either. Message-ID: <0000014b106f970e-2392ca7b-0351-4546-aec0-c44f159e8580-000... [13:29:08] 3Phabricator: Reduce or eliminate bracketed labels to shorten phabricator emails subjects - https://phabricator.wikimedia.org/T650#989915 (10Qgil) It would be useful to have someone with a vested interest in this feature leading the discussion until reaching an agreement. The implementation of the solution agree... [13:33:46] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:33:50] PROBLEM - Puppet failure on deployment-sca-cache01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:34:45] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [13:35:41] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:39:25] 3Phabricator, Wikimedia-Labs-Other: phab-01.wmflabs.org test instance's statuses are out of date - https://phabricator.wikimedia.org/T76943#989926 (10Qgil) Would it be good enough to just edit https://phab-01.wmflabs.org/config/edit/maniphest.statuses/ ? Is there documentation anywhere about how to update the L... [13:41:09] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [13:52:55] 3Engineering-Community, pywikibot-core, Phabricator: Projects for outreach programs (FOSS OPW, Google Code-in...) - https://phabricator.wikimedia.org/T1148#989940 (10Qgil) We will know whether Wikimedia has been accepted to GSoC 2015 by February 20. For updates and questions, please go to {T921} [13:56:24] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:58:44] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [13:58:49] RECOVERY - Puppet failure on deployment-sca-cache01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:59:45] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:09:00] 3Phabricator: Searchable "Reference" custom field - https://phabricator.wikimedia.org/T991#989957 (10Qgil) >>! In T991#790795, @Nemo_bis wrote: >> Would it be feasible to have the reference field indexed in the general search, so users could simply introduce i.e. "bz1234" in order to find the corresponding Phabr... [14:17:42] 3Phabricator: Change color of #Patch-For-Review project to something more unique - https://phabricator.wikimedia.org/T87226#989959 (10Qgil) I agree with Andre and Chase, and I was also thinking on red, which we still haven't used as initially planned: https://www.mediawiki.org/wiki/Phabricator/Creating_and_renam... [14:21:24] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:35:49] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:37:05] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:51:42] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:55:45] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:56:00] Project beta-scap-eqiad build #39072: FAILURE in 2 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39072/ [15:00:51] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:02:05] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [15:04:55] Yippee, build fixed! [15:04:56] Project beta-scap-eqiad build #39073: FIXED in 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39073/ [15:16:44] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:16:44] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:20:43] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:23:41] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:29:28] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:36:45] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0] [15:36:45] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:41:39] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [15:43:43] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:48:43] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [15:54:30] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [15:56:28] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [15:56:44] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [15:57:26] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:01:42] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:03:43] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:16:24] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:22:27] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:30:19] 3Continuous-Integration: Make sure everyone in RelEng has sudo on integration slaves in labs - https://phabricator.wikimedia.org/T86779#990026 (10greg) >>! In T86779#983404, @Krinkle wrote: > If you define "everyone", we can add them via https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration. "Everyone"... [16:38:40] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:41:53] 3Phabricator: Change color of #Patch-For-Review project to something more unique - https://phabricator.wikimedia.org/T87226#990030 (10matmarex) {F31151,size=full,alt=SHIP IT} [16:45:22] 3Phabricator: Change color of #Patch-For-Review project to something more unique - https://phabricator.wikimedia.org/T87226#990034 (10Jdforrester-WMF) Maybe we could make it invisible so it doesn't waste everyone's time? [17:03:42] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [17:10:12] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:18:02] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:32:41] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:34:58] Project beta-scap-eqiad build #39088: FAILURE in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39088/ [17:35:12] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [17:38:04] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [17:39:24] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:39:58] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:40:16] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:44:42] Project beta-scap-eqiad build #39089: STILL FAILING in 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39089/ [17:55:30] Yippee, build fixed! [17:55:30] Project beta-scap-eqiad build #39090: FIXED in 1 min 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39090/ [17:57:42] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [18:04:22] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [18:04:56] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [18:05:12] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [18:18:35] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #422: FAILURE in 8 min 35 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/422/ [18:20:55] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:37:39] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:39:29] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #432: FAILURE in 17 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/432/ [18:44:45] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:45:55] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [18:53:45] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:59:44] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [19:01:59] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [19:02:08] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #359: FAILURE in 1.6 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/359/ [19:02:09] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #141: FAILURE in 0.79 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/141/ [19:02:11] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #438: FAILURE in 0.68 sec: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/438/ [19:02:13] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #395: FAILURE in 0.64 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/395/ [19:02:16] Project browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce build #397: FAILURE in 1 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce/397/ [19:02:17] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #363: FAILURE in 0.5 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/363/ [19:02:21] Project browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce build #335: FAILURE in 0.54 sec: https://integration.wikimedia.org/ci/job/browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce/335/ [19:02:22] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #76: FAILURE in 0.21 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/76/ [19:02:23] Project browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce build #425: FAILURE in 0.49 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce/425/ [19:02:39] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [19:04:40] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [19:07:59] Project browsertests-VisualEditor-test2.wikipedia.org-linux-chrome-sauce build #444: FAILURE in 21 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-chrome-sauce/444/ [19:14:41] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:15:32] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #494: FAILURE in 1 hr 4 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/494/ [19:16:47] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [19:18:45] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:21:55] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [19:22:50] PROBLEM - Puppet failure on deployment-mediawiki04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:24:43] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:26:20] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #471: FAILURE in 23 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/471/ [19:34:43] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [19:41:46] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:44:40] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [19:45:58] PROBLEM - Puppet failure on deployment-cache-mobile03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [19:47:53] RECOVERY - Puppet failure on deployment-mediawiki04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:54:20] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #268: FAILURE in 59 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/268/ [19:57:48] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [19:59:42] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:03:53] PROBLEM - Puppet failure on deployment-mediawiki04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [20:06:32] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:10:54] RECOVERY - Puppet failure on deployment-cache-mobile03 is OK: OK: Less than 1.00% above the threshold [0.0] [20:12:06] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #421: FAILURE in 45 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/421/ [20:17:54] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:21:52] PROBLEM - Puppet failure on deployment-cache-mobile03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:22:47] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:28:53] RECOVERY - Puppet failure on deployment-mediawiki04 is OK: OK: Less than 1.00% above the threshold [0.0] [20:32:27] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:36:29] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [20:42:56] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [20:46:54] RECOVERY - Puppet failure on deployment-cache-mobile03 is OK: OK: Less than 1.00% above the threshold [0.0] [20:50:48] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:57:28] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:01:42] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [21:13:28] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:13:28] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [21:14:44] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #267: FAILURE in 1 hr 2 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/267/ [21:15:46] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:21:51] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:25:39] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [21:29:10] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #416: FAILURE in 15 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/416/ [21:38:24] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:38:26] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:42:41] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:45:39] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [22:00:03] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:01:43] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:02:42] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:09:57] (03Abandoned) 10Krinkle: Add .jsonlintignore feature to the json-lint.php helper script. [integration/jenkins] - 10https://gerrit.wikimedia.org/r/174047 (owner: 10Cscott) [22:11:02] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [22:13:39] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:18:39] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:20:03] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [22:26:42] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [22:35:59] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [22:38:42] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [22:43:40] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:48:49] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [22:53:45] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [23:13:48] RECOVERY - Puppet failure on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [23:13:48] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [23:18:41] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:38:46] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:42:38] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:46:31] PROBLEM - Free space - all mounts on deployment-cache-upload02 is CRITICAL: CRITICAL: deployment-prep.deployment-cache-upload02.diskspace._srv_vdb.byte_percentfree.value (<100.00%) [23:54:58] Project beta-scap-eqiad build #39127: FAILURE in 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39127/