[00:12:25] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [00:21:09] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0] [00:46:09] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [00:53:41] 3Phabricator, Release-Engineering: Answer questions about ongoing maintenance of phabricator customizations/extensions - https://phabricator.wikimedia.org/T78464#846549 (10RobLa-WMF) Thanks for laying all of this out, Mukunda. I think it's ok for us to have a temporary policy of generally postponing local custo... [01:23:28] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [01:25:38] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [01:50:40] RECOVERY - Puppet failure on deployment-parsoidcache02 is OK: OK: Less than 1.00% above the threshold [0.0] [01:53:26] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [01:55:07] Project beta-scap-eqiad build #33798: FAILURE in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33798/ [02:10:29] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<11.11%) [02:15:28] Yippee, build fixed! [02:15:28] Project beta-scap-eqiad build #33800: FIXED in 1 min 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33800/ [02:55:27] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<11.11%) [03:14:52] Project beta-scap-eqiad build #33806: FAILURE in 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33806/ [03:24:12] Project browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce build #224: FAILURE in 18 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce/224/ [03:35:15] Yippee, build fixed! [03:35:15] Project beta-scap-eqiad build #33808: FIXED in 1 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33808/ [03:40:24] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [04:11:16] PROBLEM - Puppet staleness on deployment-sca-cache01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [04:42:27] 3Phabricator, Release-Engineering: Answer questions about ongoing maintenance of phabricator customizations/extensions - https://phabricator.wikimedia.org/T78464#846628 (10mmodell) @robla-wmf: so far most of the requests have been via IRC, and I'm sure a few are being tracked in phabricator but I'll have to sear... [05:02:19] 3Phabricator, Phabricator.org, Release-Engineering: Answer questions about ongoing maintenance of phabricator customizations/extensions - https://phabricator.wikimedia.org/T78464#846640 (10mmodell) Here are a few other customization suggestions/requests that we've either declined or set aside for future consider... [05:11:20] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [05:13:01] So I c an't seem to find any way to increase the space allocated to /var/ on deployment-bastion.... it looks to me like it will continue to fill up because the /var volume is far too tiny. Is there no way to grow a volume on labs instances? [05:15:19] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [05:39:45] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [05:41:18] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [06:16:37] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0] [06:17:19] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [06:26:03] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:31:42] 3Phabricator, Phabricator.org, Release-Engineering: Answer questions about ongoing maintenance of phabricator customizations/extensions - https://phabricator.wikimedia.org/T78464#846678 (10MZMcBride) >>! In T78464#846549, @RobLa-WMF wrote: > We should be careful about "rejecting" these changes purely on the grou... [06:41:35] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:42:19] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [06:46:46] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #1139: FAILURE in 45 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/1139/ [06:49:05] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [06:55:53] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #333: FAILURE in 27 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/333/ [06:56:04] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [07:14:06] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [08:15:20] Project beta-scap-eqiad build #33836: FAILURE in 1 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33836/ [08:30:06] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [08:30:33] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #189: FAILURE in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/189/ [08:31:21] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #62: FAILURE in 2 min 6 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/62/ [08:31:50] Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #216: FAILURE in 1 min 16 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/216/ [08:32:38] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:33:18] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:35:14] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:35:23] Yippee, build fixed! [08:35:23] Project beta-scap-eqiad build #33838: FIXED in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33838/ [08:44:27] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:45:44] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #338: FAILURE in 8 min 28 sec: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/338/ [08:46:38] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:47:01] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #288: FAILURE in 1 min 16 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/288/ [08:47:04] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:22:22] Project browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce build #362: FAILURE in 1 hr 24 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce/362/ [09:35:15] Project beta-scap-eqiad build #33844: FAILURE in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33844/ [09:55:32] Yippee, build fixed! [09:55:32] Project beta-scap-eqiad build #33846: FIXED in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33846/ [11:45:20] PROBLEM - Puppet failure on deployment-parsoid04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:55:33] Project beta-scap-eqiad build #33858: FAILURE in 1 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33858/ [11:57:51] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [12:06:13] Yippee, build fixed! [12:06:14] Project beta-scap-eqiad build #33859: FIXED in 1 min 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33859/ [12:10:19] RECOVERY - Puppet failure on deployment-parsoid04 is OK: OK: Less than 1.00% above the threshold [0.0] [12:27:49] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [12:46:43] Yippee, build fixed! [12:46:44] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #1140: FIXED in 42 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/1140/ [14:35:10] Project beta-scap-eqiad build #33874: FAILURE in 1 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33874/ [14:44:07] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #86: FAILURE in 27 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/86/ [14:55:13] Yippee, build fixed! [14:55:14] Project beta-scap-eqiad build #33876: FIXED in 1 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33876/ [15:55:03] Project beta-scap-eqiad build #33882: FAILURE in 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33882/ [16:15:18] Yippee, build fixed! [16:15:18] Project beta-scap-eqiad build #33884: FIXED in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33884/ [17:15:07] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:18:18] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [17:20:24] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:34] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:15] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:27] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:31:39] RECOVERY - Puppet failure on deployment-parsoidcache02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:37:03] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [18:26:31] Yippee, build fixed! [18:26:31] Project browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce build #225: FIXED in 20 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce/225/ [18:45:26] Project beta-scap-eqiad build #33899: FAILURE in 1 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33899/ [18:55:07] Yippee, build fixed! [18:55:08] Project beta-scap-eqiad build #33900: FIXED in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33900/ [20:55:34] Project beta-scap-eqiad build #33912: FAILURE in 1 min 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33912/ [21:15:37] Yippee, build fixed! [21:15:38] Project beta-scap-eqiad build #33914: FIXED in 1 min 36 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33914/ [22:12:39] Yippee, build fixed! [22:12:39] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #334: FIXED in 42 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/334/ [22:15:07] Project beta-scap-eqiad build #33918: FAILURE in 1 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33918/ [22:35:09] Yippee, build fixed! [22:35:10] Project beta-scap-eqiad build #33920: FIXED in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33920/ [22:47:30] 3Release-Engineering, Quality-Assurance: role/phabricator.pp include a password class in the global puppet scope - https://phabricator.wikimedia.org/T78344#847061 (10Aklapper) p:5Triage>3Low [23:22:35] Yippee, build fixed! [23:22:35] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #63: FIXED in 2 min 1 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/63/ [23:23:53] Yippee, build fixed! [23:23:53] Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #217: FIXED in 1 min 17 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/217/ [23:48:55] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]