[00:18:01] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<40.00%) [02:45:43] 3Continuous-Integration, Citoid, VisualEditor: Set up CI in the mediawiki/services/citoid.git repo - https://phabricator.wikimedia.org/T76069#823699 (10Jdforrester-WMF) 5Open>3Resolved [03:22:39] Yippee, build fixed! [03:22:40] Project browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce build #210: FIXED in 16 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce/210/ [03:34:30] Yippee, build fixed! [03:34:31] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #120: FIXED in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/120/ [03:56:28] Yippee, build fixed! [03:56:28] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #174: FIXED in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/174/ [04:15:07] Project beta-scap-eqiad build #32790: FAILURE in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32790/ [04:35:10] Yippee, build fixed! [04:35:10] Project beta-scap-eqiad build #32792: FIXED in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32792/ [06:10:29] Yippee, build fixed! [06:10:29] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #368: FIXED in 54 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/368/ [06:38:04] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [08:30:13] Project browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce build #333: FAILURE in 1 hr 18 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce/333/ [10:07:22] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:35:27] Project beta-scap-eqiad build #32828: FAILURE in 1 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32828/ [10:37:23] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [10:50:02] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #330: FAILURE in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/330/ [10:55:24] Yippee, build fixed! [10:55:24] Project beta-scap-eqiad build #32830: FIXED in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32830/ [11:27:34] Project browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce build #348: FAILURE in 1 hr 21 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce/348/ [13:18:30] 3Quality-Assurance: rubocop fixes in mediawiki/selenium - https://phabricator.wikimedia.org/T75898#823816 (10Stan3) so I should rebase on env-abstraction-layer? [16:55:10] Project beta-scap-eqiad build #32866: FAILURE in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32866/ [17:15:19] Yippee, build fixed! [17:15:19] Project beta-scap-eqiad build #32868: FIXED in 1 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32868/ [17:16:42] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:41:46] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [18:22:40] Project browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce build #211: FAILURE in 16 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce/211/ [19:37:42] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:38:40] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:48:20] 3Continuous-Integration: Zuul: Implement support for customizing status_url to include the change.id - https://phabricator.wikimedia.org/T65744#823927 (10Krinkle) >>! In T65744#788834, @hashar wrote: > Has been done upstream by https://review.openstack.org/#/c/124390/ I'm not sure how that change relates to you... [19:48:35] 3Continuous-Integration: Zuul: Implement support for customizing status_url to include the change.id - https://phabricator.wikimedia.org/T65744#823928 (10Krinkle) [22:35:19] RECOVERY - Puppet failure on deployment-mx is OK: OK: Less than 1.00% above the threshold [0.0] [22:36:33] RECOVERY - Puppet failure on deployment-sca-cache01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:54:49] Krinkle|detached: all the integration puppet failures emails you were getting were just symptoms of a systemwide puppet failure that’s since been corrected. [22:54:52] just a fyi [23:15:09] Project beta-scap-eqiad build #32902: FAILURE in 1 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32902/ [23:32:24] Yippee, build fixed! [23:32:25] Project browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce build #334: FIXED in 1 hr 19 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce/334/ [23:35:12] Yippee, build fixed! [23:35:12] Project beta-scap-eqiad build #32904: FIXED in 1 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32904/ [23:46:31] PROBLEM - Free space - all mounts on deployment-cache-upload02 is CRITICAL: CRITICAL: deployment-prep.deployment-cache-upload02.diskspace._srv_vdb.byte_percentfree.value (<100.00%)