[00:05:03] (03PS9) 10KartikMistry: WIP: Add generic npm-set-env to fix npm on */deploy repos [integration/config] - 10https://gerrit.wikimedia.org/r/184609 [00:23:53] PROBLEM - Puppet failure on deployment-mediawiki04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [00:31:40] 3Continuous-Integration: Disk space "/var" full on integration-puppetmaster - https://phabricator.wikimedia.org/T87484#992405 (10Krinkle) 3NEW [00:48:51] RECOVERY - Puppet failure on deployment-mediawiki04 is OK: OK: Less than 1.00% above the threshold [0.0] [00:51:45] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [01:03:08] 3Phabricator: Update Phab on-wiki documentation about projects linking to workboards instead of project pages by default - https://phabricator.wikimedia.org/T87487#992475 (10Aklapper) 3NEW [01:04:19] 3Phabricator: Next Phabricator upgrade on YYYY-MM-DD - https://phabricator.wikimedia.org/T86772#992483 (10Aklapper) [01:04:20] 3Phabricator: Update Phab on-wiki documentation about projects linking to workboards instead of project pages by default - https://phabricator.wikimedia.org/T87487#992475 (10Aklapper) [01:05:10] 3Phabricator, operations: have any task put into ops-access-requests automatically generate an ops-access-review task - https://phabricator.wikimedia.org/T87467#992484 (10Aklapper) p:5Triage>3High [01:05:13] !log restarting Jenkins (deadlock on deployment-bastion slave) [01:05:16] Logged the message, Master [01:06:30] 3Phabricator: Add cscott to WMF_NDA. - https://phabricator.wikimedia.org/T87479#992265 (10Aklapper) [01:13:45] 3Phabricator: Searchable "Reference" custom field - https://phabricator.wikimedia.org/T991#992523 (10Aklapper) p:5Low>3Volunteer? [01:21:49] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:26:37] 3OOjs, Continuous-Integration: Publish QUnit coverage on integration.wikimedia.org - https://phabricator.wikimedia.org/T87490#992545 (10Krinkle) 3NEW [01:28:27] 3OOjs, Continuous-Integration: Publish QUnit coverage on integration.wikimedia.org - https://phabricator.wikimedia.org/T87490#992555 (10Krinkle) We'll need to proxy it through the integration-publisher labs instance since we aren't able to (and shouldn't) `rsync` directly from a Jenkins slave in labs to the `gal... [01:36:22] 3Phabricator: Add help link to explain meaning of priority levels - https://phabricator.wikimedia.org/T87411#992588 (10Aklapper) p:5Triage>3Volunteer? [01:42:07] (03PS1) 10Krinkle: [WIP] Rewrite beta-update-databases-eqiad jobs as one [integration/config] - 10https://gerrit.wikimedia.org/r/186559 [01:53:07] 3Phabricator: Update Phab on-wiki documentation about projects linking to workboards instead of project pages by default - https://phabricator.wikimedia.org/T87487#992626 (10Aklapper) ...and while we update docs, also see task 87358 [02:06:23] Does it irk anyone else that "LDAP User" uses pretty urls, but "MediaWiki User" doesn't? [02:16:46] greg-g: where are you? [02:17:38] Not seen him recently [02:18:43] Reedy: are you in the office [02:19:04] Reedy: where did you buy shoes? [03:56:02] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #435: FAILURE in 12 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/435/ [03:57:38] Yippee, build fixed! [03:57:38] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #497: FIXED in 46 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/497/ [04:31:20] Yippee, build fixed! [04:31:20] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #255: FIXED in 36 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/255/ [04:39:45] Yippee, build fixed! [04:39:46] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #463: FIXED in 35 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/463/ [04:42:13] Yippee, build fixed! [04:42:13] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #271: FIXED in 46 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/271/ [05:15:00] Project beta-scap-eqiad build #39283: FAILURE in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39283/ [05:25:06] Yippee, build fixed! [05:25:06] Project beta-scap-eqiad build #39284: FIXED in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39284/ [05:31:12] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #474: FAILURE in 20 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/474/ [05:51:01] 3Phabricator: Please remove the two-factor authentication from my Phabricator account - https://phabricator.wikimedia.org/T87495#992806 (10Aklapper) p:5Triage>3Low [05:54:49] 3Wikimedia-Fundraising-CiviCRM, Continuous-Integration: CI for Civi: provision and run tests under Jenkins/Zuul - https://phabricator.wikimedia.org/T86103#992821 (10Aklapper) [05:58:32] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<44.44%) [06:11:26] Yippee, build fixed! [06:11:27] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #426: FIXED in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/426/ [06:27:57] Project browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce build #424: FAILURE in 2 min 49 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce/424/ [06:38:30] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [06:39:46] 3Phabricator: Please remove the two-factor authentication from my Phabricator account - https://phabricator.wikimedia.org/T87495#992880 (10Aklapper) I am sorry for not following up on this earlier. There are currently no guidelines on this and how we could verify the request. Was the one-time token displayed to... [06:55:09] Project beta-scap-eqiad build #39294: FAILURE in 1 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39294/ [07:01:36] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #419: FAILURE in 15 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/419/ [07:05:35] Project beta-scap-eqiad build #39295: STILL FAILING in 1 min 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39295/ [07:07:38] 3Phabricator: Adapting Gadget-BugStatusUpdate.js to Phabricator - https://phabricator.wikimedia.org/T539#992894 (10Mattflaschen) I haven't done this yet, but I would like to keep it assigned for a little longer. [07:15:28] Yippee, build fixed! [07:15:28] Project beta-scap-eqiad build #39296: FIXED in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39296/ [08:25:54] PROBLEM - Puppet failure on deployment-cache-mobile03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:29:42] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:47:05] 3MediaWiki-Core-Team, Code-Review: Wikimedia code repository browser in Phabricator - https://phabricator.wikimedia.org/T752#992976 (10Nemo_bis) [08:50:53] RECOVERY - Puppet failure on deployment-cache-mobile03 is OK: OK: Less than 1.00% above the threshold [0.0] [08:54:24] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:55:26] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:56:36] 3MediaWiki-Core-Team, Code-Review: Wikimedia code repository browser in Phabricator - https://phabricator.wikimedia.org/T752#992979 (10Nemo_bis) [08:57:44] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [08:59:42] RECOVERY - Puppet failure on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:07:03] PROBLEM - Puppet failure on deployment-cache-mobile03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:15:10] Project beta-scap-eqiad build #39308: FAILURE in 1 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39308/ [09:17:47] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:19:25] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:20:33] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:21:06] 3Beta-Cluster, operations: Minimize differences between beta and production (Tracking) - https://phabricator.wikimedia.org/T87220#992980 (10yuvipanda) a:3yuvipanda [09:25:33] Yippee, build fixed! [09:25:34] Project beta-scap-eqiad build #39309: FIXED in 1 min 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39309/ [09:34:40] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:35:35] 3Phabricator: Please remove the two-factor authentication from my Phabricator account - https://phabricator.wikimedia.org/T87495#992986 (10zhaofengli) No, I didn't see any one-time token when activating the feature. I can provide my committed identity on [my enwiki user page](https://en.wikipedia.org/wiki/User:Z... [09:46:22] 3Phabricator: Please remove the two-factor authentication from my Phabricator account - https://phabricator.wikimedia.org/T87495#992990 (10zhaofengli) If using the committed identity is acceptable, how can I send you the source text in a secure way? [09:59:40] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [09:59:42] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:10:45] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [10:11:57] RECOVERY - Puppet failure on deployment-cache-mobile03 is OK: OK: Less than 1.00% above the threshold [0.0] [10:14:15] Project beta-code-update-eqiad build #41619: FAILURE in 1 min 14 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/41619/ [10:15:42] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:19:44] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [10:19:58] 3Quality-Assurance: use rspec-expectations expect syntax instead of should syntax - https://phabricator.wikimedia.org/T68369#993005 (10Physikerwelt) @Cmcmahon can you do a code review for this change? [10:22:11] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:24:55] PROBLEM - Puppet failure on deployment-mediawiki04 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [10:30:15] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:31:07] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [10:33:21] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [10:34:40] 3MediaWiki-Core-Team, Code-Review: Import all gerrit.wikimedia.org repositories with Diffusion - https://phabricator.wikimedia.org/T616#993011 (10Nemo_bis) Completed? I see over 800 repositories in diffusion, https://git.wikimedia.org/projects counts about 1200. [10:35:43] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [10:35:47] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:41:04] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:41:56] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:44:36] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:47:17] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [10:49:55] RECOVERY - Puppet failure on deployment-mediawiki04 is OK: OK: Less than 1.00% above the threshold [0.0] [10:53:44] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [10:53:58] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:55:14] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [10:56:01] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [10:58:20] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:01:04] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [11:03:11] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [11:04:35] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [11:06:55] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:06:57] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [11:12:07] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:17:47] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [11:18:47] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:18:58] RECOVERY - Puppet failure on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:20:41] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:22:57] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:23:11] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [11:31:40] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [11:36:55] RECOVERY - Puppet failure on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:37:03] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [11:47:11] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [11:51:32] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:56:26] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:56:38] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [12:00:12] PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [12:02:42] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [12:12:42] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [12:16:13] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [12:16:34] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [12:22:54] PROBLEM - Puppet failure on deployment-cache-mobile03 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [12:25:15] RECOVERY - Puppet failure on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [12:27:43] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [12:37:35] PROBLEM - SSH on deployment-lucid-salt is CRITICAL: Connection refused [12:37:41] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [12:38:41] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [12:41:43] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [12:44:02] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [12:47:54] RECOVERY - Puppet failure on deployment-cache-mobile03 is OK: OK: Less than 1.00% above the threshold [0.0] [12:52:46] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [13:03:43] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:03:55] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [13:03:55] PROBLEM - Puppet failure on deployment-cache-mobile03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [13:06:46] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [13:13:00] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [13:17:51] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [13:19:56] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [13:20:38] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [13:28:55] RECOVERY - Puppet failure on deployment-cache-mobile03 is OK: OK: Less than 1.00% above the threshold [0.0] [13:32:48] PROBLEM - Puppet failure on deployment-sca-cache01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [13:33:39] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:33:45] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:38:00] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [13:40:44] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [13:44:57] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [13:50:38] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [13:52:32] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [13:57:50] RECOVERY - Puppet failure on deployment-sca-cache01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:58:48] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:03:41] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [14:05:42] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:06:25] Project browsertests-Wikidata-PerformanceTests-linux-firefox-sauce build #129: FAILURE in 24 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-PerformanceTests-linux-firefox-sauce/129/ [14:12:25] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:13:48] PROBLEM - Puppet failure on deployment-sca-cache01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:17:28] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:34:38] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:35:20] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:37:29] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:38:41] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:38:51] RECOVERY - Puppet failure on deployment-sca-cache01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:40:57] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:44:43] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [14:53:26] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [14:59:42] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [15:00:13] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:00:19] RECOVERY - Puppet failure on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:02:43] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox ยป ru,contintLabsSlave && UbuntuTrusty build #10: FAILURE in 18 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=ru,label=contintLabsSlave%20&&%20UbuntuTrusty/10/ [15:03:45] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [15:04:41] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:04:57] PROBLEM - Puppet failure on deployment-cache-mobile03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:05:57] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [15:10:43] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:18:25] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:21:57] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [15:25:11] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [15:30:01] RECOVERY - Puppet failure on deployment-cache-mobile03 is OK: OK: Less than 1.00% above the threshold [0.0] [15:40:38] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [15:41:15] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:41:57] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [15:43:43] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:57:23] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:57:44] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:02:01] PROBLEM - Puppet failure on deployment-redis02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:06:18] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [16:08:46] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:13:25] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:22:24] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [16:22:42] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [16:32:03] RECOVERY - Puppet failure on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:43:27] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:55:44] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:57:11] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:12:33] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:25:41] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:21] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [17:42:33] RECOVERY - Puppet failure on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:53:50] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:01:36] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:01:40] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:14:45] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:18:47] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:19:44] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:24:54] 3MediaWiki-Core-Team, Code-Review: Import all gerrit.wikimedia.org repositories with Diffusion - https://phabricator.wikimedia.org/T616#993295 (10Chad) >>! In T616#993011, @Nemo_bis wrote: > Completed? I see over 800 repositories in diffusion, https://git.wikimedia.org/projects counts about 1200. Not quite. We'... [18:26:36] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [18:26:42] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:39:48] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:42:38] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:49:43] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [18:51:17] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [19:01:43] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [19:05:39] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #498: FAILURE in 54 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/498/ [19:06:06] Yippee, build fixed! [19:06:07] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #436: FIXED in 14 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/436/ [19:07:35] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [19:14:40] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:14:51] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [19:17:55] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:18:05] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [19:21:21] RECOVERY - Puppet failure on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:26:42] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:30:42] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:31:34] Yippee, build fixed! [19:31:34] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #426: FIXED in 1 hr 21 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/426/ [19:32:42] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [19:36:04] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #442: FAILURE in 2 min 5 sec: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/442/ [19:39:39] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [19:39:49] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:43:00] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [19:43:08] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:54:04] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:55:40] RECOVERY - Puppet failure on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:55:40] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [19:55:48] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:57:44] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:00:11] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #464: FAILURE in 42 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/464/ [20:09:58] 3pywikibot-core, Continuous-Integration: Whitelist people with +2 rights - https://phabricator.wikimedia.org/T87413#993347 (10Mpaa) If someone can explain to me how to get the +2 list, I can submit the patch. [20:14:51] 3pywikibot-core, Continuous-Integration: Whitelist people with +2 rights - https://phabricator.wikimedia.org/T87413#993348 (10Ricordisamoa) >>! In T87413#993347, @Mpaa wrote: > If someone can explain to me how to get the +2 list, I can submit the patch. https://gerrit.wikimedia.org/r/#/admin/groups/514,members [20:19:03] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [20:20:39] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [20:20:49] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:27:45] 3RESTBase, Phabricator: Create a restbase-usecase tag - https://phabricator.wikimedia.org/T87518#993358 (10GWicke) 3NEW [20:28:30] 3RESTBase, Phabricator: Create a restbase-usecase tag - https://phabricator.wikimedia.org/T87518#993365 (10GWicke) [20:29:04] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:29:22] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:50:52] 3Beta-Cluster, operations: Renumber apache user/group to uid=48 - https://phabricator.wikimedia.org/T78076#993380 (10yuvipanda) I've been talking to @faidon and @Joe about this over the last few days, hopefully we'll find a way to fix this before end of coming week. [20:53:36] Yippee, build fixed! [20:53:37] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #475: FIXED in 26 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/475/ [20:54:00] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [20:59:20] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:02:44] 3Release-Engineering: Rework beta apache config - https://phabricator.wikimedia.org/T1256#993396 (10yuvipanda) YES, we need to unify them into one set of config files that are actually templates that branch on prod vs labs (or even just use hiera to pick out the values necessary) [21:10:01] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:14:42] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:16:52] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [21:17:44] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:22:40] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:35:01] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [21:39:12] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:41:52] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:42:41] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:44:41] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [21:45:55] PROBLEM - Puppet failure on deployment-cache-mobile03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:47:41] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [21:51:01] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [21:51:37] 3Phabricator: Next Phabricator upgrade on YYYY-MM-DD - https://phabricator.wikimedia.org/T86772#993406 (10Aklapper) [21:52:11] 3MediaWiki-Core-Team, Code-Review: Import all gerrit.wikimedia.org repositories with Diffusion - https://phabricator.wikimedia.org/T616#993409 (10Aklapper) So I assume this is blocked by T87282 ? [22:04:11] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [22:10:26] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [22:10:56] RECOVERY - Puppet failure on deployment-cache-mobile03 is OK: OK: Less than 1.00% above the threshold [0.0] [22:15:59] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [22:20:11] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:35:28] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:45:11] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [22:52:41] Yippee, build fixed! [22:52:41] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #426: FIXED in 30 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/426/ [23:00:46] (03PS1) 10Mpaa: Whitelist pywikibot people with +2 rights [integration/config] - 10https://gerrit.wikimedia.org/r/186611 (https://phabricator.wikimedia.org/T87413) [23:46:30] PROBLEM - Free space - all mounts on deployment-cache-upload02 is CRITICAL: CRITICAL: deployment-prep.deployment-cache-upload02.diskspace._srv_vdb.byte_percentfree.value (<100.00%)