[00:04:09] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [00:39:09] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:23:41] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [01:46:44] 10Release-Engineering-Team, 10Operations, 10Phabricator: Viewing raw files on phab fails with ERROR_MESSAGE_MAIN - https://phabricator.wikimedia.org/T169454#3398729 (10Paladox) [01:48:46] 10Release-Engineering-Team, 10Operations, 10Phabricator: Viewing raw files on phab fails with ERROR_MESSAGE_MAIN - https://phabricator.wikimedia.org/T169454#3398741 (10Paladox) p:05Triage>03High I am filling this in #operations because the error sound unrelated to phabricator internal code. I think this... [01:52:42] 10Release-Engineering-Team, 10Operations, 10Phabricator: Viewing raw files on phab fails with ERROR_MESSAGE_MAIN - https://phabricator.wikimedia.org/T169454#3398743 (10Paladox) But the rewrites seem to work for https://phabzilla.wmflabs.org/file/data/yydp2qcdrep6lrv2vlxn/PHID-FILE-wnr4rr7iatru4jo2ityf/index.... [02:03:42] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [03:30:20] 10Release-Engineering-Team, 10Phabricator: Viewing raw files on phab fails with ERROR_MESSAGE_MAIN - https://phabricator.wikimedia.org/T169454#3398750 (10Aklapper) p:05High>03Triage Cannot reproduce, shows the raw file for me in Firefox 54 on Linux. Not #Operations territory. Resetting priority as I cannot... [03:33:46] PROBLEM - Puppet errors on integration-slave-docker-1000 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [03:35:16] Project beta-scap-eqiad build #162161: 04FAILURE in 1 min 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/162161/ [03:45:39] Yippee, build fixed! [03:45:39] Project beta-scap-eqiad build #162162: 09FIXED in 2 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/162162/ [04:13:45] RECOVERY - Puppet errors on integration-slave-docker-1000 is OK: OK: Less than 1.00% above the threshold [0.0] [05:55:28] PROBLEM - Puppet errors on deployment-ores-redis-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [05:55:50] PROBLEM - Puppet errors on deployment-apertium02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [05:59:41] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<50.00%) [06:22:41] 10Release-Engineering-Team (Kanban), 10Revision-Slider, 10TCB-Team, 10VisualEditor, and 3 others: Argument 2 passed to VisualEditorHooks::onDiffViewHeader() must be an instance of Revision, null given - https://phabricator.wikimedia.org/T169132#3398794 (10Jdforrester-WMF) [06:30:26] RECOVERY - Puppet errors on deployment-ores-redis-01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:30:50] RECOVERY - Puppet errors on deployment-apertium02 is OK: OK: Less than 1.00% above the threshold [0.0] [06:49:39] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [09:32:16] PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:54:41] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [10:12:16] RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:16:07] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:34:40] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [10:51:08] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [11:21:52] 10Release-Engineering-Team, 10Phabricator: Viewing raw files on phab fails with ERROR_MESSAGE_MAIN - https://phabricator.wikimedia.org/T169454#3398853 (10Paladox) I get this {F8592637} On iOS 11, iPhone 6 Plus. [11:26:19] 10Release-Engineering-Team, 10Phabricator: Viewing raw files on phab fails with ERROR_MESSAGE_MAIN - https://phabricator.wikimedia.org/T169454#3398854 (10Paladox) [15:09:14] PROBLEM - Puppet errors on deployment-cache-upload04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:24:12] PROBLEM - Puppet errors on deployment-jobrunner02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:25:50] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:47:25] Yippee, build fixed! [15:47:25] Project selenium-MobileFrontend ยป chrome,beta,Linux,BrowserTests build #474: 09FIXED in 25 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/474/ [16:00:50] RECOVERY - Puppet errors on deployment-kafka01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:04:11] RECOVERY - Puppet errors on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:13:12] PROBLEM - Puppet errors on deployment-ms-be03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:20:34] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:21:56] PROBLEM - Puppet errors on deployment-urldownloader is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:25:40] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [17:45:08] 10Gerrit, 10Upstream: Change Gerrit's default indentation to tabs - https://phabricator.wikimedia.org/T168590#3399104 (10Paladox) [17:45:11] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Patch-For-Review: Update gerrit to 2.14.1 - https://phabricator.wikimedia.org/T156120#3399103 (10Paladox) [17:46:26] 10Gerrit: Enable Gerrit feature to add comment when people add reviewers to a patch - https://phabricator.wikimedia.org/T168030#3399121 (10Paladox) [17:46:30] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Patch-For-Review: Update gerrit to 2.14.1 - https://phabricator.wikimedia.org/T156120#2964669 (10Paladox) [17:46:35] 10Gerrit: Enable Gerrit feature to add comment when people add reviewers to a patch - https://phabricator.wikimedia.org/T168030#3353627 (10Paladox) [17:46:38] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Patch-For-Review: Update gerrit to 2.14.1 - https://phabricator.wikimedia.org/T156120#2964669 (10Paladox) [17:47:16] 10Gerrit, 10Upstream: Change Gerrit's default indentation to tabs - https://phabricator.wikimedia.org/T168590#3399124 (10Paladox) Patch has been merged and will be released in gerrit 2.14.2. [17:47:51] 10Gerrit, 10Release-Engineering-Team: Gerrit: inline edit references are not updated - https://phabricator.wikimedia.org/T159744#3399127 (10Paladox) [17:47:51] 10Gerrit, 10Release-Engineering-Team, 10Operations: Gerrit: can lose data if it crashes - https://phabricator.wikimedia.org/T159743#3399128 (10Paladox) [17:47:53] 10Gerrit, 10Operations: Decide how to support polygerrit - https://phabricator.wikimedia.org/T158479#3399129 (10Paladox) [17:47:55] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Patch-For-Review: Update gerrit to 2.14.1 - https://phabricator.wikimedia.org/T156120#3399126 (10Paladox) [17:53:06] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Patch-For-Review: Update gerrit to 2.14.1 - https://phabricator.wikimedia.org/T156120#3399144 (10Paladox) @Peachey88 T168030 is parent to this task because the feature is in gerrit 2.14 (polygerrit (new ui)) (which means that we have to update to 2.14 to gain... [18:00:40] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [18:01:56] RECOVERY - Puppet errors on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [18:46:27] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:21:26] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [19:35:10] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [20:10:09] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:00:04] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<55.56%) [21:34:46] PROBLEM - Puppet errors on integration-slave-docker-1000 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:09:45] RECOVERY - Puppet errors on integration-slave-docker-1000 is OK: OK: Less than 1.00% above the threshold [0.0]