[05:00:13] !log deploying 5230e7d in ores beta node (T154168) [05:00:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:00:19] T154168: Quantity changes broke ORES - https://phabricator.wikimedia.org/T154168 [05:02:07] new scap, fancy [06:55:24] PROBLEM - Puppet run on deployment-puppetmaster02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:30:25] RECOVERY - Puppet run on deployment-puppetmaster02 is OK: OK: Less than 1.00% above the threshold [0.0] [13:46:23] Yippee, build fixed! [13:46:24] Project selenium-VisualEditor ยป firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #256: 09FIXED in 2 min 23 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/256/ [15:39:48] PROBLEM - Puppet run on deployment-phab02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:19:31] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T153351#2903702 (10Nikerabbit) [20:47:22] 05Gerrit-Migration, 10Differential, 10Phabricator: Consider documenting the Arcanist installer for Windows and Mac that Paladox created - https://phabricator.wikimedia.org/T131939#2903967 (10Paladox) [20:49:14] 05Gerrit-Migration, 10Differential, 10Phabricator: Consider documenting the Arcanist installer for Windows and Mac that Paladox created - https://phabricator.wikimedia.org/T131939#2903985 (10Paladox) [21:14:54] 05Gerrit-Migration, 10Differential, 10Phabricator: Consider documenting the Arcanist installer for Windows and Mac that Paladox created - https://phabricator.wikimedia.org/T131939#2904017 (10Paladox) I created a mac installer today for arcanist + libphutil :) https://github.com/paladox/Arcanist-installer-fo... [22:22:12] 10Continuous-Integration-Infrastructure, 07Jenkins: Jenkins should tolerate a one second delay - https://phabricator.wikimedia.org/T154188#2904051 (10Paladox) [22:29:01] 10Gerrit, 06Release-Engineering-Team, 06Operations: Gerrit: Restarting gerrit could lead to data loss - https://phabricator.wikimedia.org/T154205#2904053 (10Paladox) [22:31:04] 10Gerrit, 06Release-Engineering-Team, 06Operations: Gerrit: Restarting gerrit could lead to data loss - https://phabricator.wikimedia.org/T154205#2904065 (10Paladox) p:05Triage>03Unbreak! Changing priority to break as someone needs to investigate it's impact on prod, as i managed to reproduce it in labs... [22:33:33] 10Gerrit, 06Release-Engineering-Team, 06Operations: Gerrit: Restarting gerrit could lead to data loss - https://phabricator.wikimedia.org/T154205#2904069 (10Paladox) This bug may have caused T153079 since that problem happened around the time after we upgraded to gerrit 2.13. [22:34:42] 10Gerrit, 10BlueSpice, 07Upstream: Merge/Submit error on Gerrit: "org.eclipse.jgit.errors.MissingObjectException: Missing unknown" for BlueSpiceExtensions' REL1_27 branch - https://phabricator.wikimedia.org/T153079#2869000 (10Paladox) I believe this problem happened because of a buggy reindex in gerrit 2.13.... [22:34:57] 10Gerrit, 06Release-Engineering-Team, 06Operations: Gerrit: Restarting gerrit could lead to data loss - https://phabricator.wikimedia.org/T154205#2904053 (10Paladox) the data loss includes merged changes [22:48:32] (03PS1) 10Paladox: Whitelist user SamanthaNguyen [integration/config] - 10https://gerrit.wikimedia.org/r/329362 [22:52:20] (03CR) 10SamanthaNguyen: [C: 031] "Much appreciated :)" [integration/config] - 10https://gerrit.wikimedia.org/r/329362 (owner: 10Paladox) [22:53:30] (03PS2) 10Paladox: Whitelist user SamanthaNguyen [integration/config] - 10https://gerrit.wikimedia.org/r/329362 [22:55:56] (03CR) 10Paladox: "your welcome :)" [integration/config] - 10https://gerrit.wikimedia.org/r/329362 (owner: 10Paladox) [22:56:42] (03CR) 10SamanthaNguyen: [C: 031] "Just one minor thing, "codynguyen1116@gmail.com" is my old email address which I don't use anymore, could you remove that? thanks!" [integration/config] - 10https://gerrit.wikimedia.org/r/329362 (owner: 10Paladox) [22:58:39] (03PS3) 10Paladox: Whitelist user SamanthaNguyen [integration/config] - 10https://gerrit.wikimedia.org/r/329362 [22:58:55] (03CR) 10Paladox: "> Just one minor thing, "codynguyen1116@gmail.com" is my old email" [integration/config] - 10https://gerrit.wikimedia.org/r/329362 (owner: 10Paladox) [22:59:08] (03CR) 10Reedy: [C: 031] "Apparently I don't have CR+2...." [integration/config] - 10https://gerrit.wikimedia.org/r/329362 (owner: 10Paladox) [23:01:04] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Grant reedy C+2 on integration repos - https://phabricator.wikimedia.org/T154207#2904102 (10Reedy) [23:01:07] paladox: ^ :P [23:01:32] :) [23:01:33] +1 [23:01:53] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Grant reedy C+2 on integration repos - https://phabricator.wikimedia.org/T154207#2904102 (10Paladox) +1 [23:02:08] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team: Grant reedy C+2 on integration repos - https://phabricator.wikimedia.org/T154207#2904116 (10Paladox) [23:02:25] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:19:18] 10Gerrit, 06Release-Engineering-Team, 06Operations, 07Upstream: Gerrit: Restarting gerrit could lead to data loss - https://phabricator.wikimedia.org/T154205#2904128 (10Paladox) [23:23:28] 10Gerrit, 06Release-Engineering-Team, 06Operations, 07Upstream: Gerrit: Restarting gerrit could lead to data loss - https://phabricator.wikimedia.org/T154205#2904130 (10Paladox) Reported here https://bugs.chromium.org/p/gerrit/issues/detail?id=5200