[01:47:41] 10Gerrit, 10Operations: Gerrit losing slowly - https://phabricator.wikimedia.org/T215004 (10Paladox) [01:47:48] 10Gerrit, 10Operations: Gerrit losing slowly - https://phabricator.wikimedia.org/T215004 (10Paladox) p:05Triage→03Unbreak! [01:51:10] 10Gerrit, 10Operations: Gerrit loading slowly - https://phabricator.wikimedia.org/T215004 (10AntiCompositeNumber) [01:51:16] 10Gerrit, 10Operations: Gerrit loading slowly - https://phabricator.wikimedia.org/T215004 (10TheDragonFire) I get intermittent `plugin failed to load` errors, as well as this in console: ` Thu Jan 31 01:49:32 GMT+000 2019 Class$82 SEVERE: TypeError: a is null Class$S43: TypeError: a is null at Unknown.TP(htt... [02:03:29] PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_jenkins CI slave scripts] [02:08:18] 10Gerrit, 10Operations: Gerrit loading slowly - https://phabricator.wikimedia.org/T215004 (10GTirloni) It seems the issue is gone after a reboot. I didn't see anything special in /var/log/gerrit/error_log. Thanks for reporting this. [02:08:35] 10Gerrit, 10Operations: Gerrit loading slowly - https://phabricator.wikimedia.org/T215004 (10GTirloni) 05Open→03Resolved p:05Unbreak!→03High a:03GTirloni [02:29:59] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:51:10] 10Beta-Cluster-Infrastructure, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Enable CheckUser for beta cluster - https://phabricator.wikimedia.org/T214820 (10Rxy) >>! In T214820#4915381, @Legoktm wrote: > @Rxy are you looking for test data to use while testing, or just an environment to test... [07:13:50] 10Phabricator (Upstream), 10Upstream: independent column sorting in Phabricator - https://phabricator.wikimedia.org/T214988 (10Funnyjokes2019) Really your post is really very good and I appreciate it. It’s hard to sort the good from the bad sometimes.You definitely put a new spin on a topic thats been written... [07:16:17] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Discovery-Search: extensions phpunit tests time out - https://phabricator.wikimedia.org/T214978 (10Funnyjokes2019) Oh! This article has suggested to me many new ideas. I will embark on doing it. Hope you can continue to contribute your ta... [07:17:20] 10Beta-Cluster-Infrastructure, 10Page-Previews, 10Readers-Web-Backlog, 10Browser-Tests: Popups build failing consistently for a week - https://phabricator.wikimedia.org/T214974 (10Funnyjokes2019) I enjoyed over read your blog post. Your blog have nice information I got good ideas from this amazing blog. I... [11:18:14] Project beta-scap-eqiad build #236097: 04FAILURE in 17 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/236097/ [11:19:20] 10MediaWiki-Codesniffer: Enforce presence of @license tag - https://phabricator.wikimedia.org/T215011 (10Lucas_Werkmeister_WMDE) [12:04:42] Project beta-scap-eqiad build #236098: 04STILL FAILING in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/236098/ [12:17:08] Project beta-scap-eqiad build #236099: 04STILL FAILING in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/236099/ [12:29:25] Project beta-scap-eqiad build #236100: 04STILL FAILING in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/236100/ [12:41:38] Project beta-scap-eqiad build #236101: 04STILL FAILING in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/236101/ [12:53:40] Project beta-scap-eqiad build #236102: 04STILL FAILING in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/236102/ [13:05:29] Project beta-scap-eqiad build #236103: 04STILL FAILING in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/236103/ [13:10:10] Krenair: ^^ [13:11:47] I have no idea what that is paladox [13:12:44] Ok [13:49:15] Project beta-scap-eqiad build #236104: 04STILL FAILING in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/236104/ [13:55:00] PolyGerrit will now tell you if you have js disabled (as it's required) https://gerrit-review.googlesource.com/c/gerrit/+/212132 :) [14:00:11] Project beta-scap-eqiad build #236105: 04STILL FAILING in 9 min 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/236105/ [14:11:57] Krenair https://phabricator.wikimedia.org/T215012 [14:12:02] may have been caused by ^^ [14:12:06] i see deploy-01 in there [14:12:14] * deployment-deploy01 [14:13:49] Project beta-scap-eqiad build #236106: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/236106/ [14:15:12] "14:13:13 14:13:13 ['/usr/bin/scap', 'pull-master', 'deployment-deploy01.deployment-prep.eqiad.wmflabs'] on deployment-deploy02.deployment-prep.eqiad.wmflabs returned [255]: Permission denied (publickey). [14:15:12] " [14:24:44] Project beta-scap-eqiad build #236107: 04STILL FAILING in 9 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/236107/ [14:34:22] Project beta-scap-eqiad build #236108: 04STILL FAILING in 8 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/236108/ [14:44:04] Project beta-scap-eqiad build #236109: 04STILL FAILING in 8 min 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/236109/ [14:53:40] Project beta-scap-eqiad build #236110: 04STILL FAILING in 8 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/236110/ [15:03:42] !log rearm keyholder on deployment-deploy01 [15:03:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:09:58] Yippee, build fixed! [15:09:58] Project beta-scap-eqiad build #236111: 09FIXED in 15 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/236111/ [15:30:43] (03PS2) 10Paladox: Allow "Gerrit Managers" to view /monitoring [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/473636 [15:33:07] (03CR) 10Thcipriani: [V: 03+2 C: 03+2] Allow "Gerrit Managers" to view /monitoring [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/473636 (owner: 10Paladox) [15:50:46] 10Gerrit, 10Release-Engineering-Team (Watching / External), 10Operations: Add prometheus exporter to Gerrit - https://phabricator.wikimedia.org/T184086 (10Paladox) This would have been usage for debugging T215004 , should we triage as high? [15:54:21] 10Gerrit, 10Operations: Investigate why icinga did not report high cpu/load - https://phabricator.wikimedia.org/T215033 (10Paladox) [15:54:28] 10Gerrit, 10Operations: Investigate why icinga did not report high cpu/load - https://phabricator.wikimedia.org/T215033 (10Paladox) p:05Triage→03High [15:56:34] 10Gerrit, 10Operations: Investigate why icinga did not report high cpu/load for gerrit - https://phabricator.wikimedia.org/T215033 (10Paladox) [16:58:22] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Test-Coverage: generate Wikibase.git code coverage on Jenkins - https://phabricator.wikimedia.org/T88434 (10Lucas_Werkmeister_WMDE) Hm, I guess that would also work. [17:16:35] Project beta-scap-eqiad build #236120: 04FAILURE in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/236120/ [17:29:35] Project beta-scap-eqiad build #236121: 04STILL FAILING in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/236121/ [17:45:11] Project beta-scap-eqiad build #236122: 04STILL FAILING in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/236122/ [17:57:17] Project beta-scap-eqiad build #236123: 04STILL FAILING in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/236123/ [18:08:36] Yippee, build fixed! [18:08:36] Project beta-scap-eqiad build #236124: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/236124/ [19:35:00] (03PS1) 10Kosta Harlan: Sonar Scanner: Fix arguments in JJB [integration/config] - 10https://gerrit.wikimedia.org/r/487376 [19:36:57] thcipriani: https://gerrit.wikimedia.org/r/c/integration/config/+/487376 [19:37:37] (03CR) 10Thcipriani: [C: 03+2] "Pairing!" [integration/config] - 10https://gerrit.wikimedia.org/r/487376 (owner: 10Kosta Harlan) [19:40:31] (03Merged) 10jenkins-bot: Sonar Scanner: Fix arguments in JJB [integration/config] - 10https://gerrit.wikimedia.org/r/487376 (owner: 10Kosta Harlan) [20:03:37] 10Gerrit, 10Operations: Gerrit loading slowly - https://phabricator.wikimedia.org/T215004 (10greg) Thanks @GTirloni [20:40:45] 10Gerrit, 10Icinga, 10Operations, 10monitoring: Investigate why icinga did not report high cpu/load for gerrit - https://phabricator.wikimedia.org/T215033 (10Peachey88) [23:37:53] * paladox watches as the snow heads towards me https://www.netweather.tv/live-weather/radar [23:38:03] oh wrong channel [23:39:09] /1/8 [23:53:14] 10Gerrit, 10Icinga, 10Operations, 10monitoring: Investigate why icinga did not report high cpu/load for gerrit - https://phabricator.wikimedia.org/T215033 (10Aklapper) Please include context. Is this related to T215004? [23:56:30] 10Gerrit, 10Icinga, 10Operations, 10monitoring: Investigate why icinga did not report high cpu/load for gerrit - https://phabricator.wikimedia.org/T215033 (10Paladox) Yup, it's related to that.