[00:02:10] which i will try to do tommror [00:04:45] Project beta-scap-eqiad build #179044: 04STILL FAILING in 1 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179044/ [00:14:47] Project beta-scap-eqiad build #179045: 04STILL FAILING in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179045/ [00:24:40] Project beta-scap-eqiad build #179046: 04STILL FAILING in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179046/ [00:34:38] Project beta-scap-eqiad build #179047: 04STILL FAILING in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179047/ [00:44:39] Project beta-scap-eqiad build #179048: 04STILL FAILING in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179048/ [00:54:37] Project beta-scap-eqiad build #179049: 04STILL FAILING in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179049/ [01:01:59] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [01:04:23] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Document current Blubber types and functions - https://phabricator.wikimedia.org/T168000#3708676 (10dduvall) a:03dduvall [01:04:41] Project beta-scap-eqiad build #179050: 04STILL FAILING in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179050/ [01:07:55] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Document current Blubber types and functions - https://phabricator.wikimedia.org/T168000#3708681 (10dduvall) [01:14:38] Project beta-scap-eqiad build #179051: 04STILL FAILING in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179051/ [01:24:34] Project beta-scap-eqiad build #179052: 04STILL FAILING in 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179052/ [01:34:36] Project beta-scap-eqiad build #179053: 04STILL FAILING in 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179053/ [01:41:59] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:44:38] Project beta-scap-eqiad build #179054: 04STILL FAILING in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179054/ [01:54:39] Project beta-scap-eqiad build #179055: 04STILL FAILING in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179055/ [02:04:39] Project beta-scap-eqiad build #179056: 04STILL FAILING in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179056/ [02:14:36] Project beta-scap-eqiad build #179057: 04STILL FAILING in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179057/ [02:24:34] Project beta-scap-eqiad build #179058: 04STILL FAILING in 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179058/ [02:34:39] Project beta-scap-eqiad build #179059: 04STILL FAILING in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179059/ [02:43:39] Project beta-scap-eqiad build #179060: 04STILL FAILING in 0.41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179060/ [02:54:39] Project beta-scap-eqiad build #179061: 04STILL FAILING in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179061/ [03:04:43] Project beta-scap-eqiad build #179062: 04STILL FAILING in 1 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179062/ [03:14:36] Project beta-scap-eqiad build #179063: 04STILL FAILING in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179063/ [03:24:32] Project beta-scap-eqiad build #179064: 04STILL FAILING in 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179064/ [03:34:36] Project beta-scap-eqiad build #179065: 04STILL FAILING in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179065/ [03:44:35] Project beta-scap-eqiad build #179066: 04STILL FAILING in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179066/ [03:54:34] Project beta-scap-eqiad build #179067: 04STILL FAILING in 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179067/ [04:04:37] Project beta-scap-eqiad build #179068: 04STILL FAILING in 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179068/ [04:14:37] Project beta-scap-eqiad build #179069: 04STILL FAILING in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179069/ [04:24:33] Project beta-scap-eqiad build #179070: 04STILL FAILING in 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179070/ [04:24:51] https://gerrit.wikimedia.org/r/386333 is the patch for ContributionTracking/FundraiserLandingPage [04:34:38] Project beta-scap-eqiad build #179071: 04STILL FAILING in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179071/ [04:44:38] Project beta-scap-eqiad build #179072: 04STILL FAILING in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179072/ [04:54:40] Project beta-scap-eqiad build #179073: 04STILL FAILING in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179073/ [05:00:01] legoktm: Ack, merging now. [05:10:55] Yippee, build fixed! [05:10:55] Project beta-scap-eqiad build #179074: 09FIXED in 7 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/179074/ [05:13:50] woohoo [05:28:47] 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10User-Addshore: un blacklist https://integration.wikimedia.org/ci/computer/XXXX/builds - https://phabricator.wikimedia.org/T178458#3693025 (10Legoktm) >>! In T178458#3693177, @Addshore wrote: > The code has a ticket in the comment that I can't acc... [06:12:22] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:18:36] legoktm: no_justification yay! [06:41:51] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:52:22] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [06:57:17] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [07:41:49] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [08:34:08] (03CR) 10Hashar: [C: 031] "I like the idea of extracting preparing the source out of the containers that run the test command. Addshore went with that approach for p" (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/386259 (owner: 10Legoktm) [08:43:02] (03CR) 10Addshore: "> What worry me though is that the build will leave behind a bunch of material (/cache /log /src). We will circle back to the issues we h" [integration/config] - 10https://gerrit.wikimedia.org/r/386259 (owner: 10Legoktm) [08:43:05] (03CR) 10Addshore: [C: 031] Introduce ci-src-setup-simple [integration/config] - 10https://gerrit.wikimedia.org/r/386259 (owner: 10Legoktm) [08:51:48] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [09:02:02] PROBLEM - Puppet errors on deployment-trending01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [09:17:52] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [09:37:48] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [10:09:35] 10Release-Engineering-Team, 10Scap, 10Services (watching): Scap3 broken in Beta - https://phabricator.wikimedia.org/T178884#3709229 (10mobrovac) 05Open>03Resolved a:03demon All good now - Scap deployments are back. Resolving. [10:13:19] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3709235 (10WMDE-leszek) @zeljkofilipin just as heads up: @Ladsgroup was so kind to... [12:22:39] Project selenium-GettingStarted » firefox,beta,Linux,BrowserTests build #566: 04FAILURE in 39 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/566/ [12:45:12] 10Beta-Cluster-Infrastructure, 10ORES, 10Scoring-platform-team: ORESFetchScoreJob: RuntimeException No model available for [goodfaith] - https://phabricator.wikimedia.org/T178792#3709693 (10Ladsgroup) I think my patch that hasn't been merged will fix it but not sure, we should wait a little (it's not merged... [14:41:09] * paladox tested a its-base change that supports notedb and works :) [14:41:19] anyways 2.14 and 2.15 branches support soy now. [14:57:21] I’m unable to deploy the ORES submodules to our beta server, deployment-sca03. The deploy-service user doesn’t have SSH access to vcs@git-ssh.wmo. [14:58:57] I see that the .gitmodules file gets rewritten during production deploy, to point to submodule repos on tin, but that isn’t happening for wmflabs deployment. [14:59:11] thcipriani|afk: ^ if you have a moment [14:59:51] 10Release-Engineering-Team, 10Commons, 10MediaWiki-File-management, 10Multimedia: Notice: Undefined index: 1 in /srv/mediawiki/php-1.31.0-wmf.4/includes/media/FormatMetadata.php on line 744 - https://phabricator.wikimedia.org/T179004#3709948 (10Amire80) [15:03:24] awight: I am out this week (this message notwithstanding :)). no_justification or twentyafterfour may be around though. [15:04:37] Swap SSH -> HTTPS should Just Work [15:04:50] We're not writing, so no reason for the overhead of SSH [15:05:39] lol awesome thanks thcipriani|afk & no_justification [15:06:34] no_justification: Failed to connect to git-ssh.wikimedia.org port 443: Connection refused [15:06:43] Well, not git-ssh :) [15:06:45] I’ll try rewriting with a tin.wmflabs address [15:06:50] ah gotcha [15:06:54] We'd swap for whatever the https-based address is [15:07:00] What's it say on the diffusion repo? [15:08:22] like this, https://phabricator.wikimedia.org/source/draftquality.git [15:08:26] I’ll use those. [15:09:06] That should work :) [15:09:32] We should probably have a warning about that somewhere in the scap docs. [15:10:35] Maybe I should file a bug about the URL rewriting not happening in wmflabs scap? [15:11:31] Eh, I'm thinking it more likely just barfed on the phab SSH address and couldn't rewrite that. [15:11:36] I don't think we use that remote in production [15:12:09] Or, if we do, I think it's moreso that it's a firewall thing. SSH between labs/prod is...kinda a no no :) [15:12:50] https 4 life! [15:13:09] FWIW, in production scap is rewriting ssh://vcs@git-ssh.wikimedia.org/diffusion/1915/research-ores-deploy-wheels.git to http://tin.eqiad.wmnet/ores/deploy/.git/modules/submodules/wikiclass [15:13:28] We could use https in production as well, fwiw [15:13:36] yeah good point. I’ll see if I can change our master repo to use the phabricator https URLs. [15:13:41] But yeah, looks like you're right about the rewriting :) [15:14:19] Er, :( I s'pose [15:14:34] A fix for that fits nicely with our current sprint too, I'll file a task [15:15:28] Well I'd file it anyway haha [15:17:01] We’ll make the changes on our side under T179009 [15:17:02] T179009: ORES deployment submodules should point to phabricator HTTPS repos. - https://phabricator.wikimedia.org/T179009 [15:17:47] no_justification: I’m perfectly happy to not file a bug about the rewriting, since it’s not a use case we should need. [15:17:57] Thanks for the suggestions! [15:23:23] No problem! [15:24:09] Does awarding you the "Party time" token count as a blessing? haha [15:27:54] lol, a golden chaliceful of confetti [15:28:44] lol [15:30:24] no_justification: /o\, > error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504 Gateway Time-out [15:30:32] Derppppp :\ [15:30:33] fire…wall? [15:30:45] Was that an immediate fail? [15:30:48] That's weird. [15:30:55] no, a slow and painful fail ;-) [15:31:08] Firewall would fail immediately methinks [15:31:10] Works locally, though. [15:31:57] Wait, fails on master or clients? [15:32:45] I only changed the .gitmodules file (manually, on deployment-sca03), so this is a failure during git submodule update [15:32:52] Ah, ok that makes sense because we're not rewriting the URLs still (see yet-to-be-filed-bug) and they're not able to reach the outside world. [15:33:01] aha. [15:33:30] so in theory, I fix the URLs in our master repo, pull to tin.wmflabs, scap and all ist güt [15:33:47] Yeah, assuming the scap step rewrites the URLs properly [15:34:06] thanks, several mysteries were just revealed to me [15:35:03] Rewriting is actually so we don't slam gerrit/phab with a billion requests at once. But in production, it can actually route to *.wm.o from an internal host [so you don't /have/ to rewrite] [15:35:14] In beta, I bet you can't route to *.wm.o without going external first [15:35:18] Which those hosts can't do [15:35:34] (all this based on my very limited understanding of our network) [15:36:04] Well, the master isn't on a public IP in labs either, so hmmmm [15:36:13] But anyway, we should rewrite because that's what we're supposed to do! [15:36:31] This is making sense. Donno why it only started happening today, but I’m happy to make this fix regardless. [15:37:43] Well if this just started, we did land some stuff that tweaked submodule behavior [15:37:50] I'll mention that in the task, it could be a regression! [15:39:37] 10Release-Engineering-Team, 10Scap (Tech Debt Sprint FY201718-Q2): Scap failing to rewrite submodule urls in beta - https://phabricator.wikimedia.org/T179013#3710215 (10demon) [15:39:46] PROBLEM - Free space - all mounts on deployment-kafka01 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka01.diskspace.root.byte_percentfree (<100.00%) [15:40:01] awight: Filed and subscribed you. If you can/want to add any other details you think pertinent, please do. [15:40:05] * no_justification goes to walk the dog [15:55:12] no_justification i think doing this T64053 in polygerrit would be easier heh :). Just need to file a task usptream to get plugin support in commit msg. [15:55:12] T64053: Provide description of linked Phabricator objects in tooltips - https://phabricator.wikimedia.org/T64053 [15:55:28] there's no point i think in supporting gwtui. [15:58:31] filled https://bugs.chromium.org/p/gerrit/issues/detail?id=7558 [16:01:31] Actually, you could do it with an existing JS plugin [16:01:41] Doesn't need to be Polygerrit/GWTUI specific [16:01:56] But it's lowest priority for a reason :p [16:03:18] ah i see [16:03:32] * paladox dosen't know how to do a JS plugin for gwtui heh :) [16:04:02] though upstream are really active with polygerrit. [16:04:31] 10Release-Engineering-Team, 10Scap (Tech Debt Sprint FY201718-Q2): Scap failing to rewrite submodule urls in beta - https://phabricator.wikimedia.org/T179013#3710331 (10awight) Confirming this bug: scap is failing to rewrite URLs like `https://phabricator.wikimedia.org/source/editquality.git`, for the `/srv/de... [16:12:16] no_justification i guess we can close this https://phabricator.wikimedia.org/T157414 as resolved now? [16:12:46] Lemme make sure, I'm afk for a few minutes [16:13:04] ok [16:15:22] 10Release-Engineering-Team (Kanban), 10Scap, 10Services (watching): Scap3 broken in Beta - https://phabricator.wikimedia.org/T178884#3710387 (10greg) [16:24:50] no_justification deleting external id's is supported through rest api now https://gerrit-review.googlesource.com/Documentation/rest-api-accounts.html#delete-account-external-ids :) [16:24:55] should be really easy to delete them now [16:32:03] 10Release-Engineering-Team, 10GitHub-Mirrors: Enable github mirroring / diffusion for discovery parent pom - https://phabricator.wikimedia.org/T179018#3710474 (10Gehel) [16:32:26] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap: Deploy gerrit with scap3 - https://phabricator.wikimedia.org/T157414#3710486 (10demon) Ok, ran `apt-get purge gerrit` on gerrit2001 -- it removed a couple of files so had to run puppet again so everything came back up. From what I can tell -- services ar... [16:39:59] 10Release-Engineering-Team, 10GitHub-Mirrors: Enable github mirroring / diffusion for discovery parent pom - https://phabricator.wikimedia.org/T179018#3710474 (10demon) Created on github. [16:46:20] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap: Deploy gerrit with scap3 - https://phabricator.wikimedia.org/T157414#3710498 (10demon) Purged from cobalt. Also dropped git repositories for the deb package from Github, gerrit and archived in Phab. All that's left is an opsen to purge the debian packag... [16:47:29] :) :) [17:13:47] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap: Deploy gerrit with scap3 - https://phabricator.wikimedia.org/T157414#3710554 (10demon) 05Open>03Resolved @MoritzMuehlenhoff kindly dropped it from apt, as well as verified via cumin it's gone everywhere [17:17:02] 10Release-Engineering-Team, 10GitHub-Mirrors: Enable github mirroring / diffusion for discovery parent pom - https://phabricator.wikimedia.org/T179018#3710474 (10mmodell) created in diffusion: {rDPOM} and {rDMTC} [17:18:18] 10Release-Engineering-Team, 10GitHub-Mirrors: Enable github mirroring / diffusion for discovery parent pom - https://phabricator.wikimedia.org/T179018#3710574 (10mmodell) 05Open>03Resolved a:03mmodell [17:19:08] 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2): Scap failing to rewrite submodule urls in beta - https://phabricator.wikimedia.org/T179013#3710576 (10mmodell) a:03mmodell [17:22:38] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Operations, 10Patch-For-Review: Reimage gerrit2001 as stretch - https://phabricator.wikimedia.org/T168562#3710584 (10demon) This isn't really stalled, the host has indeed been reimaged as stretch and is completely working. The remaining issue is tracked in... [17:22:45] 10Gerrit, 10Operations: Upload gerrit package to stretch apt.wm.org repo - https://phabricator.wikimedia.org/T165620#3710586 (10demon) [17:22:47] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Operations, 10Patch-For-Review: Reimage gerrit2001 as stretch - https://phabricator.wikimedia.org/T168562#3710585 (10demon) 05stalled>03Resolved [17:47:42] no_justification i know why it didnt purge plugins on gerrit2001 but did on cobalt [17:48:00] it's because on gerrit2001 we recently reimaged as stretch so the package was installed without the plugins [17:48:59] Ah ok [17:49:01] makes sense [17:53:08] yep [17:59:11] no_justification heh we can do tarnery expressions in soy so i can use something as fallback. In my testing someone managed to not have a name so it will cause the template to faill so i added username as fallback. [17:59:13] :) [18:08:46] is it still possible to get a user workboard in phabricator (ala User-{{username}})? [18:09:11] and if so, what was the process, wasn't there a request template or something? [18:12:47] urandom: there's a task, but just make a task in #phabricator and someone will get to it [18:13:17] urandom: see also: https://phabricator.wikimedia.org/T555 [18:16:00] urandom: here ya go! https://phabricator.wikimedia.org/project/profile/3075/ [18:16:57] oh snap! [18:17:06] greg-g: thank you! [18:30:12] 10Release-Engineering-Team, 10Commons, 10MediaWiki-File-management, 10Multimedia: Notice: Undefined index: 1 in /srv/mediawiki/php-1.31.0-wmf.4/includes/media/FormatMetadata.php on line 744 - https://phabricator.wikimedia.org/T179004#3709948 (10Tgr) Probably the same issue as {T87572}, caused by EXIF data... [18:39:22] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:46:43] (03PS1) 10Umherirrender: Add npm jobs [integration/config] - 10https://gerrit.wikimedia.org/r/386456 [19:05:23] greg-g: haven't seen anything about fundraising in the deploy calendar; will there be a freeze period this year? [19:10:26] tgr: https://wikitech.wikimedia.org/wiki/Deployments#Upcoming and https://lists.wikimedia.org/pipermail/wikitech-l/2017-September/088927.html :) [19:10:50] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [19:12:09] greg-g: yeah but is that all? there used to be a "no major deployments" period from mid-November to mid-January [19:14:53] you asked about a freeze :) [19:15:36] re "don't do anything stupid at the end of the year" I can send an email about that [19:16:22] followed by a series of commits and patches all including a comment or !log that says "Greg said this is ok" [19:16:23] :) [19:16:35] :P [19:20:20] greg-g's sekrit plan is just to give the conductor's cap over to no_justification and run >.> [19:20:43] I've been wearing it for ages [19:26:42] Oh snap, I forgot I had ice cream. Train just got cooler today [19:26:50] 😎 [19:27:20] lol [20:31:16] 10Continuous-Integration-Infrastructure: Create pretty landing page at https://doc.wikimedia.org/cover/ - https://phabricator.wikimedia.org/T146970#3711256 (10Legoktm) a:03Legoktm I have a few ideas on how to implement this. [20:31:30] (03CR) 10Hashar: [C: 032] "That is a lot :]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]" [integration/config] - 10https://gerrit.wikimedia.org/r/386456 (owner: 10Umherirrender) [20:33:51] (03Merged) 10jenkins-bot: Add npm jobs [integration/config] - 10https://gerrit.wikimedia.org/r/386456 (owner: 10Umherirrender) [20:50:25] no_justification apparently we could do something like https://chromium.googlesource.com/infra/gerrit-plugins/landingwidget/+/refs/heads/master/src/main/resources/static/landingwidget.js :) [20:50:31] i wonder what the widget is anyways [20:50:33] * paladox tests [21:06:53] actually that's a gwtui plugin [21:07:03] i should have known by it's like of dom-module heh [21:07:03] :) [21:13:57] oh we can build a ci plugin for the top bar heh [21:25:58] 10Release-Engineering-Team (Kanban), 10GitHub-Mirrors: Enable github mirroring / diffusion for discovery parent pom - https://phabricator.wikimedia.org/T179018#3711342 (10greg) [22:13:21] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:21:12] (03PS1) 10Legoktm: Publish clover.xml files for machine-readable coverage information [integration/config] - 10https://gerrit.wikimedia.org/r/386543 (https://phabricator.wikimedia.org/T146970) [22:48:20] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [23:20:10] heh jenkins plugin updates are down https://issues.jenkins-ci.org/browse/JENKINS-47650