[00:22:33] think I've realised what's up with cache-text0 [00:22:34] 5 [00:22:37] probably missing this: [00:22:38] hieradata/role/common/cache/text.yaml:profile::cache::varnish::separate_vcl: ['misc'] [00:23:12] would explain why reload-vcl has not run vcl.label for wikimedia_misc [01:03:25] it was that + cache::alternate_domains: {} [01:04:53] puppet must've been broken there a while, I just saw it remove a 208.80.155.128/25 reference :| [01:06:15] - } elsif (req.http.Host ~ "^(?!(upload|maps)\.beta\.wmflabs\.org)") { [01:06:15] + } elsif (req.http.Host ~ "") { [01:06:16] hmmmmm [01:13:05] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/536794/ [04:58:24] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Documentation, 10User-MModell: Make PHD run on the backup phabricator server (phab2001, currently) - https://phabricator.wikimedia.org/T232883 (10Marostegui) I am removing the DBA tag from here as there i... [07:52:13] 10Project-Admins, 10translatewiki.net: Convert T41480 ("Issues affecting translatewiki.net") to a project tag - https://phabricator.wikimedia.org/T231991 (10abi_) I am not sure about the name `#translatewiki.net-production-error`. By the name itself it is difficult to understand that it contains external issue... [08:24:10] 10Project-Admins, 10translatewiki.net: Convert T41480 ("Issues affecting translatewiki.net") to a project tag - https://phabricator.wikimedia.org/T231991 (10Nikerabbit) Most of our issues are what I call logspam. Another option would be `#translatewiki.net-mediawiki-error` if the word production is troublesome... [10:10:04] 10Beta-Cluster-Infrastructure, 10Parsoid-PHP, 10RESTBase, 10Core Platform Team Workboards (Purple), 10Patch-For-Review: Make the parsoid server on the beta cluster a mediawiki app server - https://phabricator.wikimedia.org/T232538 (10hashar) Well done everyone! [11:02:51] 10Release-Engineering-Team-TODO (201909): Arrange intra-team PGP keysigning at Atlanta offsite - https://phabricator.wikimedia.org/T232990 (10LarsWirzenius) [12:21:27] thcipriani https://gerrit-review.googlesource.com/c/gerrit/+/234734 has been merged now! So we now have a migration path for our skin! [12:21:32] (which i have locally!) [12:33:53] What can be reason for https://integration.wikimedia.org/ci/blue/organizations/jenkins/service-pipeline-test/detail/service-pipeline-test-and-publish/3269/pipeline/ ? I couldn't get it. [12:33:58] (03CR) 10MSantos: layout: [kartotherian] Disable pipeline on gate as it doesn't pass (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/531134 (https://phabricator.wikimedia.org/T223275) (owner: 10Jforrester) [12:34:09] @thcipriani ^ [12:51:43] 10Release-Engineering-Team (Deployment services), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T220747 (10hashar) [13:39:36] 10Release-Engineering-Team (Deployment services), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T220747 (10hashar) 05Open→03Resolved Seems all fine! [14:12:00] (03PS1) 10Daniel Kinzler: Force value of $wgSecretKey. [integration/quibble] - 10https://gerrit.wikimedia.org/r/537118 (https://phabricator.wikimedia.org/T230340) [14:12:40] (03CR) 10jerkins-bot: [V: 04-1] Force value of $wgSecretKey. [integration/quibble] - 10https://gerrit.wikimedia.org/r/537118 (https://phabricator.wikimedia.org/T230340) (owner: 10Daniel Kinzler) [14:14:48] kart_: I'm still not quite 100% at my keyboard, but it looks like you got it figured out. It seems like it was a blip in the blubberoid service. The pipeline was trying "curl -H 'Content-type: application/yaml' --data-binary @.pipeline/blubber.yaml https://blubberoid.wikimedia.org/v1/test" and got back a 404 which is either a misconfiguration somewhere, or a momentary service interruption. Looks [14:14:50] like it was the latter. [14:14:55] (03PS2) 10Daniel Kinzler: Force value of $wgSecretKey. [integration/quibble] - 10https://gerrit.wikimedia.org/r/537118 (https://phabricator.wikimedia.org/T230340) [14:15:35] (03CR) 10jerkins-bot: [V: 04-1] Force value of $wgSecretKey. [integration/quibble] - 10https://gerrit.wikimedia.org/r/537118 (https://phabricator.wikimedia.org/T230340) (owner: 10Daniel Kinzler) [14:19:20] 10MediaWiki-Releasing, 10AbuseFilter, 10MW-1.35-release, 10Patch-For-Review, 10User-Daimona: Bundle AbuseFilter extension with MediaWiki - https://phabricator.wikimedia.org/T191740 (10Daimona) [14:30:00] (03PS3) 10Daniel Kinzler: Force value of $wgSecretKey. [integration/quibble] - 10https://gerrit.wikimedia.org/r/537118 (https://phabricator.wikimedia.org/T230340) [14:31:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10Release Pipeline, 10User-zeljkofilipin: Gerrit/Argo CI proof of concept - https://phabricator.wikimedia.org/T229246 (10zeljkofilipin) [14:41:25] 10Release-Engineering-Team-TODO, 10Packaging: gbp buildpackage with GIT_PBUILDER_AUTOCONF=no causes DIST to be ignored - https://phabricator.wikimedia.org/T233020 (10hashar) p:05Triage→03Low Maybe someday, I have just logged this task for later and to be able to look it up whenever I encounter the issue. T... [14:55:36] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: (Service Check Timed Out) [15:00:32] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 48070 bytes in 6.383 second response time [15:09:35] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Zuul: Upload zuul_2.5.1-wmf10 to apt.wikimedia.org - https://phabricator.wikimedia.org/T233025 (10hashar) [15:09:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Zuul, 10Operations: Upload zuul_2.5.1-wmf10 to apt.wikimedia.org - https://phabricator.wikimedia.org/T233025 (10hashar) [15:36:22] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Cleanup, 10Maps, and 2 others: Delete (rather than archive) the maps/kartotherian and maps/tilerator repos - https://phabricator.wikimedia.org/T228483 (10MarcoAurelio) [15:42:35] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Cleanup, 10Maps, and 2 others: Delete (rather than archive) the maps/kartotherian and maps/tilerator repos - https://phabricator.wikimedia.org/T228483 (10MarcoAurelio) [15:42:47] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Cleanup, 10Maps, and 2 others: Delete (rather than archive) the maps/kartotherian and maps/tilerator repos - https://phabricator.wikimedia.org/T228483 (10MarcoAurelio) [15:44:31] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Cleanup, 10Maps, and 2 others: Delete (rather than archive) the maps/kartotherian and maps/tilerator repos - https://phabricator.wikimedia.org/T228483 (10MarcoAurelio) I've added a To-Do list at the top of the task for yo... [15:44:34] 10Continuous-Integration-Config, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (201909), 10Scap, and 4 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) [15:47:46] 10Phabricator, 10Release-Engineering-Team-TODO (201909), 10Wikimedia-Logstash, 10observability, and 2 others: Deploy phatality into kibana - https://phabricator.wikimedia.org/T230752 (10mmodell) @fgiunchedi Indeed, either sudo or maybe we can deploy to a directory owned by the scap deploy user and symlink... [15:53:05] James_F: Afternoon! Is the to-do list at T228483 (task desc.) okay with you? [15:53:06] T228483: Delete (rather than archive) the maps/kartotherian and maps/tilerator repos - https://phabricator.wikimedia.org/T228483 [15:53:28] * James_F looks. [15:53:49] hauskatze: Yes. [15:54:04] Oh, maps/kartotherian/package is still being used? The team said it wasn't. [15:54:06] * James_F sighs. [15:54:58] Yup. We'll need to re-add to CI. [15:55:06] Yeah, trivial, just irritating [15:55:28] The commit got buried in the middle of moving the archived repos to the bottom so I'm not sure git revert will solve it [15:55:43] At least I'm not that git expert to know if that'd be possible [15:55:58] As I said, it's trivial. [15:56:05] Sure. [15:57:36] (03PS1) 10Jforrester: layout: [maps/kartotherian/package] Unarchive (back to having no CI and manual merges) [integration/config] - 10https://gerrit.wikimedia.org/r/537143 [15:57:44] (03CR) 10Jforrester: [C: 03+2] layout: [maps/kartotherian/package] Unarchive (back to having no CI and manual merges) [integration/config] - 10https://gerrit.wikimedia.org/r/537143 (owner: 10Jforrester) [15:58:19] "Don't ask." lol [15:59:32] (03Merged) 10jenkins-bot: layout: [maps/kartotherian/package] Unarchive (back to having no CI and manual merges) [integration/config] - 10https://gerrit.wikimedia.org/r/537143 (owner: 10Jforrester) [15:59:33] * hauskatze just learnt a new word: "obloquy" [16:00:05] !log Zuul: Restore maps/kartotherian/package to unarchived CI-less midden [16:00:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:04:29] (03CR) 10Jforrester: layout: [kartotherian] Disable pipeline on gate as it doesn't pass (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/531134 (https://phabricator.wikimedia.org/T223275) (owner: 10Jforrester) [16:09:42] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Cleanup, 10Maps, and 2 others: Delete (rather than archive) the maps/kartotherian and maps/tilerator repos - https://phabricator.wikimedia.org/T228483 (10Jdforrester-WMF) List looks good to me. [16:20:00] (03PS11) 10Jforrester: build: Clean up some PHPCS exclusions [tools/release] - 10https://gerrit.wikimedia.org/r/535588 (https://phabricator.wikimedia.org/T232473) (owner: 10Daimona Eaytoy) [16:20:06] (03CR) 10Jforrester: [C: 03+2] "Thank you!" [tools/release] - 10https://gerrit.wikimedia.org/r/535588 (https://phabricator.wikimedia.org/T232473) (owner: 10Daimona Eaytoy) [16:20:49] (03Merged) 10jenkins-bot: build: Clean up some PHPCS exclusions [tools/release] - 10https://gerrit.wikimedia.org/r/535588 (https://phabricator.wikimedia.org/T232473) (owner: 10Daimona Eaytoy) [16:33:32] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Cleanup, 10Maps, and 2 others: Delete (rather than archive) the maps/kartotherian and maps/tilerator repos - https://phabricator.wikimedia.org/T228483 (10MSantos) LGTM. [16:44:19] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10CirrusSearch, and 2 others: selenium-daily-beta-CirrusSearch is broken - https://phabricator.wikimedia.org/T232495 (10zeljkofilipin) Introduced by [[ https://gerrit.wikim... [16:45:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10CirrusSearch, and 2 others: selenium-daily-beta-CirrusSearch is broken - https://phabricator.wikimedia.org/T232495 (10zeljkofilipin) [16:45:51] 10Release-Engineering-Team-TODO, 10CirrusSearch, 10Discovery-Search, 10MW-1.34-notes (1.34.0-wmf.16; 2019-07-30), 10Patch-For-Review: wdio-cucumber-framework fails on NodeJS due to fibers@2.x under nodejs10 - https://phabricator.wikimedia.org/T229033 (10zeljkofilipin) [16:54:03] @James_F is there anyway for zuul to deprioritise tasks that have been merged already via the submit button in the queue? [16:54:44] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Patch-For-Review: Upgrade to Gerrit 2.16.11.1 - https://phabricator.wikimedia.org/T200739 (10Paladox) Email sent on the status of Internet Explorer support: https://lists.wikimedia.org/pipermail/wikitech-l/2019-... [16:55:20] Looking at the current queue - https://integration.wikimedia.org/zuul/ 2 of the top 3 have already been submitted to master [16:55:26] seems a bit strange to me [16:55:50] Jdlrobson: How have they been merged already? [16:55:56] James_F: manually, as far as I can tell [16:55:59] eg https://gerrit.wikimedia.org/r/c/mediawiki/extensions/BlueSpiceFoundation/+/537122 [16:55:59] Eurgh. [16:56:04] Manual merges break the world. [16:56:04] :S [16:56:21] We should take away the rights for that. [16:57:08] I guess the manual merge was possible as soon as the main test build gave Verified+2 [16:57:13] without the merger even needing to override that [16:57:50] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Cleanup, 10Maps, and 2 others: Delete (rather than archive) the maps/kartotherian and maps/tilerator repos - https://phabricator.wikimedia.org/T228483 (10MarcoAurelio) [16:58:10] no, wait, I forgot gate-and-submit starting removes the Verified vote again [16:58:29] It's meant to, but sometimes jenkins is slow to do that. [18:25:31] are there known issues with beta cluster updates? [18:25:50] it looks like https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/ has been “waiting for next available executor” for almost 7 hours [18:26:02] not sure if that’s due to Zuul being generally busy or if those executors are supposed to be separate [18:26:39] Lucas_WMDE: Looks like it just unlocked a bunch of them? [18:26:55] I’m not sure what the “pending” means [18:26:56] I killed the stuck beta-scap-eqiad job [18:26:59] ah ok [18:27:06] But https://integration.wikimedia.org/ci/view/Beta/job/beta-mediawiki-config-update-eqiad/ just went through [18:27:18] And https://integration.wikimedia.org/ci/view/Beta/job/beta-code-update-eqiad/ is going [18:27:28] So we'll see if beta-scap-eqiad works again when the code update is done [18:28:18] Yeah, it looks to be off [18:28:24] I guess something got upset with itself [18:28:35] ok, thanks for fixing it [18:28:36] https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/267250/console [18:33:21] * James_F looks. [18:33:52] Nothing immediately obvious. [18:51:33] !log Gerrit: tweaked members of the Analytics group # T233026 [18:51:33] stashbot: pong [18:51:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:51:35] See https://wikitech.wikimedia.org/wiki/Tool:Stashbot for help. [18:51:58] 10Continuous-Integration-Config, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (201909), 10Scap, and 5 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Anomie) [18:52:27] !log Gerrit: tweaked members of the Analytics group # T233026 [18:52:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:53:38] 10Continuous-Integration-Config, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (201909), 10Scap, and 5 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) [19:20:20] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10CirrusSearch, and 3 others: selenium-daily-beta-CirrusSearch is broken - https://phabricator.wikimedia.org/T232495 (10hashar) From my review on https://gerrit.wikimedia.o... [19:25:37] 10Continuous-Integration-Config, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (201909), 10Scap, and 5 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Anomie) > * InitialiseSettings.php (and much of CommonS... [19:28:42] :O https://gerrit.wikimedia.org/r/monitoring [19:28:51] someone's query is taking... forever [19:28:53] thcipriani ^ [19:29:18] i guess it's expected [19:30:23] though, i would have thought "n=25" would stop that. [19:32:19] 11797.604s = 196.626733333 [19:33:00] appears the query has been running for over 3+ hours [19:35:37] 10Gerrit, 10Release-Engineering-Team-TODO, 10serviceops-radar: Gerrit GC thrashing during branch cut - https://phabricator.wikimedia.org/T231872 (10thcipriani) posted upstream as https://groups.google.com/forum/#!topic/repo-discuss/Ah6ftyVe39c [19:39:16] thcipriani should we kill the query? I doin't think anyone would be waiting for that query to complete. [19:39:23] (03PS1) 10Krinkle: mediawiki-core-doxygen: Remove --no-extensions option to mwdocgen.php [integration/config] - 10https://gerrit.wikimedia.org/r/537189 (https://phabricator.wikimedia.org/T232104) [19:40:03] (03PS2) 10Krinkle: mediawiki-core-doxygen: Remove --no-extensions option to mwdocgen.php [integration/config] - 10https://gerrit.wikimedia.org/r/537189 (https://phabricator.wikimedia.org/T232104) [19:41:39] (03CR) 10Krinkle: [C: 03+2] "Deployed 1 updated job (mediawiki-core-doxygen-docker)." [integration/config] - 10https://gerrit.wikimedia.org/r/537189 (https://phabricator.wikimedia.org/T232104) (owner: 10Krinkle) [19:43:44] paladox: we can probably kill it, I'm fiddling with threaddumps to see if it's locked or waiting on a lock [19:43:51] ok [19:44:03] (03Merged) 10jenkins-bot: mediawiki-core-doxygen: Remove --no-extensions option to mwdocgen.php [integration/config] - 10https://gerrit.wikimedia.org/r/537189 (https://phabricator.wikimedia.org/T232104) (owner: 10Krinkle) [19:45:45] thcipriani luca replied to your thread. [19:45:52] that was fast [19:46:21] yup [19:46:46] heh, oh good, the issue he points to begins, "I am able to systematically reproduce a fatal memory leak that causes the complete collapse of the JVM." weeee [19:46:57] heh [19:47:02] jgit memory leak :P [19:47:11] yeah [19:50:02] paladox: thread killed [19:50:08] thcipriani thanks! [19:50:24] thanks for pointing it out [19:50:51] yw :) [20:10:40] 10Release-Engineering-Team-TODO, 10MW-1.34-notes (1.34.0-wmf.16; 2019-07-30), 10Patch-For-Review: wdio-cucumber-framework fails on NodeJS due to fibers@2.x under nodejs10 - https://phabricator.wikimedia.org/T229033 (10EBernhardson) CirrusSearch tests upgraded in https://gerrit.wikimedia.org/r/#/c/mediawiki/e... [20:11:02] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10CirrusSearch, and 3 others: selenium-daily-beta-CirrusSearch is broken - https://phabricator.wikimedia.org/T232495 (10EBernhardson) cindy was recently upgraded (patch mer... [20:22:35] 10Continuous-Integration-Config, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (201909), 10Scap, and 5 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) >>! In T223602#5496973, @Anomie wrote:... [20:22:53] 10Continuous-Integration-Config, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (201909), 10Scap, and 5 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) [21:13:45] https://review.opendev.org/#/c/682487/1 may interest hashar [21:47:34] https://review.opendev.org/#/c/682473/ is something we would be interested in i think [21:52:13] 10Gerrit, 10Patch-For-Review: Create a plugin that makes ci comments pretty - https://phabricator.wikimedia.org/T215658 (10Paladox) 05Open→03Declined https://github.com/dburm/pg-test-result-plugin/blob/master/test_result-0.1.html is preferable. [21:53:21] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations: Reimage cobalt and gerrit2001 as buster - https://phabricator.wikimedia.org/T176774 (10Paladox) Cobalt is being replaced with Gerrit1001. [21:55:39] 10MediaWiki-Releasing, 10MediaWiki-extensions-Scribunto, 10MW-1.34-release, 10Patch-For-Review: Bundle Scribunto extension with MediaWiki - https://phabricator.wikimedia.org/T191737 (10CCicalese_WMF) >>! In T191737#5492008, @Jdforrester-WMF wrote: > I think it's reasonable to proceed and let Debian do what... [21:55:50] 10MediaWiki-Releasing, 10MediaWiki-extensions-Scribunto, 10MW-1.34-release, 10Patch-For-Review: Bundle Scribunto extension with MediaWiki - https://phabricator.wikimedia.org/T191737 (10CCicalese_WMF) [21:56:33] (03CR) 10Cicalese: [C: 03+2] make-release: Include Scribunto in 1.34+ [tools/release] - 10https://gerrit.wikimedia.org/r/536690 (https://phabricator.wikimedia.org/T191737) (owner: 10Jforrester) [21:58:26] (03Merged) 10jenkins-bot: make-release: Include Scribunto in 1.34+ [tools/release] - 10https://gerrit.wikimedia.org/r/536690 (https://phabricator.wikimedia.org/T191737) (owner: 10Jforrester) [22:06:43] 10MediaWiki-Releasing, 10MediaWiki-extensions-Scribunto, 10MW-1.34-release, 10Patch-For-Review: Bundle Scribunto extension with MediaWiki - https://phabricator.wikimedia.org/T191737 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF This will happen with 1.34. [22:06:45] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.34-release: Expand the set of bundled extensions and skins in MediaWiki 1.34 - https://phabricator.wikimedia.org/T209220 (10Jdforrester-WMF) [22:07:35] 10Project-Admins: Create a Project Hash Checking - https://phabricator.wikimedia.org/T233060 (10Thargrovewmf) [22:09:03] oh hi corvus! [22:16:17] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.34-release: Expand the set of bundled extensions and skins in MediaWiki 1.34 - https://phabricator.wikimedia.org/T209220 (10CCicalese_WMF) [22:22:38] paladox: o/ [22:23:45] :) [22:40:00] what happened to the poor gate-and-submit queue, it's 4 hours long :( [22:40:47] MatmaRex: Too many long-running jobs combined with people force-merging and resetting reality repeatedly. :-( [22:41:07] It's been > 2 hours long continuously since I started work eight hours ago. [22:44:47] 10Release-Engineering-Team (Deployment services), 10Release, 10Train Deployments: 1.34.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T220748 (10thcipriani) a:03mmodell [22:46:11] 10Release-Engineering-Team (Deployment services), 10Release, 10Train Deployments: 1.34.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T220749 (10thcipriani) a:03mmodell [23:46:22] (03PS1) 10Jforrester: layout: [WebToolsManager] Add CI for this repo, set to just quibble for now [integration/config] - 10https://gerrit.wikimedia.org/r/537226 [23:48:09] (03CR) 10Jforrester: [C: 03+2] layout: [WebToolsManager] Add CI for this repo, set to just quibble for now [integration/config] - 10https://gerrit.wikimedia.org/r/537226 (owner: 10Jforrester) [23:49:43] RECOVERY - Host integration-agent-docker-1001 is UP: PING OK - Packet loss = 0%, RTA = 0.81 ms [23:53:14] (03Merged) 10jenkins-bot: layout: [WebToolsManager] Add CI for this repo, set to just quibble for now [integration/config] - 10https://gerrit.wikimedia.org/r/537226 (owner: 10Jforrester) [23:54:24] !log Zuul: Initial CI for WebToolsManager [23:54:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:56:31] PROBLEM - Host integration-agent-docker-1001 is DOWN: CRITICAL - Host Unreachable (172.16.3.36)