[07:59:08] PROBLEM - Parsoid on deployment-parsoid09 is CRITICAL: connect to address 172.16.5.63 and port 8000: Connection refused [08:22:37] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+2] Add @property to work with UnusedUseStatementSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/536326 (owner: 10Umherirrender) [08:23:44] (03Merged) 10jenkins-bot: Add @property to work with UnusedUseStatementSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/536326 (owner: 10Umherirrender) [08:35:06] (03CR) 10jenkins-bot: Add @property to work with UnusedUseStatementSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/536326 (owner: 10Umherirrender) [08:53:59] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Composer, 10Upstream, 10Wikimedia-production-error (Shared Build Failure): Jenkins jobs failing with Composer TransportException: 404 Not Found (September 2019) - https://phabricator.wikimedia.org/T233264 (10hashar) [[ https://github.... [08:54:07] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Composer, 10Upstream, 10Wikimedia-production-error (Shared Build Failure): Jenkins jobs failing with Composer TransportException: 404 Not Found (September 2019) - https://phabricator.wikimedia.org/T233264 (10hashar) 05Open→03Resolv... [09:20:26] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Operations, 10serviceops: Upload docker-ce 18.06.3 upstream package for Stretch - https://phabricator.wikimedia.org/T226236 (10MoritzMuehlenhoff) Sorry, no. We're not going to intentionally downgrade to an old, known-insecure version. If there's a... [09:22:39] 10Beta-Cluster-Infrastructure: Fatal error: Invalid operand type was used: cannot perform this operation with arrays in /srv/mediawiki/php-master/languages/Language.php on line 514 - https://phabricator.wikimedia.org/T233155 (10MarcoAurelio) Per @Urbanecm and @jeena. It looks this was just a beta cluster instanc... [09:31:59] !log Upgrading Jenkins plugins on CI and release hosts [09:32:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:45:26] 10Continuous-Integration-Config, 10Operations, 10Release Pipeline, 10local-charts, and 2 others: Set up CI for the deployment-charts repository - https://phabricator.wikimedia.org/T233291 (10MarcoAurelio) [09:51:02] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909): Rebuild integration-slave-docker-* instances to use less RAM, new name and Stretch - https://phabricator.wikimedia.org/T226233 (10hashar) [09:51:05] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Operations, 10serviceops: Upload docker-ce 18.06.3 upstream package for Stretch - https://phabricator.wikimedia.org/T226236 (10hashar) 05Open→03Declined It is not about downgrading Docker, but rather to keep the same version we are currently u... [09:51:08] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Operations: Migrate contint* hosts to Stretch/Buster - https://phabricator.wikimedia.org/T224591 (10hashar) [09:52:01] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909): Rebuild integration-slave-docker-* instances to use less RAM, new name - https://phabricator.wikimedia.org/T226233 (10hashar) [09:52:30] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909): Rebuild integration-slave-docker-* instances to use less RAM, new name - https://phabricator.wikimedia.org/T226233 (10hashar) Will do the Stretch upgrade l... [10:03:49] 10Continuous-Integration-Config, 10Operations, 10Release Pipeline, 10local-charts, and 2 others: Set up CI for the deployment-charts repository - https://phabricator.wikimedia.org/T233291 (10hashar) @Jdforrester-WMF has added an experimental `helm-lint` job to the repository: T216049. It runs `help lint --... [10:05:25] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Quibble: Quibble should fatal out on clone/fetch failure"ERROR:zuul.Repo:Unable to initialize repo for npm-test.git" - https://phabricator.wikimedia.org/T233143 (10hashar) [10:07:04] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Operations, 10serviceops: Upload docker-ce 18.06.3 upstream package for Stretch - https://phabricator.wikimedia.org/T226236 (10MoritzMuehlenhoff) >>! In T226236#5505608, @hashar wrote: > Anywa,y I am declining this and postpone the migration to St... [10:12:42] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Quibble: Quibble should fatal out on clone/fetch failure"ERROR:zuul.Repo:Unable to initialize repo for npm-test.git" - https://phabricator.wikimedia.org/T233143 (10hashar) I think that is due to //Support to clone repositories in parallel// (5f58... [10:15:24] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Quibble: Quibble should fatal out on clone/fetch failure"ERROR:zuul.Repo:Unable to initialize repo for npm-test.git" - https://phabricator.wikimedia.org/T233143 (10hashar) ` name=quibble/zuul.py with ThreadPoolExecutor(max_workers=workers) as... [10:23:58] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Cleanup, 10Maps, and 2 others: Delete (rather than archive) the maps/kartotherian and maps/tilerator repos - https://phabricator.wikimedia.org/T228483 (10MarcoAurelio) [10:25:08] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Cleanup, 10Maps, and 2 others: Delete (rather than archive) the maps/kartotherian and maps/tilerator repos - https://phabricator.wikimedia.org/T228483 (10MarcoAurelio) I have deleted `maps/kartotherian` and `maps/tilerato... [10:33:57] 10Continuous-Integration-Config, 10Wikidata-Tainted-References-Sprint1: Make vue linting and testing run with CI - https://phabricator.wikimedia.org/T233107 (10Tarrow) Current sequential install time with `npm install` takes: npm install 144.30s user 63.94s system 75% cpu 4:35.22 total Current sequential ins... [10:34:58] 10Continuous-Integration-Config, 10Operations, 10Release Pipeline, 10local-charts, and 2 others: Set up CI for the deployment-charts repository - https://phabricator.wikimedia.org/T233291 (10Joe) >>! In T233291#5505631, @hashar wrote: > @Jdforrester-WMF has added an experimental `helm-lint` job to the repo... [10:35:19] 10Continuous-Integration-Config, 10Operations, 10Release Pipeline, 10local-charts, and 2 others: Set up CI for the deployment-charts repository - https://phabricator.wikimedia.org/T233291 (10Joe) p:05Triage→03High a:03Joe [10:36:14] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO, 10Developer Productivity, 10local-charts: Add tests to local-charts / configure local-charts for CI - https://phabricator.wikimedia.org/T217868 (10Joe) [10:36:16] 10Continuous-Integration-Config, 10Operations, 10Release Pipeline, 10local-charts, and 2 others: Set up CI for the deployment-charts repository - https://phabricator.wikimedia.org/T233291 (10Joe) [10:36:19] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10serviceops, 10Kubernetes, 10User-fsero: add CI job into operations/deployments-charts repo that helm lint packages and perform the helm index after merge. - https://phabricator.wikimedia.org/T216049 (10Joe) [10:43:15] 10Project-Admins: Wiki Club West project - https://phabricator.wikimedia.org/T231416 (10JarrahTree) Wiki Club West project - the Wiki Club West organisation is basically an extension of Wikimedia Australia - in the state of Western Australia, we have had GLAM collaborations within the state, and the capital city... [11:25:44] 10Phabricator, 10Project-Admins, 10Growth-Team (Current Sprint): Convert GrowthExperiments-* project to subprojects of GrowthExperiments - https://phabricator.wikimedia.org/T230831 (10JTannerWMF) Thanks for doing this @Aklapper @mmodell [12:34:15] 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Make Scrum of Scrums meeting notes easier to read - https://phabricator.wikimedia.org/T227176 (10zeljkofilipin) [13:13:34] (03PS1) 10Hashar: Exit on cloning failure [integration/quibble] - 10https://gerrit.wikimedia.org/r/538020 (https://phabricator.wikimedia.org/T233143) [13:13:46] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Quibble, 10Patch-For-Review: Quibble should fatal out on clone/fetch failure"ERROR:zuul.Repo:Unable to initialize repo for npm-test.git" - https://phabricator.wikimedia.org/T233143 (10hashar) a:03hashar [13:18:01] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T220748 (10Reedy) [13:18:18] (03CR) 10Hashar: "/me invokes awight python wisdom!" (034 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/538020 (https://phabricator.wikimedia.org/T233143) (owner: 10Hashar) [13:18:52] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Quibble, 10Patch-For-Review: Quibble should fatal out on clone/fetch failure"ERROR:zuul.Repo:Unable to initialize repo for npm-test.gi... - https://phabricator.wikimedia.org/T233143 [13:19:35] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Quibble, 10Patch-For-Review: Quibble should fatal out on clone/fetch failure"ERROR:zuul.Repo:Unable to initialize repo for npm-test.gi... - https://phabricator.wikimedia.org/T233143 [13:19:43] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Quibble, 10Patch-For-Review: Quibble should fatal out on clone/fetch failure"ERROR:zuul.Repo:Unable to initialize repo for npm-test.gi... - https://phabricator.wikimedia.org/T233143 [13:20:58] <_joe_> anyone willing to review https://gerrit.wikimedia.org/r/#/c/integration/config/+/538023 and https://gerrit.wikimedia.org/r/#/c/integration/config/+/538024 for me? [13:21:33] (03PS1) 10Giuseppe Lavagetto: helm-linter: run rake instead of helm lint directly [integration/config] - 10https://gerrit.wikimedia.org/r/538023 (https://phabricator.wikimedia.org/T233291) [13:21:35] (03PS1) 10Giuseppe Lavagetto: Switch the experimental job for deployment-charts to the new image [integration/config] - 10https://gerrit.wikimedia.org/r/538024 [13:21:47] <_joe_> wow wikibugs has a lot of bugs [13:21:52] <_joe_> err lag [13:32:00] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10serviceops-radar: Proposal: simplify set up of basic CI jobs for new projects - https://phabricator.wikimedia.org/T233298 (10Joe) [13:32:34] <_joe_> liw, hashar out of the toil I had to endure today, alex and I came up with the proposal above ^^ [13:32:41] <_joe_> I would like to hear your opinions [13:32:51] _joe_: of the toil? [13:33:16] <_joe_> hashar: performing repetitive actions that scale linearly, more or less [13:33:33] <_joe_> like the N steps we need to do to set up / modify CI for a project [13:34:02] <_joe_> https://landing.google.com/sre/sre-book/chapters/eliminating-toil/ :) [13:41:39] (03PS2) 10Hashar: Exit on cloning failure [integration/quibble] - 10https://gerrit.wikimedia.org/r/538020 (https://phabricator.wikimedia.org/T233143) [13:42:55] (03CR) 10Hashar: Exit on cloning failure (032 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/538020 (https://phabricator.wikimedia.org/T233143) (owner: 10Hashar) [13:45:29] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T220748 (10Reedy) [13:48:50] 10Project-Admins: Wiki Club West project - https://phabricator.wikimedia.org/T231416 (10Aklapper) 05Stalled→03Resolved a:03Aklapper Thanks! Requested public project #wiki-club-west has been created: https://phabricator.wikimedia.org/project/view/4286/ Please encourage interested people to visit the projec... [13:52:08] _joe_: that would be in the future CI I guess. iirc we made that proposal a hard requirement [13:56:14] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10serviceops-radar: Proposal: simplify set up of basic CI jobs for new projects - https://phabricator.wikimedia.org/T233298 (10thcipriani) p:05Triage→03Normal Nice proposal! This dovetails with a lot of our curre... [13:57:32] 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Make Scrum of Scrums meeting notes easier to read - https://phabricator.wikimedia.org/T227176 (10zeljkofilipin) [14:00:38] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10serviceops-radar: Proposal: simplify set up of basic CI jobs for new projects - https://phabricator.wikimedia.org/T233298 (10thcipriani) Also, I should say, feel free to yell at us on the [[https://www.mediawiki.or... [14:04:55] 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Make Scrum of Scrums meeting notes easier to read - https://phabricator.wikimedia.org/T227176 (10zeljkofilipin) [14:05:04] 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Make Scrum of Scrums meeting notes easier to read - https://phabricator.wikimedia.org/T227176 (10zeljkofilipin) [14:17:00] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201909), 10CirrusSearch, 10Discovery-Search (Current work): Unit tests in mediawiki-config repo don't pass under PHP72, only HHVM - https://phabricator.wikimedia.org/T232691 (10dcausse) [14:20:27] (03CR) 10Hashar: helm-linter: run rake instead of helm lint directly (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/538023 (https://phabricator.wikimedia.org/T233291) (owner: 10Giuseppe Lavagetto) [14:21:24] (03CR) 10Hashar: [C: 03+1] Switch the experimental job for deployment-charts to the new image [integration/config] - 10https://gerrit.wikimedia.org/r/538024 (owner: 10Giuseppe Lavagetto) [14:25:04] 10MediaWiki-Releasing, 10AbuseFilter, 10MW-1.35-release, 10Patch-For-Review, 10User-Daimona: Bundle AbuseFilter extension with MediaWiki - https://phabricator.wikimedia.org/T191740 (10Daimona) [14:28:59] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (201909), 10Release Pipeline, 10Maps (Kartotherian): Deployment Pipeline fails with CPS error for Kartotherian - https://phabricator.wikimedia.org/T233316 (10thcipriani) [14:29:32] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (201909), 10Release Pipeline, 10Maps (Kartotherian): Deployment Pipeline fails with CPS error for Kartotherian - https://phabricator.wikimedia.org/T233316 (10thcipriani) p:05Triage→03Normal [14:31:13] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10MinervaNeue, 10Readers-Web-Backlog, 10User-zeljkofilipin: Need RelEng help: selenium-daily-beta-Minerva and MobileFrontend has failed for 6 days - https://phabricator.wikimedia.org/T232708 (10zeljkofilipi... [14:31:21] 10Release-Engineering-Team-TODO (201909), 10MobileFrontend, 10Readers-Web-Backlog, 10User-zeljkofilipin: MF Selenium tests failing with "Promise timeout" - https://phabricator.wikimedia.org/T233193 (10zeljkofilipin) p:05Triage→03Normal [14:43:59] (03CR) 10Giuseppe Lavagetto: "Thanks for the review!" (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/538023 (https://phabricator.wikimedia.org/T233291) (owner: 10Giuseppe Lavagetto) [14:45:50] (03PS2) 10Giuseppe Lavagetto: helm-linter: run rake instead of helm lint directly [integration/config] - 10https://gerrit.wikimedia.org/r/538023 (https://phabricator.wikimedia.org/T233291) [14:45:52] (03PS2) 10Giuseppe Lavagetto: Switch the experimental job for deployment-charts to the new image [integration/config] - 10https://gerrit.wikimedia.org/r/538024 [15:41:31] (03CR) 10Jforrester: [C: 03+1] Exit on cloning failure [integration/quibble] - 10https://gerrit.wikimedia.org/r/538020 (https://phabricator.wikimedia.org/T233143) (owner: 10Hashar) [15:48:51] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Cleanup, 10Maps, and 2 others: Delete (rather than archive) the maps/kartotherian and maps/tilerator repos - https://phabricator.wikimedia.org/T228483 (10Jdforrester-WMF) 05Open→03Resolved [16:34:24] (03CR) 10Jforrester: [C: 03+2] helm-linter: run rake instead of helm lint directly [integration/config] - 10https://gerrit.wikimedia.org/r/538023 (https://phabricator.wikimedia.org/T233291) (owner: 10Giuseppe Lavagetto) [16:36:17] (03Merged) 10jenkins-bot: helm-linter: run rake instead of helm lint directly [integration/config] - 10https://gerrit.wikimedia.org/r/538023 (https://phabricator.wikimedia.org/T233291) (owner: 10Giuseppe Lavagetto) [16:39:52] _joe_, I see nothing bad in those changes, but I'm afraid I don't understand the current CI or integration-config well enough to vote on them [16:40:44] <_joe_> liw: oh I was asking your feedback on a task I wrote actually [16:41:33] _joe_, I'm confused? [16:43:14] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10serviceops-radar: Proposal: simplify set up of basic CI jobs for new projects - https://phabricator.wikimedia.org/T233298 (10Joe) Just to clarify - this wasn't an attempt to imagine a future, perfect system, but ju... [16:43:21] <_joe_> liw: https://phabricator.wikimedia.org/T233298 :) [16:45:11] _joe_, check, I'll read and comment on the task (though not today, I have a meeting in 45 mins) [16:45:26] <_joe_> yeah no rush at all :) [16:45:33] <_joe_> just wanted to get it on your radar [16:48:34] (03CR) 10Hashar: helm-linter: run rake instead of helm lint directly (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/538023 (https://phabricator.wikimedia.org/T233291) (owner: 10Giuseppe Lavagetto) [16:49:01] Do we have a way to know which Diffusion repos are configured to Mirror (push to elsewhere) contents? [16:49:55] (03PS1) 10Hashar: docker: fix user root in helm-linter [integration/config] - 10https://gerrit.wikimedia.org/r/538065 (https://phabricator.wikimedia.org/T233291) [16:50:11] (03CR) 10Hashar: [C: 03+2] docker: fix user root in helm-linter [integration/config] - 10https://gerrit.wikimedia.org/r/538065 (https://phabricator.wikimedia.org/T233291) (owner: 10Hashar) [16:51:39] (03Merged) 10jenkins-bot: docker: fix user root in helm-linter [integration/config] - 10https://gerrit.wikimedia.org/r/538065 (https://phabricator.wikimedia.org/T233291) (owner: 10Hashar) [16:52:50] !log Building releng/helm-linter:0.2.0 T233291 [16:52:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:52:53] T233291: Set up CI for the deployment-charts repository - https://phabricator.wikimedia.org/T233291 [16:52:58] (03CR) 10Jforrester: [C: 03+2] "Oh, hah." [integration/config] - 10https://gerrit.wikimedia.org/r/538065 (https://phabricator.wikimedia.org/T233291) (owner: 10Hashar) [16:53:38] (03PS3) 10Hashar: Switch the experimental job for deployment-charts to the new image [integration/config] - 10https://gerrit.wikimedia.org/r/538024 (owner: 10Giuseppe Lavagetto) [16:53:47] Successfully tagged docker-registry.discovery.wmnet/releng/helm-linter:0.2.0 (image.py:179) [16:53:50] better :] [16:54:45] (03CR) 10Hashar: [C: 03+2] "INFO:jenkins_jobs.builder:Reconfiguring jenkins job helm-lint" [integration/config] - 10https://gerrit.wikimedia.org/r/538024 (owner: 10Giuseppe Lavagetto) [16:56:52] (03Merged) 10jenkins-bot: Switch the experimental job for deployment-charts to the new image [integration/config] - 10https://gerrit.wikimedia.org/r/538024 (owner: 10Giuseppe Lavagetto) [16:57:34] (03PS1) 10Umherirrender: [OAuth] Run phan job [integration/config] - 10https://gerrit.wikimedia.org/r/538067 (https://phabricator.wikimedia.org/T224761) [17:01:59] (03CR) 10Hashar: [C: 03+2] [OAuth] Run phan job [integration/config] - 10https://gerrit.wikimedia.org/r/538067 (https://phabricator.wikimedia.org/T224761) (owner: 10Umherirrender) [17:03:32] (03Merged) 10jenkins-bot: [OAuth] Run phan job [integration/config] - 10https://gerrit.wikimedia.org/r/538067 (https://phabricator.wikimedia.org/T224761) (owner: 10Umherirrender) [17:04:12] (03CR) 10Hashar: [C: 03+2] "deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/538067 (https://phabricator.wikimedia.org/T224761) (owner: 10Umherirrender) [17:05:00] 10Release-Engineering-Team-TODO (201909), 10MobileFrontend, 10Readers-Web-Backlog, 10User-zeljkofilipin: MF and Minerva Selenium tests failing with "Promise timeout" - https://phabricator.wikimedia.org/T233193 (10Jdlrobson) [17:18:47] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10Wikimedia-General-or-Unknown, 10Epic, and 2 others: Enable mediawiki/mediawiki-phan-config on all Wikimedia-deployed repositories - https://phabricator.wikimedia.org/T224783 (10Umherirrender) [17:22:43] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (201909), 10Release Pipeline, 10Maps (Kartotherian): Deployment Pipeline fails with CPS error for Kartotherian - https://phabricator.wikimedia.org/T233316 (10dduvall) a:03dduvall [17:28:36] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:33:31] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 48070 bytes in 6.347 second response time [17:35:55] 10Gerrit, 10Phabricator, 10Wikimedia-Logstash, 10observability, and 2 others: Include Gerrit apache logs in Logstash - https://phabricator.wikimedia.org/T219271 (10thcipriani) I do see the gerrit apache logs in logstash; however, I don't see the log4j logs for gerrit in logstash (https://gerrit.wikimedia.o... [17:36:17] thcipriani we no longer send log4j logs to logstash. [17:36:28] Rather i think herron converted it to use kafka [17:36:43] also the log4j plugin was deprecated in logstash [17:36:48] not sure if it has been removed [17:38:48] what does "converted it to use kafka" entail? I think the gist of my questions are: I don't see gerrit log4j events in logstash anymore, but it sure would be nice to be able to :) [17:40:01] thcipriani https://phabricator.wikimedia.org/T213899 [17:40:41] thcipriani also https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/492633/ [17:41:24] i [17:41:44] i'm not sure if recursivly gets the log, or if it's using the old file. [17:43:27] thcipriani https://github.com/wikimedia/puppet/commit/b8d0a9764c9465d853d9aec37d9383da31b83b28#diff-3c513f2f4046f814a8fd23ae0bfbc476 [17:43:31] so we've removed the non-kafka logstash input in that patch; however, I can't seem to find the new logs... [17:43:49] that's using the old files :( [17:45:34] thcipriani i'm not sure how to find it in logstash :( [17:45:48] though... it's not using gerrit.log thus no recent data from that [17:45:56] it is indeed using an old file. Which might explain the emptiness of program:input-file-gerrit-multiline... [17:46:04] (if I'm even searching correctly) [17:46:48] yeh [18:05:49] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (201909), 10MediaWiki-extensions-CentralAuth, 10User-zeljkofilipin: [betalabs] Cannot create a new user account - https://phabricator.wikimedia.org/T232796 (10mmodell) @tgr: where did you see these logs specifically? There are two versions of... [18:15:26] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10cloud-services-team: Check bandwidth limitation on integration-castor03.integration.eqiad.wmflabs / cloudvirt1002 - https://phabricator.wikimedia.org/T232644 (10hashar) ... [18:17:40] 10Continuous-Integration-Infrastructure: castor rsync's taking 3-5 minutes for mwgate-npm jobs - https://phabricator.wikimedia.org/T188375 (10hashar) The network is barely capped anymore, it got bumped to 800Mbits for egress traffic. [18:18:28] 10Continuous-Integration-Infrastructure: castor rsync's taking 3-5 minutes for mwgate-npm jobs - https://phabricator.wikimedia.org/T188375 (10hashar) [18:18:30] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10cloud-services-team: Move integration-castor03.integration.eqiad.wmflabs to a newer cloudvirt machine - https://phabricator.wikimedia.org/T232646 (10hashar) [18:18:32] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10cloud-services-team: Check bandwidth limitation on integration-castor03.integration.eqiad.wmflabs / cloudvirt1002 - https://phabricator.wikimedia.org/T232644 (10hashar) [18:19:54] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Cloud-VPS, 10cloud-services-team: Move integration-castor03.integration.eqiad.wmflabs to a newer cloudvirt machine - https://phabricator.wikimedia.org/T232646 (10hashar... [19:04:21] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [19:07:18] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (201909), 10Release Pipeline, 10Maps (Kartotherian): Deployment Pipeline fails with CPS error for Kartotherian - https://phabricator.wikimedia.org/T233316 (10dduvall) There are a few things going on here. First, the `execution` field f... [19:13:37] 10Continuous-Integration-Config, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (201909), 10Scap, and 5 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) [19:13:59] (03PS1) 10Dduvall: Default to locahost for JENKINS_HOST in Makefile [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/538086 [19:14:05] 10Continuous-Integration-Config, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (201909), 10Scap, and 5 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) [19:14:18] (03CR) 10jerkins-bot: [V: 04-1] Default to locahost for JENKINS_HOST in Makefile [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/538086 (owner: 10Dduvall) [19:17:31] (03PS1) 10Dduvall: Annotate ExecutionGraph.toString as NonCPS [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/538088 [19:17:46] (03CR) 10jerkins-bot: [V: 04-1] Annotate ExecutionGraph.toString as NonCPS [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/538088 (owner: 10Dduvall) [19:18:55] 10Release-Engineering-Team, 10Wikimedia-Site-requests: "empty" InitialiseSettings.php breaks a ton of links and references to that file - https://phabricator.wikimedia.org/T233069 (10Krinkle) Per T223602, this is only a transitional stage. Unless it's easy and worthwhile updating twice in a short period, I thi... [19:21:30] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [19:23:03] 10Release-Engineering-Team, 10Wikimedia-Site-requests: "empty" InitialiseSettings.php breaks a ton of links and references to that file - https://phabricator.wikimedia.org/T233069 (10Jdforrester-WMF) I mean, it's temporary that there's a file called VariantSettings and no (useful) file called InitialiseSetting... [19:24:02] (03PS1) 10Umherirrender: [PropertySuggester] Add phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/538090 (https://phabricator.wikimedia.org/T224774) [19:33:58] (03PS1) 10Dduvall: Validate that `execution` configuration is a list of lists [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/538093 (https://phabricator.wikimedia.org/T233316) [19:34:11] (03CR) 10jerkins-bot: [V: 04-1] Validate that `execution` configuration is a list of lists [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/538093 (https://phabricator.wikimedia.org/T233316) (owner: 10Dduvall) [19:34:29] (03PS2) 10Dduvall: Annotate ExecutionGraph.toString as NonCPS [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/538088 (https://phabricator.wikimedia.org/T233316) [19:34:51] (03CR) 10jerkins-bot: [V: 04-1] Annotate ExecutionGraph.toString as NonCPS [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/538088 (https://phabricator.wikimedia.org/T233316) (owner: 10Dduvall) [19:41:49] (03CR) 10Dduvall: "retest" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/538088 (https://phabricator.wikimedia.org/T233316) (owner: 10Dduvall) [19:43:59] (03CR) 10Dduvall: "recheck" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/538088 (https://phabricator.wikimedia.org/T233316) (owner: 10Dduvall) [19:44:09] (03CR) 10jerkins-bot: [V: 04-1] Annotate ExecutionGraph.toString as NonCPS [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/538088 (https://phabricator.wikimedia.org/T233316) (owner: 10Dduvall) [19:46:13] (03PS1) 10Dduvall: Upgrade blubber config to v4 [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/538096 [19:46:48] (03PS3) 10Dduvall: Annotate ExecutionGraph.toString as NonCPS [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/538088 (https://phabricator.wikimedia.org/T233316) [19:46:54] (03CR) 10jerkins-bot: [V: 04-1] Upgrade blubber config to v4 [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/538096 (owner: 10Dduvall) [19:47:13] (03CR) 10jerkins-bot: [V: 04-1] Annotate ExecutionGraph.toString as NonCPS [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/538088 (https://phabricator.wikimedia.org/T233316) (owner: 10Dduvall) [20:12:35] 10Continuous-Integration-Config, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (201909), 10Scap, and 5 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Krinkle) >>! In T223602#5455095, @gerritbot wrote: > Ch... [20:13:44] 10Release-Engineering-Team, 10Wikimedia-Site-requests: "empty" InitialiseSettings.php breaks a ton of links and references to that file - https://phabricator.wikimedia.org/T233069 (10Krinkle) 05Open→03Declined Declining then. Short version: The code in VariantSettings will soon move back to InitialiseSetti... [22:15:38] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Cleanup, 10Maps, and 2 others: Delete (rather than archive) the maps/kartotherian and maps/tilerator repos - https://phabricator.wikimedia.org/T228483 (10MarcoAurelio) Thanks to whoever did the GitHub deletions. I was goi... [22:25:05] 10Release-Engineering-Team-TODO, 10serviceops: Blubberoid endpoint intermittently routing to MediaWiki backend - https://phabricator.wikimedia.org/T233369 (10dduvall) [22:28:44] 10Release-Engineering-Team-TODO, 10serviceops: Blubberoid endpoint intermittently routing to MediaWiki backend - https://phabricator.wikimedia.org/T233369 (10dduvall) p:05Triage→03High [22:38:57] 10Release-Engineering-Team-TODO, 10Operations, 10Traffic: Blubberoid endpoint intermittently routing to MediaWiki backend - https://phabricator.wikimedia.org/T233369 (10dduvall) [23:08:59] 10Gerrit, 10Operations: setup/install gerrit1001 - https://phabricator.wikimedia.org/T231046 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` gerrit1001.wikimedia.org ` The log can be found in `/var/log/wmf-auto-reimage/201909192308_dzahn_39474_gerrit10... [23:26:48] 10Gerrit, 10Operations: setup/install gerrit1001 - https://phabricator.wikimedia.org/T231046 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['gerrit1001.wikimedia.org'] ` and were **ALL** successful. [23:37:10] 10Project-Admins, 10WikiClubWest: Wiki Club West project - https://phabricator.wikimedia.org/T231416 (10Samwilson) [23:46:32] 10Continuous-Integration-Config, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (201909), 10Scap, and 5 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) >>! In T223602#5507683, @Krinkle wrote...