[00:17:23] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs, 10Patch-For-Review, 10Regression: FlaggedRevs: Automatic user promotion stopped working on some wikis on June 24, 2019 - https://phabricator.wikimedia.org/T237191 (10DannyS712) @matmarex the rights should probably also be removed from the user... [00:30:26] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Math, 10Mathoid: Migrate mathoid and other remaining services to use PipelineLib - https://phabricator.wikimedia.org/T261809 (10jeena) [00:30:30] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Wikidata-Termbox, 10Patch-For-Review: Migrate termbox to pipelineLib CI - https://phabricator.wikimedia.org/T262987 (10jeena) [01:12:29] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10User-brennen: Write proposal for personal development solution - https://phabricator.wikimedia.org/T242258 (10jeena) 05Open→03Resolved Proposal was written in Google docs. I've now moved it to my us... [01:18:25] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Release Pipeline: Modify PipelineLib to work with WMF chart museum - https://phabricator.wikimedia.org/T261346 (10jeena) 05Open→03Resolved [07:03:11] (03CR) 10Ladsgroup: [C: 03+1] jjb: Migrate termbox pipeline to pipelinelib [integration/config] - 10https://gerrit.wikimedia.org/r/627631 (https://phabricator.wikimedia.org/T262987) (owner: 10Jeena Huneidi) [07:07:27] hashar: Hey, good morning. I want to bother you for this change https://gerrit.wikimedia.org/r/c/integration/config/+/627631 :D [07:07:48] Amir1: hi [07:08:04] bah I don't have access to Gerrit somehow [07:08:10] hmm no, was just "slow" [07:08:52] :D [07:09:40] Amir1: deploying it [07:10:05] Thanks \o/ [07:10:08] grrb [07:10:13] stupid trailing whitespaces [07:10:55] (03CR) 10Hashar: jjb: Migrate termbox pipeline to pipelinelib (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/627631 (https://phabricator.wikimedia.org/T262987) (owner: 10Jeena Huneidi) [07:11:09] (03PS6) 10Hashar: jjb: Migrate termbox pipeline to pipelinelib [integration/config] - 10https://gerrit.wikimedia.org/r/627631 (https://phabricator.wikimedia.org/T262987) (owner: 10Jeena Huneidi) [07:14:20] (03CR) 10Hashar: [C: 03+2] "I have fixed some whitespaces." (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/627631 (https://phabricator.wikimedia.org/T262987) (owner: 10Jeena Huneidi) [07:15:07] Amir1: the dependency should probably have been the other way around: first the CI config change in integration/config and once deployed the change that introduces the .pipeline/config.xml file ;] [07:16:03] xml? [07:16:04] :D [07:16:05] (03Merged) 10jenkins-bot: jjb: Migrate termbox pipeline to pipelinelib [integration/config] - 10https://gerrit.wikimedia.org/r/627631 (https://phabricator.wikimedia.org/T262987) (owner: 10Jeena Huneidi) [07:16:40] The final change is https://gerrit.wikimedia.org/r/c/wikibase/termbox/+/627414 [07:16:59] It hasn't been merged yet (I'm not sure if it's fully done) [07:17:31] it was referring to the merged change https://gerrit.wikimedia.org/r/c/wikibase/termbox/+/627632 [07:17:38] but yeah that is the not end of the world ;] [07:38:22] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs, 10Patch-For-Review, 10Regression: FlaggedRevs: Automatic user promotion stopped working on some wikis on June 24, 2019 - https://phabricator.wikimedia.org/T237191 (10Count_Count) >>! In T237191#6464683, @matmarex wrote: > It looks like it does... [07:58:25] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Enable avatars in gerrit - https://phabricator.wikimedia.org/T191183 (10hashar) 05Open→03Declined The summary is: We can't use the third party service gravatar.com since that leaks... [08:13:06] I'd like to do a security backport for https://phabricator.wikimedia.org/T262628 , is there anyone here who wants to hear more? :-) [08:13:30] (03CR) 10Hashar: [C: 03+2] dockerfiles: [java8-sonar-scanner] Excludes resources/lib [integration/config] - 10https://gerrit.wikimedia.org/r/627420 (https://phabricator.wikimedia.org/T256942) (owner: 10Kosta Harlan) [08:14:35] (03Merged) 10jenkins-bot: dockerfiles: [java8-sonar-scanner] Excludes resources/lib [integration/config] - 10https://gerrit.wikimedia.org/r/627420 (https://phabricator.wikimedia.org/T256942) (owner: 10Kosta Harlan) [08:15:27] kostajh: I am deploying your sonar related chagne for resources/libs [08:15:36] hashar: thx [08:17:04] While making the patch I was thinking it might be nice to just add a sonar-project.properties file to MediaWiki core. That way, making changes to exclusion patterns etc is manageable without an image rebuild. OTOH the rules don't have to be changed often if at all, so maybe it's just fine as it is. [08:18:46] !log Successfully tagged docker-registry.discovery.wmnet/releng/java8-sonar-scanner:0.6.2-s4 # T256942 [08:18:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:18:50] T256942: Stop SonarQube spam - https://phabricator.wikimedia.org/T256942 [08:21:10] (03CR) 10Hashar: [C: 03+2] "Jobs refreshed:" [integration/config] - 10https://gerrit.wikimedia.org/r/627421 (https://phabricator.wikimedia.org/T256942) (owner: 10Kosta Harlan) [08:22:17] (03Merged) 10jenkins-bot: jjb: Update sonar-scanner image [integration/config] - 10https://gerrit.wikimedia.org/r/627421 (https://phabricator.wikimedia.org/T256942) (owner: 10Kosta Harlan) [09:03:34] hashar: gehel pointed me to you to ask about a potential use of JUnit Jenkins plugin in our project (wikidata-query-rdf-maven-java8-docker) - would that be possible? [09:06:40] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Discovery-Search, 10Wikimedia-Logstash: scap on beta fails canary check: KeyError: 'aggregations' - https://phabricator.wikimedia.org/T260667 (10hashar) Th... [09:08:10] zpapierski: yup should be. You can file a task against #continuous-integration-config . I guess the devil is to find in which directory the junit reports are written to and we can then adjust the job to process them. [09:09:34] hashar: thanks! I'll do that [09:11:31] zpapierski: I guess all the build artifacts under a specific directory. Something such as ./build [09:12:09] so in Docker we would create a bind mount to have the files to be written on the host and Jenkins will then be able to process them [09:13:11] is there something I can do with that, other than providing where the reports live? [09:13:41] (our project is modular and generally reports are under **/target/surefire-reports/*.xml) [09:14:28] the whole setup will be done by adjusting the Jenkins job [09:15:14] ok, thx [09:15:47] its configuration will be in integration/config.git in ./jjb/job-templates.yaml [09:15:52] name: '{name}-maven-java8-docker' [09:15:56] has all the configuration [09:16:12] fun thing, it already mounts /log/ [09:16:35] maybe the reports can be written there when running under CI, but I guess it might be complicated to set that up with maven [09:17:03] hmm, this may actually be more complicated - I can't just mount **/target/surefire-reports :( [09:17:17] need to think more on this, maybe single place for reports is doable [09:17:20] maybe we can change maven target directory using a parameter? [09:17:33] PROBLEM - SSH access on gerrit1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Gerrit [09:18:01] I'm sure it's generally possible, but I have no idea if gettting around multimodule setup here is [09:18:27] we would need to invoke our java gurus [09:18:34] crazy idea - can we just mount it all? [09:18:43] I think some repositories use a specific maven profile for CI [09:18:53] so CI invoeks something like mvn -PCI [09:19:02] and the magic tricks are all done directly in the repo [09:19:21] then yeah [09:19:22] hmm [09:19:26] on second thought you are right [09:19:35] the source code is cloned/checked out in a "src" directory [09:19:57] which is volume binded iinto the container running maven [09:20:06] so it's already available? [09:20:14] so surely the report will be available and it might be just about configure the Junit jenkins plugin to look at: [09:20:24] yay :) [09:20:28] artifacts: src/**/target/surefire-reports/* [09:20:33] or maybe [09:20:52] artifacts: src/**/target/surefire-reports/**/junit*.xml or whatever selector would end up selecting the junit files [09:20:58] yeah that will be easier ;] [09:21:19] cool, I'm on it then [09:21:54] bah my yaml example above is not even proper [09:21:57] should be something such as: [09:21:59] publishers: [09:22:02] - junit: [09:22:37] pattern: 'src/**/target/surefire-reports/junit*.xml' [09:22:51] with 'junit*.xml' needing to be adjusted to match the actual junit filename :] [09:25:08] RECOVERY - SSH access on gerrit1001 is OK: SSH OK - GerritCodeReview_3.2.3-1-g185bdc3a69 (APACHE-SSHD-2.4.0) (protocol 2.0) https://wikitech.wikimedia.org/wiki/Gerrit [09:32:00] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T257977 (10LarsWirzenius) {T262900} is supposed to be fixed, though not closed yet. Backported by @hash... [09:34:04] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)): Gerrit out of heap - https://phabricator.wikimedia.org/T263008 (10hashar) [09:37:15] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)): Gerrit out of heap - https://phabricator.wikimedia.org/T263008 (10hashar) [09:43:57] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)): Gerrit out of heap - https://phabricator.wikimedia.org/T263008 (10hashar) [09:44:53] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)): Gerrit out of heap - https://phabricator.wikimedia.org/T263008 (10hashar) [09:45:34] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)): Gerrit out of heap - https://phabricator.wikimedia.org/T263008 (10hashar) From {F32352401} It looks like memory usage is slowing growing since mid august [09:48:10] SRE has restarted Gerrit [09:48:16] java went out of memory :-\ [10:07:54] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T257977 (10LarsWirzenius) This week's train is powered by Assam, Lapsan souchong, and Creme caramel tea... [10:15:45] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T257977 (10LarsWirzenius) [10:15:56] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T257977 (10LarsWirzenius) Deployment to testwikis worked. Going for group0 now. [10:21:05] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T257977 (10LarsWirzenius) We are at group0 now. [10:27:37] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T257977 (10LarsWirzenius) Reported {T263014} but not making it a train blocker at this rate. [10:36:36] (03PS1) 10ZPapierski: Report junit tests for wikidata mvn jobs [integration/config] - 10https://gerrit.wikimedia.org/r/627783 [10:37:22] (03CR) 10jerkins-bot: [V: 04-1] Report junit tests for wikidata mvn jobs [integration/config] - 10https://gerrit.wikimedia.org/r/627783 (owner: 10ZPapierski) [10:39:03] now that's mildly dissapointing... [10:40:04] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T257977 (10LarsWirzenius) [10:41:20] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T257977 (10LarsWirzenius) Made {T263014} a blocker as it happens a lot now. [10:51:39] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T257977 (10LarsWirzenius) [10:52:34] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T257978 (10Jdforrester-WMF) [11:19:56] (03PS2) 10ZPapierski: Report junit tests for wikidata mvn jobs [integration/config] - 10https://gerrit.wikimedia.org/r/627783 [11:22:29] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs, 10Patch-For-Review, 10Regression: FlaggedRevs: Automatic user promotion stopped working on some wikis on June 24, 2019 - https://phabricator.wikimedia.org/T237191 (101234qwer1234qwer4) >>! In T237191#6464905, @DannyS712 wrote: > @matmarex the... [11:24:55] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MW-1.34-notes (1.34.0-wmf.5; 2019-05-14), 10MW-1.35-notes (1.35.0-wmf.16; 2020-01-21), and 3 others: Stop using jsonlint and instead use eslint-plugin-json for the linting - https://phabricator.wikimedia.org/T220036... [11:30:06] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs, 10Patch-For-Review, 10Regression: FlaggedRevs: Automatic user promotion stopped working on some wikis on June 24, 2019 - https://phabricator.wikimedia.org/T237191 (10Udo_T) In de.wikt, it has only occurred to one user so far and I've already t... [11:40:37] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T257977 (10LarsWirzenius) Reported {T263020} but not as a train blocker: only happens on nostalgiawiki. [12:14:07] 10phan, 10MediaWiki-extensions-Other: Phan related tests for SpamDiffTool extension fails - https://phabricator.wikimedia.org/T263021 (10Zoranzoki21) [12:14:53] 10phan, 10MediaWiki-extensions-Other: Phan related tests for SpamDiffTool extension fails - https://phabricator.wikimedia.org/T263021 (10Zoranzoki21) I had to revert my "moves to the includes/" so tests can pass. But when PHP files are in includes/ or src/ tests are failing. [12:39:48] 10phan, 10MediaWiki-extensions-Other: Phan related tests for SpamDiffTool extension fails - https://phabricator.wikimedia.org/T263021 (10Daimona) >>! In T263021#6466001, @Zoranzoki21 wrote: > I had to revert my "moves to the includes/" so tests can pass. But when PHP files are in includes/ or src/ tests are fa... [13:08:40] 10Release-Engineering-Team (CI & Testing services), 10MinervaNeue, 10Readers-Web-Backlog, 10ci-test-error: Can't call getText on element with selector ".section-heading" because element wasn't found - https://phabricator.wikimedia.org/T262617 (10zeljkofilipin) Just to make sure I've understood the problem:... [13:14:03] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T257977 (10LarsWirzenius) Train promoted to group1. [15:30:22] 10MediaWiki-Releasing, 10AbuseFilter (Overhaul-2020), 10MW-1.36-release, 10Patch-For-Review: Bundle AbuseFilter extension with MediaWiki - https://phabricator.wikimedia.org/T191740 (10Daimona) Hoping that we'll be able to resolve all blockers within the overhaul. [15:33:12] (03CR) 10Ahmon Dancy: [V: 03+2 C: 03+2] doc: add a definition of done for train-dev [tools/train-dev] - 10https://gerrit.wikimedia.org/r/627286 (owner: 10Lars Wirzenius) [15:34:25] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T257977 (10brennen) [16:03:58] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10MW-on-K8s: Sketch MediaWiki production image composition and workflows - https://phabricator.wikimedia.org/T259817 (10dduvall) Thanks for the feedback, @Joe. >>! In T259817#6461335, @Joe wrote: > Let me... [16:09:18] (03PS1) 10Ahmon Dancy: sync -> sync-world in warning message [tools/scap] - 10https://gerrit.wikimedia.org/r/627875 [16:10:08] (03CR) 10Ahmon Dancy: "We may also consider removing the warning now." [tools/scap] - 10https://gerrit.wikimedia.org/r/627875 (owner: 10Ahmon Dancy) [16:12:07] (03CR) 10jerkins-bot: [V: 04-1] sync -> sync-world in warning message [tools/scap] - 10https://gerrit.wikimedia.org/r/627875 (owner: 10Ahmon Dancy) [16:16:28] (03CR) 10Jforrester: "`E501 line too long (81 > 79 characters)`" [tools/scap] - 10https://gerrit.wikimedia.org/r/627875 (owner: 10Ahmon Dancy) [16:17:01] 10Release-Engineering-Team (Pipeline): Dev images registry - https://phabricator.wikimedia.org/T263038 (10jeena) [16:25:19] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Scap, 10EngProd-Virtual-Hackathon, 10Patch-For-Review, 10Sustainability (Incident Followup): Rename `scap sync` to `scap sync-world` - https://phabricator.wikimedia.org/T250302 (10LarsWirzenius) [16:25:41] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Scap, 10EngProd-Virtual-Hackathon, 10Patch-For-Review, 10Sustainability (Incident Followup): Rename `scap sync` to `scap sync-world` - https://phabricator.wikimedia.org/T250302 (10LarsWirzenius) All known cases fixed, but leaving task ope... [16:25:43] hi all! does anyone know if we have somewhere a docker image like this one, but with based on buster rather than stretch? https://dockerregistry.toolforge.org/dev/stretch-php72-fpm-apache2-xdebug/tags/ [16:27:31] (03PS2) 10Ahmon Dancy: sync -> sync-world in warning message [tools/scap] - 10https://gerrit.wikimedia.org/r/627875 (https://phabricator.wikimedia.org/T250302) [16:30:11] (03CR) 10Lars Wirzenius: [C: 03+2] sync -> sync-world in warning message [tools/scap] - 10https://gerrit.wikimedia.org/r/627875 (https://phabricator.wikimedia.org/T250302) (owner: 10Ahmon Dancy) [16:32:21] (03Merged) 10jenkins-bot: sync -> sync-world in warning message [tools/scap] - 10https://gerrit.wikimedia.org/r/627875 (https://phabricator.wikimedia.org/T250302) (owner: 10Ahmon Dancy) [16:44:13] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Scap, 10EngProd-Virtual-Hackathon, 10Patch-For-Review, 10Sustainability (Incident Followup): Rename `scap sync` to `scap sync-world` - https://phabricator.wikimedia.org/T250302 (10LarsWirzenius) p:05Medium→03Low [16:44:18] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10MediaWiki Train Development Environment: train-dev has no "definition of done" - https://phabricator.wikimedia.org/T261394 (10LarsWirzenius) 05Open→03Resolved a DONE.md has been merged now. [16:49:06] greg-g: ^ ? or any ideas whom I might ping? Thanks in advance! [16:55:24] AndyRussG: you've found the browser (https://dockerregistry.toolforge.org/) so if it's not there no :/ Use case? [16:56:48] greg-g: it's for a fundraising specific docker dev setup, trying to mimic fundraising cluster, which is on buster, production setup as closely as possible [16:57:44] I was imagining others might be working on something similar, but maybe not then? Is prod not on buster generally? [16:58:17] AndyRussG, I don't think prod is generally on buster at this time, but I could be wrong [16:59:33] liw: greg-g: maybe someone knows how the stretch-php72-fpm-apache2-xdebug base image, which is recommended for mw dev setup, is generally created and maintained? [16:59:50] AndyRussG, I'm afraid I don't [17:01:20] liw: this is the docker-compose.yml file that is in the core git repo, which references that image: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/refs/heads/master/docker-compose.yml [17:02:03] That specific line authored by Brennen Bearnes, others by Kosta Harlan [17:02:25] hmm I should also figure out what's in the jobrunner image, dunno if I need that too [17:02:35] stretch-php72-jobrunner:0.0.1 [17:03:13] liw greg-g so other than the registry, we don't have doc or repository for how these images for developer usage are created and maintained? [17:03:57] basically for fr-tech we're clear it's time to move away from vagrant, and should stay in sync with what the rest of the organization does as much as possible [17:04:17] AndyRussG: have a look at DEVELOPERS.md in the mediawiki core repo [17:04:35] kostajh: hi! yes that's what I've done [17:04:45] I’d be happy to help you get the fr-tech stack set up, do you mind filing a task and tagging the #MediaWiki-Docker project [17:04:54] kostajh: basically we want to start with something as similar as possible, but based on buster images instead [17:04:56] And adding info about what services and misc things you need? [17:05:09] ok sure! [17:05:11] Ok. I know brennen is working on new images [17:05:12] (i see a ping in here and will catch up with scrollback eventually, but i'm in a meeting and have an UBN to deploy so it might be a minute.) [17:05:17] But I don’t think they are ready yet [17:05:28] brennen: thanks! [17:05:37] I am also away from my computer for the next few hours but will look later tonight / tomorrow [17:06:53] kostajh: basically for fundraising, we actually need a pretty involved stack, that will provide the same or better than the current vagrant fundraising role + https role + mailcatcher role... I'll make a Phab task and link the existing fr-tech tasks, then, and post the link here in a sec! Thanks so much! [17:18:20] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs, 10Patch-For-Review, 10Regression: FlaggedRevs: Automatic user promotion stopped working on some wikis on June 24, 2019 - https://phabricator.wikimedia.org/T237191 (10matmarex) >>! In T237191#6464905, @DannyS712 wrote: > @matmarex the rights sh... [17:21:47] 10Gerrit, 10User-zeljkofilipin: Do not expand large files (like package-lock.json) by default when `Expand All` is clicked - https://phabricator.wikimedia.org/T262944 (10hashar) For the .gitattributes support, I am guessing the logic to determine whether a file is binary comes from jgit so one would have to di... [17:32:29] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10GitLab-Test, 10Patch-For-Review, 10User-brennen: Unable to push repository to remote, hostname is inaccessible - https://phabricator.wikimedia.org/T262516 (10thcipriani) a:05brennen→0... [17:36:47] 10Gerrit, 10User-zeljkofilipin: Do not expand large files (like package-lock.json) by default when `Expand All` is clicked - https://phabricator.wikimedia.org/T262944 (10Krinkle) >>! In T262944#6467291, @hashar wrote: > The issue is that EXPAND ALL does what is requested: expands every single diff, much like a... [17:38:19] kostajh: brennen: Just added the Mediawiki-Docker tag to this existing task, and added a comment explaining what we're trying to accomplish and what I think we could use help with... thanks again! https://phabricator.wikimedia.org/T262976 [17:40:43] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10git-protocol-v2: Gerrit out of heap - https://phabricator.wikimedia.org/T263008 (10hashar) That might be related to the deployment of #git-protocol-v2 on clients. For CI that was... [17:45:48] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Scap: Make `scap sync` prompt before proceeding - https://phabricator.wikimedia.org/T250223 (10thcipriani) 05Open→03Resolved [17:47:46] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Elasticsearch: Set up data storage to collect loosely structured data from CI - https://phabricator.wikimedia.org/T211904 (10mmodell) a:03mmodell [17:55:09] AndyRussG: just had a look, seems doable. I’ll comment more when I’m at my computer later [18:08:45] (03PS21) 10Jeena Huneidi: Enable deployment-charts update for published image [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/622231 (https://phabricator.wikimedia.org/T255835) [18:10:23] still in meetings, but will likewise get caught up in a bit. [18:14:06] kostajh: brennen: thx!!!!! [18:17:14] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T257977 (10Jdlrobson) [18:20:29] (03CR) 10Dduvall: [C: 03+2] "Really nice work!" (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/622231 (https://phabricator.wikimedia.org/T255835) (owner: 10Jeena Huneidi) [18:21:05] (03Merged) 10jenkins-bot: Enable deployment-charts update for published image [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/622231 (https://phabricator.wikimedia.org/T255835) (owner: 10Jeena Huneidi) [18:26:31] (03PS1) 10Jeena Huneidi: .pipeline/config.yaml: add promote step [blubber] - 10https://gerrit.wikimedia.org/r/627893 (https://phabricator.wikimedia.org/T255835) [18:41:13] (03CR) 10Dduvall: [C: 03+2] .pipeline/config.yaml: add promote step [blubber] - 10https://gerrit.wikimedia.org/r/627893 (https://phabricator.wikimedia.org/T255835) (owner: 10Jeena Huneidi) [18:44:14] (03Merged) 10jenkins-bot: .pipeline/config.yaml: add promote step [blubber] - 10https://gerrit.wikimedia.org/r/627893 (https://phabricator.wikimedia.org/T255835) (owner: 10Jeena Huneidi) [18:48:53] 10Release-Engineering-Team (Logspam), 10MediaWiki-extensions-LiquidThreads, 10Wikimedia-production-error: PHP Notice: unserialize(): Error at offset 65519 of 65535 bytes - https://phabricator.wikimedia.org/T259111 (10Krinkle) No known owner per mw:Maintainers.. [18:50:50] (03PS1) 10Jeena Huneidi: fix typo [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/627900 (https://phabricator.wikimedia.org/T255835) [18:51:33] (03CR) 10Dduvall: [C: 03+2] fix typo [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/627900 (https://phabricator.wikimedia.org/T255835) (owner: 10Jeena Huneidi) [18:53:22] (03Merged) 10jenkins-bot: fix typo [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/627900 (https://phabricator.wikimedia.org/T255835) (owner: 10Jeena Huneidi) [18:54:19] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T257977 (10brennen) [18:54:26] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T257977 (10LarsWirzenius) a:05LarsWirzenius→03brennen Over to you, with thanks. [18:55:43] Project beta-scap-eqiad build #317485: 04FAILURE in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/317485/ [19:01:20] Project beta-scap-eqiad build #317486: 04STILL FAILING in 4 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/317486/ [19:02:00] looks like deployment-snapshot01 has filled up [19:02:11] It keeps doing it [19:02:17] I did a bit of cleanup last night... [19:02:25] But ended up basically just rm -rf php-master/cache [19:02:42] isn't big enough [19:03:16] / isn't big enough [19:04:45] !log deployment-snapshot01 removed existing .~tmp~ directories [19:04:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:05:00] ^ rsync leaves those behind and sometimes they're huge [19:05:20] Yeah, I think that was the problem last night [19:08:26] (03PS1) 10Jeena Huneidi: PipelineStage: pass CHARTSREPO as String [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/627904 [19:09:39] thanks for tossing those [19:10:00] let's see how things look tomorrow [19:13:23] Yippee, build fixed! [19:13:23] Project beta-scap-eqiad build #317487: 09FIXED in 9 min 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/317487/ [19:59:18] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10git-protocol-v2: Gerrit out of heap - https://phabricator.wikimedia.org/T263008 (10hashar) I guess we can check the memory usage again in a week / two weeks at https://gerrit.wiki... [19:59:38] (03PS1) 10Ahmon Dancy: Use /etc/wikimedia-cluster to choose the wikiversion file [tools/release] - 10https://gerrit.wikimedia.org/r/627907 [20:00:24] (03PS2) 10Ahmon Dancy: Use /etc/wikimedia-cluster to choose the wikiversion file [tools/release] - 10https://gerrit.wikimedia.org/r/627907 [20:05:54] brennen: would it be feasible to use the deployment pipeline for building the MediaWiki-Docker images, so that making updates to those images is done by merging the patch rather than requiring RelEng to build and publish them? [20:11:21] kostajh: we had mw/core doing that for a while on patches landing in master, so yeah. [20:12:03] there are probably some wrinkles i'm not remembering, but _in principle_ at least. [20:20:14] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs, 10Patch-For-Review, 10Regression: FlaggedRevs: Automatic user promotion stopped working on some wikis on June 24, 2019 - https://phabricator.wikimedia.org/T237191 (10Ameisenigel) >>! In T237191#6467282, @matmarex wrote: >>>! In T237191#646490... [20:54:44] Project beta-scap-eqiad build #317497: 04FAILURE in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/317497/ [20:55:47] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs, 10Patch-For-Review, 10Regression: FlaggedRevs: Automatic user promotion stopped working on some wikis on June 24, 2019 - https://phabricator.wikimedia.org/T237191 (10DannyS712) [21:00:17] Project beta-scap-eqiad build #317498: 04STILL FAILING in 4 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/317498/ [21:07:30] Project beta-scap-eqiad build #317499: 04STILL FAILING in 3 min 37 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/317499/ [21:12:42] when i search Phabricator for the string "mw2256" i get "Cannot use object of type PhutilSafeHTML as array". when i search for "foo" it does not happen. [21:14:34] string: mw1234 = no issue, string: mw2234 = crash [21:15:20] mw222 = OK mw2222 = NOT OK ... odd , isn't it [21:17:35] Project beta-scap-eqiad build #317500: 04STILL FAILING in 3 min 37 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/317500/ [21:21:57] 10Phabricator: Phabricator search - Cannot use object of type PhutilSafeHTML as array - https://phabricator.wikimedia.org/T263063 (10Dzahn) [21:27:28] Project beta-scap-eqiad build #317501: 04STILL FAILING in 3 min 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/317501/ [21:28:11] 10Release-Engineering-Team (Logspam), 10Collection, 10Wikimedia-production-error: Uncaught ReferenceError: collectionCall is not defined - https://phabricator.wikimedia.org/T259809 (10Jdlrobson) [21:28:25] 10Release-Engineering-Team (Logspam), 10Collection, 10Wikimedia-production-error: Uncaught ReferenceError: collectionCall is not defined on slow connections - https://phabricator.wikimedia.org/T259809 (10Jdlrobson) [21:37:32] Project beta-scap-eqiad build #317502: 04STILL FAILING in 3 min 36 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/317502/ [21:47:26] Project beta-scap-eqiad build #317503: 04STILL FAILING in 3 min 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/317503/ [21:50:56] kostajh: See https://phabricator.wikimedia.org/T263038 for why it was stopped [21:57:25] Project beta-scap-eqiad build #317504: 04STILL FAILING in 3 min 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/317504/ [22:07:38] Project beta-scap-eqiad build #317505: 04STILL FAILING in 3 min 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/317505/ [22:17:32] Project beta-scap-eqiad build #317506: 04STILL FAILING in 3 min 37 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/317506/ [22:27:33] Project beta-scap-eqiad build #317507: 04STILL FAILING in 3 min 37 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/317507/ [22:35:47] deployment-snapshot01 again [22:36:03] We should really stop doing cdb and php l10n on beta... [22:36:18] Cause it takes 9G+ [22:37:08] PHP is 1.8G [22:37:16] 1.9G for CDB [22:37:30] upstream/*.json is 2.0G [22:38:08] hmm. no. it's ~5.7G [22:38:13] plus whatever rsync adds when it screws up [22:38:58] (03PS4) 10Jeena Huneidi: Use git credential helper [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/627904 [22:42:01] (03PS5) 10Jeena Huneidi: Use git credential helper [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/627904 [22:43:36] Yippee, build fixed! [22:43:36] Project beta-scap-eqiad build #317508: 09FIXED in 9 min 37 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/317508/ [22:43:59] (03PS6) 10Jeena Huneidi: Use git credential helper [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/627904 [22:54:10] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs, 10Patch-For-Review, 10Regression: FlaggedRevs: Automatic user promotion stopped working on some wikis on June 24, 2019 - https://phabricator.wikimedia.org/T237191 (101234qwer1234qwer4) There have been [[https://en.wikibooks.org/wiki/Special:Lo... [22:56:48] 10MediaWiki-Releasing, 10MW-1.36-release, 10MinervaNeue (Tracking), 10Readers-Web-Backlog (Tracking): Bundle Minerva Neue skin with MediaWiki - https://phabricator.wikimedia.org/T191743 (10Jdlrobson) [23:26:48] !log Cherry picking https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/626241 in beta. [23:26:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:28:01] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T257977 (10brennen) a:05brennen→03LarsWirzenius End-of-day here, passing back to Lars. [23:54:14] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs, 10Patch-For-Review, 10Regression: FlaggedRevs: Automatic user promotion stopped working on some wikis on June 24, 2019 - https://phabricator.wikimedia.org/T237191 (10matmarex)