[01:01:10] PROBLEM - PHD should be supervising processes on phab1001 is CRITICAL: PROCS CRITICAL: 2 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [01:03:04] RECOVERY - PHD should be supervising processes on phab1001 is OK: PROCS OK: 4 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [02:34:09] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10SRE Observability, 10Wikimedia-Logstash, 10observability, 10User-brennen: Experiment with automating error log triage - https://phabricator.wikimedia.org/T290267 (10lmata) @brennen I've sent an invite with proposed timing for the 20th. Please feel free to pr... [02:34:58] PROBLEM - PHD should be supervising processes on phab1001 is CRITICAL: PROCS CRITICAL: 2 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [02:38:48] RECOVERY - PHD should be supervising processes on phab1001 is OK: PROCS OK: 13 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [04:54:26] PROBLEM - PHD should be supervising processes on phab1001 is CRITICAL: PROCS CRITICAL: 2 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [04:56:20] RECOVERY - PHD should be supervising processes on phab1001 is OK: PROCS OK: 13 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [05:18:58] PROBLEM - PHD should be supervising processes on phab1001 is CRITICAL: PROCS CRITICAL: 2 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [05:20:48] RECOVERY - PHD should be supervising processes on phab1001 is OK: PROCS OK: 4 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [05:35:31] 10Project-Admins, 10Turkish-Sites: trwiki Twinkle project - https://phabricator.wikimedia.org/T289095 (10Aklapper) Could you please answer my previous comment? Also, I don't see any previous work on gadgets on Turkish Wikipedia by the author of this task, which confuses me. Could someone elaborate, please? [05:59:24] PROBLEM - PHD should be supervising processes on phab1001 is CRITICAL: PROCS CRITICAL: 2 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [06:01:16] RECOVERY - PHD should be supervising processes on phab1001 is OK: PROCS OK: 12 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [07:35:13] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.37-release: Expand the set of bundled extensions and skins in MediaWiki 1.37 - https://phabricator.wikimedia.org/T279842 (10Krabina) Is there a chance to bundle the chameleon skin? It is widely used in third-part... [08:18:41] hello! puppet on integration cloud vps projects seems to be again having troubles [08:29:52] 10Release-Engineering-Team (Radar), 10SRE, 10SRE-Access-Requests: Requesting exec access to pods in 'ci' namespace staging kubernetes - https://phabricator.wikimedia.org/T290360 (10akosiaris) a:03akosiaris [08:49:37] Is anyone aware of anyone ever looking at or thinking about https://backstage.io/ for Wikimedia? [08:49:44] I'm going to give it a brief look at and try out for wmde teams and software components and see how it feels [08:50:50] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10mwcli: mwcli: Automate upload of new version to releases server - https://phabricator.wikimedia.org/T290335 (10Addshore) So, do we actually think we need them to land on releases.wikimedia.org? Perhaps we would be fine with releases from gitlab itself? [08:59:30] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10mwcli: mwcli: Automate upload of new version to releases server - https://phabricator.wikimedia.org/T290335 (10kostajh) >>! In T290335#7347544, @Addshore wrote: > So, do we actually think we need them to land on releases.wikimedia.org? > Perhaps we would be fine w... [09:01:09] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10mwcli: mwcli: Automate upload of new version to releases server - https://phabricator.wikimedia.org/T290335 (10Addshore) That sounds like something I'd be interested in and also will aim to look at as one of the next things, as it would allow be to fully retire th... [09:30:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10Cloud-VPS, 10Patch-For-Review, 10cloud-services-team (Kanban): Support Cinder for CI docker workers - https://phabricator.wikimedia.org/T277078 (10hashar) https://gerrit.wikimedia.org/r/c/operations/puppet/+/670524 got applied... [09:33:52] !log Castor cache: nuked files that were last changed more than six months ago to free up disk space [09:33:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:45:44] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.37-release: Expand the set of bundled extensions and skins in MediaWiki 1.37 - https://phabricator.wikimedia.org/T279842 (10Aklapper) @Krabina: See discussion in T178349 instead [09:58:08] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Tracking-Neverending: Expand the set of bundled extensions and skins to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349 (10Krabina) Maybe this thread is to... [10:02:06] 10Deployments, 10Math, 10Mathoid, 10Platform Team Workboards (Platform Engineering Reliability): Deploy new Mathoid version to production - https://phabricator.wikimedia.org/T289674 (10Lucas_Werkmeister_WMDE) >>! In T289674#7324369, @hnowlan wrote: > Documentation has been updated - the process is relative... [10:24:13] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T281164 (10hashar) I am the one running the train this week :-] [10:31:10] 10Project-Admins, 10Turkish-Sites: trwiki Twinkle project - https://phabricator.wikimedia.org/T289095 (10Vikipolimer) Hi, I have just added this request to create a new work team for the twinkle gadget by the way @ TmY_e12 knows the details. Thanks. [10:38:48] 10Deployments, 10Math, 10Mathoid, 10Platform Team Workboards (Platform Engineering Reliability): Deploy new Mathoid version to production - https://phabricator.wikimedia.org/T289674 (10akosiaris) >>! In T289674#7324488, @Physikerwelt wrote: > @hnowlan This was extremely helpful. > > A side remark: I just... [10:52:59] (03CR) 10Hashar: [C: 03+2] Zuul: [mediawiki/extensions/BlockInactive] Enable CI [integration/config] - 10https://gerrit.wikimedia.org/r/720464 (owner: 10Zoranzoki21) [10:54:35] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/BlockInactive] Enable CI [integration/config] - 10https://gerrit.wikimedia.org/r/720464 (owner: 10Zoranzoki21) [10:55:29] (03CR) 10Hashar: "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/720464 (owner: 10Zoranzoki21) [11:35:35] 10Release-Engineering-Team, 10Voice & Tone: Rename backport & deployment windows - https://phabricator.wikimedia.org/T290859 (10kostajh) [11:38:21] PROBLEM - SSH on contint2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [11:49:52] 10Release-Engineering-Team, 10Voice & Tone: Rename backport & deployment windows - https://phabricator.wikimedia.org/T290859 (10Aklapper) Could stick to a timezone, like "UTC evening" etc? [12:04:37] 10Continuous-Integration-Infrastructure, 10DC-Ops, 10Infrastructure-Foundations, 10netops, 10serviceops: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ) - https://phabricator.wikimedia.org/T283582 (10hashar) That is still happening from time to time. Any person or team I can raise th... [12:36:56] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10mwcli: mwcli: Automate upload of new version to releases server - https://phabricator.wikimedia.org/T290335 (10kostajh) >>! In T290335#7347564, @Addshore wrote: > That sounds like something I'd be interested in and also will aim to look at as one of the next thing... [12:39:11] RECOVERY - SSH on contint2001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [12:49:58] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10mwcli: mwcli: Automate upload of new version to releases server - https://phabricator.wikimedia.org/T290335 (10Addshore) That last quote you use refers to public clouds though. The current runner for the mwcli is only used by mwcli, is part of the integration clou... [12:52:57] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10mwcli: mwcli: Automate upload of new version to releases server - https://phabricator.wikimedia.org/T290335 (10kostajh) >>! In T290335#7348175, @Addshore wrote: > That last quote you use refers to public clouds though. > The current runner for the mwcli is only us... [13:20:19] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Scap, 10Patch-For-Review: scap sync-file --force warns "sudo: no tty present and no askpass program specified" - https://phabricator.wikimedia.org/T290038 (10Joe) I merged this change but I just noticed that, this way, services are just restarted without depooli... [13:22:17] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Scap, 10Patch-For-Review: scap sync-file --force warns "sudo: no tty present and no askpass program specified" - https://phabricator.wikimedia.org/T290038 (10Joe) Nevermind, I'm clearly rusty; I just re-read scap's code and this seems allright :) [13:55:56] dancy: somehow ssh to the qemu VM works again :] It is blocked later on but that is a different matter [13:56:14] I hvaent changed anything, I would blame cosmic rays / files corruption of some sort [13:57:05] it is blocked further in the test suite that docker run an image inside the VM. Apparently due to slow disk IO or download but I haven't jumped into that rabbit hole https://integration.wikimedia.org/ci/job/fresh-test/171/console [13:57:08] we will see ;] [13:58:56] and https://gerrit.wikimedia.org/r/c/operations/puppet/+/720241 would let us get a newer nodejs version under bullseye (12.22.5 from bullseye security updates intead of 12.21.0 ). [14:48:17] hashar: Interesting. [14:48:42] I tried creating a VM in the deployment-prep project over the weekend to test it out but I couldn't figure out how to log into the VM after it was created. :-) [14:59:16] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10mwcli: mwcli: Automate upload of new version to releases server - https://phabricator.wikimedia.org/T290335 (10Addshore) >>! In T290335#7348195, @kostajh wrote: > What is the integration cloud project? https://openstack-browser.toolforge.org/project/integration [15:12:04] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Scap: scap sync-file --force warns "sudo: no tty present and no askpass program specified" - https://phabricator.wikimedia.org/T290038 (10dancy) 05Open→03Resolved [15:12:06] 10Release-Engineering-Team (Radar), 10Scap, 10Patch-For-Review, 10User-jijiki: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10dancy) [15:19:23] (03PS1) 10Ahmon Dancy: Clarify the need to roll back if canary checks fail [tools/scap] - 10https://gerrit.wikimedia.org/r/720775 (https://phabricator.wikimedia.org/T290037) [15:23:50] 10Release-Engineering-Team, 10Voice & Tone: Rename backport & deployment windows - https://phabricator.wikimedia.org/T290859 (10thcipriani) >>! In T290859#7347998, @Aklapper wrote: > Could stick to a timezone, like "UTC evening" etc? +1 not let the perfect be the enemy of the good enough: * UTC Morning *... [15:27:54] hashar: twentyafterfour before I forget to mention it. hashar could you not do group0 tomorrow and twentyafterfour can you do group0 tomorrow? The DC switchover is happening this week and we need to do train at a time that won't interfere. Sorry for short notice, calendar already reflects this change. hashar twentyafterfour will this work for you both? [15:29:20] (03CR) 10Thcipriani: [C: 03+2] Clarify the need to roll back if canary checks fail [tools/scap] - 10https://gerrit.wikimedia.org/r/720775 (https://phabricator.wikimedia.org/T290037) (owner: 10Ahmon Dancy) [15:30:46] (03Merged) 10jenkins-bot: Clarify the need to roll back if canary checks fail [tools/scap] - 10https://gerrit.wikimedia.org/r/720775 (https://phabricator.wikimedia.org/T290037) (owner: 10Ahmon Dancy) [15:32:08] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Scap, 10Patch-For-Review: Scap should be clearer about the need for a revert after a failed canary check - https://phabricator.wikimedia.org/T290037 (10dancy) 05Open→03Resolved a:03dancy Marking this as resolved although the change will not be realized unt... [15:37:05] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T281164 (10thcipriani) a:05hashar→03mmodell Flipping conductor and backup for group0 deployment. Let's swap back after group0. Group0 would normally happen this week... [15:39:24] 10Continuous-Integration-Infrastructure, 10Browser-Tests, 10Patch-For-Review, 10User-Addshore, 10User-zeljkofilipin: Centrally look for flakey browser tests - https://phabricator.wikimedia.org/T277205 (10Addshore) [15:42:31] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10SRE Observability, 10Wikimedia-Logstash, 10observability, 10User-brennen: Experiment with automating error log triage - https://phabricator.wikimedia.org/T290267 (10thcipriani) >>! In T290267#7347221, @lmata wrote: > @brennen I've sent an invite with propose... [15:44:05] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Observability-Logging, 10User-brennen: Experiment with automating error log triage - https://phabricator.wikimedia.org/T290267 (10lmata) thanks, @thcipriani, invite updated. [15:45:34] (03PS1) 10Dduvall: TEST: verifying pipelinelib validation works [blubber] - 10https://gerrit.wikimedia.org/r/720780 [15:46:33] 10Release-Engineering-Team (Radar), 10Quality-and-Test-Engineering-Team (QTE), 10serviceops-radar, 10CommRel-Specialists-Support (Jul-Sep-2021), and 2 others: Expand the list of group 1 wikis to contain at least one (preferably 2) smaller "top ten size" wikis - https://phabricator.wikimedia.org/T286664 (10X... [15:46:51] 10Continuous-Integration-Infrastructure, 10Browser-Tests, 10Patch-For-Review, 10User-Addshore, 10User-zeljkofilipin: Centrally look for flakey browser tests - https://phabricator.wikimedia.org/T277205 (10Addshore) [15:47:29] (03PS1) 10Ahmon Dancy: jjb: set DEBEMAIL in beta-build-scap-deb job [integration/config] - 10https://gerrit.wikimedia.org/r/720781 (https://phabricator.wikimedia.org/T273896) [15:48:09] greg-g: hello! you need to grant ircservserv access to this channel if you want it to manage it, see https://meta.wikimedia.org/wiki/IRC/Bots/ircservserv#Setup_a_new_channel for instructions [15:48:56] (03CR) 10Ahmon Dancy: "This change is already live." [integration/config] - 10https://gerrit.wikimedia.org/r/720781 (https://phabricator.wikimedia.org/T273896) (owner: 10Ahmon Dancy) [16:00:28] 10Continuous-Integration-Infrastructure, 10Browser-Tests, 10Patch-For-Review, 10User-Addshore, 10User-zeljkofilipin: Centrally look for flakey browser tests - https://phabricator.wikimedia.org/T277205 (10Addshore) I collected data again today, and realized that some patterns still existed that I was expe... [16:00:31] (03CR) 10Dduvall: "I verified that the ajv output would show in a real-world scenario by running a recent blubber patch with invalid .pipeline/config.yaml ag" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719382 (https://phabricator.wikimedia.org/T225335) (owner: 10Dduvall) [16:06:01] majavah: ack, thanks! [16:06:30] ok to merge the ircservserv patch now? [16:16:03] (03PS1) 10Jforrester: Zuul: Add Hjfocs to the CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/720784 [16:17:20] 10Continuous-Integration-Infrastructure, 10Browser-Tests, 10Patch-For-Review, 10User-Addshore, 10User-zeljkofilipin: Centrally look for flakey browser tests - https://phabricator.wikimedia.org/T277205 (10Addshore) With the new command we get a much shorter list: - 13 ✖ can save with tag - 12 ✖ sav... [16:17:29] (03CR) 10Jforrester: [C: 03+2] Zuul: Add Hjfocs to the CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/720784 (owner: 10Jforrester) [16:18:23] 10Continuous-Integration-Infrastructure, 10Browser-Tests, 10Patch-For-Review, 10User-Addshore, 10User-zeljkofilipin: Centrally look for flakey browser tests - https://phabricator.wikimedia.org/T277205 (10Addshore) [16:18:58] (03Merged) 10jenkins-bot: Zuul: Add Hjfocs to the CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/720784 (owner: 10Jforrester) [16:26:24] greg-g: if you're ready, I can merge the config patch and deploy it [16:26:31] after that you still need to run !issync to apply the changes [16:50:37] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Tracking-Neverending: Expand the set of bundled extensions and skins to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349 (10Tgr) @Krabina 1.37 is branched t... [16:58:44] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Tracking-Neverending: Expand the set of bundled extensions and skins to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349 (10Jdforrester-WMF) 05Open→03Dec... [16:59:12] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Tracking-Neverending: Expand the set of bundled extensions and skins to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349 (10Jdforrester-WMF) 05Declined→0... [16:59:24] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Tracking-Neverending: Expand the set of bundled extensions and skins to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349 (10Jdforrester-WMF) [16:59:57] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.37-release: Expand the set of bundled extensions and skins in MediaWiki 1.37 - https://phabricator.wikimedia.org/T279842 (10Jdforrester-WMF) 05Open→03Declined We're a few months late to be adding things now.... [17:11:18] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Tracking-Neverending: Expand the set of bundled extensions and skins to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349 (10Jdforrester-WMF) (But I'd encour... [17:12:28] 10Phabricator: Evaluate adding "In progress" status to Phabricator. - https://phabricator.wikimedia.org/T288956 (10greg) 👍 from me. [17:16:25] (03CR) 10Thcipriani: [C: 03+2] "Let's get this in before the next branch cut tonight" [tools/release] - 10https://gerrit.wikimedia.org/r/720359 (https://phabricator.wikimedia.org/T290759) (owner: 10Jforrester) [17:17:49] dancy: thx for the qemu debugging last friday. What I suspect is that we might had TWO vm running and listening on :22 [17:18:04] or something weird. Cause I tried again earlier this afternoon and there was no trouble whatsoever [17:18:09] so hmm it is a mystery [17:18:28] the job I was trying to get fixed (for fresh.git) is still broken but that is apparently due to nodejs 12 [17:18:33] so I guess it is {completed} [17:18:56] (03CR) 10Reedy: Stop branching VipsScaler for Wikimedia production (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/720359 (https://phabricator.wikimedia.org/T290759) (owner: 10Jforrester) [17:20:10] (03CR) 10Jforrester: Stop branching VipsScaler for Wikimedia production (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/720359 (https://phabricator.wikimedia.org/T290759) (owner: 10Jforrester) [17:24:51] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T281164 (10hashar) @Daniel and I have a weekly check in on Tuesday morning so we will be able to talk about the change to page protection. I guess we might catch a few low... [17:25:11] twentyafterfour: about the train, well there are a couple risky ones mentioned [17:25:34] notably page protection that got factored out of the `Title` class. I have a meeting with Daniel Kinzler tomorrow already [17:25:45] and shellbox, I guess it is straightforward [17:25:46] also [17:26:02] I'll definitely be around in case the shellbox one goes wrong :) [17:26:04] also last minute is that I cant' run the train at the usual european time [17:26:32] cause there is a DC switchother scheduled there so it is overlapping with the usual train window. But I will sync that with SRE tomorrow and see what I can push [17:26:42] my .plan is to at least all the prep work [17:27:17] then either I flip version just after the dc switch other, or do it during the usual noon PST window [17:27:19] * bd808 always smiles when hashar mentions a .plan [17:27:24] ;D [17:28:10] legoktm: I guess shellbox/pygments being broken is easy to verify and will easily standout in the log [17:28:31] the page protection one / Title refactoring, who knows what kind of side effects it might have [17:28:36] yep, also very easy to test :) [17:28:40] yeah [17:28:47] and we know about those risks [17:28:51] so I will check with Daniel tomorrow [17:29:24] twentyafterfour: that was my tldr for this week :-] [17:29:25] hashar: twentyafterfour: as a general note, CentralAuth has had a ton of internal changes since the last train, no single particularly risky patch but the extension is sometimes fragile [17:29:51] ahhh [17:29:56] joy [17:30:05] "risky extension" [17:30:18] exactly :D [17:30:36] 10Phabricator: Evaluate adding "In progress" status to Phabricator. - https://phabricator.wikimedia.org/T288956 (10kchapman) I think this is an excellent addition. As teams attempt to reduce Work in Progress (WIP) it would be great to have a status that allows them to see what is currently being worked on. [17:31:09] do you prefer that in phab too or is this enough? [17:31:18] +1 on phab if you don't mind [17:31:46] doesn't not necessarily to be five pages long. Just a poke that stuff got changed in centralauth and eventually a few contacts will surely help if something goes bad [17:33:47] I am off for dinner [17:37:58] (03Abandoned) 10Dduvall: WIP: Encapsulate auto-stage and rsync within multi-stage Dockerfile [tools/release] - 10https://gerrit.wikimedia.org/r/711019 (owner: 10Dduvall) [17:38:06] done [17:38:22] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T281164 (10Majavah) ##### Risky Extension! 🚂🔥 * **Extension**: CentralAuth * **Summary**: ** CentralAuth is in charge of user authentication and parts of authorization.... [17:39:15] (03Abandoned) 10Dduvall: TEST: verifying pipelinelib validation works [blubber] - 10https://gerrit.wikimedia.org/r/720780 (owner: 10Dduvall) [17:44:20] 10Release-Engineering-Team, 10Voice & Tone: Rename backport & deployment windows - https://phabricator.wikimedia.org/T290859 (10kostajh) >>! In T290859#7348669, @thcipriani wrote: > > >>>! In T290859#7347998, @Aklapper wrote: >> Could stick to a timezone, like "UTC evening" etc? > > +1 not let the perfect... [18:01:02] hashar: can you summarize why we would not allow a verified-contributor-only test to run outside docker or sub-VM? I'd like to re-think our solution on https://phabricator.wikimedia.org/T250808 since testing Docker-tools (Fresh) inside docker, inside a VM, inside another VM, inside WMCS seems like a lot, and I don't want to maintain that by myself. [18:01:20] its also quite slow given no docker cache inside the VM since it's a new VM each time [18:02:13] it seems like maybe it would be okay to run directly on those VM agents, assuming proper firewall and worst case scenario we un-verify the bad actor and re-create the agent in a few clicks. [18:06:01] thcipriani: Did you want me to rush out the switching off of VipsScaler? [18:13:11] 10Gerrit: Gerrit Reviewer Bot is unreliable - https://phabricator.wikimedia.org/T290905 (10Tgr) [18:38:02] 10Gerrit: Gerrit Reviewer Bot is unreliable - https://phabricator.wikimedia.org/T290905 (10Jdforrester-WMF) I've been wanting to kill off the terrible bot for ages but people bewilderingly seem to prefer it to the built-in feature. Of course, with the move to GitLab it'll need substantial re-working or retiremen... [18:50:28] 10Beta-Cluster-Infrastructure: Temporary interface admin right to beta commons for testing UploadWizard CSS gadget - https://phabricator.wikimedia.org/T290124 (10Multichill) 05Open→03Resolved a:03Multichill Added you to the group, see https://commons.wikimedia.beta.wmflabs.org/wiki/Special:ListUsers?userna... [18:52:47] 10Release-Engineering-Team (Radar), 10MW-on-K8s, 10SRE, 10serviceops: The restricted/mediawiki-webserver image should include skins and resources - https://phabricator.wikimedia.org/T285232 (10dancy) Still not working: https://foundation.wikimedia.org/static/current/skins/Timeless/resources/print.css [18:58:54] James_F: nope, wasn't my intention, noticed the patch with a +1 without noticing the depends-on: gerrit saves me from my trigger-happy nature [19:00:08] (03PS12) 10Dduvall: Perform validation using JSON schema and ajv-cli [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719382 (https://phabricator.wikimedia.org/T225335) [19:00:20] (03CR) 10Thcipriani: Stop branching VipsScaler for Wikimedia production (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/720359 (https://phabricator.wikimedia.org/T290759) (owner: 10Jforrester) [19:06:17] jeena: thanks for the quick +2s on those pipeline config fixes. once those merge, i think https://gerrit.wikimedia.org/r/c/integration/pipelinelib/+/719382 should be good to merge as well (cc dancy) [19:06:34] Am I remembering correctly that deprecation warnings are blocking the next weeks train? [19:08:40] zabe, that's the current policy: they're not user-facing breakage, so when we see a deprecation in the wild it blocks the next train [19:09:32] I was asking because of T290871 and T290909 [19:09:32] T290909: CirrusSearch is using hard deprecated Title::getBacklinkCache - https://phabricator.wikimedia.org/T290909 [19:09:32] T290871: Deprecated: Use of Title::getBacklinkCache was deprecated in MediaWiki 1.37. [Called from DeleteAction::tempConfirmDelete in /var/www/html/core/includes/actions/DeleteAction.php at line 230] - https://phabricator.wikimedia.org/T290871 [19:09:44] thcipriani: thanks [19:22:28] majavah: thank you for the CentralAuth notice for this week train :] [19:24:25] Krinkle: I guess the rules of thumb are: not executing anything on the agent but have everything inside container. There are a few other use cases for having test suite running containers, and I would rather not build a solution on top of the legacy ci (jenkins/zuul/wmcs agents) [19:24:50] so I guess we should actively look at how gitlab can fulfill that. Namely have a gitlab runner in a VM that has Docker up and running [19:25:49] and get said vm to be accelerated with /dev/kvm on the host + find out whether the vm can come prepopulated with the docker image/ use a cache [19:26:09] I am not sure what went wrong with the qemu vm last friday though [19:26:37] and the new agent with bullseye has some troubles cause WMCS now relies on cinder for storage (what ever it can be) so there are some puppet adjustements required [19:27:04] also found out that the node12-* images based on SRE Bullseye image are lacking the bullseye image and hence don't have the latest nodejs 12 provided by debian [19:27:21] but I have no idea whether that new node would fix the utf8 buffer issue :-\ [19:29:32] Krinkle: re the docker testing, addshore got it working for our repo in gitlab recently https://gitlab.wikimedia.org/releng/cli [19:29:58] readme at https://gitlab.wikimedia.org/addshore/test for how it is tied together right now [19:30:04] not sure if its the best way, but it is a way [19:30:50] do we have a fleet of baremetal / prod vm to run gitlab-runners on? [19:31:12] (I confess I have to catch up on the gitlab infra side) [19:31:38] this one is in a vm [19:31:44] I'm not sure, but maybe dduvall know something? I think he was working on some sort of gitlab runner thing with brenne.n [19:32:15] hashar: I assumed that the idea was to re-use integration for the GitLab runners, seeing that addshore's was there. [19:32:27] If we're using something else, that'd be fine of course. [19:32:38] I think there is some other project already for gitlab runners? [19:32:55] but I 1 runner was created randomly by me before that was setup [19:33:07] https://openstack-browser.toolforge.org/project/gitlab-runners [19:33:40] Aha. Can we kill your test runner in integration then? It's using up our buffer of CPUs so I had to decom 1001 to build 1021. [19:33:51] I'd rather not use WMCS, that adds a whole layer of complication. But maybe it has been decided otherwise [19:34:08] T285913 implied we decided to use WMCS. [19:34:09] T285913: Request creation of gitlab-runners VPS project - https://phabricator.wikimedia.org/T285913 [19:34:42] James_F: I sure can if I am told where to run it, i dont currently have access to the gitlab-runners project etc [19:34:58] Neither do I. [19:34:59] Also not sure if that would be the right place for this 1 runner, as this 1 runner exists only for this 1 project (mwcli) right now [19:35:23] Ack. [19:35:39] I wonder if we are using both of the wemu instances on integration though? [19:36:00] mwcli used to use the qmeu hosts for CI, but now it is on the gitlab runner perhaps we dont need so much smeu space there? [19:36:28] K.rinkle was talking about killing those and running such things direct on contint* earlier. [19:37:55] 10Gerrit: Gerrit Reviewer Bot is unreliable - https://phabricator.wikimedia.org/T290905 (10Tgr) Do people prefer it because of the regexp support? In theory Gerrit's built-in notifications can do that (and more) but practice I rarely manage to get it to work. (Also I think there's value in having a publicly vis... [19:39:17] ^ thcipriani greg-g: ircservserv is now configured for this channel, going to leave the initial sync (!issync) to you [19:41:02] majavah: <3 [19:41:11] !issync [19:41:13] Error: You don't have permission to update channel settings [19:41:16] bah [19:41:18] :D [19:41:35] 10Gerrit: Gerrit Reviewer Bot is unreliable - https://phabricator.wikimedia.org/T290905 (10Legoktm) The bot's issue tracker is https://github.com/valhallasw/gerrit-reviewer-bot/issues - I don't see anyone having reported any issues recently? [19:41:51] ^ greg-g I may need your assistance [19:42:28] only a +F (founder) can use the sync command [19:42:35] correct, you need chanserv +F for it [19:45:49] 10Release-Engineering-Team, 10User-jeena: Organize a deployment training for k8s - https://phabricator.wikimedia.org/T290915 (10jeena) [19:46:51] jeena: gitlab-runners is the wmcs project for runners. there's a puppet profile for provisioning and a handful of vms atm [19:48:13] https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner [19:49:47] if people have a need for it, they can use that same profile to start runners in there own projects [19:50:27] *their [19:50:50] * dduvall shamelessly blames android keyboard [19:51:24] James_F: addshore: ^ the source of trust :] [19:51:47] *looks at the profile* [19:51:54] + the ticket James referred to which was brennen requesting the creation of the gitlab-runner wmcs project [19:52:15] What is setup within this new "new shared runner" [19:52:27] as part of the profile I mean.... [19:53:23] Aha, https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner#Future_Gitlab_Runner_setup_(T286958) [19:53:24] T286958: Document long-term requirements for GitLab job runners - https://phabricator.wikimedia.org/T286958 [19:53:39] * James_F subscribes. [19:54:15] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Tracking-Neverending: Expand the set of bundled extensions and skins to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349 (10Aklapper) 05Open→03Invalid C... [19:57:41] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release: Branch REL1_37 for MediaWiki and all extensions and skins - https://phabricator.wikimedia.org/T289586 (10Jdforrester-WMF) a:03Jdforrester-WMF [19:59:24] 10Phabricator: Evaluate adding "In progress" status to Phabricator. - https://phabricator.wikimedia.org/T288956 (10Aklapper) Thanks everyone for the feedback, appreciated! I //guess// next would be someone cooking up a patch to update `maniphest.statuses` in https://gerrit.wikimedia.org/g/operations/puppet/+/ref... [20:07:12] (03CR) 10Ahmon Dancy: [C: 03+2] Perform validation using JSON schema and ajv-cli [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719382 (https://phabricator.wikimedia.org/T225335) (owner: 10Dduvall) [20:07:52] (03Merged) 10jenkins-bot: Perform validation using JSON schema and ajv-cli [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719382 (https://phabricator.wikimedia.org/T225335) (owner: 10Dduvall) [20:08:40] 10Phabricator, 10Patch-For-Review: Evaluate adding "In progress" status to Phabricator. - https://phabricator.wikimedia.org/T288956 (10Jdforrester-WMF) >>! In T288956#7349849, @Aklapper wrote: > Thanks everyone for the feedback, appreciated! I //guess// next would be someone cooking up a patch to update `manip... [20:15:51] 10Release-Engineering-Team, 10MediaWiki-extensions-VipsScaler, 10Structured Data Engineering, 10Structured-Data-Backlog, and 2 others: Undeploy VipsScaler from Wikimedia wikis - https://phabricator.wikimedia.org/T290759 (10Legoktm) [20:15:57] 10Release-Engineering-Team, 10MediaWiki-extensions-VipsScaler, 10Structured Data Engineering, 10Structured-Data-Backlog, and 2 others: Undeploy VipsScaler from Wikimedia wikis - https://phabricator.wikimedia.org/T290759 (10Legoktm) >>! In T290759#7346446, @Krinkle wrote: >>>! In T199938, @Krinkle wrote: >>... [20:18:48] 10Release-Engineering-Team, 10MediaWiki-extensions-VipsScaler, 10Structured Data Engineering, 10Structured-Data-Backlog, and 2 others: Undeploy VipsScaler from Wikimedia wikis - https://phabricator.wikimedia.org/T290759 (10Jdforrester-WMF) One this is done should we revert https://gerrit.wikimedia.org/r/c/... [20:23:14] 10Release-Engineering-Team, 10MediaWiki-extensions-VipsScaler, 10Structured Data Engineering, 10Structured-Data-Backlog, and 2 others: Undeploy VipsScaler from Wikimedia wikis - https://phabricator.wikimedia.org/T290759 (10Legoktm) >>! In T290759#7349879, @Jdforrester-WMF wrote: > One this is done should w... [20:27:59] 10Release-Engineering-Team, 10MediaWiki-extensions-VipsScaler, 10Structured Data Engineering, 10Structured-Data-Backlog, and 2 others: Undeploy VipsScaler from Wikimedia wikis - https://phabricator.wikimedia.org/T290759 (10Jdforrester-WMF) >>! In T290759#7349899, @Legoktm wrote: >>>! In T290759#7349879, @J... [20:30:21] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10User-jeena (Sprint 8/18-8/31): Create WMF CI image for Node.js 14 - https://phabricator.wikimedia.org/T267888 (10jeena) a:05jeena→03None [20:31:27] [operations/mediawiki-config@master] Undeploy VipsScaler: I \u2013 Disable on all wikis [20:31:39] James_F: nice to see proper use of en-dash :P [20:32:02] too bad the Phab integration still encodes it doubly so though [20:34:30] 10Release-Engineering-Team, 10MediaWiki-extensions-VipsScaler, 10Structured Data Engineering, 10Structured-Data-Backlog, and 2 others: Undeploy VipsScaler from Wikimedia wikis - https://phabricator.wikimedia.org/T290759 (10Krinkle) >>! In T290759#7349872, @Legoktm wrote: >>>! In T290759#7346446, @Krinkle w... [20:36:03] Krinkle: Indeed. [20:36:27] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.37-release: Expand the set of bundled extensions and skins in MediaWiki 1.37 - https://phabricator.wikimedia.org/T279842 (10Tgr) >>! In T279842#7349142, @Jdforrester-WMF wrote: > We're a few months late to be add... [20:38:49] 10Gerrit: Gerrit Reviewer Bot is unreliable - https://phabricator.wikimedia.org/T290905 (10Tgr) I'd assume it's more of an infrastructure issue than an issue with the bot code as I haven't seen any pattern in when the bot does/doesn't work. Are there any logs that can be checked for how a specific Gerrit changes... [20:40:11] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.37-release: Expand the set of bundled extensions and skins in MediaWiki 1.37 - https://phabricator.wikimedia.org/T279842 (10Jdforrester-WMF) >>! In T279842#7349944, @Tgr wrote: >>>! In T279842#7349142, @Jdforrest... [20:44:19] 10Deployments, 10SRE, 10Stashbot: [[wikitech:Server_admin_log]] should not rely on IRC for logmsgbot entries - https://phabricator.wikimedia.org/T46791 (10Legoktm) [20:55:09] 10Deployments, 10SRE, 10Stashbot: [[wikitech:Server_admin_log]] should not rely on IRC for logmsgbot entries - https://phabricator.wikimedia.org/T46791 (10Jdforrester-WMF) [20:55:23] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.37-release: Expand the set of bundled extensions and skins in MediaWiki 1.37 - https://phabricator.wikimedia.org/T279842 (10Tgr) >>! In T279842#7349965, @Jdforrester-WMF wrote: > I would say at bare minimum the a... [21:00:49] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.37-release: Expand the set of bundled extensions and skins in MediaWiki 1.37 - https://phabricator.wikimedia.org/T279842 (10Jdforrester-WMF) >>! In T279842#7350004, @Tgr wrote: >>>! In T279842#7349965, @Jdforrest... [21:05:44] addshore: the profile is fairly sparse. it installs docker and the gitlab-runner package, and executes the registration command [21:08:21] Looking at the role one of the differences between that and how my runner for mwcli was setup is docker-privileged [21:08:50] And thats because I followed https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#docker-in-docker-with-tls-enabled [21:10:09] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Release Pipeline, 10Patch-For-Review: Implement pipeline config validation - https://phabricator.wikimedia.org/T225335 (10dduvall) 05Open→03Resolved [21:10:43] oh yeah, we don't want docker-privileged on shared runners [21:11:48] image building is a whole other part of the new platform to tackle but the tl;dr is that we very likely will not support building from raw Dockerfiles [21:13:18] obliquely relevant is https://phabricator.wikimedia.org/T287211 [21:14:31] and https://phabricator.wikimedia.org/T290339 [21:48:15] 10Phabricator, 10Patch-For-Review: Evaluate adding "In progress" status to Phabricator. - https://phabricator.wikimedia.org/T288956 (10mmodell) >>! In T288956#7349858, @Jdforrester-WMF wrote: > I imagine we'd want @gerritbot to also state-change tasks into "In Progress" when a Patch-To-Review tag gets added?... [22:11:17] 10Release-Engineering-Team (Radar), 10MW-on-K8s, 10SRE, 10serviceops, 10Patch-For-Review: The restricted/mediawiki-webserver image should include skins and resources - https://phabricator.wikimedia.org/T285232 (10dancy) Can someone give me an example of a curl command that exercises the /w/static.php cod... [22:21:03] 10Phabricator, 10Patch-For-Review: Evaluate adding "In progress" status to Phabricator. - https://phabricator.wikimedia.org/T288956 (10mmodell) What icon should we use? These are the ones that jumped out at me: * {icon tachometer} tachometer * {icon thumb-tack} thumb-tack * {icon play-circle} play-circle * {... [22:22:25] 10Phabricator, 10Patch-For-Review: Evaluate adding "In progress" status to Phabricator. - https://phabricator.wikimedia.org/T288956 (10mmodell) Oh I hadn't noticed https://gerrit.wikimedia.org/r/720811... That is unfortunately out of date / obsolete. Phabricator gets that setting from the database: `lang=json... [22:25:30] 10Phabricator, 10Patch-For-Review: Evaluate adding "In progress" status to Phabricator. - https://phabricator.wikimedia.org/T288956 (10lmata) FWIW this would help with reporting on my end to better track work in progress. Apologies for commenting late to the party [22:39:29] 10Release-Engineering-Team (Seen), 10MediaWiki-Release-Tools: Branch script doesn't respect `branchpoint` when creating submodules, resulting in "implicit merges" error from gerrit - https://phabricator.wikimedia.org/T279718 (10thcipriani) > The do_core_work method runs git('checkout', '-B', branch) against (i... [23:00:22] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10User-jeena (Sprint 8/18-8/31): Create WMF CI image for Node.js 14 - https://phabricator.wikimedia.org/T267888 (10thcipriani) I see [[ https://docker-registry.wikimedia.org/releng/node14/tags/ | t... [23:01:38] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10User-jeena (Sprint 8/18-8/31): Create WMF CI image for Node.js 14 - https://phabricator.wikimedia.org/T267888 (10jeena) [23:07:17] 10Release-Engineering-Team, 10User-jeena: Organize a deployment training for k8s - https://phabricator.wikimedia.org/T290915 (10thcipriani) [23:30:09] 10Phabricator, 10Patch-For-Review: Evaluate adding "In progress" status to Phabricator. - https://phabricator.wikimedia.org/T288956 (10Jdforrester-WMF) >>! In T288956#7350247, @mmodell wrote: > Oh I hadn't noticed https://gerrit.wikimedia.org/r/720811... That is unfortunately out of date / obsolete. Phabricato... [23:35:50] 10Release-Engineering-Team (Seen), 10MediaWiki-Release-Tools: Branch script doesn't respect `branchpoint` when creating submodules, resulting in "implicit merges" error from gerrit - https://phabricator.wikimedia.org/T279718 (10Jdforrester-WMF) >>! In T279718#7350329, @thcipriani wrote: >> The do_core_work met... [23:37:08] (03PS1) 10Thcipriani: fix: accuracy of deployment window names [tools/release] - 10https://gerrit.wikimedia.org/r/720826 (https://phabricator.wikimedia.org/T290859) [23:38:02] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Voice & Tone: Rename backport & deployment windows - https://phabricator.wikimedia.org/T290859 (10thcipriani) a:03thcipriani [23:38:33] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Voice & Tone: Rename backport & deployment windows - https://phabricator.wikimedia.org/T290859 (10thcipriani) p:05Triage→03Medium Example output: https://wikitech.wikimedia.org/wiki/User:Thcipriani/Deployments [23:44:20] 10Release-Engineering-Team (Seen), 10MediaWiki-Release-Tools: Branch script doesn't respect `branchpoint` when creating submodules, resulting in "implicit merges" error from gerrit - https://phabricator.wikimedia.org/T279718 (10thcipriani) >>! In T279718#7350422, @Jdforrester-WMF wrote: >>>! In T279718#7350329... [23:46:09] (03CR) 10Jforrester: [C: 03+2] "…" [tools/release] - 10https://gerrit.wikimedia.org/r/720359 (https://phabricator.wikimedia.org/T290759) (owner: 10Jforrester) [23:47:07] (03Merged) 10jenkins-bot: Stop branching VipsScaler for Wikimedia production [tools/release] - 10https://gerrit.wikimedia.org/r/720359 (https://phabricator.wikimedia.org/T290759) (owner: 10Jforrester) [23:47:16] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10User-jeena (Sprint 8/18-8/31): Create WMF CI image for Node.js 14 - https://phabricator.wikimedia.org/T267888 (10thcipriani) >>! In T267888#7350353, @thcipriani wrote: > I see [[ https://docker-r... [23:49:43] !issync [23:49:44] Syncing #wikimedia-releng (requested by greg-g) [23:49:46] Set /cs flags #wikimedia-releng litharge +o [23:49:47] Set /cs flags #wikimedia-releng jeena +Vv [23:49:49] Set /cs flags #wikimedia-releng hashar +Vv [23:49:51] Set /cs flags #wikimedia-releng wmopbot +o [23:49:53] Set /cs flags #wikimedia-releng thcipriani +Afiortv [23:49:55] Set /cs flags #wikimedia-releng legoktm +FRefrs [23:49:57] Set /cs flags #wikimedia-releng brennen +Afiortv [23:49:59] Set /cs flags #wikimedia-releng twentyafterfour +Vv [23:50:01] Set /cs flags #wikimedia-releng Az1568 -AFRefiorstv [23:50:03] Set /cs flags #wikimedia-releng dancy +Vv [23:50:05] Set /cs flags #wikimedia-releng *!*@libera/staff/* +o [23:50:07] Set /cs flags #wikimedia-releng dduvall +Vv [23:50:09] Set /mode #wikimedia-releng +b $j:#wikimedia-bans [23:50:14] sorry for the pings, all [23:50:14] 10Release-Engineering-Team (Seen), 10MediaWiki-Release-Tools: Branch script doesn't respect `branchpoint` when creating submodules, resulting in "implicit merges" error from gerrit - https://phabricator.wikimedia.org/T279718 (10Jdforrester-WMF) >>! In T279718#7350475, @thcipriani wrote: >>>! In T279718#7350422... [23:50:28] yay pings [23:51:01] thanks greg-g [23:51:23] 10Release-Engineering-Team, 10MediaWiki-extensions-VipsScaler, 10Structured Data Engineering, 10Structured-Data-Backlog, and 2 others: Undeploy VipsScaler from Wikimedia wikis - https://phabricator.wikimedia.org/T290759 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF [23:51:37] I assume litharge is from "libera_staff = true" ? [23:52:07] no: [litharge] (litharge@libera/bot/litharge): #libera-bots / https://tinyurl.com/chantracker [23:52:54] 10Release-Engineering-Team (Seen), 10MediaWiki-Release-Tools: Branch script doesn't respect `branchpoint` when creating submodules, resulting in "implicit merges" error from gerrit - https://phabricator.wikimedia.org/T279718 (10thcipriani) >>! In T279718#7350519, @Jdforrester-WMF wrote: >>>! In T279718#7350475...