[00:09:35] 10Release-Engineering-Team, 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: switch releases.wikimedia.org from eqiad to codfw - https://phabricator.wikimedia.org/T330960 (10Dzahn) I confirmed the files in /srv/org/wikimedia/releases/ (the document root) are identical on both backends and... [00:10:27] I am switching releases.wikimedia.org from eqiad to codfw. ticket is https://phabricator.wikimedia.org/T330960 and I will send an email shortly to all users in "releasers-*" admin classes. [00:11:00] files on both sides confirmed to be identical (constant rsync is automatic, but switching directions) [00:20:03] Project beta-update-databases-eqiad build #65530: 04STILL FAILING in 2.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65530/ [00:37:21] !log take deployment-prep out of read-only # T331019 [00:37:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:37:27] T331019: Edits not saving on beta cluster (db replication error, corrupted table) - https://phabricator.wikimedia.org/T331019 [00:48:35] Yippee, build fixed! [00:48:36] Project beta-update-databases-eqiad build #65531: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65531/ [02:04:36] 10Release-Engineering-Team, 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: switch releases.wikimedia.org from eqiad to codfw - https://phabricator.wikimedia.org/T330960 (10Dzahn) This is switched! I am sending an email to all members in admin groups on releases* servers. I am just keepi... [02:26:52] 10Continuous-Integration-Config, 10Patch-For-Review: diffConfig broken due to git stash failure - https://phabricator.wikimedia.org/T331020 (10Krinkle) 05Openβ†’03Resolved p:05Triageβ†’03High [03:44:28] (03CR) 10Krinkle: [C: 03+1] jjb: Bump almost all Quibble jobs to 1.1.0 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/726880 (https://phabricator.wikimedia.org/T291549) (owner: 10Kosta Harlan) [03:54:50] (03CR) 10Krinkle: Docker: [node16] Switch npm from pinned 7.21.0 to upstream 8.19.3+ (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/891649 (owner: 10Jforrester) [05:20:37] (03CR) 10Hashar: "docker-pkg requires an update to the `changelog` file in order to notice it has to be build a new image :] I will amend and build then upd" [integration/config] - 10https://gerrit.wikimedia.org/r/893827 (https://phabricator.wikimedia.org/T329770) (owner: 10Jeena Huneidi) [05:20:49] (03PS2) 10Hashar: dockerfiles/release-notes: releases branch rename [integration/config] - 10https://gerrit.wikimedia.org/r/893827 (https://phabricator.wikimedia.org/T329770) (owner: 10Jeena Huneidi) [05:20:59] (03CR) 10Hashar: [C: 03+2] dockerfiles/release-notes: releases branch rename [integration/config] - 10https://gerrit.wikimedia.org/r/893827 (https://phabricator.wikimedia.org/T329770) (owner: 10Jeena Huneidi) [05:22:07] (03Merged) 10jenkins-bot: dockerfiles/release-notes: releases branch rename [integration/config] - 10https://gerrit.wikimedia.org/r/893827 (https://phabricator.wikimedia.org/T329770) (owner: 10Jeena Huneidi) [05:26:22] (03PS1) 10Hashar: jjb: update for releases repo branch rename [integration/config] - 10https://gerrit.wikimedia.org/r/893844 (https://phabricator.wikimedia.org/T329770) [05:38:50] (03CR) 10Hashar: [C: 03+2] "Successfully published image docker-registry.discovery.wmnet/releng/release-notes:0.0.9" [integration/config] - 10https://gerrit.wikimedia.org/r/893827 (https://phabricator.wikimedia.org/T329770) (owner: 10Jeena Huneidi) [05:40:50] (03CR) 10Hashar: [C: 03+2] "Given the main branch exist:" [integration/config] - 10https://gerrit.wikimedia.org/r/893844 (https://phabricator.wikimedia.org/T329770) (owner: 10Hashar) [05:41:57] (03Merged) 10jenkins-bot: jjb: update for releases repo branch rename [integration/config] - 10https://gerrit.wikimedia.org/r/893844 (https://phabricator.wikimedia.org/T329770) (owner: 10Hashar) [07:23:58] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity: Capture output from failed command and transmit to earlywarningbot - https://phabricator.wikimedia.org/T331061 (10kostajh) Thinking about it some more, there's another way that doesn't involve any changes to Quibble. The bot can ge... [07:34:40] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: switch releases.wikimedia.org from eqiad to codfw - https://phabricator.wikimedia.org/T330960 (10hashar) From my mail earlier today, that broke [[ https://releases-jenkin... [07:51:56] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity: Capture output from failed command and transmit to earlywarningbot - https://phabricator.wikimedia.org/T331061 (10hashar) I would like to investigate having Gerrit to fetch the output from the Jenkins build and insert the result to... [08:56:30] !log Updating Jenkins jobs with JJB 4.0.0 | https://gerrit.wikimedia.org/r/c/integration/config/+/893667/ [08:56:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:56:50] (03CR) 10Hashar: [C: 03+2] "Lets try it, I have refreshed all jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/893667 (owner: 10Hashar) [08:57:15] 10Release-Engineering-Team (GitLab V: Event Horizon πŸŒ„), 10Scap, 10Patch-For-Review: Avoid local venv when distributing Scap - https://phabricator.wikimedia.org/T329622 (10jnuche) 05In progressβ†’03Resolved [08:57:59] (03CR) 10Hashar: "Yup it is a noop which is a good sign. I have split the updates in small chunks so it is easier to catch potential issue and still have s" [integration/config] - 10https://gerrit.wikimedia.org/r/893764 (owner: 10Hashar) [08:59:17] (03Merged) 10jenkins-bot: Upgrade to JJB from 3.12.0 to 4.0.0 [integration/config] - 10https://gerrit.wikimedia.org/r/893667 (owner: 10Hashar) [09:04:08] 10Release-Engineering-Team (Deployment Autopilot πŸ›©οΈ), 10Scap, 10Infrastructure-Foundations, 10serviceops, 10Patch-For-Review: Use scap to deploy itself to scap targets - https://phabricator.wikimedia.org/T303559 (10jnuche) [09:04:21] 10Release-Engineering-Team (GitLab V: Event Horizon πŸŒ„), 10Scap, 10SRE: Scap fails on debian bullseye targets - https://phabricator.wikimedia.org/T326668 (10jnuche) 05Openβ†’03Resolved Bullseye hosts now get specific Python wheels created on our base bullseye image. [09:15:45] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity, 10Patch-For-Review: Capture output from failed command and transmit to earlywarningbot - https://phabricator.wikimedia.org/T331061 (10kostajh) >>! In T331061#8662796, @hashar wrote: > I would like to investigate having Gerrit to... [09:16:29] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity, 10Patch-For-Review: Capture output from failed command and transmit to earlywarningbot - https://phabricator.wikimedia.org/T331061 (10kostajh) >>! In T331061#8662882, @gerritbot wrote: > Change 893994 had a related patch set uplo... [09:16:35] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity, 10Patch-For-Review: Capture output from failed command and transmit to earlywarningbot - https://phabricator.wikimedia.org/T331061 (10hashar) Well that can still be explored :-] [09:41:28] 10Continuous-Integration-Config, 10affects-translatewiki.net: jenkins.job=mediawiki-i18n-check-docker fails without a real error - https://phabricator.wikimedia.org/T331098 (10Raymond) [09:47:12] GitLab needs a short maintenance break in around 30 minutes. [09:53:16] 10Beta-Cluster-Infrastructure, 10Chinese-Sites: Request for sysop right on zh.wikipedia.beta.wmflabs - https://phabricator.wikimedia.org/T330976 (10Shizhao) @Zabe I also must interface admin right. Thx! [09:54:42] 10Beta-Cluster-Infrastructure, 10Chinese-Sites: Request for sysop right on zh.wikipedia.beta.wmflabs - https://phabricator.wikimedia.org/T330976 (10Zabe) Done [10:23:30] jelto: needs to comply with german arbeitszeitgesetz? :-P [10:25:40] jayme: :D haha yes.. btw the break should be less than 10 minutes, so not fully compliant [10:31:08] done [10:59:30] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10Browser-Tests, 10User-zeljkofilipin: Jenkins job should fail when all (selenium) tests are skipped - https://phabricator.wikimedia.org/T324480 (10zeljkofilipin) It's for selenium tests in general. It has happened at least once (see [[ https://gerri... [14:57:42] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: switch releases.wikimedia.org from eqiad to codfw - https://phabricator.wikimedia.org/T330960 (10hashar) https://releases-jenkins.wikimedia.org/ is back on releases1002 w... [15:12:35] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: switch releases.wikimedia.org from eqiad to codfw - https://phabricator.wikimedia.org/T330960 (10Dzahn) 05Openβ†’03Stalled [15:12:54] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: switch releases.wikimedia.org from eqiad to codfw - https://phabricator.wikimedia.org/T330960 (10Dzahn) a:03Dzahn [15:18:45] (03PS3) 10JHathaway: helm-linter: add semver-cli [integration/config] - 10https://gerrit.wikimedia.org/r/893074 (https://phabricator.wikimedia.org/T320554) [15:32:11] (03PS4) 10JHathaway: helm-linter: add semver-cli [integration/config] - 10https://gerrit.wikimedia.org/r/893074 (https://phabricator.wikimedia.org/T320554) [15:45:03] does anyone with +2 on integration/config have a moment to look at this patch, https://gerrit.wikimedia.org/r/c/integration/config/+/893074? [15:56:07] 10phabricator maintenance bot: Maintenance bot creates duplicates for new wiki tasks - https://phabricator.wikimedia.org/T286740 (10JJMC89) [16:13:52] (03CR) 10Lucas Werkmeister (WMDE): "FYI, Michael figured out how to update All The Thingsβ„’ in Termbox without breaking `npm dedupe` (see I1b6ea2f4f5), so we no longer have an" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/893028 (https://phabricator.wikimedia.org/T329553) (owner: 10Lucas Werkmeister (WMDE)) [16:18:57] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: switch releases.wikimedia.org from eqiad to codfw - https://phabricator.wikimedia.org/T330960 (10Dzahn) >>! In T330960#8662750, @hashar wrote: > From my mail earlier toda... [16:20:57] 10phabricator maintenance bot: Maintenance bot creates duplicates for new wiki tasks - https://phabricator.wikimedia.org/T286740 (10JJMC89) [17:52:04] (03PS1) 10Hashar: wm-checks-api: support the Early Warning bot [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/894099 (https://phabricator.wikimedia.org/T330850) [17:55:03] 10Gerrit, 10Patch-For-Review: [wm-checks-api] support EarlyWarningBot - https://phabricator.wikimedia.org/T330850 (10hashar) When browsing the Check UI that looks like: {F36890482 size=full} And clicking to uncollapse the message gives: {F36890480 size=full} Which shows the whole message. [18:09:53] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: switch releases.wikimedia.org from eqiad to codfw - https://phabricator.wikimedia.org/T330960 (10Dzahn) ^ merging the change above and then reverting it again (after real... [18:15:09] 10GitLab (Integrations), 10Phabricator-Bot-Requests, 10Release-Engineering-Team (GitLab V: Event Horizon πŸŒ„), 10User-brennen: Create CodeReviewBot for use by GitLab-Phabricator integrations - https://phabricator.wikimedia.org/T330923 (10brennen) 05Openβ†’03Resolved @CodeReviewBot exists now. [18:15:13] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (GitLab V: Event Horizon πŸŒ„): GitLab comments should come from a CodeReviewBot instead of gerritbot - https://phabricator.wikimedia.org/T327424 (10brennen) [18:45:15] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (GitLab III: GitLab in LA πŸͺƒ), 10User-brennen: Sandbox task for gitlab-phabricator comment integration - https://phabricator.wikimedia.org/T324164 (10CodeReviewBot) brennen closed https://gitlab.wikimedia.org/brennen/test/-/merge_requests/6... [18:49:16] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (GitLab V: Event Horizon πŸŒ„): GitLab comments should come from a CodeReviewBot instead of gerritbot - https://phabricator.wikimedia.org/T327424 (10brennen) [19:11:07] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (GitLab V: Event Horizon πŸŒ„): GitLab comments should come from a CodeReviewBot instead of gerritbot - https://phabricator.wikimedia.org/T327424 (10brennen) I confirmed the new CodeReviewBot user works here - T324164#8664674 - but haven't perma... [19:41:37] 10Release-Engineering-Team (GitLab V: Event Horizon πŸŒ„), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T325588 (10Jdforrester-WMF) 05Openβ†’03Resolved [19:52:36] !log deployment-db13: import dump into mariadb # T331019 [19:52:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:52:41] T331019: Edits not saving on beta cluster (db replication error, corrupted table) - https://phabricator.wikimedia.org/T331019 [19:52:56] !log deployment-db13: start replication from deployment-db11 # T331019 [19:53:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:24:38] 10Gerrit, 10serviceops-collab, 10Patch-For-Review: Issues with Gerrit test instance - https://phabricator.wikimedia.org/T329444 (10Dzahn) 05Openβ†’03Resolved @Ameisenigel Sorry for the delay. Let me get back to this. So, I think we can break this down into multiple parts: - instance was down due to some u... [20:29:53] 10Phabricator, 10Release-Engineering-Team, 10Diffusion-Repository-Administrators, 10serviceops-collab: Audit Diffusion-Repository-Administrators group membership and rights - https://phabricator.wikimedia.org/T324171 (10Dzahn) Well, we have reduced the number of admins but we have also heard from some of t... [20:47:36] (03CR) 10Jeena Huneidi: "Thanks Antoine!" [integration/config] - 10https://gerrit.wikimedia.org/r/893844 (https://phabricator.wikimedia.org/T329770) (owner: 10Hashar) [22:16:07] 10phabricator maintenance bot: Maintenance bot creates duplicates for new wiki tasks - https://phabricator.wikimedia.org/T286740 (10JJMC89) [23:23:15] 10Continuous-Integration-Config, 10Patch-For-Review: Upgrade all CI jobs for WMF-deployed projects from Node 14 to Node 16 - https://phabricator.wikimedia.org/T314470 (10Jdforrester-WMF) [23:23:37] 10Continuous-Integration-Config: Upgrade all CI jobs for WMF-deployed projects from Node 16 to Node 18 - https://phabricator.wikimedia.org/T331180 (10Jdforrester-WMF) [23:24:29] 10Continuous-Integration-Infrastructure: Create WMF CI images and jobs for Node.js 18 - https://phabricator.wikimedia.org/T331181 (10Jdforrester-WMF) [23:24:42] 10Continuous-Integration-Infrastructure: Create WMF CI images and jobs for Node.js 18 - https://phabricator.wikimedia.org/T331181 (10Jdforrester-WMF) 05Openβ†’03Stalled a:03Jdforrester-WMF [23:24:44] 10Continuous-Integration-Config: Upgrade all CI jobs for WMF-deployed projects from Node 16 to Node 18 - https://phabricator.wikimedia.org/T331180 (10Jdforrester-WMF) [23:25:32] 10Continuous-Integration-Config, 10Patch-For-Review: Upgrade all CI jobs for WMF-deployed projects from Node 14 to Node 16 - https://phabricator.wikimedia.org/T314470 (10Jdforrester-WMF) [23:35:52] (03PS3) 10Jforrester: Docker: [node16] Switch npm from pinned 7.21.0 to upstream 8.19.3+ [integration/config] - 10https://gerrit.wikimedia.org/r/891649 [23:35:54] (03PS1) 10Jforrester: [DNM] Docker: [quibble] Switch from node14 to node16 [integration/config] - 10https://gerrit.wikimedia.org/r/894125 [23:35:56] (03PS1) 10Jforrester: [DNM] Docker: Provide node18 images [integration/config] - 10https://gerrit.wikimedia.org/r/894126 (https://phabricator.wikimedia.org/T331181) [23:37:14] (03CR) 10CI reject: [V: 04-1] [DNM] Docker: Provide node18 images [integration/config] - 10https://gerrit.wikimedia.org/r/894126 (https://phabricator.wikimedia.org/T331181) (owner: 10Jforrester)