[00:00:04] When do we get a sneak peak of the results of the developer survey then? :P [00:15:41] addshore: T321331 could use feedback from you. :) [00:15:42] T321331: Grant TheresNoTime membership in the Huggle Cloud VPS project - https://phabricator.wikimedia.org/T321331 [00:17:56] addshore: oh, and T329405 too [00:17:57] T329405: Grant Phuzion membership in the Huggle Cloud VPS project - https://phabricator.wikimedia.org/T329405 [00:52:55] 10Release-Engineering-Team, 10serviceops-collab: switch releases.wikimedia.org from eqiad to codfw - https://phabricator.wikimedia.org/T330960 (10Dzahn) [00:53:45] 10Release-Engineering-Team, 10serviceops-collab, 10Datacenter-Switchover: switch releases.wikimedia.org from eqiad to codfw - https://phabricator.wikimedia.org/T330960 (10Dzahn) [00:54:25] 10Release-Engineering-Team, 10serviceops-collab, 10Datacenter-Switchover: switch releases.wikimedia.org from eqiad to codfw - https://phabricator.wikimedia.org/T330960 (10Dzahn) [releases2002:~] $ host releases.discovery.wmnet releases.discovery.wmnet is an alias for releases1002.eqiad.wmnet. [01:02:49] 10Release-Engineering-Team, 10serviceops-collab, 10Datacenter-Switchover: switch releases.wikimedia.org from eqiad to codfw - https://phabricator.wikimedia.org/T330960 (10Dzahn) [03:10:54] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10serviceops-collab, 10serviceops-radar: Set up mirror of the docker hub registry for gitlab-runners - https://phabricator.wikimedia.org/T329679 (10dancy) [07:16:54] (03CR) 10Kosta Harlan: "Should be good for wider rollout, thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/893530 (owner: 10Hashar) [07:30:10] 10Beta-Cluster-Infrastructure, 10Chinese-Sites: Request for sysop right on zh.wikipedia.beta.wmflabs - https://phabricator.wikimedia.org/T330976 (10Shizhao) [08:07:54] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10serviceops-collab, 10serviceops-radar: Set up mirror of the docker hub registry for gitlab-runners - https://phabricator.wikimedia.org/T329679 (10Jelto) One of the WMCS Shared Runners `runner-29` has a dedicated disk for... [08:43:08] (03CR) 10Hashar: [C: 03+1] Allow more BuildKit frontend image names (032 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/893028 (https://phabricator.wikimedia.org/T329553) (owner: 10Lucas Werkmeister (WMDE)) [09:00:31] 10Project-Admins: Create project tag for EImage - https://phabricator.wikimedia.org/T330919 (10Aklapper) 05Open→03Stalled Hi @Want, thanks for taking the time to report this! https://www.mediawiki.org/wiki/Extension:EImage currently says it's archived. https://www.mediawiki.org/w/index.php?oldid=4058612 poin... [09:06:44] (03PS1) 10Hashar: (DO NOT SUBMIT) try JJB 4.0.0 [integration/config] - 10https://gerrit.wikimedia.org/r/893667 [09:12:34] 10Continuous-Integration-Infrastructure, 10RelEng-Archive-FY201718-Q1, 10SRE, 10Patch-For-Review, 10User-Joe: CI for operations/puppet is taking too long - https://phabricator.wikimedia.org/T166888 (10hashar) [09:14:34] Project mediawiki-core-doxygen-docker build #41435: 04FAILURE in 10 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/41435/ [09:26:04] 10Continuous-Integration-Config, 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10SRE: Fix operations/puppet.git "rebase hell" - https://phabricator.wikimedia.org/T224033 (10hashar) For future reference, the configuration change in Gerrit is under `refs/meta... [09:38:22] <_joe_> good mornig, https://phabricator.wikimedia.org/T330942 seems like UBN! to me [09:40:47] _joe_: lets flag it as such so :] [09:42:49] <_joe_> yeah I didn't want to make you all work on this without discussion :) [09:45:08] 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T325588 (10hashar) [09:47:00] hi there, the issue was already happening on the Russian wiki (group1) yesterday [09:47:10] so wmf.25 is probably not the main suspect [09:47:19] sorry, group2 [09:54:34] 10Project-Admins: Create project tag for EImage - https://phabricator.wikimedia.org/T330919 (10Want) git clone https://www.thewoodcraft.org/pub/wiki/eimage.git I have not final repository for now. Change for MW version 1.35 was only cosmetic and for MW 1.39 too. Now I do more changes, because original version u... [09:55:14] <_joe_> jnuche: actually the image is from commons [09:56:18] ah, true that [10:11:59] 10Beta-Cluster-Infrastructure, 10Chinese-Sites: Request for sysop right on zh.wikipedia.beta.wmflabs - https://phabricator.wikimedia.org/T330976 (10Zabe) 05Open→03Resolved a:03Zabe https://zh.wikipedia.beta.wmflabs.org/w/index.php?title=Special:%E6%97%A5%E5%BF%97&logid=9932 [10:12:21] Yippee, build fixed! [10:12:22] Project mediawiki-core-doxygen-docker build #41436: 09FIXED in 8 min 2 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/41436/ [11:05:03] 10Phabricator: Reset Phabricator 2FA for matthiasmullie - https://phabricator.wikimedia.org/T330073 (10Aklapper) 05Open→03Resolved a:03Aklapper Sorry, this fell through the cracks. Should be done now (verified via a Slack conversation) [11:06:39] 10Phabricator: Reset Phabricator 2FA for matthiasmullie - https://phabricator.wikimedia.org/T330073 (10matthiasmullie) Thanks a lot! [11:10:38] 10Scap: Improve behavior around global Scal lock + communicate changes - https://phabricator.wikimedia.org/T330756 (10jnuche) [11:20:44] 10Project-Admins: Create project tag for EImage - https://phabricator.wikimedia.org/T330919 (10Want) >>! In T330919#8659679, @Aklapper wrote: > Hi @Want, thanks for taking the time to report this! https://www.mediawiki.org/wiki/Extension:EImage currently says it's archived. https://www.mediawiki.org/w/index.php?... [12:13:10] (03PS2) 10Lucas Werkmeister (WMDE): Allow more BuildKit frontend image names [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/893028 (https://phabricator.wikimedia.org/T329553) [12:13:22] (03CR) 10Lucas Werkmeister (WMDE): Allow more BuildKit frontend image names (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/893028 (https://phabricator.wikimedia.org/T329553) (owner: 10Lucas Werkmeister (WMDE)) [12:13:35] !log delete deploymentwiki # T198673 [12:13:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:13:41] T198673: Remove deployment.wikimedia.beta.wmflabs.org wiki (deploymentwiki) - https://phabricator.wikimedia.org/T198673 [12:20:58] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Seen), 10Patch-For-Review, 10Technical-Debt, 10User-DannyS712: Remove deployment.wikimedia.beta.wmflabs.org wiki (deploymentwiki) - https://phabricator.wikimedia.org/T198673 (10Zabe) 05Open→03Resolved [12:21:00] 10Beta-Cluster-Infrastructure, 10SRE, 10Technical-Debt, 10Tracking-Neverending: Minimize infrastructure differences between Beta Cluster and production - https://phabricator.wikimedia.org/T87220 (10Zabe) [12:39:16] !log wikidata/query-builder/deploy.git: changed HEAD from master to production # T330752 [12:39:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:39:21] T330752: Remove master branch from wikidata/query-builder/deploy.git - https://phabricator.wikimedia.org/T330752 [12:39:51] * Luca is apparently using the wrong IRC nick [12:41:10] !log wikidata/query-builder/deploy.git: deleted master branch (was ed287d58a1e9a2ea39f2eb5bbba31d8fc54269c2 ) # T330752 [12:41:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:42:07] 10Gerrit, 10Wikidata, 10Wikidata Query Builder, 10wdwb-tech, 10wmde-wikidata-tech: Remove master branch from wikidata/query-builder/deploy.git - https://phabricator.wikimedia.org/T330752 (10Lucas_Werkmeister_WMDE) 05Open→03Resolved a:03Lucas_Werkmeister_WMDE >>! In T330752#8660288, @Stashbot wrote:... [12:46:02] (03PS1) 10Hashar: jjb: update Quibble jobs to 1.5.1 [integration/config] - 10https://gerrit.wikimedia.org/r/893715 (https://phabricator.wikimedia.org/T330750) [12:46:18] (03CR) 10Hashar: [C: 03+2] jjb: update Quibble jobs to 1.5.1 [integration/config] - 10https://gerrit.wikimedia.org/r/893715 (https://phabricator.wikimedia.org/T330750) (owner: 10Hashar) [12:46:52] 10Project-Admins: Create project tag for EImage - https://phabricator.wikimedia.org/T330919 (10Aklapper) > I have not final repository for now. Do you plan to [host it in Wikimedia Gerrit](https://www.mediawiki.org/wiki/Gerrit/New_repositories) or [host it in Wikimedia GitLab](https://www.mediawiki.org/wiki/Gi... [12:47:30] (03Merged) 10jenkins-bot: jjb: update Quibble jobs to 1.5.1 [integration/config] - 10https://gerrit.wikimedia.org/r/893715 (https://phabricator.wikimedia.org/T330750) (owner: 10Hashar) [12:48:26] 10Gerrit, 10Upstream: OpenSSH 9 scp does not work with Gerrit due to lack of sftp subsystem - https://phabricator.wikimedia.org/T330740 (10hashar) [12:49:48] 10Scap, 10MW-on-K8s, 10serviceops, 10Upstream: Kubernetes configuration file is group-readable - https://phabricator.wikimedia.org/T329899 (10Clement_Goubert) p:05Triage→03Low [13:01:01] (03PS1) 10Hashar: Merge tag 'v3.5.5' into wmf/stable-3.5 [software/gerrit] (wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/893723 (https://phabricator.wikimedia.org/T330663) [13:05:00] Project beta-update-databases-eqiad build #65516: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65516/ [13:05:51] (03PS2) 10Hashar: Merge tag 'v3.5.5' into wmf/stable-3.5 [software/gerrit] (wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/893723 (https://phabricator.wikimedia.org/T330663) [13:18:02] (03CR) 10Hashar: [C: 03+2] Merge tag 'v3.5.5' into wmf/stable-3.5 [software/gerrit] (wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/893723 (https://phabricator.wikimedia.org/T330663) (owner: 10Hashar) [13:28:34] (03Merged) 10jenkins-bot: Merge tag 'v3.5.5' into wmf/stable-3.5 [software/gerrit] (wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/893723 (https://phabricator.wikimedia.org/T330663) (owner: 10Hashar) [13:35:44] maintenance-disconnect-full-disks build 470043 integration-agent-docker-1033 (/: 29%, /srv: 97%, /var/lib/docker: 58%): OFFLINE due to disk space [13:38:39] (Queue (Jenkins jobs + Zuul functions) alert) firing: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [13:40:47] maintenance-disconnect-full-disks build 470044 integration-agent-docker-1033 (/: 29%, /srv: 43%, /var/lib/docker: 54%): RECOVERY disk space OK [13:41:50] 10Project-Admins: Create project tag for EImage - https://phabricator.wikimedia.org/T330919 (10Want) Gerrit. AccessControl (maintained by me) has repo in Gerrit too. At least I will work with it more often and reduce the likelihood of forgetting how to do what again. ;-) For it my request, create new project EIm... [13:51:58] (03PS1) 10Hashar: Update Gerrit to v3.5.5 and rebuild plugins [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/893743 (https://phabricator.wikimedia.org/T330663) [13:52:35] (03CR) 10CI reject: [V: 04-1] Update Gerrit to v3.5.5 and rebuild plugins [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/893743 (https://phabricator.wikimedia.org/T330663) (owner: 10Hashar) [13:52:59] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [13:53:00] Project beta-update-databases-eqiad build #65517: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65517/ [13:54:55] (03CR) 10Hashar: "recheck" [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/893743 (https://phabricator.wikimedia.org/T330663) (owner: 10Hashar) [13:57:17] (03CR) 10Hashar: "recheck" [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/893743 (https://phabricator.wikimedia.org/T330663) (owner: 10Hashar) [13:58:39] (Queue (Jenkins jobs + Zuul functions) alert) resolved: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [14:00:13] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [14:00:37] !log gerrit: removed old replication tasks from /var/lib/gerrit2/review_site/data/replication/ref-updates/waiting (most from August 5th 2022) [14:00:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:02:52] is it me, or are edits not saving on the beta cluster.. [14:08:12] 10Beta-Cluster-Infrastructure: Edits not saving on beta cluster - https://phabricator.wikimedia.org/T331019 (10TheresNoTime) [14:09:27] 10Beta-Cluster-Infrastructure: Edits not saving on beta cluster - https://phabricator.wikimedia.org/T331019 (10TheresNoTime) Huh, was also trying the above logged out (editing https://en.wikipedia.beta.wmflabs.org/w/index.php?title=Sandbox&action=edit) and got an edit conflict — the "conflicting edit" is not dis... [14:20:28] 10Continuous-Integration-Config: diffConfig broken due to git stash failure - https://phabricator.wikimedia.org/T331020 (10Lucas_Werkmeister_WMDE) [14:22:14] 10Continuous-Integration-Config: diffConfig broken due to git stash failure - https://phabricator.wikimedia.org/T331020 (10Lucas_Werkmeister_WMDE) CC @Krinkle and @Ladsgroup from [build: Change diffConfig to use git-stash instead of git-add](https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/893088)... [14:24:53] 10Continuous-Integration-Config: diffConfig broken due to git stash failure - https://phabricator.wikimedia.org/T331020 (10Krinkle) a:03Krinkle [14:24:56] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: Edits not saving on beta cluster - https://phabricator.wikimedia.org/T331019 (10TheresNoTime) p:05Triage→03Unbreak! //UBN!// given this interrupts testing (nb. wonder why more CI isn't complaining...) [14:32:12] (03CR) 10Jforrester: "In practical terms, the only changes are:" [integration/config] - 10https://gerrit.wikimedia.org/r/893667 (owner: 10Hashar) [14:32:35] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: Edits not saving on beta cluster - https://phabricator.wikimedia.org/T331019 (10TheresNoTime) Consistent `Wikimedia\Rdbms\LoadMonitor::computeServerStates: host deployment-db12 is not replicating?` / `Wikimedia\Rdbms\LoadMonitor::computeServerStates: h... [14:38:01] Project beta-update-databases-eqiad build #65518: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65518/ [14:43:21] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: Edits not saving on beta cluster - https://phabricator.wikimedia.org/T331019 (10TheresNoTime) ` Mar 02 12:13:09 deployment-db11 mysqld[14965]: 2023-03-02 12:13:09 27 [ERROR] Incorrect definition of table mysql.proc: expected column 'definer' at positio... [14:47:08] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10DBA: Edits not saving on beta cluster - https://phabricator.wikimedia.org/T331019 (10TheresNoTime) Tagging #dba as I'm afraid my familiarity with repairing //whatever happened there// is zero — any assistance or suggestions would be greatly appreci... [14:49:13] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10DBA: Edits not saving on beta cluster (db replication error, corrupted table) - https://phabricator.wikimedia.org/T331019 (10TheresNoTime) [14:50:17] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: Edits not saving on beta cluster (db replication error, corrupted table) - https://phabricator.wikimedia.org/T331019 (10Ladsgroup) Sorry, DBAs don't maintain beta cluster DBs, I'm currently busy with this UBN T330942 otherwise I would have helped [14:50:43] (03CR) 10Hashar: (DO NOT SUBMIT) try JJB 4.0.0 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/893667 (owner: 10Hashar) [14:51:18] (03PS2) 10Hashar: Upgrade to JJB from 3.12.0 to 4.0.0 [integration/config] - 10https://gerrit.wikimedia.org/r/893667 [14:52:53] 10Project-Admins: Create project tag for EImage - https://phabricator.wikimedia.org/T330919 (10Aklapper) 05Stalled→03Resolved Oh, that sounds all great! Thanks! Requested public project #eimage has been created: https://phabricator.wikimedia.org/project/view/6421/ (In case you need to edit the project or pr... [15:04:11] James_F: naming things is hard, to list the versions of nodejs/python/php for the JJB jobs, I thought about the name [[Continuous integration/Language Versions Policy]] [15:04:53] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) >>! In T329931#8652138, @Jelto wrote: > The rsync jobs between production host and replica are only created but not... [15:12:42] (03PS1) 10Hashar: Upgrade JJB from 4.0.0 to 4.1.0 [integration/config] - 10https://gerrit.wikimedia.org/r/893764 [15:14:26] (03PS1) 10Hashar: Upgrade JJB from 4.1.0 to 4.2.0 [integration/config] - 10https://gerrit.wikimedia.org/r/893765 [15:14:28] (03PS1) 10Hashar: Upgrade JJB from 4.2.0 to 4.3.0 [integration/config] - 10https://gerrit.wikimedia.org/r/893786 [15:23:02] Project beta-update-databases-eqiad build #65519: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65519/ [15:41:31] Project beta-update-databases-eqiad build #65520: 15ABORTED in 18 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65520/ [15:42:45] hashar: sorry for the ping — I think you have jenkins access to pause the timer for https://integration.wikimedia.org/ci/view/Beta/job/beta-update-databases-eqiad/ pending T331019 being resolved [15:42:46] T331019: Edits not saving on beta cluster (db replication error, corrupted table) - https://phabricator.wikimedia.org/T331019 [15:43:21] TheresNoTime: like disable the job entirely right? [15:43:28] yeah please [15:45:21] checking :] [15:45:43] thank you [15:47:32] (03PS1) 10Hashar: jjb: disable beta-update-databases-eqiad [integration/config] - 10https://gerrit.wikimedia.org/r/893791 (https://phabricator.wikimedia.org/T331019) [15:48:30] TheresNoTime: ^ if you can +1 one that change. I have deployed it and the job shows as disabled now https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/ [15:48:45] so later one can revert that patch then run `./jjb-update beta-update-databases-eqiad` ;] [15:48:52] (03CR) 10Samtar: [C: 03+1] "thx :)" [integration/config] - 10https://gerrit.wikimedia.org/r/893791 (https://phabricator.wikimedia.org/T331019) (owner: 10Hashar) [15:49:05] (03CR) 10Hashar: [C: 03+2] "You are welcome!" [integration/config] - 10https://gerrit.wikimedia.org/r/893791 (https://phabricator.wikimedia.org/T331019) (owner: 10Hashar) [15:49:44] now I just need to talk someone into recovering the DBs :) [15:50:02] hashar: Sounds fine. [15:50:16] (03Merged) 10jenkins-bot: jjb: disable beta-update-databases-eqiad [integration/config] - 10https://gerrit.wikimedia.org/r/893791 (https://phabricator.wikimedia.org/T331019) (owner: 10Hashar) [15:50:26] hashar: We can always move it later: https://www.mediawiki.org/wiki/Continuous_integration/Language_Versions_Policy [15:51:13] very true [16:07:25] damn table wikitext [16:07:27] I screwed it up again [16:11:23] James_F: https://www.mediawiki.org/wiki/Continuous_integration/Language_Versions_Policy [16:11:31] we still have container based on stretch :-\ [16:13:13] Boo Java. [16:13:53] !log failover deployment-prep master from deployment-db09 to deployment-db11 # T331019 [16:13:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:13:59] T331019: Edits not saving on beta cluster (db replication error, corrupted table) - https://phabricator.wikimedia.org/T331019 [16:20:17] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: Edits not saving on beta cluster (db replication error, corrupted table) - https://phabricator.wikimedia.org/T331019 (10Zabe) 05Open→03Resolved a:03Zabe [16:22:04] (03PS1) 10Samtar: Revert "jjb: disable beta-update-databases-eqiad" [integration/config] - 10https://gerrit.wikimedia.org/r/893773 [16:22:06] James_F: I am pretty sure we can get java 8 from Buster via a port we had to do at wmf [16:22:15] Eww. [16:22:25] But yeah, that's probably better than getting teams to migrate to Java 11. [16:22:48] (03CR) 10Hashar: [C: 03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/893773 (owner: 10Samtar) [16:22:54] Or, gasp, Java 17. [16:23:08] TheresNoTime: deployed [16:23:22] thank you hashar :) (and thank you zabe!) [16:23:48] dockerfiles/typos/Dockerfile.template:FROM {{ "ci-stretch" | image_tag }} [16:23:48] :D [16:24:03] (03Merged) 10jenkins-bot: Revert "jjb: disable beta-update-databases-eqiad" [integration/config] - 10https://gerrit.wikimedia.org/r/893773 (owner: 10Samtar) [16:24:35] !log shutoff deployment-db09 # T331019 [16:24:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:24:41] T331019: Edits not saving on beta cluster (db replication error, corrupted table) - https://phabricator.wikimedia.org/T331019 [16:28:26] yw, no idea what happened to deployment-db09, but tbh I don't really care. Maybe it was also corrupted curing the cloud incedent [16:29:58] !log create deployment-db13 as g3.cores8.ram16.disk20 # T331019 [16:30:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:30:03] T331019: Edits not saving on beta cluster (db replication error, corrupted table) - https://phabricator.wikimedia.org/T331019 [16:35:03] !log create volume db13 and attach to deployment-db13 # T331019 [16:35:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:35:08] T331019: Edits not saving on beta cluster (db replication error, corrupted table) - https://phabricator.wikimedia.org/T331019 [16:45:58] 10Release-Engineering-Team, 10Gerrit (Gerrit 3.5), 10Patch-For-Review: Upgrade Gerrit from 3.5.4 to 3.5.5 - https://phabricator.wikimedia.org/T330663 (10hashar) a:03hashar I have [[ https://wikitech.wikimedia.org/w/index.php?title=Deployments&diff=prev&oldid=2057803 | scheduled it ]] for Monday March 6th a... [16:46:15] 10Release-Engineering-Team, 10Gerrit (Gerrit 3.5), 10Patch-For-Review: Upgrade Gerrit from 3.5.4 to 3.5.5 - https://phabricator.wikimedia.org/T330663 (10hashar) [16:58:36] 10Gerrit, 10Release-Engineering-Team (Seen): Can't `git pull` mediawiki/core from Gerrit: "fatal: the remote end hung up unexpectedly" SSH_MSG_CHANNEL_WINDOW_ADJUST - https://phabricator.wikimedia.org/T263293 (10hashar) I do not see any occurence of `SSH_MSG_CHANNEL_WINDOW_ADJUST` or `unassigned` in the Gerrit... [17:00:09] 10Gerrit, 10Release-Engineering-Team (Seen): Can't `git pull` mediawiki/core from Gerrit: "fatal: the remote end hung up unexpectedly" SSH_MSG_CHANNEL_WINDOW_ADJUST - https://phabricator.wikimedia.org/T263293 (10hashar) 05Open→03Resolved a:03hashar Marking solved as far as we are concerned. Upstream will... [18:20:18] 10Phabricator, 10Release-Engineering-Team (Blocking 🧱), 10Security-Team, 10User-AKlapper, 10user-sbassett: Establish a workflow that scales for requesting Phab 2FA resets - https://phabricator.wikimedia.org/T306708 (10sbassett) >>! In T306708#8657402, @Aklapper wrote: > @sbassett: Thanks. Any updates? Th... [20:42:26] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity: Capture output from failed command and transmit to earlywarningbot - https://phabricator.wikimedia.org/T331061 (10kostajh) [20:43:37] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity: Capture output from failed command and transmit to earlywarningbot - https://phabricator.wikimedia.org/T331061 (10kostajh) [21:22:40] (03PS2) 10JHathaway: helm-linter: add semver-cli [integration/config] - 10https://gerrit.wikimedia.org/r/893074 (https://phabricator.wikimedia.org/T320554) [21:29:14] (03CR) 10Jforrester: [C: 03+1] Upgrade to JJB from 3.12.0 to 4.0.0 [integration/config] - 10https://gerrit.wikimedia.org/r/893667 (owner: 10Hashar) [21:35:05] (03CR) 10Jforrester: [C: 03+1] "No-op in practice for us?" [integration/config] - 10https://gerrit.wikimedia.org/r/893764 (owner: 10Hashar) [21:37:17] (03CR) 10Jforrester: "This has warnings of" [integration/config] - 10https://gerrit.wikimedia.org/r/893765 (owner: 10Hashar) [21:46:03] 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T325588 (10Jdforrester-WMF) [21:54:42] !log install mariadb 10.6 via role::mariadb::beta on deployment-db12 # T331019 [21:54:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:54:47] T331019: Edits not saving on beta cluster (db replication error, corrupted table) - https://phabricator.wikimedia.org/T331019 [22:01:40] !log enable read-only mode and create dump of all databases # T331019 [22:01:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:01:45] T331019: Edits not saving on beta cluster (db replication error, corrupted table) - https://phabricator.wikimedia.org/T331019 [22:20:07] Project beta-update-databases-eqiad build #65527: 04FAILURE in 6.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65527/ [22:47:37] Project beta-update-databases-eqiad build #65528: 04STILL FAILING in 2.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65528/ [22:48:02] (03PS1) 10Jeena Huneidi: dockerfiles/release-notes: releases branch rename [integration/config] - 10https://gerrit.wikimedia.org/r/893827 (https://phabricator.wikimedia.org/T329770) [23:03:59] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Patch-For-Review, 10User-brennen: Rename mainline branch from "master" to "main" in GitLab:repos/releng/release - https://phabricator.wikimedia.org/T329770 (10jeena) [23:20:02] Project beta-update-databases-eqiad build #65529: 04STILL FAILING in 2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65529/