[02:25:37] 10Continuous-Integration-Infrastructure, 10PHP 8.2 support, 10Patch-For-Review: Create PHP 8.2 CI images and jobs for early testing - https://phabricator.wikimedia.org/T314093 (10Legoktm) Thanks James, you're awesome. Debian Bookworm is currently planning to release with 8.2 in June 2023ish, so this is a pri... [07:51:34] 10Deployments, 10Pywikibot, 10Pywikibot-tests, 10Regression, and 2 others: Retrieving embeddedin pages fails with internal_api_error_DBQueryError on testwiki since yesterday - https://phabricator.wikimedia.org/T314112 (10Xqt) [09:23:59] 10GitLab (Project Migration), 10Release-Engineering-Team (Doing), 10User-brennen, 10Voice & Tone: Rename mainline Git branch from "master" to "main" on all WMF-hosted repositories during GitLab migration - https://phabricator.wikimedia.org/T281593 (10Tacsipacsi) Are projects currently stored (and not just... [10:53:13] 10Deployments, 10Pywikibot, 10Pywikibot-tests, 10Regression, and 2 others: Retrieving embeddedin pages fails with internal_api_error_DBQueryError on testwiki since yesterday - https://phabricator.wikimedia.org/T314112 (10Xqt) [13:29:39] 10Phabricator, 10Triagers: add MPhamWMF to Triagers group - https://phabricator.wikimedia.org/T314146 (10MPhamWMF) [15:24:20] 10Release-Engineering-Team, 10Discovery, 10Discovery-Search, 10Elasticsearch, 10Developer Productivity: Developer productivity: Shared ElasticSearch instance for local development environments, Quibble CI, and Patch Demo - https://phabricator.wikimedia.org/T312198 (10kostajh) [15:35:13] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10Patch-For-Review: Add gerrit2002 as a replica of gerrit1001 - https://phabricator.wikimedia.org/T313972 (10Volans) @Dzahn I've assigned in Netbox an additional IPv4 and an additional IPv6 to `gerrit2002` like for gitlab not marked as VIP, m... [15:36:54] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10Patch-For-Review: Add gerrit2002 as a replica of gerrit1001 - https://phabricator.wikimedia.org/T313972 (10Dzahn) @Volans Thank you! That looks good. I will do the manual DNS change for that (amending to the existing one now) [15:52:43] 10GitLab (Infrastructure), 10serviceops: Reduce usage of public IPv4 addresses on GitLab hosts - https://phabricator.wikimedia.org/T310265 (10Volans) One small problem that we have... the additional IPs are puppetized in a way that assigns them as /32 on the host: ` inet 208.80.154.15/32 scope global eno1... [16:12:06] 10GitLab (Project Migration), 10Release-Engineering-Team (Doing), 10User-brennen, 10Voice & Tone: Rename mainline Git branch from "master" to "main" on all WMF-hosted repositories during GitLab migration - https://phabricator.wikimedia.org/T281593 (10Jdforrester-WMF) >>! In T281593#8114606, @Tacsipacsi wro... [16:58:39] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn) @Arnoldokoth The DNS change for `gitlab-replica-new` has been merged and deployed on DNS servers now: ` [authdns1001:~] $ host gitlab-replica-... [17:59:30] 10Phabricator, 10SRE, 10Sustainability (Incident Followup): Unable to view tasks in DB read-only mode - https://phabricator.wikimedia.org/T313879 (10Dzahn) p:05Triageβ†’03High [17:59:57] 10Phabricator, 10Release-Engineering-Team (Radar), 10SRE, 10serviceops-radar, 10Sustainability (Incident Followup): Unable to view tasks in DB read-only mode - https://phabricator.wikimedia.org/T313879 (10Dzahn) [18:00:05] !log using standalone puppetmaster in devtools to test phabricator scap3 changes [18:00:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:00:14] 10Phabricator, 10Release-Engineering-Team (Radar), 10SRE, 10serviceops-radar, 10Sustainability (Incident Followup): Phabricator: Unable to view tasks in DB read-only mode - https://phabricator.wikimedia.org/T313879 (10Dzahn) [18:19:41] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10serviceops, 10serviceops-collab, 10Patch-For-Review: move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Dzahn) members of the shell admin group "phabricator-roots" (@20after4, @thcipriani, @b... [18:19:45] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10serviceops, 10serviceops-collab, 10Patch-For-Review: Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn) members of the shell admin group "phabricator-roots" (@20after4, @thcipriani, @brennen, @j... [18:37:03] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10serviceops, 10serviceops-collab, 10Patch-For-Review: Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn) p:05Triageβ†’03High [18:42:17] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10serviceops, 10serviceops-collab, 10Patch-For-Review: Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn) With the changes above (T313360#8105754) it's now possible to rsync from old to new phab... [19:09:17] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10Patch-For-Review: Add gerrit2002 as a replica of gerrit1001 - https://phabricator.wikimedia.org/T313972 (10Dzahn) a:03Dzahn [19:09:29] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10Patch-For-Review: Bring up gerrit2002 - https://phabricator.wikimedia.org/T313250 (10Dzahn) [19:09:33] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10SRE, 10serviceops, and 2 others: replacement for gerrit2001 - https://phabricator.wikimedia.org/T243027 (10Dzahn) [19:10:06] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10Patch-For-Review: Add gerrit2002 as a replica of gerrit1001 - https://phabricator.wikimedia.org/T313972 (10Dzahn) 05Openβ†’03In progress p:05Triageβ†’03High [19:36:51] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10User-brennen: Deploy Phabricator with scap - https://phabricator.wikimedia.org/T313259 (10brennen) [19:38:58] rsync from gerrit1001 to gerrit2002 is still running. it's at like 86GB of 242GB.. a bit too conservative with the bwlimit..but whatever.. it will just run over weekend and nobody notices a difference [20:19:28] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10Patch-For-Review: Bring up gerrit2002 - https://phabricator.wikimedia.org/T313250 (10Dzahn) new in DNS: [authdns1001:~] $ host gerrit-replica-new.wikimedia.org gerrit-replica-new.wikimedia.org has address 208.80.153.104 gerrit-replica-new.... [20:19:47] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10Patch-For-Review: Add gerrit2002 as a replica of gerrit1001 - https://phabricator.wikimedia.org/T313972 (10Dzahn) new in DNS: [authdns1001:~] $ host gerrit-replica-new.wikimedia.org gerrit-replica-new.wikimedia.org has address 208.80.153.1... [20:19:59] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10SRE, 10serviceops, and 2 others: replacement for gerrit2001 - https://phabricator.wikimedia.org/T243027 (10Dzahn) new in DNS: [authdns1001:~] $ host gerrit-replica-new.wikimedia.org gerrit-replica-new.wikimedia.org has address 208.80.153... [20:20:50] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10Patch-For-Review: Add gerrit2002 as a replica of gerrit1001 - https://phabricator.wikimedia.org/T313972 (10Dzahn) [20:48:11] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10Patch-For-Review: Bring up gerrit2002 - https://phabricator.wikimedia.org/T313250 (10Dzahn) 05Openβ†’03In progress p:05Triageβ†’03High [20:52:00] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10Patch-For-Review: Add gerrit2002 as a replica of gerrit1001 - https://phabricator.wikimedia.org/T313972 (10Dzahn) @thcipriani and all: We got the IP address, we tied it to gerrit-replica-new.wikimedia.org. (v4 + v6) I added in Hiera via g... [20:53:29] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10Patch-For-Review: Bring up gerrit2002 - https://phabricator.wikimedia.org/T313250 (10Dzahn) T313972#8116463 [20:54:21] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10SRE, 10serviceops, and 2 others: replacement for gerrit2001 - https://phabricator.wikimedia.org/T243027 (10Dzahn) T313972#8116463 [20:59:15] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10Patch-For-Review: Bring up gerrit2002 - https://phabricator.wikimedia.org/T313250 (10Dzahn) rsync of `/srv/gerrit` from gerrit1001 to gerrit2002 is still running (with bandwidth limit). I will just let it run over the weekend, it's in a scr... [21:14:33] thcipriani: Oops, wmf.32? [21:14:58] I was thinking we'd stop at wmf.28 and switch to 1.40.0-wmf.1. [21:15:21] whoop, script went to far [21:15:24] I can update manually [21:15:27] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission πŸ’€): Configure keyholder on devtools deploy host for phabricator deployment - https://phabricator.wikimedia.org/T314195 (10dduvall) [21:15:37] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission πŸ’€): Configure keyholder on devtools deploy host for phabricator deployment - https://phabricator.wikimedia.org/T314195 (10dduvall) [21:16:04] Thanks! [21:20:08] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.39.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T314187 (10thcipriani) p:05Triageβ†’03Medium a:03hashar [21:41:20] When trying to view T314191, I am getting "A non well formed numeric value encountered". [21:41:21] T314191: 1.40.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T314191 [21:41:54] zabe: same. I need to file it. I think it has to do with a phabricator extension we maintain for making pretty train tasks :\ [21:42:06] it's picky about version numbers [21:43:25] I messed up the version number somehow, I guess :( [21:44:09] Oops. [21:44:19] personally I blame James_F :D [21:45:01] I blame myself too. [21:45:07] <3 [21:45:12] When in doubt, blame James_F. Good principle. [21:45:27] it's on my family crest [21:49:02] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10serviceops, 10serviceops-collab, 10Patch-For-Review: move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Dzahn) https://gerrit.wikimedia.org/r/c/operations/puppet/+/818513/ https://puppet-com... [21:56:23] (CR) Dzahn: [V: +1 C: +2] "noop confirmed on phab1001, phab2001 - adds all the "base" things to phab1004, phab2002" [puppet] - [21:56:28] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Release, 10Train Deployments: 1.39.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T308076 (10daniel) Heads up: We made a change that may cause a blip in the `html2wt.size.input` metrics: we changed the computation of the size from being in... [22:00:01] thcipriani: Want to send T313918 out? :-) [22:00:01] T313918: Announce 1.39 forthcoming "pencils down" and likely timeline to wikitech-l etc. - https://phabricator.wikimedia.org/T313918 [22:00:21] :) [22:00:23] sure [22:00:27] <3 [22:03:22] signed: Love, Jamesβ€”oh! no, I mean, Tyler [22:04:20] thcipriani: executive summary before weekend: both gerrit and phabricator have 'migration roles' applied that do the rsync/ferm/shell access for people. currently there are screen sessions that slowly sync phab1001>phab1004 data and gerrit1001>gerrit2002 another path. also the hosts are now in Icinga monitoring, got the base things installed and we have "gerrit-replica-new.wikimedia.org" in DNS [22:04:26] ready to be used. next change we can do together is to add it to the gerrit config in a standup or so [22:04:39] \o/ [22:04:42] nice mutante [22:04:44] (the new hardware I mean. of course noop on prod) [22:05:04] cheers [22:11:31] 10MediaWiki-Releasing, 10MW-1.39-release: Announce 1.39 forthcoming "pencils down" and likely timeline to wikitech-l etc. - https://phabricator.wikimedia.org/T313918 (10Jdforrester-WMF) 05Openβ†’03Resolved a:03thcipriani Posted: https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thr... [22:11:35] 10MediaWiki-Releasing, 10MW-1.39-notes, 10MW-1.39-release: Release MW 1.39.0 - https://phabricator.wikimedia.org/T313916 (10Jdforrester-WMF) [22:12:32] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10serviceops, 10serviceops-collab, 10Patch-For-Review: Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn) [22:14:06] (03CR) 10Jforrester: [C: 03+2] Zuul: Configure CI for the forthcoming REL1_39 branches [integration/config] - 10https://gerrit.wikimedia.org/r/817825 (https://phabricator.wikimedia.org/T313919) (owner: 10Jforrester) [22:15:19] James_F: sent. Thank you, as always, for being a hyper-competent and thoughtful human being. [22:15:58] (03Merged) 10jenkins-bot: Zuul: Configure CI for the forthcoming REL1_39 branches [integration/config] - 10https://gerrit.wikimedia.org/r/817825 (https://phabricator.wikimedia.org/T313919) (owner: 10Jforrester) [22:16:04] Thank you for being kind and dedicated and supportive. :_) [22:16:22] !log Zuul: Configure CI for the forthcoming REL1_39 branches for T313919 [22:16:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:16:24] T313919: Configure CI to support REL1_39 - https://phabricator.wikimedia.org/T313919 [22:18:09] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/817826 (https://phabricator.wikimedia.org/T313919) (owner: 10Jforrester) [22:18:31] 10Continuous-Integration-Config, 10MediaWiki-Releasing, 10MW-1.39-release, 10Patch-For-Review: Configure CI to support REL1_39 - https://phabricator.wikimedia.org/T313919 (10Jdforrester-WMF) 05Openβ†’03Resolved a:03Jdforrester-WMF Deployed. [22:18:35] 10MediaWiki-Releasing, 10MW-1.39-notes, 10MW-1.39-release: Release MW 1.39.0 - https://phabricator.wikimedia.org/T313916 (10Jdforrester-WMF) [22:20:13] (03Merged) 10jenkins-bot: jjb: [mediawiki-php-security] and [quibble-integration] Add REL1_39 branch [integration/config] - 10https://gerrit.wikimedia.org/r/817826 (https://phabricator.wikimedia.org/T313919) (owner: 10Jforrester)