[00:04:21] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12007556 (10thcipriani) Feedback from our #release-engineering-team team meeting today: - Preference is always for standard git forge behavior: port 22... [00:08:12] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 07Jenkins, and 3 others: Move beta cluster automatic deployment to a dedicated infrastructure - https://phabricator.wikimedia.org/T256168#12007559 (10bd808) >>! In T256168#11985947, @dancy wrote: > In p... [00:38:42] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 07Jenkins, and 3 others: Move beta cluster automatic deployment to a dedicated infrastructure - https://phabricator.wikimedia.org/T256168#12007606 (10thcipriani) Added the sender to the accepted senders... [02:01:47] (03CR) 10BCornwall: "Moving attention set to sukhe since he mentioned wanting to try building 😊" [integration/config] - 10https://gerrit.wikimedia.org/r/1288954 (owner: 10BCornwall) [02:05:28] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for TehKittyCat - https://phabricator.wikimedia.org/T428836 (10TehKittyCat) 03NEW [04:27:56] 10MediaWiki-Releasing, 07Documentation, 05MW-1.46-release: Finish up RELEASE-NOTES for 1.46.0 - https://phabricator.wikimedia.org/T423270#12008053 (10Bugreporter) >>! In T423270#12001934, @matmarex wrote: > I know what it says in 1.45, but I don't know whether that is any more correct. For example, "json" de... [05:50:59] (03PS1) 10Vaughn Walters: jjb: [catalyst-daily-AdvancedSearch] Add AdvancedSearch job [integration/config] - 10https://gerrit.wikimedia.org/r/1299812 (https://phabricator.wikimedia.org/T427013) [05:59:26] (03CR) 10Vaughn Walters: "Okay go ahead and deploy, and we can chat about it in watercooler I'll be there too. I'm going to have more of these too that will likely" [integration/config] - 10https://gerrit.wikimedia.org/r/1299812 (https://phabricator.wikimedia.org/T427013) (owner: 10Vaughn Walters) [06:24:13] 10Phabricator, 10Phabricator maintenance bot: Replace deprecated Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T428847 (10Aklapper) 03NEW [06:26:05] 10Phabricator, 10ReleaseTaggerBot, 13Patch-For-Review: Replace deprecated Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T402453#12008257 (10Aklapper) [06:26:45] (03Abandoned) 10Phedenskog: Dockerfiles: Remove Chromedriver start/stop from browser images [integration/config] - 10https://gerrit.wikimedia.org/r/1201546 (https://phabricator.wikimedia.org/T408327) (owner: 10Phedenskog) [06:29:01] 10Phabricator, 10LibUp: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T427620#12008260 (10Aklapper) [06:50:35] 10GitLab (Project and group requests), 06Release-Engineering-Team, 07Essential-Work: GitLab Private Repository Request for: production access meta data - https://phabricator.wikimedia.org/T411642#12008336 (10MoritzMuehlenhoff) @brennen Sorry, I missed your reply until now! Yes, that is still needed, repos/sr... [06:50:37] 10Phabricator: Replace deprecated Phabricator Conduit API calls with their stable equivalents (PatchDemoBot) - https://phabricator.wikimedia.org/T428850 (10Aklapper) 03NEW [06:50:41] 10Phabricator, 10Tool-Phabricator-bug-status: Replace deprecated Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T428851 (10Aklapper) 03NEW [06:51:39] 10Phabricator, 10Tool-Phabricator-bug-status: Replace deprecated Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T428851#12008358 (10Aklapper) [06:54:58] 10Phabricator, 10Wikibugs, 13Patch-For-Review: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T402454#12008371 (10Aklapper) Thanks a lot for the maniphest.info patch, very appreciated! Now I wonder who could review that. Also, that... [08:09:48] 10Phabricator Translations Extension, 06translatewiki.net: Drop fund and phortune from Phabricator Translations - https://phabricator.wikimedia.org/T428697#12008582 (10Nikerabbit) p:05Triageβ†’03Medium @Pppery Planning to submit a patch or should we do it? [08:40:18] !log gerrit: deleted mediawiki/core branch "development" that pointed to 7f622781cc31053b121f6f4ddbff506cba10d38e (which is contained by master). Had probably been created by a direct push. [08:40:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:47:18] 10Phabricator, 10Catalyst (PatchDemo): Replace deprecated Phabricator Conduit API calls with their stable equivalents (PatchDemoBot) - https://phabricator.wikimedia.org/T428850#12008719 (10matmarex) Relevant code: https://gitlab.wikimedia.org/repos/test-platform/catalyst/patchdemo/-/blob/2ede986aef0ceaa36dab77... [08:49:09] 10MediaWiki-Releasing, 06Release-Engineering-Team, 10MediaWiki-Release-Tools: Convert old release branches to tags - https://phabricator.wikimedia.org/T428864 (10hashar) 03NEW [08:51:30] 10MediaWiki-Releasing, 06Release-Engineering-Team, 10MediaWiki-Release-Tools: Convert old release branches to tags - https://phabricator.wikimedia.org/T428864#12008739 (10hashar) [08:56:57] !log Converting REL1_39 branches to tags # T428864 [08:56:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:57:00] T428864: Convert old release branches to tags - https://phabricator.wikimedia.org/T428864 [09:05:19] !log Converting REL1_40 branches to tags # T428864 [09:05:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:05:22] T428864: Convert old release branches to tags - https://phabricator.wikimedia.org/T428864 [09:05:33] good thing is the script automates the mass conversion [09:05:56] and I guess it should be part of whatever runbook we have when we close/obsolete a release branch [09:18:30] !log Converting REL1_41 branches to tags # T428864 [09:18:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:18:32] T428864: Convert old release branches to tags - https://phabricator.wikimedia.org/T428864 [09:18:56] !log Converting REL1_42 branches to tags # T428864 [09:18:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:35:00] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for TehKittyCat - https://phabricator.wikimedia.org/T428836#12008944 (10Aklapper) @TehKittyCat: Thanks for the request! Which project(s) in GitLab do you plan to work on? [09:37:07] 10MediaWiki-Releasing, 06Release-Engineering-Team, 10MediaWiki-Release-Tools: Convert old release branches to tags - https://phabricator.wikimedia.org/T428864#12008948 (10hashar) Logs | REL1_39 | P94064 | REL1_40 | P94065 | REL1_41 | P94066 | REL1_42 | P94067 The `wmf_core` bundle might not cover all exten... [09:39:20] 10Phabricator (Search), 06Release-Engineering-Team (Priority Backlog πŸ“₯): Have a more scope-restricted default global (primary) search - https://phabricator.wikimedia.org/T397558#12008962 (10Aklapper) 05Stalledβ†’03Open This should be unblocked as the deployment in T410849 expanded the maximum width of the to... [09:45:10] !log Converted mediawiki/core branches REL1_39, REL1_40, REL1_41, REL1_42 to tags # T428864 [09:45:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:45:13] T428864: Convert old release branches to tags - https://phabricator.wikimedia.org/T428864 [09:47:34] 10MediaWiki-Releasing, 06Release-Engineering-Team, 10MediaWiki-Release-Tools: Convert old release branches to tags - https://phabricator.wikimedia.org/T428864#12008994 (10hashar) mediawiki/extensions and mediawiki/skins do not have release branches. For core I reran the script with --core and an empty bundl... [10:08:54] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Infrastructure-Foundations: Upgrade releng/operations-puppet CI image from Bullseye to Bookworm - https://phabricator.wikimedia.org/T417976#12009047 (10elukey) @hashar hi! Should we start planning this move? Lemme know if we can help! [10:12:27] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#12009055 (10hashar) ` upstream connect error or disconnect/reset before headers. retried and the latest reset r... [10:50:58] (03PS1) 10Sergio Gimeno: Update phan dependencies for Echo [integration/config] - 10https://gerrit.wikimedia.org/r/1300747 (https://phabricator.wikimedia.org/T424205) [11:11:30] (03PS2) 10Sergio Gimeno: phan: add TestKitchen as Echo phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1300747 (https://phabricator.wikimedia.org/T424205) [11:12:59] (03CR) 10Hashar: [C:03+2] phan: add TestKitchen as Echo phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1300747 (https://phabricator.wikimedia.org/T424205) (owner: 10Sergio Gimeno) [11:13:03] :) [11:16:22] (03Merged) 10jenkins-bot: phan: add TestKitchen as Echo phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1300747 (https://phabricator.wikimedia.org/T424205) (owner: 10Sergio Gimeno) [12:03:52] (03PS1) 10Hslater: Zuul: [mediawiki/extensions/VisualEditorPlus] Disable selenium tests [integration/config] - 10https://gerrit.wikimedia.org/r/1300765 [12:14:32] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-kafka-main-5 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [12:25:22] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/VisualEditorPlus] Disable selenium tests [integration/config] - 10https://gerrit.wikimedia.org/r/1300765 (owner: 10Hslater) [12:29:12] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/VisualEditorPlus] Disable selenium tests [integration/config] - 10https://gerrit.wikimedia.org/r/1300765 (owner: 10Hslater) [12:29:29] (03CR) 10Hashar: [C:03+2] "deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/1300765 (owner: 10Hslater) [12:41:02] FIRING: [5x] PuppetAgentNoResources: No Puppet resources found on instance deployment-kafka-main-5 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [12:44:32] FIRING: [7x] PuppetAgentNoResources: No Puppet resources found on instance deployment-imagescaler04 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:02:02] FIRING: [7x] PuppetAgentNoResources: No Puppet resources found on instance deployment-imagescaler04 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:15:32] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [13:15:43] 10Beta-Cluster-Infrastructure: Widespread puppet agent failures in project deployment-prep - https://phabricator.wikimedia.org/T428901 (10wmcs-alerts) 03NEW [13:17:47] FIRING: [7x] PuppetAgentNoResources: No Puppet resources found on instance deployment-imagescaler04 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:20:32] RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [13:24:08] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 06Traffic: gitlab behind CDN: serve gitlab.wm.o via text-lb instead of dedicated IPs? - https://phabricator.wikimedia.org/T428903 (10ABran-WMF) 03NEW [13:25:01] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 06Traffic: gitlab behind CDN: serve gitlab.wm.o via text-lb instead of dedicated IPs? - https://phabricator.wikimedia.org/T428903#12009971 (10ABran-WMF) 05Openβ†’03In progress p:05Triageβ†’03High [13:28:17] FIRING: [7x] PuppetAgentNoResources: No Puppet resources found on instance deployment-imagescaler04 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:29:32] FIRING: [7x] PuppetAgentNoResources: No Puppet resources found on instance deployment-imagescaler04 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:29:40] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-poolcounter07 on project deployment-prep - https://phabricator.wikimedia.org/T428905 (10wmcs-alerts) 03NEW [13:30:09] 06Project-Admins, 10WikimediaCustomizations: Split WikimediaCustomizations Phabricator project into subprojects - https://phabricator.wikimedia.org/T428727#12010010 (10Tgr) The immediate motivation was the donor badge project having a lot of Phabricator activity. [13:32:03] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 06Traffic: gitlab behind CDN: serve gitlab.wm.o via text-lb instead of dedicated IPs? - https://phabricator.wikimedia.org/T428903#12010021 (10ABran-WMF) [13:33:32] RESOLVED: [7x] PuppetAgentNoResources: No Puppet resources found on instance deployment-imagescaler04 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:44:14] 10Phabricator Translations Extension, 06translatewiki.net: Drop fund and phortune from Phabricator Translations - https://phabricator.wikimedia.org/T428697#12010062 (10Pppery) I'm planning to submit a patch. (I intended to do this yesterday but then life got in the way) [13:44:53] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T423915#12010066 (10Jdforrester-WMF) Should {T428849} be a train blocker? [14:09:21] 10Beta-Cluster-Infrastructure, 10MediaWiki-libs-Rdbms: Beta Cluster MariaDB is still 10.6.17, MW now requires 10.11 - https://phabricator.wikimedia.org/T428910 (10Jdforrester-WMF) 03NEW [14:11:52] 10Beta-Cluster-Infrastructure, 10MediaWiki-libs-Rdbms: Beta Cluster MariaDB is still 10.6.17, MW now requires 10.11 - https://phabricator.wikimedia.org/T428910#12010251 (10Jdforrester-WMF) Fall-out from {T366644}. [14:12:32] 10Beta-Cluster-Infrastructure, 10MediaWiki-libs-Rdbms: Beta Cluster MariaDB is still 10.6.17, MW now requires 10.11 - https://phabricator.wikimedia.org/T428910#12010256 (10Jdforrester-WMF) p:05Triageβ†’03High At least High, but I've not been doing the work on Beta Cluster maintenance in any real sense for ye... [14:38:33] 10MediaWiki-Releasing, 06Release-Engineering-Team, 10MediaWiki-Release-Tools: Convert old release branches to tags - https://phabricator.wikimedia.org/T428864#12010371 (10hashar) 05Openβ†’03Resolved To get rid of the legacy branch from a local clone: ` git fetch origin --prune ` For the superprojects... [14:46:11] !log On all MediaWiki repos, converting old release branches up to REL1_42 included to tags. Last time I missed non wmf repo # T380841 | T428864 [14:46:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:46:16] T380841: Convert old release branches to tags - https://phabricator.wikimedia.org/T380841 [14:46:16] T428864: Convert old release branches to tags - https://phabricator.wikimedia.org/T428864 [14:46:51] !log for minor in $(seq 21 42); do ./.tox/make-release/bin/python -u ./make-release/branch.py --delete --abandon --bundle '*' "REL1_$minor"; done; [14:46:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:47:13] Created https://gerrit.wikimedia.org/g/mediawiki/extensions/Description2/+/refs/tags/REL1_21 [3fe7637d164c20a8734fc58214df444d4ed16fa6] [14:47:13] Deleting branch REL1_21 in mediawiki/extensions/Description2 [14:47:14] fun times [14:47:45] 10Beta-Cluster-Infrastructure, 10MediaWiki-libs-Rdbms: Beta Cluster MariaDB is still 10.6.17, MW now requires 10.11 - https://phabricator.wikimedia.org/T428910#12010403 (10bd808) [14:47:49] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Priority Backlog πŸ“₯), 10Cloud-VPS (Debian Bullseye Deprecation), 07Epic: Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#12010402 (10bd808) [14:48:44] 10Beta-Cluster-Infrastructure, 10MediaWiki-libs-Rdbms: Beta Cluster MariaDB is still 10.6.17, MW now requires 10.11 - https://phabricator.wikimedia.org/T428910#12010405 (10bd808) [15:05:30] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T423915#12010525 (10jnuche) [15:20:18] 10Phabricator, 10Catalyst (PatchDemo): Replace deprecated Phabricator Conduit API calls with their stable equivalents (PatchDemoBot) - https://phabricator.wikimedia.org/T428850#12010657 (10Aklapper) @matmarex: I can update that! I assume QTE Team does not exist anymore (that's what I make out of internal https... [15:27:59] (03CR) 10Hashar: [C:03+2] "I have deployed the job and started a build https://integration.wikimedia.org/ci/job/catalyst-daily-AdvancedSearch/ 😎" [integration/config] - 10https://gerrit.wikimedia.org/r/1299812 (https://phabricator.wikimedia.org/T427013) (owner: 10Vaughn Walters) [15:28:57] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/ReadingLists] Add SiteMatrix dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1300220 (https://phabricator.wikimedia.org/T428780) (owner: 10Aude) [15:30:28] (03Merged) 10jenkins-bot: jjb: [catalyst-daily-AdvancedSearch] Add AdvancedSearch job [integration/config] - 10https://gerrit.wikimedia.org/r/1299812 (https://phabricator.wikimedia.org/T427013) (owner: 10Vaughn Walters) [15:30:57] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/ReadingLists] Add SiteMatrix dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1300220 (https://phabricator.wikimedia.org/T428780) (owner: 10Aude) [15:35:28] (03PS1) 10Jaime Nuche: Wikilambda Catalyst e2e: Pass `ZUUL_PATCHSET` to Catalyst client [integration/config] - 10https://gerrit.wikimedia.org/r/1300828 (https://phabricator.wikimedia.org/T424815) [15:38:14] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 10Development-Metrics: GrimoireLab's Perceval fails pulling Maniphest transaction data since 20260403 - https://phabricator.wikimedia.org/T428300#12010767 (10Aklapper) [15:38:15] (03CR) 10Hashar: [C:03+2] "Deployed, and I have done a `recheck`Β on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ReadingLists/+/1300209" [integration/config] - 10https://gerrit.wikimedia.org/r/1300220 (https://phabricator.wikimedia.org/T428780) (owner: 10Aude) [15:38:52] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 10Development-Metrics: GrimoireLab's Perceval fails pulling Maniphest transaction data since 20260403 - https://phabricator.wikimedia.org/T428300#12010768 (10Aklapper) Perceval logs are in the `/docker/mordred/instances/wikimedia/logs` directory on the `mordred`... [15:43:07] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 10Development-Metrics: GrimoireLab's Perceval fails pulling Maniphest transaction data since 20260403 - https://phabricator.wikimedia.org/T428300#12010804 (10Aklapper) [15:43:08] 10Phabricator (Upstream), 07Upstream: EXCEPTION: (Exception) Policy identifier is an object PHID (''), but no object handle was provided. A handle must be provided for object policies. - https://phabricator.wikimedia.org/T361459#12010805 (10Aklapper) [15:43:31] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: EXCEPTION: (Exception) Policy identifier is an object PHID (''), but no object handle was provided. A handle must be provided for object policies. - https://phabricator.wikimedia.org/T361459#12010807 (10Aklapper) p:05Lowβ†’03High a... [15:47:03] 10Beta-Cluster-Infrastructure, 10MediaWiki-libs-Rdbms: Beta Cluster MariaDB is still 10.6.17, MW now requires 10.11 - https://phabricator.wikimedia.org/T428910#12010836 (10dancy) Notes: From operations/mediawiki-config/wmf-config/db-labs.php: ` 'hostsByName' => [ // depl... [15:47:25] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: EXCEPTION: (Exception) Policy identifier is an object PHID (''), but no object handle was provided. A handle must be provided for object policies. - https://phabricator.wikimedia.org/T361459#12010838 (10Aklapper) This crashes mordred... [15:57:45] 10MediaWiki-Releasing, 05MW-1.46-notes, 05MW-1.46-release: Release MW 1.46.0-rc.0 - https://phabricator.wikimedia.org/T423267#12010915 (10matmarex) 05Openβ†’03Resolved https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/QFJVF2DRRP2NM2K5ALN3Q6EEXKGWCJUM/ [16:06:08] 10Beta-Cluster-Infrastructure: Set up deployment-db15 with Trixie and wmf-mariadb1011 - https://phabricator.wikimedia.org/T428930 (10dancy) 03NEW [16:06:35] 10Beta-Cluster-Infrastructure: Set up deployment-db15 with Trixie and wmf-mariadb1011 - https://phabricator.wikimedia.org/T428930#12010966 (10dancy) [16:06:37] 10Beta-Cluster-Infrastructure, 10MediaWiki-libs-Rdbms: Beta Cluster MariaDB is still 10.6.17, MW now requires 10.11 - https://phabricator.wikimedia.org/T428910#12010965 (10dancy) [16:07:51] 10Phabricator, 10Catalyst (PatchDemo): Replace deprecated Phabricator Conduit API calls with their stable equivalents (PatchDemoBot) - https://phabricator.wikimedia.org/T428850#12010972 (10matmarex) Ugh, well, I mean the team that owns Catalyst/Patchdemo… I don't think they got disbanded, so I hope they'll tel... [16:08:08] 10Beta-Cluster-Infrastructure: Set up deployment-db15 with Trixie and wmf-mariadb1011 - https://phabricator.wikimedia.org/T428930#12010973 (10dancy) p:05Triageβ†’03High [16:09:26] 10Beta-Cluster-Infrastructure: Set up deployment-db15 with Trixie and wmf-mariadb1011 - https://phabricator.wikimedia.org/T428930#12010991 (10dancy) [16:11:40] 10Beta-Cluster-Infrastructure: Set up deployment-db15 with Trixie and wmf-mariadb1011 - https://phabricator.wikimedia.org/T428930#12011010 (10dancy) [16:19:26] (03PS3) 10Hashar: Add coverage job for Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/1299569 (https://phabricator.wikimedia.org/T428642) [16:19:47] (03CR) 10Hashar: [C:03+2] jjb: create tox coverage publishing job template [integration/config] - 10https://gerrit.wikimedia.org/r/1299568 (https://phabricator.wikimedia.org/T244650) (owner: 10Hashar) [16:19:56] (03CR) 10Hashar: [C:03+2] Add coverage job for Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/1299569 (https://phabricator.wikimedia.org/T428642) (owner: 10Hashar) [16:25:22] (03Merged) 10jenkins-bot: jjb: create tox coverage publishing job template [integration/config] - 10https://gerrit.wikimedia.org/r/1299568 (https://phabricator.wikimedia.org/T244650) (owner: 10Hashar) [16:25:26] (03Merged) 10jenkins-bot: Add coverage job for Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/1299569 (https://phabricator.wikimedia.org/T428642) (owner: 10Hashar) [16:27:32] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-db15 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [16:27:37] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-db15 in project deployment-prep - https://phabricator.wikimedia.org/T428934 (10wmcs-alerts) 03NEW [16:29:21] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: EXCEPTION: (Exception) Policy identifier is an object PHID (''), but no object handle was provided. A handle must be provided for object policies. - https://phabricator.wikimedia.org/T361459#12011105 (10Aklapper) I can reproduce now,... [16:32:32] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-db15 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [16:48:45] 10Beta-Cluster-Infrastructure: Set up deployment-db15 with Trixie and wmf-mariadb1011 - https://phabricator.wikimedia.org/T428930#12011243 (10dancy) Current `run-pupet-agent` output: {F87993735} [16:49:14] (03update) 10dduvall: magnum: Use the Magnum Cluster API (CAPI) [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/58 (https://phabricator.wikimedia.org/T424061) [16:49:20] (03update) 10dduvall: magnum: Use the Magnum Cluster API (CAPI) [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/58 (https://phabricator.wikimedia.org/T424061) [16:49:20] (03open) 10dduvall: magnum: Use the Magnum Cluster API (CAPI) [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/58 (https://phabricator.wikimedia.org/T424061) [16:52:19] bd808: magnum/capi success! ^ [16:53:54] 10Phabricator Translations Extension: Export source strings with PLURAL where appropriate - https://phabricator.wikimedia.org/T412421#12011275 (10Pppery) 05Openβ†’03Resolved [16:54:04] (03update) 10dduvall: magnum: Use the Magnum Cluster API (CAPI) [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/58 (https://phabricator.wikimedia.org/T424061) [17:04:05] 10Phabricator Translations Extension: Reduce dirty diffs in qqq.json - https://phabricator.wikimedia.org/T412649#12011318 (10Pppery) 05Openβ†’03Resolved Given the lack of irrelevant diffs between patches 1 and 2 of https://gerrit.wikimedia.org/r/c/phabricator/translations/+/1300823 (in which I cleared all... [17:04:24] (03PS2) 10Jforrester: jjb: [wikilambda-catalyst-end-to-end*] Pass `ZUUL_PATCHSET` to Catalyst client [integration/config] - 10https://gerrit.wikimedia.org/r/1300828 (https://phabricator.wikimedia.org/T424815) (owner: 10Jaime Nuche) [17:04:27] dduvall: sweet! I see the linter is yelling at you, but that should be a quick fix [17:04:31] (03CR) 10Jforrester: [C:03+2] jjb: [wikilambda-catalyst-end-to-end*] Pass `ZUUL_PATCHSET` to Catalyst client [integration/config] - 10https://gerrit.wikimedia.org/r/1300828 (https://phabricator.wikimedia.org/T424815) (owner: 10Jaime Nuche) [17:05:14] PROBLEM - jenkins_service_running on releases2003 is CRITICAL: PROCS CRITICAL: 0 processes with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [17:07:48] fixing this [17:07:57] (03Merged) 10jenkins-bot: jjb: [wikilambda-catalyst-end-to-end*] Pass `ZUUL_PATCHSET` to Catalyst client [integration/config] - 10https://gerrit.wikimedia.org/r/1300828 (https://phabricator.wikimedia.org/T424815) (owner: 10Jaime Nuche) [17:10:28] 10Phabricator Translations Extension, 13Patch-For-Review: Make WMF translations extension pass `bin/i18n validate` - https://phabricator.wikimedia.org/T412651#12011349 (10Pppery) (This is still nowhere close to done; it needs, in addition to the above patch being merged, an import/export on the translatewiki.n... [17:14:01] 10Phabricator Translations Extension, 06translatewiki.net: Support PLURAL in a more reasonable manner in Phabricator translations - https://phabricator.wikimedia.org/T411837#12011364 (10Pppery) 05Openβ†’03Resolved This is basically done now. T412423 is still a good idea and I may get to it once the dust... [17:24:28] (03update) 10dduvall: magnum: Use the Magnum Cluster API (CAPI) [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/58 (https://phabricator.wikimedia.org/T424061) [17:27:39] bd808: grr. how do i update the tofu docs? [17:28:08] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: EXCEPTION: (Exception) Policy identifier is an object PHID (''), but no object handle was provided. A handle must be provided for object policies. - https://phabricator.wikimedia.org/T361459#12011425 (10Aklapper) I am puzzled. `SELEC... [17:31:20] dduvall: `make docs` [17:31:31] oh :) [17:31:33] very nice [17:32:12] (03update) 10dduvall: magnum: Use the Magnum Cluster API (CAPI) [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/58 (https://phabricator.wikimedia.org/T424061) [17:34:24] bd808: k, all green now [17:39:04] (03open) 10aklapper: Work around missing old value after Edit Policy change [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/122 (https://phabricator.wikimedia.org/T361459) [17:40:10] (03approved) 10bd808: magnum: Use the Magnum Cluster API (CAPI) [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/58 (https://phabricator.wikimedia.org/T424061) (owner: 10dduvall) [17:40:13] (03merge) 10aklapper: Work around missing old value after Edit Policy change [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/122 (https://phabricator.wikimedia.org/T361459) [17:43:09] 10Phabricator (phabricator-next), 06Release-Engineering-Team (Doing 😎), 10Development-Metrics: GrimoireLab's Perceval fails pulling Maniphest transaction data since 20260403 - https://phabricator.wikimedia.org/T428300#12011487 (10Aklapper) a:03Aklapper I hope that https://gitlab.wikimedia.org/repos/phabric... [17:45:32] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-db15 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [17:45:39] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-db15 in project deployment-prep - https://phabricator.wikimedia.org/T428944 (10wmcs-alerts) 03NEW [17:55:42] 10Phabricator, 10Tool-Phabricator-bug-status: Replace deprecated Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T428851#12011578 (10bd808) β†’14Duplicate dup:03T418328 [18:03:51] (03approved) 10thcipriani: gitlab-runner: set nofile ulimit of 4096 for runners [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/610 (https://phabricator.wikimedia.org/T426827) (owner: 10dancy) [18:03:56] 10Phabricator, 10Wikibugs, 13Patch-For-Review: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T402454#12011605 (10bd808) >>! In T402454#12008371, @Aklapper wrote: > Now I wonder who could review that. I'm sure you did not mean this... [18:14:25] 06Release-Engineering-Team (Radar), 10Cloud-VPS (Quota-requests): Quota increase request for zuul - https://phabricator.wikimedia.org/T428515#12011627 (10Andrew) As I understand it, recent Magnum innovations/discoveries may make this unnecessary. @dduvall if that's incorrect please respond and you will have a... [18:47:00] 10Phabricator, 10Wikibugs, 13Patch-For-Review: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T402454#12011768 (10Aklapper) @bd808: Meh, sorry. No it was not meant like that; I was honestly wondering who may find time at some point,... [18:48:06] 10Phabricator, 10Tool-Phabricator-bug-status: Move away from deprecated `maniphest.query` Conduit API - https://phabricator.wikimedia.org/T418328#12011773 (10Aklapper) [18:56:44] (03merge) 10dduvall: magnum: Use the Magnum Cluster API (CAPI) [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/58 (https://phabricator.wikimedia.org/T424061) [19:03:36] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T423915#12011808 (10dduvall) [19:42:04] 10Phabricator Translations Extension: Deal with "AM"/"PM" messages - https://phabricator.wikimedia.org/T428957 (10Pppery) 03NEW [19:42:37] 10Phabricator Translations Extension: Deal with "AM"/"PM" messages - https://phabricator.wikimedia.org/T428957#12011949 (10Pppery) (I could explain what I'm going to do here in more detail but since I'm assigning this to myself and plan to right a patch in a few days I don't see the point) [19:43:41] 10Phabricator Translations Extension: Drop "Variable types" spew from autogenerated documentation - https://phabricator.wikimedia.org/T428958 (10Pppery) 03NEW [19:44:00] 10Phabricator Translations Extension: Drop "Variable types" spew from autogenerated documentation - https://phabricator.wikimedia.org/T428958#12011965 (10Pppery) a:03Pppery [19:46:33] 10Beta-Cluster-Infrastructure: Set up deployment-db15 with Trixie and wmf-mariadb1011 - https://phabricator.wikimedia.org/T428930#12011977 (10dancy) @Zabe I saw that you handled T329577 a few years ago and I'm wondering if you can help me bring deployment-db15 online to take over for deployment-db14 (xref T42891... [20:00:47] 10Beta-Cluster-Infrastructure: Set up deployment-db15 with Trixie and wmf-mariadb1011 - https://phabricator.wikimedia.org/T428930#12011989 (10dancy) I have a 120Gib `db15` volume mounted on `/srv`. [20:31:22] 10Beta-Cluster-Infrastructure: Set up deployment-db15 with Trixie and wmf-mariadb1011 - https://phabricator.wikimedia.org/T428930#12012104 (10dancy) Initialize db stuff: ` sudo -u mysql /opt/wmf-mariadb1011/scripts/mariadb-install-db \ --basedir=/opt/wmf-mariadb1011 \ --datadir=/srv/sqldata ` [20:34:12] 10Beta-Cluster-Infrastructure: Set up deployment-db15 with Trixie and wmf-mariadb1011 - https://phabricator.wikimedia.org/T428930#12012134 (10dancy) ` root@deployment-db11:~# time /opt/wmf-mariadb106/bin/mariadb-dump --all-databases --single-transaction --gtid --triggers | gzip > /srv/db11-seed.sql.gz ` [20:46:00] 10Phabricator-Bot-Requests, 06Release-Engineering-Team (Doing 😎), 06Fundraising-Backlog: Phabricator Bot Request: Fundraising Tech Bug Reporter - https://phabricator.wikimedia.org/T426459#12012216 (10AKanji-WMF) Hi @Aklapper - the FR-Tech string inside "FR-Tech-Request-Bot" is causing unanticipated alert... [20:56:43] 10Phabricator-Bot-Requests, 06Release-Engineering-Team (Doing 😎), 06Fundraising-Backlog: Phabricator Bot Request: Fundraising Tech Bug Reporter - https://phabricator.wikimedia.org/T426459#12012238 (10Aklapper) @AKanji-WMF Sure sure! Renamed to @Fundraising-Task-Bot now. [21:00:32] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance deployment-db15 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [21:30:26] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T423915#12012350 (10dduvall) 05Openβ†’03Resolved [21:34:14] 10Phabricator, 10Wikibugs, 13Patch-For-Review: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T402454#12012364 (10bd808) Wikibugs has a lot of maintainers in theory: https://toolsadmin.wikimedia.org/tools/id/wikibugs In practice, i... [21:43:25] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Set up deployment-db15 with Trixie and wmf-mariadb1011 - https://phabricator.wikimedia.org/T428930#12012391 (10dancy) I copied `/srv/db11-seed.sql.gz` from `deployment-db11` to `deployment-db15` and I'm running ` zcat /srv/db11-seed.sql.gz | sudo mysql ` [21:59:19] (03PS3) 10Jforrester: jjb: [wikilambda-catalyst-end-to-end*] Add catalyst api-tests step [integration/config] - 10https://gerrit.wikimedia.org/r/1289415 (https://phabricator.wikimedia.org/T343378) [21:59:22] (03PS4) 10Jforrester: jjb: [wikilambda-catalyst-end-to-end*] Add catalyst api-tests step [integration/config] - 10https://gerrit.wikimedia.org/r/1289415 (https://phabricator.wikimedia.org/T343378) [21:59:47] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services: Update Jenkins hosts from Java 17 to Java 21 - https://phabricator.wikimedia.org/T418109#12012451 (10Dzahn) Both releases* machines and https://releases-jenkins.wikimedia.org are now on Trixie and Java 21. [22:02:47] (03CR) 10CI reject: [V:04-1] jjb: [wikilambda-catalyst-end-to-end*] Add catalyst api-tests step [integration/config] - 10https://gerrit.wikimedia.org/r/1289415 (https://phabricator.wikimedia.org/T343378) (owner: 10Jforrester) [22:04:39] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Set up deployment-db15 with Trixie and wmf-mariadb1011 - https://phabricator.wikimedia.org/T428930#12012467 (10Zabe) >>! In T428930#12011975, @dancy wrote: > @Zabe I saw that you handled T329577 a few years ago and I'm wondering if you can help me bring deploy... [22:05:09] 10Beta-Cluster-Infrastructure, 10MediaWiki-libs-Rdbms: Beta Cluster MariaDB is still 10.6.17, MW now requires 10.11 - https://phabricator.wikimedia.org/T428910#12012468 (10Jdforrester-WMF) [22:07:15] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-poolcounter07 on project deployment-prep - https://phabricator.wikimedia.org/T428905#12012474 (10bd808) 05Openβ†’03Invalid `lang=shell-session bd808@deployment-poolcounter07:~$ sudo -i puppet agent -tv Info: Using environment '... [22:08:06] 10Beta-Cluster-Infrastructure: Widespread puppet agent failures in project deployment-prep - https://phabricator.wikimedia.org/T428901#12012481 (10bd808) 05Openβ†’03Invalid Resolved per https://prometheus-alerts.wmcloud.org/?q=%40state%3Dactive&q=project%3Ddeployment-prep monitoring. [22:08:58] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-db15 in project deployment-prep - https://phabricator.wikimedia.org/T428934#12012487 (10bd808) β†’14Duplicate dup:03T428930 [22:09:01] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Set up deployment-db15 with Trixie and wmf-mariadb1011 - https://phabricator.wikimedia.org/T428930#12012489 (10bd808) [22:09:31] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-db15 in project deployment-prep - https://phabricator.wikimedia.org/T428944#12012492 (10bd808) β†’14Duplicate dup:03T428930 [22:09:32] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Set up deployment-db15 with Trixie and wmf-mariadb1011 - https://phabricator.wikimedia.org/T428930#12012494 (10bd808) [22:15:41] (03PS5) 10Jforrester: jjb: [wikilambda-catalyst-end-to-end*] Add catalyst api-tests step [integration/config] - 10https://gerrit.wikimedia.org/r/1289415 (https://phabricator.wikimedia.org/T343378) [22:15:41] (03CR) 10Jforrester: jjb: [wikilambda-catalyst-end-to-end*] Add catalyst api-tests step (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/1289415 (https://phabricator.wikimedia.org/T343378) (owner: 10Jforrester) [22:22:20] (03PS1) 10Vaughn Walters: jjb: [catalyst-daily-Popups] Add Popups job [integration/config] - 10https://gerrit.wikimedia.org/r/1300926 (https://phabricator.wikimedia.org/T427011) [22:29:29] (03CR) 10Vaughn Walters: "This is ready for review." [integration/config] - 10https://gerrit.wikimedia.org/r/1300926 (https://phabricator.wikimedia.org/T427011) (owner: 10Vaughn Walters) [22:29:38] 06Release-Engineering-Team (Priority Backlog πŸ“₯): Allow configuration of canary and production checks based on deployment target - https://phabricator.wikimedia.org/T428971 (10bd808) 03NEW [22:31:06] 06Release-Engineering-Team (Priority Backlog πŸ“₯): Allow configuration of canary and production checks based on deployment target - https://phabricator.wikimedia.org/T428971#12012544 (10bd808) p:05Triageβ†’03Medium [22:36:19] 06Release-Engineering-Team (Priority Backlog πŸ“₯): Allow configuration of canary and production checks based on deployment target - https://phabricator.wikimedia.org/T428971#12012572 (10bd808) @Scott_French Does it seem reasonable that what I called the "control group" will always be all logs minus the target depl... [22:45:40] 06Release-Engineering-Team (Priority Backlog πŸ“₯): Configure optional httpbb checks to perform during a deployment - https://phabricator.wikimedia.org/T428972 (10bd808) 03NEW [22:47:21] 06Release-Engineering-Team (Priority Backlog πŸ“₯): Configure optional httpbb checks to perform during a deployment - https://phabricator.wikimedia.org/T428972#12012600 (10bd808) p:05Triageβ†’03Medium [22:47:47] 06Release-Engineering-Team (Priority Backlog πŸ“₯): Configure additional httpbb checks to perform during a deployment - https://phabricator.wikimedia.org/T428972#12012601 (10bd808) [22:59:18] 06Release-Engineering-Team (Priority Backlog πŸ“₯): Investigate dynamic threshold model comparisons for canary and production log volume checks - https://phabricator.wikimedia.org/T428976 (10bd808) 03NEW [22:59:56] 06Release-Engineering-Team (Priority Backlog πŸ“₯): Investigate dynamic threshold model comparisons for canary and production log volume checks - https://phabricator.wikimedia.org/T428976#12012670 (10bd808) p:05Triageβ†’03Medium [23:40:29] 10Phabricator Translations Extension: Make WMF translations extension pass `bin/i18n validate` - https://phabricator.wikimedia.org/T412651#12012769 (10Pppery) Current validation errors: P94100 Still needs another export/generate cycle [23:44:22] 10Gerrit: Gerrit keeps returning 503 errors - https://phabricator.wikimedia.org/T428981 (10SomeRandomDeveloper) 03NEW [23:44:53] 10Gerrit: Gerrit keeps returning 503 errors - https://phabricator.wikimedia.org/T428981#12012797 (10SomeRandomDeveloper) Different error: {F88042221} [23:57:33] 10Gerrit, 07ci-test-error (WMF-deployed Build Failure): Gerrit keeps returning 503 errors - https://phabricator.wikimedia.org/T428981#12012816 (10SomeRandomDeveloper)