[04:16:05] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#11987429 (10Jelto) Persisting some discussion out of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1282428: It is possible to modify the ssh host... [05:07:34] 10Phabricator: Consider stop using Phame - https://phabricator.wikimedia.org/T428225 (10Bugreporter) 03NEW [06:07:53] 10GitLab (Pipeline Services Migration🐀), 10Wikispeech-Jobrunner, 10Wikispeech-Text-to-Speech: Migrate Wikispeech services to GitLab - https://phabricator.wikimedia.org/T344741#11987543 (10Sebastian_Berlin-WMSE) Relevant repos have been moved [[ https://gitlab.wikimedia.org/repos/mediawiki/services/speechoid... [06:10:57] 10GitLab (Pipeline Services Migration🐀), 10Wikispeech-Jobrunner, 10Wikispeech-Text-to-Speech: Migrate Wikispeech services to GitLab - https://phabricator.wikimedia.org/T344741#11987548 (10Jelto) Thank you for the migration! > Is there a proper way to archive the Gerrit repos? Yes Gerrit admins can do that.... [07:42:45] (03update) 10kineticpelagic: feat(linter): Improve linting by validating examples in nested schemas (T424002) [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/11 [07:46:20] (03update) 10kineticpelagic: feat(linter): Improve linting by validating examples in nested schemas (T424002) [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/11 [08:52:22] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Remove buildkit helper image docker/dockerfile-copy from build pipeline - https://phabricator.wikimedia.org/T321316#11987827 (10Jelto) [09:57:58] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-docker-mobileapps02 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [09:58:03] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-docker-mobileapps02 on project deployment-prep - https://phabricator.wikimedia.org/T428239 (10wmcs-alerts) 03NEW [11:36:31] (03update) 10aghirelli: feat(linter): Improve linting by validating examples in nested schemas (T424002) [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/11 (owner: 10kineticpelagic) [12:07:22] 10Phabricator (2026-05-26), 10Phabricator Antivandalism Extension, 06Release-Engineering-Team (Doing 😎): Make it more discoverable that a Phab user account triggered AVA countermeasures - https://phabricator.wikimedia.org/T426972#11988372 (10Aklapper) 05Openβ†’03Resolved [12:30:16] 10Phabricator: Consider stop using Phame - https://phabricator.wikimedia.org/T428225#11988527 (10Aklapper) Hmm, where would be a central place/URI where I'd put a very boring list of changes from Phabricator deployments? I don't feel like this is in scope for Diff. [12:35:54] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Remove buildkit helper image docker/dockerfile-copy from build pipeline - https://phabricator.wikimedia.org/T321316#11988556 (10jnuche) [12:36:12] 06Release-Engineering-Team, 06ServiceOps new, 05FY2025-26 KR 5.1, 06MediaWiki-Platform-Team (Kanban Board), and 3 others: api-gateway: run make test in CI - https://phabricator.wikimedia.org/T424824#11988560 (10daniel) 05Openβ†’03Resolved [12:49:08] 06Release-Engineering-Team, 06ServiceOps new, 05FY2025-26 KR 5.1, 06MediaWiki-Platform-Team (Kanban Board), and 2 others: rest-gateway: run system tests via helm test - https://phabricator.wikimedia.org/T424825#11988588 (10daniel) [12:51:14] 10Phabricator: Consider stop using Phame - https://phabricator.wikimedia.org/T428225#11988596 (10Peachey88) Phame did seem to be the good middle ground, but it also appears to be missing from the default dashboard view now? [12:57:53] 06Release-Engineering-Team, 06ServiceOps new, 05FY2025-26 KR 5.1, 06MediaWiki-Platform-Team (Kanban Board), and 2 others: rest-gateway: run system tests via helm test - https://phabricator.wikimedia.org/T424825#11988648 (10daniel) [13:22:49] 10Phabricator: Consider stop using Phame - https://phabricator.wikimedia.org/T428225#11988774 (10Bugreporter) >>! In T428225#11988526, @Aklapper wrote: > Hmm, where would be a central place/URI where I'd put a very boring list of changes from Phabricator deployments? I don't feel like this is in scope for Diff.... [13:24:38] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-06-05 - 2026-06-26): Write lightweight OCI-image-based Puppet plans for beta cluster - https://phabricator.wikimedia.org/T425585#11988804 (10Gehel) [13:25:33] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-06-05 - 2026-06-26): No Puppet resources found on instance deployment-cirrussearch14 on project deployment-prep - https://phabricator.wikimedia.org/T424100#11988825 (10Gehel) [13:32:21] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-06-05 - 2026-06-26): Migrate beta cluster to OpenSearch 2.x - https://phabricator.wikimedia.org/T421763#11988966 (10Gehel) [13:35:50] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team (Doing 😎): Add PyPy 3.11 to Wikimedia CI - https://phabricator.wikimedia.org/T423607#11989056 (10Xqt) Thanks a lot @hashar [13:43:27] 10Continuous-Integration-Infrastructure, 10Castor, 06Test Platform (Tallinn 27): Speedup mwext-codehealth-master-non-voting Castor job - https://phabricator.wikimedia.org/T427471#11989076 (10Peter) a:05Peterβ†’03None My suggestion: Disable this job for a week. Compare the metrics between the two weeks to h... [13:59:24] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team (Doing 😎): Add PyPy 3.11 to Wikimedia CI - https://phabricator.wikimedia.org/T423607#11989123 (10hashar) @Xqt you are very welcome, and thank you for asking those Python/pypy upgrades! :-] [14:29:30] hi releng folks! [14:30:00] Trying to figure out best practices for dev dbs now defaulting to wanting SSL [14:31:34] does releng have some internal-only CA cert that is used for mariadb TLS, and added to the trusted certs on CI images? [14:34:14] at least one of the tools in the fundraising dev stack is currently incapable of passing extra parameters to its mysql client calls, e.g. ssl-mode = DISABLE [14:34:27] (that's amp, used in building CiviCRM instances) [14:35:34] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services: Update Jenkins hosts from Java 17 to Java 21 - https://phabricator.wikimedia.org/T418109#11989242 (10Dzahn) Hi @jnuche and @hashar I have upgraded the non-active releases hosts to trixie (T418299) and puppet worked without needin... [14:42:53] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T423914#11989267 (10dancy) 05Openβ†’03Resolved [14:47:46] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-docker-mobileapps02 on project deployment-prep - https://phabricator.wikimedia.org/T428239#11989292 (10bd808) `lang=shell-session,counterexample bd808@deployment-docker-mobileapps02:~$ sudo -i puppet agent -tv Info: Using environment... [14:55:02] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services: Update Jenkins hosts from Java 17 to Java 21 - https://phabricator.wikimedia.org/T418109#11989335 (10jnuche) >>! In T418109#11989242, @Dzahn wrote: > Hi @jnuche and @hashar > > I have upgraded the non-active releases hosts to tri... [14:55:42] (03update) 10aghirelli: feat(linter): Improve linting by validating examples in nested schemas (T424002) [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/11 (owner: 10kineticpelagic) [14:55:44] (03approved) 10aghirelli: feat(linter): Improve linting by validating examples in nested schemas (T424002) [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/11 (owner: 10kineticpelagic) [14:56:26] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-docker-mobileapps02 on project deployment-prep - https://phabricator.wikimedia.org/T428239#11989340 (10bd808) `lang=shell-session bd808@deployment-docker-mobileapps02:/var/log$ sudo du -sh *| sort -h | tail -10 28M syslog.27.gz 2... [14:58:34] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-docker-mobileapps02 on project deployment-prep - https://phabricator.wikimedia.org/T428239#11989343 (10bd808) 05Openβ†’03Resolved a:03bd808 `lang=shell-session bd808@deployment-docker-mobileapps02:/var/log$ sudo rm syslog.1 b... [14:59:53] ejegg: Can you point me to the failing job? [15:00:39] dancy: it's not failing in CI yet as we're still on bookworm in CI [15:01:07] (03merge) 10aghirelli: feat(linter): Improve linting by validating examples in nested schemas (T424002) [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/11 (owner: 10kineticpelagic) [15:01:08] but we're upgrading our local docker dev environments to trixie since the fundraising cluster is going to be upgraded in a couple weeks [15:01:43] and the build process (which is the same as in CI) is now failing b/c of the new more secure defaults of the mariadb client [15:02:04] so I figured we should probably solve it locally in a way that will work for CI as well [15:02:23] I see. [15:02:38] as we will soon want to update the CI image to trixie to match local + fundraising cluster [15:03:25] This sounds like a situation that warrants a ticket. [15:03:44] To answer your original question, releng does not have some sort of internal-only CA. [15:04:18] (nor do we want to) [15:04:52] ok, cool. So I guess we want to fix amp to allow it to pass ssl disablement options to its mariadb calls [15:04:59] Agreed. [15:06:07] unfortunately giving the user a my.cnf file doesn't work in this case because amp generates custom my.cnf files for each connection and uses the --defaults-file option to point to those (which makes it ignore the global user my.cnf file) [15:06:42] thanks for the clarification! [15:07:30] Good luck! Let us know where we can assist. [15:12:58] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance deployment-docker-mobileapps02 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [16:29:51] (03update) 10aghirelli: feat(T424210): add enum-description checks for requestBody and response [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/8 (https://phabricator.wikimedia.org/T424210) [16:36:53] 06Gerrit-Privilege-Requests: Request membership in extension-SphinxSearch group for NicJansma - https://phabricator.wikimedia.org/T396496#11989828 (10lcawte) I know this bump is almost a year after this conversation started, but this extension is effectively without a maintainer. It would be great if someone (po... [16:55:47] 10Phabricator: Requests for changes of the automated weekly Phabricator data for Tech News - https://phabricator.wikimedia.org/T428290 (10STei-WMF) 03NEW [17:07:06] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services: Update Jenkins hosts from Java 17 to Java 21 - https://phabricator.wikimedia.org/T418109#11989969 (10Dzahn) Thanks :) Per IRC chat: Upgrade planned for coming Tuesday because releases-jenkins is needed on Monday preparing for the... [17:33:28] (03CR) 10LazyShrey: "CI is green now and the previous inline comment has been addressed in PS2. Could you take another look when you have time?" [fresh] - 10https://gerrit.wikimedia.org/r/1287901 (https://phabricator.wikimedia.org/T424867) (owner: 10LazyShrey) [19:04:29] 06Release-Engineering-Team, 06collaboration-services: install a service on phab1005 - https://phabricator.wikimedia.org/T377889#11990163 (10Dzahn) [19:04:31] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services: Upgrade phab (phorge) hosts to bookworm - https://phabricator.wikimedia.org/T372619#11990164 (10Dzahn) [19:04:48] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services: Upgrade phab (phorge) hosts to bookworm - https://phabricator.wikimedia.org/T372619#11990166 (10Dzahn) [19:04:52] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-devtools, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11990165 (10Dzahn) [19:05:10] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services: Upgrade phab (phorge) hosts to bookworm - https://phabricator.wikimedia.org/T372619#11990167 (10Dzahn) [20:05:44] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 10Development-Metrics: GrimoireLab fails pulling Maniphest transaction data since 20260403 - https://phabricator.wikimedia.org/T428300 (10Aklapper) 03NEW p:05Triageβ†’03High [20:12:11] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 10Development-Metrics: GrimoireLab fails pulling Maniphest transaction data since 20260403 - https://phabricator.wikimedia.org/T428300#11990238 (10Aklapper) In case that there are logs somewhere, then I don't know yet where. [20:14:51] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 10Development-Metrics: GrimoireLab fails pulling Maniphest transaction data since 20260403 - https://phabricator.wikimedia.org/T428300#11990242 (10Aklapper) [20:38:38] 10Phabricator: Requests for changes of the automated weekly Phabricator data for Tech News - https://phabricator.wikimedia.org/T428290#11990306 (10Aklapper) 1A: Maybe we could just exclude all accounts marked as bots as author? That should be adding `AND autr.isSystemAgent = 0` condition in the SQL query at http... [20:38:49] 10Phabricator: Requests for changes of the automated weekly Phabricator data for Tech News - https://phabricator.wikimedia.org/T428290#11990310 (10Aklapper) @Stei-WMF: Would you like to propose a patch for these changes in Gerrit? See https://www.mediawiki.org/wiki/Gerrit/Tutorial , the repository is `operations... [21:52:34] whyisbetabroken.com was about to expire, so I (sadly) renewed it for a couple more years and updated the content to be more accurate [22:15:02] taavi: thank you for your service [22:53:46] 06Release-Engineering-Team (Radar), 06cloud-services-team, 10Cloud-VPS: Magnum cluster stuck in DELETE_FAILED status - https://phabricator.wikimedia.org/T428312 (10dduvall) 03NEW [22:58:06] (03PS1) 10Vaughn Walters: jjb: [catalyst-daily-Echo] Add Echo job [integration/config] - 10https://gerrit.wikimedia.org/r/1298364 (https://phabricator.wikimedia.org/T427007) [23:45:41] 10Scap: scap update-patch can abort and leave /srv/patches in a mess - https://phabricator.wikimedia.org/T428316 (10Reedy) 03NEW