[00:13:58] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T418120#11643011 (10bd808) I applied this LDIF to create the user and group in our LDAP directory: `lang=ldif,name=T418120-analytics-sre.ldif dn: uid=a... [00:15:28] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [00:17:50] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T418120#11643020 (10bd808) 05Open→03Resolved a:03bd808 `lang=shell-session bd808@deployment-puppetserver-1.deployment-prep.eqiad1:~$ sudo -i... [00:52:29] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T413808#11643096 (10SecurityPatchBot) [01:45:13] (03update) 10thcipriani: JobCard: Format time and date [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1106 [01:45:24] (03update) 10thcipriani: JobCard: Format time and date [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1106 [04:08:26] 06Release-Engineering-Team (Doing), 10MediaWiki-extensions-InterwikiSorting, 05MW-1.46-notes (1.46.0-wmf.16; 2026-02-17), 13Patch-For-Review, and 2 others: Undeploy the InterwikiSorting extension from Wikipedia production - https://phabricator.wikimedia.org/T253764#11643395 (10Legoktm) This feels like... [07:28:26] 06Release-Engineering-Team (Doing 😎), 06MediaWiki-Gerrit-Group-Requests: Update members of wmde-mediawiki Gerrit group (early 2026) - https://phabricator.wikimedia.org/T418177#11643531 (10Aklapper) 05Open→03Resolved a:03Aklapper Done. [07:38:51] 06Release-Engineering-Team (Doing), 10MediaWiki-extensions-InterwikiSorting, 05MW-1.46-notes (1.46.0-wmf.16; 2026-02-17), 13Patch-For-Review, and 2 others: Undeploy the InterwikiSorting extension from Wikipedia production - https://phabricator.wikimedia.org/T253764#11643568 (10Addshore) From my vague m... [08:03:24] 06Release-Engineering-Team, 10Scap: git pull on a scap deployment repository deletes scap sync tags - https://phabricator.wikimedia.org/T418085#11643587 (10hashar) [09:00:00] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit: nft_throttling_denylists triggers NodeTextfileStale alert - https://phabricator.wikimedia.org/T418139#11643784 (10ABran-WMF) [09:02:01] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit: nft_throttling_denylists triggers NodeTextfileStale alert - https://phabricator.wikimedia.org/T418139#11643820 (10ABran-WMF) 05In progress→03Resolved @Dzahn I forgot but @Jelto [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1239... [09:13:31] (03open) 10hashar: Configure repositories to avoind pruning tags by default [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1107 (https://phabricator.wikimedia.org/T418085) [09:18:31] (03close) 10hashar: Configure repositories to avoind pruning tags by default [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1107 (https://phabricator.wikimedia.org/T418085) [09:18:45] 06Release-Engineering-Team (Doing 😎), 06MediaWiki-Gerrit-Group-Requests: Update members of wmde-mediawiki Gerrit group (early 2026) - https://phabricator.wikimedia.org/T418177#11643956 (10WMDE-leszek) Many thanks [09:19:42] (03reopen) 10hashar: Configure repositories to avoind pruning tags by default [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1107 (https://phabricator.wikimedia.org/T418085) [09:25:18] 06Release-Engineering-Team, 10Scap, 13Patch-For-Review: git pull on a scap deployment repository deletes scap sync tags - https://phabricator.wikimedia.org/T418085#11643985 (10hashar) p:05Triage→03Low a:03hashar [09:32:43] Project mediawiki-core-doxygen build #18019: 04FAILURE in 14 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/18019/ [09:42:36] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit: replication monitoring improvement - https://phabricator.wikimedia.org/T418084#11644061 (10ABran-WMF) >>! In T418084#11642858, @hashar wrote: > Another thing I looked at tonight is the delay/latency graph. They show the values for different quan... [09:50:58] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit: replication monitoring improvement - https://phabricator.wikimedia.org/T418084#11644081 (10ABran-WMF) Nothing obvious on neither of these for the Feb. 09 issue: - https://grafana.wikimedia.org/goto/OXBzDOOvg?orgId=1 - https://grafana.wikimed... [09:53:23] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services: Update gerrit replication plugin with new metrics - https://phabricator.wikimedia.org/T418215 (10ABran-WMF) 03NEW [09:54:43] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services: Create alerts using the new replication metrics exposed in Gerrit - https://phabricator.wikimedia.org/T418216 (10ABran-WMF) 03NEW [09:55:04] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services: Create alerts using the new replication metrics exposed in Gerrit - https://phabricator.wikimedia.org/T418216#11644116 (10ABran-WMF) p:05Triage→03Low [09:55:44] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services: Update gerrit replication plugin with new metrics - https://phabricator.wikimedia.org/T418215#11644118 (10ABran-WMF) p:05Triage→03Medium [09:56:19] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit: replication monitoring improvement - https://phabricator.wikimedia.org/T418084#11644121 (10ABran-WMF) 05Open→03Resolved this can be considered done, follow up tasks have been created [10:10:03] maintenance-disconnect-full-disks build 783866 integration-agent-docker-1046 (/: 36%, /srv: 100%, /var/lib/docker: 28%): OFFLINE due to disk space [10:10:24] Could anyone, please, abandon the change https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GlobalWatchlist/+/1221985 with a reason "Continued in Iedf953a2b654f471e8895e97b519a24076decc57"? Thank you. I finally found a way to be an owner of changes a couple of weeks ago, so this is the penultimate time I'm asking this, sorry. [10:13:15] jeez, how can that have 289 patchsets [10:15:03] maintenance-disconnect-full-disks build 783867 integration-agent-docker-1046 (/: 36%, /srv: 56%, /var/lib/docker: 28%): RECOVERY disk space OK [10:16:55] andre: Thanks a lot. Unfortunately, as for now I can't work on Mediawiki Gerrit on local computer from personal reasons, so there is no choice but to compile it there every time. [10:16:57] (03merge) 10aklapper: Uninstall Countdown application [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/95 (https://phabricator.wikimedia.org/T418033) [10:18:03] 10Phabricator (phabricator-next), 06Release-Engineering-Team (Doing 😎): Uninstall Countdown (Phabricator application) - https://phabricator.wikimedia.org/T418033#11644217 (10Aklapper) a:03Aklapper [10:18:11] 10Phabricator (phabricator-next), 06Release-Engineering-Team (Doing 😎): Uninstall Countdown (Phabricator application) - https://phabricator.wikimedia.org/T418033#11644219 (10Aklapper) [10:18:55] (03update) 10aklapper: Remove / disable diffusion.allow-http-auth [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/96 (https://phabricator.wikimedia.org/T418045) [10:19:26] (03merge) 10aklapper: Remove / disable diffusion.allow-http-auth [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/96 (https://phabricator.wikimedia.org/T418045) [10:19:50] 10Diffusion, 10Phabricator (phabricator-next), 06Release-Engineering-Team (Doing 😎): Disable "diffusion.allow-http-auth" - https://phabricator.wikimedia.org/T418045#11644230 (10Aklapper) a:03Aklapper [10:20:00] 10Diffusion, 10Phabricator (phabricator-next), 06Release-Engineering-Team (Doing 😎): Disable "diffusion.allow-http-auth" - https://phabricator.wikimedia.org/T418045#11644234 (10Aklapper) [10:22:29] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): Update to Phorge/Arcanist upstream (2026.xx) - https://phabricator.wikimedia.org/T410849#11644237 (10Aklapper) [10:27:46] 10Beta-Cluster-Infrastructure: "Sender address rejected: Domain not found" for emails sent from the beta cluster - https://phabricator.wikimedia.org/T291679#11644275 (10taavi) >>! In T291679#11642995, @matmarex wrote: > It depends on the recipient address. I can receive the emails at an @gmail.com address (and I... [10:33:28] Yippee, build fixed! [10:33:28] Project mediawiki-core-doxygen build #18020: 09FIXED in 15 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/18020/ [10:35:03] maintenance-disconnect-full-disks build 783871 integration-agent-docker-1054 (/: 52%, /srv: 100%, /var/lib/docker: 27%): OFFLINE due to disk space [10:40:02] maintenance-disconnect-full-disks build 783872 integration-agent-docker-1054 (/: 52%, /srv: 65%, /var/lib/docker: 27%): RECOVERY disk space OK [11:17:45] 06Gerrit-Privilege-Requests, 06Release-Engineering-Team, 06Security-Team, 06SRE, 10SRE-Access-Requests: Request membership in wmf-deployment group for Rsilvola - https://phabricator.wikimedia.org/T418004#11644447 (10IBerker-WMF) As Riku's manager, I approve. [11:33:20] Looks like wikibase-client-php83 has been failing since yesterday afternoon, and it doesn't seem like it has anything to do with changes in Wikibase: https://integration.wikimedia.org/ci/job/wikibase-client-php83/ https://phabricator.wikimedia.org/T418217 [12:15:37] (and Scribunto CI also seems to be failing) [12:34:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-kafka-main-5 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [12:39:28] FIRING: [6x] PuppetAgentNoResources: No Puppet resources found on instance deployment-db11 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [12:44:28] FIRING: [10x] PuppetAgentNoResources: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [12:49:28] FIRING: [16x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief06 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [12:55:28] FIRING: [17x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief06 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [12:59:28] FIRING: [25x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief06 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:04:28] FIRING: [28x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief06 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:19:28] FIRING: [34x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:21:43] FIRING: [36x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:24:28] FIRING: [39x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:26:58] FIRING: [41x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:29:28] FIRING: [43x] PuppetAgentNoResources: No Puppet resources found on instance deployment-acme-chief05 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:43:11] Gerrit question (as anyone here knows Gerrit better than me): How to best delete a git master branch? Feedback welcome in https://phabricator.wikimedia.org/T413890#11644967 - thanks in advance! [14:14:11] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Reimage gerrit2002 - https://phabricator.wikimedia.org/T417247#11645200 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by arnaudb@cumin1003 for host gerrit2002.wikimedia.org with OS bookworm completed: - gerrit2002 (**PASS**) - Do... [14:17:51] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Reimage gerrit2002 - https://phabricator.wikimedia.org/T417247#11645227 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by arnaudb@cumin1003 for host gerrit2002.wikimedia.org with OS bookworm [14:18:36] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Rename gerrit2 unix user to gerrit and assign a fixed uid - https://phabricator.wikimedia.org/T338470#11645229 (10ABran-WMF) 05In progress→03Resolved [14:23:29] 06Release-Engineering-Team, 10Math, 10Mathoid, 10Texvcjs: update mathoid to node22 - https://phabricator.wikimedia.org/T408665#11645279 (10Dreamy_Jazz) [14:26:08] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Reimage gerrit2002 - https://phabricator.wikimedia.org/T417247#11645317 (10ABran-WMF) I've updated the relation chain here: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1242275 We were putting `gerrit2002` back in the `role(gerrit)` puppet group... [14:36:25] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit: replication monitoring improvement - https://phabricator.wikimedia.org/T418084#11645410 (10hashar) 05Resolved→03Open I am reopening because I would like to: - Polish up the [[ https://grafana.wikimedia.org/d/d4a4da73-c27f-4ce6-a9e5-ab84dd7a4... [14:45:01] (03CR) 10Jforrester: [C:03+2] build: Updating ajv to 6.14.0 [performance/fresnel] - 10https://gerrit.wikimedia.org/r/1241894 (owner: 10Libraryupgrader) [15:01:25] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Reimage gerrit2002 - https://phabricator.wikimedia.org/T417247#11645637 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by arnaudb@cumin1003 for host gerrit2002.wikimedia.org with OS bookworm completed: - gerrit2002 (**PASS**) - Do... [15:07:42] 06Release-Engineering-Team, 10Release Pipeline (Blubber): blubber: Pin Python to setuptools 81 for now? - https://phabricator.wikimedia.org/T418253 (10MoritzMuehlenhoff) 03NEW [15:10:19] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Reimage gerrit2002 - https://phabricator.wikimedia.org/T417247#11645737 (10ABran-WMF) sync-instances cookbook running [15:19:26] 06Release-Engineering-Team, 10ChangeProp, 06Data-Engineering, 10EventStreams, and 16 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11645938 (10Dreamy_Jazz) [15:20:12] 06Release-Engineering-Team, 10ChangeProp, 06Data-Engineering, 10EventStreams, and 15 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11645944 (10Dreamy_Jazz) Per {T416623}, #ipoid-service will soon be undeployed. #similarusers is archived so I think t... [15:20:20] 06Release-Engineering-Team, 10ChangeProp, 06Data-Engineering, 10EventStreams, and 15 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11645946 (10Dreamy_Jazz) [15:24:00] 06Release-Engineering-Team, 10ChangeProp, 06Data-Engineering, 10EventStreams, and 15 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11645965 (10brouberol) [15:28:44] 06Release-Engineering-Team, 10dev-images, 10Catalyst (Radar), 13Patch-For-Review: dev-images: bookworm-php-sury children fail to build on sury key - https://phabricator.wikimedia.org/T417711#11645990 (10dancy) 05In progress→03Resolved `bookworm-php-sury` and its dependent images have been updated. [15:33:51] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services: Deploy Phab/Phorge 2026-02-04 - https://phabricator.wikimedia.org/T418256 (10brennen) 03NEW [15:34:13] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services: Deploy Phab/Phorge 2026-02-24 - https://phabricator.wikimedia.org/T418256#11646046 (10brennen) [15:36:25] (03open) 10brennen: update phabricator submodule for 2026-02-24 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/97 (https://phabricator.wikimedia.org/T418256) [15:41:21] 10MediaWiki-Releasing, 10AntiSpoof, 06Product Safety and Integrity: Bundle AntiSpoof extension with MediaWiki - https://phabricator.wikimedia.org/T191736#11646199 (10Dreamy_Jazz) [15:42:18] 10Beta-Cluster-Infrastructure, 10CheckUser, 06Product Safety and Integrity, 06Trust-and-Safety, 06WMF-Legal: Install CheckUser in Beta Cluster - https://phabricator.wikimedia.org/T357959#11646213 (10Dreamy_Jazz) [15:46:58] 06Gerrit-Privilege-Requests, 06Release-Engineering-Team, 06Security-Team, 06SRE, 10SRE-Access-Requests: Request membership in deployment (and wmf-deployment group) for Rsilvola - https://phabricator.wikimedia.org/T418004#11646285 (10Dzahn) [15:47:32] 06Gerrit-Privilege-Requests, 06Release-Engineering-Team, 06Security-Team, 06SRE, 10SRE-Access-Requests: Request membership in deployment (and wmf-deployment group) for Rsilvola - https://phabricator.wikimedia.org/T418004#11646296 (10Dzahn) [15:52:29] 10Phabricator: Reset MFA for EMcFarland-WMF on Phabricator - https://phabricator.wikimedia.org/T418260 (10Urbanecm_WMF) 03NEW [15:53:09] 10Phabricator: Reset MFA for EMcFarland-WMF on Phabricator - https://phabricator.wikimedia.org/T418260#11646385 (10Urbanecm_WMF) 05Open→03Resolved Resolving. [15:54:05] 06Gerrit-Privilege-Requests, 06Release-Engineering-Team, 06Security-Team, 06SRE, 10SRE-Access-Requests: Request membership in deployment (and wmf-deployment group) for Rsilvola - https://phabricator.wikimedia.org/T418004#11646397 (10Dzahn) [15:54:35] 06Gerrit-Privilege-Requests, 06Release-Engineering-Team, 06Security-Team, 06SRE, 10SRE-Access-Requests: Request membership in deployment (and wmf-deployment group) for Rsilvola - https://phabricator.wikimedia.org/T418004#11646421 (10Dzahn) [15:54:51] 06Gerrit-Privilege-Requests, 06Release-Engineering-Team, 06Security-Team, 06SRE, 10SRE-Access-Requests: Request membership in deployment (and wmf-deployment group) for Rsilvola - https://phabricator.wikimedia.org/T418004#11646432 (10Dzahn) Thanks. Most things here are done. The SSH key needs to be verifi... [15:55:39] !log devtools: test deploy phab/phorge to test instance (T418256) [15:55:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:55:42] T418256: Deploy Phab/Phorge 2026-02-24 - https://phabricator.wikimedia.org/T418256 [15:56:16] 06Gerrit-Privilege-Requests, 06Release-Engineering-Team, 06Security-Team, 06SRE, 10SRE-Access-Requests: Request membership in deployment (and wmf-deployment group) for Rsilvola - https://phabricator.wikimedia.org/T418004#11646452 (10Rsilvola) Thank you! At the moment, I only expect to do occasional depl... [15:56:36] 10Phabricator, 06Community-Tech, 06MediaWiki-Platform-Team, 06Product Safety and Integrity (Sprint Flower (Feb 9 - Feb 27)): Potential changes to team-tagging Herald rules as a result of recent code-steward updates (Feb 2026) - https://phabricator.wikimedia.org/T417936#11646456 (10A_smart_kitten) Follo... [15:57:05] (03merge) 10brennen: update phabricator submodule for 2026-02-24 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/97 (https://phabricator.wikimedia.org/T418256) [15:57:21] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Reimage gerrit2002 - https://phabricator.wikimedia.org/T417247#11646495 (10ABran-WMF) replication has been warmed up, @Dzahn will handle the last step: ` sudo cookbook sre.gerrit.restart-gerrit -t T417247 --host gerrit2003 ` because gerrit replication... [15:58:45] 10Gerrit, 06collaboration-services: Fix gerrit-restart cookbook - https://phabricator.wikimedia.org/T418264 (10ABran-WMF) 03NEW [15:59:01] 10Phabricator, 06Community-Tech, 06MediaWiki-Platform-Team, 06Product Safety and Integrity (Sprint Flower (Feb 9 - Feb 27)): Potential changes to team-tagging Herald rules as a result of recent code-steward updates (Feb 2026) - https://phabricator.wikimedia.org/T417936#11646550 (10kostajh) Updated! [15:59:15] 10Gerrit, 06collaboration-services: Fix gerrit-restart cookbook - https://phabricator.wikimedia.org/T418264#11646554 (10ABran-WMF) p:05Triage→03Medium [16:04:11] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Reimage gerrit2002 - https://phabricator.wikimedia.org/T417247#11646599 (10ABran-WMF) 05In progress→03Resolved `gerrit2002` is now `gerrit-spare` https://grafana.wikimedia.org/goto/Zoho3OOvg?orgId=1, replication is catching up [16:05:57] 10Continuous-Integration-Config, 10ReaderExperiments: (CI) MobileFrontend test failures when used as a dep for ReaderExperiments - https://phabricator.wikimedia.org/T417818#11646623 (10A_smart_kitten) >>! In T417818#11630050, @bvibber wrote: > may be a dupe of T415451 Yeah, seems like the same test failures f... [16:11:21] 10Diffusion, 10Phabricator (2026-02-24), 06Release-Engineering-Team (Doing 😎): Disable "diffusion.allow-http-auth" - https://phabricator.wikimedia.org/T418045#11646712 (10Aklapper) [16:12:32] 06Release-Engineering-Team (Doing), 10MediaWiki-extensions-InterwikiSorting, 05MW-1.46-notes (1.46.0-wmf.16; 2026-02-17), 13Patch-For-Review, and 2 others: Undeploy the InterwikiSorting extension from Wikipedia production - https://phabricator.wikimedia.org/T253764#11646718 (10Ladsgroup) >>! In T253764... [16:13:48] 10Phabricator (2026-02-24), 06Release-Engineering-Team (Doing 😎): Uninstall Countdown (Phabricator application) - https://phabricator.wikimedia.org/T418033#11646732 (10Aklapper) [16:14:02] 10Phabricator (2026-02-24), 06Release-Engineering-Team (Doing 😎): Uninstall Countdown (Phabricator application) - https://phabricator.wikimedia.org/T418033#11646740 (10Aklapper) 05Open→03Resolved Done. [16:14:15] 10Diffusion, 10Phabricator (2026-02-24), 06Release-Engineering-Team (Doing 😎): Disable "diffusion.allow-http-auth" - https://phabricator.wikimedia.org/T418045#11646748 (10Aklapper) 05Open→03Resolved Config file change just got deployed as part of T418256; I now also changed our custom config value to... [16:14:37] 10Phabricator (2026-02-24), 06Release-Engineering-Team (Doing 😎): Hide unneeded "SSH Public Keys" personal settings panel - https://phabricator.wikimedia.org/T418044#11646755 (10Aklapper) 05Open→03Resolved Confirming that our deployment in T418256 hid that panel [16:18:14] 10Phabricator (2026-02-24), 06Release-Engineering-Team (Doing 😎), 06collaboration-services, 13Patch-For-Review: Deploy Phab/Phorge 2026-02-24 - https://phabricator.wikimedia.org/T418256#11646795 (10Aklapper) 05Open→03Resolved a:03brennen * {T418044} via https://gitlab.wikimedia.org/repos/phabrica... [16:18:38] 10Phabricator, 06Community-Tech, 06MediaWiki-Platform-Team, 06Product Safety and Integrity (Sprint Flower (Feb 9 - Feb 27)): Potential changes to team-tagging Herald rules as a result of recent code-steward updates (Feb 2026) - https://phabricator.wikimedia.org/T417936#11646807 (10A_smart_kitten) Thanks! [16:22:06] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Rename gerrit2 unix user to gerrit and assign a fixed uid - https://phabricator.wikimedia.org/T338470#11646832 (10hashar) 05Resolved→03Open There are still a bunch of occurences for `gerrit2` in Puppet, notably the back... [17:06:59] 06Gerrit-Privilege-Requests, 06Release-Engineering-Team, 06Security-Team, 06SRE, 10SRE-Access-Requests: Request membership in deployment (and wmf-deployment group) for Rsilvola - https://phabricator.wikimedia.org/T418004#11647253 (10Dzahn) @Rsilvola Gotcha! We just need to verify it's really you and your... [17:19:34] 06Release-Engineering-Team (Doing), 10MediaWiki-extensions-InterwikiSorting, 05MW-1.46-notes (1.46.0-wmf.16; 2026-02-17), 13Patch-For-Review, and 2 others: Undeploy the InterwikiSorting extension from Wikipedia production - https://phabricator.wikimedia.org/T253764#11647335 (10Iniquity) >>! In T253764#... [17:21:25] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Reimage gerrit2002 - https://phabricator.wikimedia.org/T417247#11647343 (10jcrespo) Backups from gerrit2002 are failing with: ` Could not stat "/var/lib/gerrit": ERR=No such file or directory ` Same for: gerrit2003.wikimedia.org-Hourly-Tue-ReposEq... [17:21:37] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Reimage gerrit2002 - https://phabricator.wikimedia.org/T417247#11647345 (10jcrespo) 05Resolved→03Open [17:30:54] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Reimage gerrit2002 - https://phabricator.wikimedia.org/T417247#11647397 (10Dzahn) It would make sense if it was "/var/lib/gerrit**2**: No such file or directory". ` [gerrit2002:/var/lib] $ cd /var/lib/gerrit2 -bash: cd: /var/lib/gerrit2: No such file... [17:33:28] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Reimage gerrit2002 - https://phabricator.wikimedia.org/T417247#11647408 (10Dzahn) same on gerrit2003. `/var/lib/gerrit2` is gone, `/var/lib/gerrit` is not. the exact opposite of what that error message seems to say. [17:34:14] (03open) 10dancy: fab: Minimize ssh calls [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/95 [17:34:16] (03update) 10dancy: fab: Minimize ssh calls [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/95 [17:34:57] (03update) 10dancy: fab: Minimize ssh calls [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/95 [17:35:00] (03update) 10dancy: fab: Minimize ssh calls [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/95 [17:36:38] (03update) 10dancy: fab: Minimize ssh calls [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/95 [18:07:00] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Reimage gerrit2002 - https://phabricator.wikimedia.org/T417247#11647660 (10jcrespo) Then that message may be misleading and not the cause of the issues, but the error is real: `Terminated Jobs: JobId Level Files Bytes Status Finished... [18:11:13] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Reimage gerrit2002 - https://phabricator.wikimedia.org/T417247#11647679 (10jcrespo) This is the real error: ` 24-Feb 15:00 backup1014.eqiad.wmnet JobId 684390: Start Backup JobId 684390, Job=gerrit2002.wikimedia.org-Hourly-Tue-ReposEqiad-gerrit-repo-da... [18:12:16] 10Continuous-Integration-Config, 06Release-Engineering-Team (Priority Backlog 📥), 10Browser Test Platform, 10Wikibase (3rd party installations), and 3 others: Move some Wikibase selenium tests to a standalone job - https://phabricator.wikimedia.org/T287582#11647683 (10zeljkofilipin) [18:25:20] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Reimage gerrit2002 - https://phabricator.wikimedia.org/T417247#11647749 (10Dzahn) In that case.. I would think it only happened because it attempted to connect while the host was being reimaged. current status of bacula-fd looks normal: ` [gerrit2002... [18:47:33] 06Gerrit-Privilege-Requests, 06Release-Engineering-Team, 06Security-Team, 06SRE, 10SRE-Access-Requests: Request membership in deployment (and wmf-deployment group) for Rsilvola - https://phabricator.wikimedia.org/T418004#11647908 (10Dzahn) [18:47:42] 06Gerrit-Privilege-Requests, 06Release-Engineering-Team, 06Security-Team, 06SRE, 10SRE-Access-Requests: Request membership in deployment (and wmf-deployment group) for Rsilvola - https://phabricator.wikimedia.org/T418004#11647909 (10Dzahn) Thanks! SSH key confirmed out-of-band. [20:46:18] (03update) 10dduvall: digitalocean: Separate management of cluster and in-cluster resources [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/552 (https://phabricator.wikimedia.org/T416260) [21:30:22] 06Release-Engineering-Team (Doing), 10MediaWiki-extensions-InterwikiSorting, 05MW-1.46-notes (1.46.0-wmf.16; 2026-02-17), 13Patch-For-Review, and 2 others: Undeploy the InterwikiSorting extension from Wikipedia production - https://phabricator.wikimedia.org/T253764#11648668 (10Johannnes89) >>! In T2537... [21:55:49] 06Release-Engineering-Team, 10Quibble, 13Patch-For-Review: Quibble should emit a report of each time it took to run the steps/stages - https://phabricator.wikimedia.org/T417399#11648716 (10thcipriani) Aside: @ArthurTaylor unsure what you're aiming to do, but in case it's useful, I gather some of this data to... [21:56:02] (03update) 10dduvall: digitalocean: Separate management of cluster and in-cluster resources [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/552 (https://phabricator.wikimedia.org/T416260) [22:03:23] (03PS11) 10Hashar: Report Chronometer durations at end of plan [integration/quibble] - 10https://gerrit.wikimedia.org/r/1235326 (https://phabricator.wikimedia.org/T417399) [22:09:11] (03CR) 10Hashar: [C:04-1] "I think I want to use named tuples instead:" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1235315 (https://phabricator.wikimedia.org/T417399) (owner: 10Hashar) [22:35:06] Project beta-code-update-eqiad build #589073: 04FAILURE in 2 min 6 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/589073/ [22:38:37] (03approved) 10dancy: digitalocean: Separate management of cluster and in-cluster resources [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/552 (https://phabricator.wikimedia.org/T416260) (owner: 10dduvall) [22:39:57] fatal: unable to access 'https://gerrit.wikimedia.org/r/mediawiki/vendor/': The requested URL returned error: 503 [22:40:37] that doesn't sound right... is Jenkins getting rate limited talking to gerrit? [22:44:13] Yippee, build fixed! [22:44:13] Project beta-code-update-eqiad build #589074: 09FIXED in 2 min 11 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/589074/ [22:44:25] sunspots I guess [23:19:49] (03update) 10dancy: fab: Minimize ssh calls [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/95 [23:20:15] (03update) 10dancy: fab: Minimize ssh calls [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/95 [23:21:54] (03update) 10dancy: fab: Minimize ssh calls [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/95 [23:22:17] (03update) 10dancy: fab: Minimize ssh calls [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/95 [23:30:02] maintenance-disconnect-full-disks build 784026 integration-agent-docker-1063 (/: 25%, /srv: 97%, /var/lib/docker: 33%): OFFLINE due to disk space [23:35:02] maintenance-disconnect-full-disks build 784027 integration-agent-docker-1063 (/: 25%, /srv: 50%, /var/lib/docker: 33%): RECOVERY disk space OK [23:55:51] (03update) 10dancy: fab: Minimize ssh calls [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/95