[00:08:52] (03PS1) 10Pwangai: Fix sonar authentication [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/1310210 (https://phabricator.wikimedia.org/T429547) [03:27:57] 06Release-Engineering-Team (Radar), 10QS-Test-Automation, 07Epic: [EPIC] Create Critical User Journey test automation for CI - https://phabricator.wikimedia.org/T422555#12117703 (10Peter) We had similar tasks for performance test in the past where we wanted to test critical user journeys (see for example T37... [03:39:12] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Castor, 06Test Platform, 07Spike: POC: Try to make Castor save async (to remove queue time from job times) - https://phabricator.wikimedia.org/T425411#12117740 (10Peter) a:05Peter→03None [05:07:10] GitLab needs a short maintenance restart [05:15:17] done [06:10:41] (03merge) 10jelto: add repos/sre/etherpad to Trusted Runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/180 [07:53:32] (03approved) 10aghirelli: Update DEVELOPERS.md and the OAD example [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/16 (https://phabricator.wikimedia.org/T422927) (owner: 10kbach) [07:53:40] (03merge) 10aghirelli: Update DEVELOPERS.md and the OAD example [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/16 (https://phabricator.wikimedia.org/T422927) (owner: 10kbach) [08:37:01] !log sudo -u jenkins-deploy rm -fR /srv/castor/castor-mw-ext-and-skins/master/mwext-node24-rundoc/ # run on integration-castor06.integration.eqiad1.wikimedia.cloud [08:37:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:41:04] 10Continuous-Integration-Infrastructure, 10doc.wikimedia.org, 07Software-Licensing: integration.wikimedia.org should have a FOSS license - https://phabricator.wikimedia.org/T431663#12118232 (10hashar) The code we wrote is now placed under GPL2.0 or later The Zuul status page was borrowed from upstream and i... [08:54:06] 10GitLab (Infrastructure), 06collaboration-services: Investigate GitLab backup size change Jun 2026 - https://phabricator.wikimedia.org/T431966#12118253 (10Jelto) 05Open→03Resolved I restored an old backup from bacular and there is a significant size difference in the LFS backup: ` jelto@gitlab1004:~$... [08:58:09] 10Continuous-Integration-Infrastructure, 10Castor, 06Test Platform (Tallinn 27): Speedup mwext-codehealth-master-non-voting Castor job - https://phabricator.wikimedia.org/T427471#12118293 (10Peter) [09:00:35] 06Release-Engineering-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12118300 (10CWilliams-WMF) [09:05:27] 06Release-Engineering-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12118310 (10CWilliams-WMF) [09:23:49] RECOVERY - jenkins_service_running on contint1002 is OK: PROCS OK: 1 process with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [09:26:49] PROBLEM - jenkins_service_running on contint1002 is CRITICAL: PROCS CRITICAL: 0 processes with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [09:41:18] (03CR) 10Kosta Harlan: "@hashar@free.fr this follows up from I381ad6ee86dad50b96ae55c60ad3e8308eb44f3d. Are you open to this approach? It would support testing wi" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1309235 (https://phabricator.wikimedia.org/T342763) (owner: 10Kosta Harlan) [10:26:01] 06Gerrit-Privilege-Requests: Request membership in toollabs-trusted & wmcs-trusted groups for tlepage - https://phabricator.wikimedia.org/T431014#12118546 (10taavi) 05Open→03Resolved a:03taavi [11:08:19] 06Release-Engineering-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12118704 (10cmooney) [11:08:58] 06Release-Engineering-Team, 10Cloud-VPS, 06collaboration-services, 06Data-Persistence, and 8 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12118705 (10taavi) [11:45:36] 06Release-Engineering-Team, 10Cloud-VPS, 06collaboration-services, 06Data-Persistence, and 8 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12118824 (10cmooney) [11:48:59] 06Release-Engineering-Team, 10Cloud-VPS, 06collaboration-services, 06Data-Persistence, and 8 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12118846 (10BTullis) I have fully drained dse-k8s-worker2003 and depooled cirrussearch2110. Good to go from our side. [11:49:02] (03PS2) 10Jforrester: docker(commit-message-validator): Update to v3.0.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1310186 (https://phabricator.wikimedia.org/T431799) (owner: 10BryanDavis) [11:49:09] (03CR) 10Jforrester: [C:03+2] tox: Bump commit-message-validator to 3.0.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1310140 (https://phabricator.wikimedia.org/T431799) (owner: 10BryanDavis) [11:49:31] (03PS3) 10Jforrester: Docker: [commit-message-validator] Update to v3.0.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1310186 (https://phabricator.wikimedia.org/T431799) (owner: 10BryanDavis) [11:49:34] (03CR) 10Jforrester: [C:03+2] Docker: [commit-message-validator] Update to v3.0.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1310186 (https://phabricator.wikimedia.org/T431799) (owner: 10BryanDavis) [11:50:52] (03Merged) 10jenkins-bot: tox: Bump commit-message-validator to 3.0.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1310140 (https://phabricator.wikimedia.org/T431799) (owner: 10BryanDavis) [11:51:24] (03Merged) 10jenkins-bot: Docker: [commit-message-validator] Update to v3.0.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1310186 (https://phabricator.wikimedia.org/T431799) (owner: 10BryanDavis) [11:53:53] !log Docker: [commit-message-validator] Update to v3.0.0, for T431799 [11:53:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:53:56] T431799: Migrate to a newer Python than 3.9 so CI doesn't have a bullseye-based image still. - https://phabricator.wikimedia.org/T431799 [11:54:56] (03PS1) 10Jforrester: jjb: [commit-message-validator] Switch users to 3.0.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1310556 (https://phabricator.wikimedia.org/T431799) [11:55:18] (03CR) 10Jforrester: [C:03+2] jjb: [commit-message-validator] Switch users to 3.0.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1310556 (https://phabricator.wikimedia.org/T431799) (owner: 10Jforrester) [11:55:43] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team (Doing 😎), 10commit-message-validator, 13Patch-For-Review: Migrate to a newer Python than 3.9 so CI doesn't have a bullseye-based image still. - https://phabricator.wikimedia.org/T431799#12118913 (10Jdforrester-WMF) 05In progress→03... [11:56:15] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate all CI jobs from Bullseye to Bookworm or later and drop Bullseye testing support - https://phabricator.wikimedia.org/T383337#12118916 (10Jdforrester-WMF) [11:57:12] (03Merged) 10jenkins-bot: jjb: [commit-message-validator] Switch users to 3.0.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1310556 (https://phabricator.wikimedia.org/T431799) (owner: 10Jforrester) [11:58:55] (03PS2) 10Jforrester: jjb: [catalyst-daily-*] Replace castor-save to stop transient build failures [integration/config] - 10https://gerrit.wikimedia.org/r/1310184 (https://phabricator.wikimedia.org/T432047) (owner: 10Vaughn Walters) [11:59:00] (03CR) 10Jforrester: [C:03+2] jjb: [catalyst-daily-*] Replace castor-save to stop transient build failures [integration/config] - 10https://gerrit.wikimedia.org/r/1310184 (https://phabricator.wikimedia.org/T432047) (owner: 10Vaughn Walters) [12:00:57] (03Merged) 10jenkins-bot: jjb: [catalyst-daily-*] Replace castor-save to stop transient build failures [integration/config] - 10https://gerrit.wikimedia.org/r/1310184 (https://phabricator.wikimedia.org/T432047) (owner: 10Vaughn Walters) [12:43:54] RECOVERY - jenkins_service_running on contint1002 is OK: PROCS OK: 1 process with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [12:46:54] PROBLEM - jenkins_service_running on contint1002 is CRITICAL: PROCS CRITICAL: 0 processes with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [12:55:38] 06Release-Engineering-Team, 10Cloud-VPS, 06collaboration-services, 06Data-Persistence, and 8 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12119177 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=8b130fc6-8cf0-455b-b0d3-cfde4861384c) set by cmooney@cumin... [12:57:32] 06Release-Engineering-Team, 10Cloud-VPS, 06collaboration-services, 06Data-Persistence, and 8 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12119202 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=bdeaaae5-a950-451d-a6e0-365a469fd764) set by cmooney@cumin... [13:48:49] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T432120 (10phaultfinder) 03NEW [13:51:45] 06Release-Engineering-Team, 10Cloud-VPS, 06collaboration-services, 06Data-Persistence, and 8 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12119568 (10cmooney) Upgrade is now complete, all looks healthy. Thanks everyone for their assistance. [13:54:59] RESOLVED: [4x] ProbeDown: Service gerrit2003:29418 has failed probes (tcp_gerrit_ssh_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:59:03] 06Release-Engineering-Team, 06collaboration-services: ProbeDown (gerrit2003) - https://phabricator.wikimedia.org/T432120#12119630 (10Jelto) [13:59:16] 06Release-Engineering-Team, 06collaboration-services: ProbeDown (gerrit2003) - https://phabricator.wikimedia.org/T432120#12119632 (10Jelto) 05Open→03Resolved a:03Jelto Resolved after 5 minutes [14:23:46] 10Continuous-Integration-Infrastructure, 07Code-Health, 07JavaScript, 07Test-Coverage: Generate JavaScript code coverage reports for extensions - https://phabricator.wikimedia.org/T184657#12119842 (10Esanders) 05Declined→03Open Now that ResourceLoader has sourcemaps, and Chrome V8 reports coverage, thi... [14:48:01] (03merge) 10dancy: logstash: Remove logstash_host config option [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1223 (https://phabricator.wikimedia.org/T431635) [14:48:53] (03open) 10dancy: Release 4.274.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1224 [14:51:09] (03merge) 10dancy: Release 4.274.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1224 [15:04:20] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 07Jenkins, and 3 others: Move beta cluster automatic deployment to a dedicated infrastructure - https://phabricator.wikimedia.org/T256168#12120028 (10dancy) @bd808 Seems like this ticket may be ready to... [15:17:43] 06Release-Engineering-Team (Radar), 10QS-Test-Automation, 07Epic: [EPIC] Create Critical User Journey test automation for CI - https://phabricator.wikimedia.org/T422555#12120109 (10bd808) >>! In T422555#12117703, @Peter wrote: > it was really hard to get product to define the journeys. I think we should star... [15:18:05] 10GitLab (Project and group requests), 06Release-Engineering-Team: GitLab Private Repository Request for: paws-private - https://phabricator.wikimedia.org/T432136 (10dcaro) 03NEW [15:24:58] 10GitLab (Project and group requests), 06Release-Engineering-Team: GitLab Private Repository Request for: paws-private - https://phabricator.wikimedia.org/T432136#12120163 (10brennen) 05Open→03Resolved a:03brennen Created: https://gitlab.wikimedia.org/repos/cloud/paws-private > the members of the pa... [15:46:47] 10Continuous-Integration-Config, 10MediaWiki-extensions-ReadingLists, 06Reader Experience Team, 05FY252627 Reading Lists (Graduating from Beta): ReadingLists tests not run in jenkins wmf-quibble-core-vendor-mysql-php81 - https://phabricator.wikimedia.org/T403560#12120301 (10aude) We want to address this wi... [15:52:15] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-devtools, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#12120342 (10Arnoldokoth) Thanks @Dzahn I got this from @brennen https://... [16:01:56] PROBLEM - jenkins_service_running on contint1003 is CRITICAL: PROCS CRITICAL: 2 processes with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [16:02:56] RECOVERY - jenkins_service_running on contint1003 is OK: PROCS OK: 1 process with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [16:06:07] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-devtools, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#12120397 (10Arnoldokoth) I also dumped the the database from `phabricator-... [16:06:23] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-devtools, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#12120399 (10Dzahn) cool! congrats getting it to work! [17:01:04] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 07Jenkins, and 2 others: Move beta cluster automatic deployment to a dedicated infrastructure - https://phabricator.wikimedia.org/T256168#12120667 (10bd808) 05In progress→03Resolved >>! In T2561... [18:23:45] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 13Patch-For-Review, 05Release, 05Train Deployments: 1.47.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T430830#12121110 (10brennen) [18:24:09] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 13Patch-For-Review, 05Release, 05Train Deployments: 1.47.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T430830#12121118 (10brennen) Adding self as backup, since hashar is AFK this week. [20:03:02] 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: install a service on phab1005 - https://phabricator.wikimedia.org/T377889#12121418 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by aokoth@cumin1003 for host phab1005.eqiad.wmnet with OS trixie [20:03:35] 10Gerrit, 10Phabricator: Gerritbot is not commenting on tasks with merged patches - https://phabricator.wikimedia.org/T432168 (10matmarex) 03NEW [20:03:57] 10GerritBot, 10Phabricator: Gerritbot is not commenting on tasks with merged patches - https://phabricator.wikimedia.org/T432168#12121436 (10matmarex) [20:12:02] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10BlueSpice: BlueSpice/HalloWelt force merging to branches and messing up mediawiki/extensions.git - https://phabricator.wikimedia.org/T431845#12121466 (10Reedy) ` fatal: remote error: want 58785c7111cf3a1bd4cca16779a730aba2e59ef6 not valid fatal:... [20:15:11] 10GerritBot, 10Phabricator, 06Release-Engineering-Team, 06collaboration-services: Gerritbot is not commenting on tasks with merged patches - https://phabricator.wikimedia.org/T432168#12121469 (10brennen) [20:23:33] 10GerritBot, 10Phabricator, 06Release-Engineering-Team, 06collaboration-services: Gerritbot is not commenting on tasks with merged patches - https://phabricator.wikimedia.org/T432168#12121500 (10Dzahn) Well... that was quick :/ I just had to rotate the token used by this bot. [20:24:45] 10GerritBot, 10Phabricator, 06Release-Engineering-Team, 06collaboration-services: Gerritbot is not commenting on tasks with merged patches - https://phabricator.wikimedia.org/T432168#12121505 (10Dzahn) a:03Dzahn [20:38:06] 10GitLab (Project and group requests), 06Release-Engineering-Team, 07Essential-Work: GitLab Private Repository Request for: paws-private - https://phabricator.wikimedia.org/T432136#12121547 (10brennen) [20:42:03] 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: install a service on phab1005 - https://phabricator.wikimedia.org/T377889#12121555 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by aokoth@cumin1003 for host phab1005.eqiad.wmnet with OS trixie completed: - phab10... [21:46:10] 10GerritBot, 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Gerritbot is not commenting on tasks with merged patches - https://phabricator.wikimedia.org/T432168#12121858 (10Dzahn) 05Open→03Resolved actually merged something linked to this ticket - and you... [21:54:03] 10Gerrit, 06collaboration-services, 06Infrastructure-Foundations, 06SRE: gerrit.w.o is not included in https://config-master.wikimedia.org/known_hosts - https://phabricator.wikimedia.org/T340947#12121883 (10Dzahn) [21:54:12] 10Gerrit, 06collaboration-services, 06Infrastructure-Foundations, 06SRE: gerrit.w.o is not included in https://config-master.wikimedia.org/known_hosts - https://phabricator.wikimedia.org/T340947#12121884 (10Dzahn) 05Open→03Resolved [22:06:25] 06Release-Engineering-Team (Doing 😎): How MediaWiki code gets deployed and things we are changing - https://phabricator.wikimedia.org/T424780#12121908 (10bd808) 05Open→03Resolved Giving up on the dream of publishing slides. Thanks to everyone who came to the session. [22:44:42] 10Beta-Cluster-Infrastructure, 07Kubernetes: Re-consider setting up a Kubernetes cluster on the Beta cluster - https://phabricator.wikimedia.org/T276650#12122013 (10bd808) p:05Triage→03Medium The topic of deploying MediaWiki with Kubernetes in the Beta Cluster came up today in a Pretrain meeting. @dancy wo...