[08:04:02] 10MediaWiki-Releasing, 06translatewiki.net, 06LPL Essential (2025 Oct-Dec), 10LPL Projects (Other), and 2 others: Configure Translatewiki.net for REL1_45 translation backports - https://phabricator.wikimedia.org/T408469#11334346 (10abi_) 05Open→03In progress [09:00:35] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/AutoCreatePage] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1200654 (owner: 10SomeRandomDeveloper) [09:00:46] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/EmbedSpotify] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1200655 (owner: 10SomeRandomDeveloper) [09:00:58] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/JavascriptSlideshow] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1200656 (owner: 10SomeRandomDeveloper) [09:01:09] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/MachineTranslation] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1200657 (owner: 10SomeRandomDeveloper) [09:02:12] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/AutoCreatePage] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1200654 (owner: 10SomeRandomDeveloper) [09:02:21] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/EmbedSpotify] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1200655 (owner: 10SomeRandomDeveloper) [09:02:37] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/JavascriptSlideshow] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1200656 (owner: 10SomeRandomDeveloper) [09:02:44] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/MachineTranslation] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1200657 (owner: 10SomeRandomDeveloper) [09:16:26] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 06Data-Persistence: Occasional database errors when using/browsing Phabricator - https://phabricator.wikimedia.org/T408956#11334472 (10Marostegui) I am checking the proxies and I am seeing errors like: ` syslog.1:2025-11-02T21:05:27.96837... [09:19:19] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 06Data-Persistence: Occasional database errors when using/browsing Phabricator - https://phabricator.wikimedia.org/T408956#11334477 (10Marostegui) Sorry, hit enter too fast above. There seems to be occasional bursts of connections to our... [09:47:34] 06Project-Admins, 10PendingChangesBot: PendingChangesBot has two issue trackers - https://phabricator.wikimedia.org/T406300#11334626 (10Aklapper) 05Open→03Declined So far we've interpreted `#affects-foo` as an issue //in another codebase than foo`. I'm okay'ish with the comment, thus I'm going to decli... [10:25:59] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 06Data-Persistence: Occasional database errors when using/browsing Phabricator - https://phabricator.wikimedia.org/T408956#11334882 (10Marostegui) I am going to switch to the stand by proxy, just to discard issues dbproxy1026 itself. [10:38:45] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 06Data-Persistence, 13Patch-For-Review: Occasional database errors when using/browsing Phabricator - https://phabricator.wikimedia.org/T408956#11334904 (10LSobanski) I can't see anything obvious in Superset. [10:45:07] (03PS1) 10Bartosz Wójtowicz: inference-services: Add CI stage for Revise Tone Task Generator. [integration/config] - 10https://gerrit.wikimedia.org/r/1201019 [10:46:49] (03CR) 10CI reject: [V:04-1] inference-services: Add CI stage for Revise Tone Task Generator. [integration/config] - 10https://gerrit.wikimedia.org/r/1201019 (owner: 10Bartosz Wójtowicz) [10:51:27] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 06Data-Persistence, 13Patch-For-Review: Occasional database errors when using/browsing Phabricator - https://phabricator.wikimedia.org/T408956#11334962 (10Marostegui) Most of the traffic is now going through dbproxy1028. I will keep an... [10:52:41] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 06Data-Persistence, 13Patch-For-Review: Occasional database errors when using/browsing Phabricator - https://phabricator.wikimedia.org/T408956#11334973 (10Marostegui) Woops, there we go: ` Nov 03 10:51:55 dbproxy1028 haproxy[751121]: Co... [10:55:56] (03PS2) 10Bartosz Wójtowicz: inference-services: Add CI stage for Revise Tone Task Generator. [integration/config] - 10https://gerrit.wikimedia.org/r/1201019 (https://phabricator.wikimedia.org/T408538) [11:39:30] (03CR) 10AikoChou: [C:03+1] inference-services: Add CI stage for Revise Tone Task Generator. [integration/config] - 10https://gerrit.wikimedia.org/r/1201019 (https://phabricator.wikimedia.org/T408538) (owner: 10Bartosz Wójtowicz) [11:59:13] 10Gerrit, 06Diffusion-Repository-Administrators, 10Projects-Cleanup, 10Wikimedia-GitHub: Archive the operations/software/transferpy repository and migrate phabricator and github mirrors - https://phabricator.wikimedia.org/T409062 (10jcrespo) 03NEW [12:05:03] maintenance-disconnect-full-disks build 751364 integration-agent-docker-1040 (/: 36%, /srv: 98%, /var/lib/docker: 23%): OFFLINE due to disk space [12:10:03] maintenance-disconnect-full-disks build 751365 integration-agent-docker-1040 (/: 36%, /srv: 56%, /var/lib/docker: 24%): RECOVERY disk space OK [12:18:04] 10Gerrit, 06Diffusion-Repository-Administrators, 10Projects-Cleanup, 10Wikimedia-GitHub: Archive the operations/software/transferpy repository and migrate phabricator and github mirrors - https://phabricator.wikimedia.org/T409062#11335283 (10jcrespo) [12:19:12] 10Gerrit, 06Diffusion-Repository-Administrators, 10Projects-Cleanup, 10Wikimedia-GitHub: Archive the operations/software/transferpy repository and migrate phabricator and github mirrors - https://phabricator.wikimedia.org/T409062#11335285 (10jcrespo) [12:30:25] 10Gerrit, 10Projects-Cleanup, 10Wikimedia-GitHub: Archive the operations/software/transferpy repository and migrate phabricator and github mirrors - https://phabricator.wikimedia.org/T409062#11335326 (10Aklapper) [12:31:25] 10Gerrit, 10Projects-Cleanup, 10Wikimedia-GitHub: Archive the operations/software/transferpy repository and migrate phabricator and github mirrors - https://phabricator.wikimedia.org/T409062#11335332 (10jcrespo) [12:31:44] 10Gerrit, 10Projects-Cleanup, 10Wikimedia-GitHub: Archive the operations/software/transferpy repository and migrate phabricator and github mirrors - https://phabricator.wikimedia.org/T409062#11335334 (10jcrespo) [12:38:35] 10Gerrit, 10Projects-Cleanup, 10Wikimedia-GitHub: Archive the operations/software/transferpy repository and migrate phabricator and github mirrors - https://phabricator.wikimedia.org/T409062#11335347 (10jcrespo) [13:30:03] maintenance-disconnect-full-disks build 751381 integration-agent-docker-1051 (/: 36%, /srv: 97%, /var/lib/docker: 24%): OFFLINE due to disk space [13:30:48] 10Scap, 07Sustainability (Incident Followup): scap only one version - https://phabricator.wikimedia.org/T100575#11335488 (10LSobanski) Is this still relevant in the age of SpiderPig? [13:33:45] 06Release-Engineering-Team (Seen), 10Scap, 10WMF-JobQueue, 07Sustainability (Incident Followup): Add a jobrunner server to the Scap canary pool - https://phabricator.wikimedia.org/T172480#11335493 (10LSobanski) Is this still relevant with the new K8s infrastructure? [13:35:03] maintenance-disconnect-full-disks build 751382 integration-agent-docker-1051 (/: 36%, /srv: 43%, /var/lib/docker: 24%): RECOVERY disk space OK [14:06:22] (03CR) 10Hashar: [C:03+2] "Thanks! I have deployed the jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/1201019 (https://phabricator.wikimedia.org/T408538) (owner: 10Bartosz Wójtowicz) [14:08:15] (03Merged) 10jenkins-bot: inference-services: Add CI stage for Revise Tone Task Generator. [integration/config] - 10https://gerrit.wikimedia.org/r/1201019 (https://phabricator.wikimedia.org/T408538) (owner: 10Bartosz Wójtowicz) [14:10:57] (03PS2) 10Hashar: jjb/zuul: use Python 3.9 instead of 3.8 for deeptest [integration/config] - 10https://gerrit.wikimedia.org/r/1200389 (https://phabricator.wikimedia.org/T401802) (owner: 10Xqt) [14:11:35] (03CR) 10Hashar: [C:03+2] "INFO:jenkins_jobs.builder:Number of jobs generated: 1" [integration/config] - 10https://gerrit.wikimedia.org/r/1200389 (https://phabricator.wikimedia.org/T401802) (owner: 10Xqt) [14:13:18] (03Merged) 10jenkins-bot: jjb/zuul: use Python 3.9 instead of 3.8 for deeptest [integration/config] - 10https://gerrit.wikimedia.org/r/1200389 (https://phabricator.wikimedia.org/T401802) (owner: 10Xqt) [14:19:51] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/PageViewInfo] Add Scribunto phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1199736 (https://phabricator.wikimedia.org/T362937) (owner: 10SD0001) [14:19:52] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: The 'Other Assignee' task field is unsearchable using the web UI - https://phabricator.wikimedia.org/T408524#11335737 (10Aklapper) [14:22:20] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/PageViewInfo] Add Scribunto phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1199736 (https://phabricator.wikimedia.org/T362937) (owner: 10SD0001) [14:48:15] 10MediaWiki-Releasing, 06translatewiki.net, 06LPL Essential (2025 Oct-Dec), 10LPL Projects (Other), 05MW-1.45-release: Configure Translatewiki.net for REL1_45 translation backports - https://phabricator.wikimedia.org/T408469#11335928 (10abi_) Change deployed. Backports should happen on Thursday (06 Nov,... [14:59:13] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Kulupu), 07Essential-Work: Conflicts not detected when editing a wiki - https://phabricator.wikimedia.org/T406211#11335963 (10thcipriani) [15:07:40] (03approved) 10jnuche: scap remove-patch: support removing multiple patches [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1012 (https://phabricator.wikimedia.org/T404593) (owner: 10dancy) [15:43:02] 10Scap, 07Sustainability (Incident Followup): scap only one version - https://phabricator.wikimedia.org/T100575#11336136 (10dancy) 05Open→03Declined >>! In T100575#11335488, @LSobanski wrote: > Is this still relevant in the age of SpiderPig? I'm going to boldly say no. Since this ticket was filed in... [15:53:19] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work, 13Patch-For-Review: Set cpu limits on gitlab-cloud-runner workloads - https://phabricator.wikimedia.org/T408740#11336184 (10dancy) 05Open→03In progress p:05Triage→03Low a:03dancy [16:02:46] (03open) 10dancy: chart: Separate registry and nginx resources requests/limits [repos/releng/docker-hub-mirror] - 10https://gitlab.wikimedia.org/repos/releng/docker-hub-mirror/-/merge_requests/36 (https://phabricator.wikimedia.org/T408740) [16:02:48] (03update) 10dancy: chart: Separate registry and nginx resources requests/limits [repos/releng/docker-hub-mirror] - 10https://gitlab.wikimedia.org/repos/releng/docker-hub-mirror/-/merge_requests/36 (https://phabricator.wikimedia.org/T408740) [16:06:26] (03open) 10dancy: .pipeline/blubber.yaml: bullseye -> bookworm [repos/releng/docker-hub-mirror] - 10https://gitlab.wikimedia.org/repos/releng/docker-hub-mirror/-/merge_requests/37 [16:06:29] (03update) 10dancy: .pipeline/blubber.yaml: bullseye -> bookworm [repos/releng/docker-hub-mirror] - 10https://gitlab.wikimedia.org/repos/releng/docker-hub-mirror/-/merge_requests/37 [16:08:26] !log Unblock 124.168.248.0/21 (T409001) [16:08:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:08:58] (03merge) 10dancy: .pipeline/blubber.yaml: bullseye -> bookworm [repos/releng/docker-hub-mirror] - 10https://gitlab.wikimedia.org/repos/releng/docker-hub-mirror/-/merge_requests/37 [16:09:31] (03update) 10dancy: chart: Separate registry and nginx resources requests/limits [repos/releng/docker-hub-mirror] - 10https://gitlab.wikimedia.org/repos/releng/docker-hub-mirror/-/merge_requests/36 (https://phabricator.wikimedia.org/T408740) [16:09:54] (03merge) 10dancy: chart: Separate registry and nginx resources requests/limits [repos/releng/docker-hub-mirror] - 10https://gitlab.wikimedia.org/repos/releng/docker-hub-mirror/-/merge_requests/36 (https://phabricator.wikimedia.org/T408740) [16:19:14] 10Beta-Cluster-Infrastructure, 07Chinese-Sites: Request interface admin and admin rights for PexEric on zh.wikipedia.beta.wmcloud.org - https://phabricator.wikimedia.org/T408685#11336422 (10bd808) 05Open→03Resolved a:03bd808 https://zh.wikipedia.beta.wmcloud.org/w/index.php?title=Special:Log&logid=11... [16:57:33] 10Gerrit, 07Upstream: Gerrit fudges multiple suggestions in comments - https://phabricator.wikimedia.org/T405625#11336654 (10Paladox) 3.10.9 is out with the fix [16:58:28] 10Gerrit, 07Upstream: Gerrit does not send email when a new patchset is created using the web interface - https://phabricator.wikimedia.org/T343471#11336658 (10Paladox) 3.10.9 is out with the fix [17:33:38] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T408271#11336886 (10Jdlrobson-WMF) [18:10:03] maintenance-disconnect-full-disks build 751437 integration-agent-docker-1040 (/: 36%, /srv: 97%, /var/lib/docker: 24%): OFFLINE due to disk space [18:15:03] maintenance-disconnect-full-disks build 751438 integration-agent-docker-1040 (/: 36%, /srv: 48%, /var/lib/docker: 23%): RECOVERY disk space OK [19:33:35] (03open) 10dancy: docker-hub-mirror: Separate nginx and registry resources [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/525 (https://phabricator.wikimedia.org/T408740) [19:33:39] (03update) 10dancy: docker-hub-mirror: Separate nginx and registry resources [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/525 (https://phabricator.wikimedia.org/T408740) [19:35:00] (03merge) 10dancy: docker-hub-mirror: Separate nginx and registry resources [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/525 (https://phabricator.wikimedia.org/T408740) [19:49:19] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikiEditor/+/1200640 seems stuck, any ideas why? [19:49:23] the patch it depends on merged already [19:50:55] It's moving now [19:51:13] ah, ok. I just had to ask about it for it to start moving :) [19:51:47] Exactly [19:51:57] I guess it was waiting for the wmf.25 backport to finish https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ConfirmEdit/+/1201185 [20:08:38] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work, 13Patch-For-Review: Set cpu limits on gitlab-cloud-runner workloads - https://phabricator.wikimedia.org/T408740#11337521 (10dancy) [20:27:11] 10MediaWiki-Releasing, 10MediaWiki-Vendor, 05MW-1.45-release: Prune /vendor for REL1_45 - https://phabricator.wikimedia.org/T408470#11337663 (10ABreault-WMF) a:03ABreault-WMF [20:29:42] (03open) 10dancy: Set resource requests/limits on gitlab-runner manager pods [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/526 (https://phabricator.wikimedia.org/T408740) [20:29:44] (03update) 10dancy: Set resource requests/limits on gitlab-runner manager pods [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/526 (https://phabricator.wikimedia.org/T408740) [20:34:00] (03update) 10dancy: Set resource requests/limits on gitlab-runner manager pods [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/526 (https://phabricator.wikimedia.org/T408740) [20:40:07] (03update) 10dancy: Set resource requests/limits on gitlab-runner manager pods [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/526 (https://phabricator.wikimedia.org/T408740) [20:41:21] (03merge) 10dancy: Set resource requests/limits on gitlab-runner manager pods [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/526 (https://phabricator.wikimedia.org/T408740) [20:47:29] !log gitlab-cloud-runner runners restarted (T408740) [20:47:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:47:32] T408740: Set cpu limits on gitlab-cloud-runner workloads - https://phabricator.wikimedia.org/T408740 [20:54:34] (03PS2) 10Brennen Bearnes: WIP: Parallel runs: Add a group count [integration/quibble] - 10https://gerrit.wikimedia.org/r/1200415 (https://phabricator.wikimedia.org/T384517) [20:57:23] (03CR) 10CI reject: [V:04-1] WIP: Parallel runs: Add a group count [integration/quibble] - 10https://gerrit.wikimedia.org/r/1200415 (https://phabricator.wikimedia.org/T384517) (owner: 10Brennen Bearnes) [21:04:22] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T408271#11337755 (10Jdlrobson-WMF) [21:08:31] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work, 13Patch-For-Review: Set cpu limits on gitlab-cloud-runner workloads - https://phabricator.wikimedia.org/T408740#11337793 (10dancy) [21:13:33] (03PS3) 10Brennen Bearnes: WIP: Parallel runs: Add a group count [integration/quibble] - 10https://gerrit.wikimedia.org/r/1200415 (https://phabricator.wikimedia.org/T384517) [21:32:00] (03CR) 10CI reject: [V:04-1] WIP: Parallel runs: Add a group count [integration/quibble] - 10https://gerrit.wikimedia.org/r/1200415 (https://phabricator.wikimedia.org/T384517) (owner: 10Brennen Bearnes) [22:34:08] (03open) 10dancy: gitlab-runner: Set helper_cpu_limit [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/527 (https://phabricator.wikimedia.org/T408740) [22:34:08] (03update) 10dancy: gitlab-runner: Set helper_cpu_limit [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/527 (https://phabricator.wikimedia.org/T408740) [22:35:43] (03merge) 10dancy: gitlab-runner: Set helper_cpu_limit [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/527 (https://phabricator.wikimedia.org/T408740) [22:42:51] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work, 13Patch-For-Review: Set cpu limits on gitlab-cloud-runner workloads - https://phabricator.wikimedia.org/T408740#11338061 (10dancy) [23:00:04] maintenance-disconnect-full-disks build 751495 integration-agent-docker-1040 (/: 36%, /srv: 98%, /var/lib/docker: 24%): OFFLINE due to disk space [23:05:03] maintenance-disconnect-full-disks build 751496 integration-agent-docker-1040 (/: 36%, /srv: 38%, /var/lib/docker: 23%): RECOVERY disk space OK