[07:16:45] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/Sanctions] Add CommunityConfiguration dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1369551 (owner: 10Lens0021) [07:17:56] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Sanctions] Add CommunityConfiguration dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1369551 (owner: 10Lens0021) [07:29:24] GitLab needs a short maintenance break in a few minutes [07:41:51] 06Release-Engineering-Team, 10Wikidata, 06Wikidata-Omega, 07ci-test-error: termbox CI: docker-build "trying to send message larger than max (19888840 vs. 16777216)" - https://phabricator.wikimedia.org/T444456#12563820 (10ArthurTaylor) 05Open→03Resolved a:03ArthurTaylor Seems to be resolved now -... [07:48:54] (03CR) 10Awight: "I have read that the root misbehavior is fixed in containerd 2.x: https://github.com/containerd/containerd/pull/11441" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/1369513 (https://phabricator.wikimedia.org/T444811) (owner: 10Ahmon Dancy) [07:52:00] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Machine-Learning-Team (Q2 FY2026-27): Publish Pipeline Issue: publishing large images fails - https://phabricator.wikimedia.org/T444811#12563827 (10ArthurTaylor) [07:52:01] 06Release-Engineering-Team, 10Wikidata, 06Wikidata-Omega, 07ci-test-error: termbox CI: docker-build "trying to send message larger than max (19888840 vs. 16777216)" - https://phabricator.wikimedia.org/T444456#12563830 (10ArthurTaylor) →14Duplicate dup:03T444811 [07:52:39] 06Release-Engineering-Team, 10Wikidata, 06Wikidata-Omega, 07ci-test-error: termbox CI: docker-build "trying to send message larger than max (19888840 vs. 16777216)" - https://phabricator.wikimedia.org/T444456#12563832 (10awight) Likely fixed by {T444811} (leaving this breadcrumb in case it regresses in... [07:56:25] Gerrit needs a short maintenance break in a few minutes [09:59:50] (03merge) 10addshore: build(deps): bump golang.org/x/crypto from 0.58.0 to 0.59.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/816 (owner: 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6) [10:00:17] (03merge) 10addshore: Bump traefik from v3.7.18 to v3.7.19 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/815 (owner: 10project_16_bot_841eade40a22c56a6272172b8cb67384) [10:00:45] (03merge) 10addshore: Bump cloudflare/cloudflared from 2026.10.7 to 2026.11.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/814 (owner: 10project_16_bot_841eade40a22c56a6272172b8cb67384) [10:01:09] (03merge) 10addshore: Bump docker-registry.wikimedia.org/repos/data-engineering/eventgate-wikimedia from v1.38.0 to v1.39.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/813 (owner: 10project_16_bot_841eade40a22c56a6272172b8cb67384) [10:32:48] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Machine-Learning-Team (Q2 FY2026-27): Publish Pipeline Issue: publishing large images fails - https://phabricator.wikimedia.org/T444811#12564571 (10gkyziridis) Thank you very much @dancy ! I am closing this task [10:33:01] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Machine-Learning-Team (Q2 FY2026-27): Publish Pipeline Issue: publishing large images fails - https://phabricator.wikimedia.org/T444811#12564573 (10gkyziridis) 05Open→03Resolved [10:37:13] (03CR) 10Hashar: [C:03+2] "I was wondering why the AbuseFilter code was proposed in IPInfo. That is how that was done for IPReputation as well ( IPReputation/src/Ho" [integration/config] - 10https://gerrit.wikimedia.org/r/1369716 (owner: 10DatGuy) [10:39:24] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/IPInfo] Add AbuseFilter phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1369716 (owner: 10DatGuy) [10:58:30] (03CR) 10Hashar: [C:03+2] "Thank you! 😊" [fresh] - 10https://gerrit.wikimedia.org/r/1369134 (https://phabricator.wikimedia.org/T439828) (owner: 10Krinkle) [10:59:00] (03CR) 10Hashar: [C:03+2] Zuul: Add waki285 to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1369688 (owner: 10Peter Li) [11:01:05] (03Merged) 10jenkins-bot: Zuul: Add waki285 to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1369688 (owner: 10Peter Li) [11:01:40] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Machine-Learning-Team (Q2 FY2026-27): Publish Pipeline Issue: publishing large images fails - https://phabricator.wikimedia.org/T444811#12564706 (10gkyziridis) a:03gkyziridis [11:02:59] (03CR) 10Hashar: [C:03+2] dockerfiles: load PHP pcov but disable it by default (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/1369431 (https://phabricator.wikimedia.org/T420009) (owner: 10Hashar) [11:04:28] (03Merged) 10jenkins-bot: docs: Fix welcome text instruction [fresh] - 10https://gerrit.wikimedia.org/r/1369134 (https://phabricator.wikimedia.org/T439828) (owner: 10Krinkle) [11:08:01] 10Gerrit, 06Collaboration-Services, 07Incident Severity 1, 13Patch-For-Review, 07Sustainability (Incident Followup): Add the ability to warm caches to Gerrit switchover cookbook - https://phabricator.wikimedia.org/T440177#12564733 (10MLechvien-WMF) [11:20:03] 10Gerrit, 06Collaboration-Services, 10SRE-SLO: Implement service level indicator measurement for Gerrit - https://phabricator.wikimedia.org/T396979#12564813 (10ABran-WMF) 05Open→03Stalled https://wikitech.wikimedia.org/wiki/SLO/Gerrit has been published with a pair of 99.5% SLOs, https://grafana-rw.wikim... [11:20:12] 10Gerrit, 06Collaboration-Services, 10SRE-SLO: Implement service level indicator measurement for Gerrit - https://phabricator.wikimedia.org/T396979#12564817 (10ABran-WMF) [11:23:51] 10Gerrit, 06Collaboration-Services, 10SRE-SLO: Implement service level indicator measurement for Gerrit - https://phabricator.wikimedia.org/T396979#12564823 (10ABran-WMF) [11:25:14] (03CR) 10Hashar: [C:03+2] Merge branch 'upstream/stable-3.11' into wmf/stable-3.11 (032 comments) [software/gerrit] (wmf/stable-3.11) - 10https://gerrit.wikimedia.org/r/1360360 (https://phabricator.wikimedia.org/T379714) (owner: 10Hashar) [11:25:58] (03PS1) 10Hashar: wm-pcc: add a Rerun callback on CheckRun [software/gerrit] (deploy/wmf/stable-3.11) - 10https://gerrit.wikimedia.org/r/1369875 [11:36:01] (03Merged) 10jenkins-bot: Merge branch 'upstream/stable-3.11' into wmf/stable-3.11 [software/gerrit] (wmf/stable-3.11) - 10https://gerrit.wikimedia.org/r/1360360 (https://phabricator.wikimedia.org/T379714) (owner: 10Hashar) [11:41:16] 10Gerrit, 06Collaboration-Services, 10SRE-SLO, 13Patch-For-Review: Implement service level indicator measurement for Gerrit - https://phabricator.wikimedia.org/T396979#12564912 (10ABran-WMF) [11:52:38] 10Gerrit (Gerrit 3.11), 06Release-Engineering-Team (Priority Backlog 📥), 06Collaboration-Services, 13Patch-For-Review: Upgrade to Gerrit 3.11 - https://phabricator.wikimedia.org/T379714#12564962 (10hashar) > https://gerrit-review.googlesource.com/c/gerrit/+/430619 Update Plugin API from HTMLElement to Plug... [11:54:10] 10Gerrit (Gerrit 3.11), 06Release-Engineering-Team (Priority Backlog 📥), 06Collaboration-Services, 13Patch-For-Review: Upgrade to Gerrit 3.11 - https://phabricator.wikimedia.org/T379714#12564974 (10hashar) [12:02:07] 10Gerrit (Gerrit 3.11): Enforce project configuration changes for code review - https://phabricator.wikimedia.org/T445067 (10hashar) 03NEW [12:03:11] 10Gerrit (Gerrit 3.11), 06Release-Engineering-Team (Priority Backlog 📥), 06Collaboration-Services, 13Patch-For-Review: Upgrade to Gerrit 3.11 - https://phabricator.wikimedia.org/T379714#12565059 (10hashar) [12:04:58] (03PS2) 10Hashar: Gerrit 3.11.x rebuild plugins and update TypeScript API [software/gerrit] (deploy/wmf/stable-3.11) - 10https://gerrit.wikimedia.org/r/1360417 (https://phabricator.wikimedia.org/T379714) [12:04:58] (03CR) 10Hashar: "One last remaining action was to sync our JavaScript plugins with a change I made upstream:" [software/gerrit] (deploy/wmf/stable-3.11) - 10https://gerrit.wikimedia.org/r/1360417 (https://phabricator.wikimedia.org/T379714) (owner: 10Hashar) [12:20:19] Project mediawiki-core-doxygen build #24348: 04FAILURE in 2 min 17 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/24348/ [12:51:20] 06Project-Admins, 10database-orchestrator, 06DBA: Re-label "#orchestrator" to be less generic so it's less confusing - https://phabricator.wikimedia.org/T444624#12565283 (10Marostegui) I think it is fine to drop [13:09:35] Yippee, build fixed! [13:09:35] Project mediawiki-core-doxygen build #24349: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/24349/ [15:36:30] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team (Radar), 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, and 2 others: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12566247 (10Lars) I think we can close this one up now. [15:50:41] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Machine-Learning-Team (Q2 FY2026-27): Publish Pipeline Issue: publishing large images fails - https://phabricator.wikimedia.org/T444811#12566431 (10dancy) @dduvall: In https://gerrit.wikimedia.org/r/c/integration/pipelinelib/+/1369513... [15:57:13] (03CR) 10Ahmon Dancy: [C:03+1] jjb: point Composer at integration-cache-proxy when Zuul asks (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1369164 (https://phabricator.wikimedia.org/T444738) (owner: 10Mhurd) [16:55:15] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Machine-Learning-Team (Q2 FY2026-27): Publish Pipeline Issue: publishing large images fails - https://phabricator.wikimedia.org/T444811#12567003 (10dduvall) >>! In T444811#12566431, @dancy wrote: > @dduvall: In https://gerrit.wikimedi... [17:06:31] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.48.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T442223#12567050 (10thcipriani) a:05jeena→03brennen Reassigning to @brennen for Tue/Thu while we get prod access resorted for @jeena. [17:07:58] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/gophercloud/gophercloud/v2 from 2.15.0 to 2.16.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/817 [17:08:22] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/docker/cli from 29.9.3+incompatible to 29.10.0+incompatible [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/818 [17:08:38] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump gopkg.in/ini.v1 from 1.67.3 to 1.68.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/819 [17:08:40] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump gopkg.in/ini.v1 from 1.67.3 to 1.68.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/819 [17:09:32] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/moby/moby/client from 0.6.2 to 0.6.3 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/820 [17:09:41] 06Release-Engineering-Team (Doing 😎), 06Infrastructure-Foundations, 10Packaging: Re-enable thirdparty/docker updates for bookworm/trixie - https://phabricator.wikimedia.org/T445121 (10dduvall) 03NEW [17:11:24] 06Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review: Refactor `build-images.py` to use a common code image and `docker buildx` - https://phabricator.wikimedia.org/T392526#12567110 (10dduvall) 05In progress→03Stalled p:05Triage→03Low [17:16:56] 06Release-Engineering-Team (Doing 😎), 06Collaboration-Services, 10Release Pipeline, 13Patch-For-Review: ST6.1.4 Refactor Pipelinelib to use buildx and generate SBOMs - https://phabricator.wikimedia.org/T440668#12567139 (10dduvall) 05Resolved→03Open Reopened due to edge cases where SBOMs are large enoug... [17:17:00] 06Release-Engineering-Team (Doing 😎), 06Infrastructure-Foundations, 10Packaging: Re-enable thirdparty/docker updates for bookworm/trixie - https://phabricator.wikimedia.org/T445121#12567143 (10dduvall) [17:17:04] 06Release-Engineering-Team (Doing 😎), 06Collaboration-Services, 10Release Pipeline, 13Patch-For-Review: ST6.1.4 Refactor Pipelinelib to use buildx and generate SBOMs - https://phabricator.wikimedia.org/T440668#12567144 (10dduvall) [17:17:38] 06Release-Engineering-Team (Doing 😎), 06Infrastructure-Foundations, 10Packaging: Re-enable thirdparty/docker updates for bookworm/trixie - https://phabricator.wikimedia.org/T445121#12567146 (10dduvall) 05Open→03In progress p:05Triage→03High [17:20:00] 10Beta-Cluster-Infrastructure: unblock.py crashes when network picked to unblock is superset of blocked network - https://phabricator.wikimedia.org/T445124 (10bd808) 03NEW [17:39:03] 10WikimediaDebug, 06Content-Platform-Team, 06ServiceOps: k8s-mw-parsoid-eqiad / k8s-mw-parsoid-codfw no longer available for selection in WikimediaDebug extension - https://phabricator.wikimedia.org/T442935#12567348 (10MLechvien-WMF) 05Open→03Declined As mentioned on Slack, I'll close this for now, p... [17:56:47] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: [Oct 2026] Upgrade K3s cluster to most recent stable version - https://phabricator.wikimedia.org/T426866#12567417 (10brennen) [18:00:10] FIRING: [4x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [18:15:16] 06Release-Engineering-Team (Priority Backlog 📥): Draft a report on "path to production" tools/workflows currently known to be in use - https://phabricator.wikimedia.org/T445136 (10bd808) 03NEW [18:16:30] 06Release-Engineering-Team (Priority Backlog 📥): Draft a report on "path to production" tools/workflows currently known to be in use - https://phabricator.wikimedia.org/T445136#12567607 (10bd808) [18:16:37] 06Release-Engineering-Team (Priority Backlog 📥): Draft a report on "path to production" tools/workflows currently known to be in use - https://phabricator.wikimedia.org/T445136#12567608 (10bd808) p:05Triage→03Medium [18:26:26] FIRING: [4x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [18:28:07] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-zookeeper04 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [19:00:16] Hi, could I get a +1 for https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1369713? [19:02:59] Southparkfan: {{done}} [19:03:04] Thanks. :) [19:03:18] * bd808 is a good meat puppet [19:03:49] Haha [19:14:31] 10Beta-Cluster-Infrastructure, 06Data-Engineering, 10Kafka-Infrastructure: beta - Run a single event platform Kafka cluster in beta - https://phabricator.wikimedia.org/T437541#12567871 (10tchin) >>! In T437541#12563222, @Southparkfan wrote: > Hi @tchin, I am working on T436481. I noticed kafka_clusters still... [19:23:51] 10Beta-Cluster-Infrastructure: unblock.py crashes when network picked to unblock is superset of blocked network - https://phabricator.wikimedia.org/T445124#12567907 (10bd808) 05Open→03In progress p:05Triage→03High a:03bd808 [19:25:41] 10Beta-Cluster-Infrastructure: unblock.py crashes when network picked to unblock is superset of blocked network - https://phabricator.wikimedia.org/T445124#12567917 (10bd808) I wrote a patch that made this case work and unblocked the IP range that first found this problem. I need to write some tests to prove tha... [20:33:20] (03merge) 10addshore: build(deps): bump github.com/moby/moby/client from 0.6.2 to 0.6.3 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/820 (owner: 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6) [20:33:47] (03merge) 10addshore: build(deps): bump github.com/gophercloud/gophercloud/v2 from 2.15.0 to 2.16.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/817 (owner: 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6) [20:34:04] (03merge) 10addshore: build(deps): bump github.com/docker/cli from 29.9.3+incompatible to 29.10.0+incompatible [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/818 (owner: 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6) [20:38:10] (03open) 10dancy: deploy-service: Deploy one release of a service [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1435 (https://phabricator.wikimedia.org/T442586) [20:41:24] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team (Radar), 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, and 2 others: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12568251 (10Lars) 05Open→03Resolved [21:00:55] (03update) 10addshore: Draft: Add initial Enterprise commands [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/800 [21:01:57] (03close) 10addshore: build(deps): bump golang.org/x/sys from 0.49.0 to 0.50.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/804 (owner: 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6) [21:02:16] (03update) 10addshore: Add initial Enterprise commands [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/800 [21:09:08] (03update) 10addshore: Add initial Enterprise commands [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/800 [21:29:25] (03merge) 10addshore: Add initial Enterprise commands [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/800 [22:05:16] (03update) 10addshore: Draft: feat(toolforge): add authenticated local Wiki Replica proxy [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/794 (https://phabricator.wikimedia.org/T443528) [22:05:57] (03update) 10addshore: feat(toolforge): add authenticated local Wiki Replica & toolsdb proxy [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/794 (https://phabricator.wikimedia.org/T443528 https://phabricator.wikimedia.org/T444068) [22:10:06] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [22:10:12] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T445160 (10wmcs-alerts) 03NEW [22:17:40] (03merge) 10addshore: feat(toolforge): add authenticated local Wiki Replica & toolsdb proxy [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/794 (https://phabricator.wikimedia.org/T443528 https://phabricator.wikimedia.org/T444068) [22:21:08] (03PS2) 10Abijeet Patro: Zuul: [mediawiki/extensions/Translate] Add ConfirmEdit dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1370134 (https://phabricator.wikimedia.org/T442735) [22:50:06] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [22:51:08] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [22:57:41] (03CR) 10Dreamy Jazz: "Probably needs to be listed in https://gerrit.wikimedia.org/g/integration/config/+/0e816cf58187ac8378da34b87836dcc5008cf11d/zuul/phan_depe" [integration/config] - 10https://gerrit.wikimedia.org/r/1370134 (https://phabricator.wikimedia.org/T442735) (owner: 10Abijeet Patro) [23:01:32] (03open) 10brennen: rollback: add $no_extensions param to initializeScriptEnvironment() [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/73