[00:32:17] Project mediawiki-core-doxygen build #15075: 04STILL FAILING in 14 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/15075/ [01:32:34] Yippee, build fixed! [01:32:34] Project mediawiki-core-doxygen build #15076: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/15076/ [05:15:06] 06Project-Admins: Create project tag for - https://phabricator.wikimedia.org/T409887 (10Cw95yt5) 03NEW [06:30:56] (03PS1) 10QChris: Allow β€œGerrit Managers” to import history [extensions/WandaScribe] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1204109 [06:30:56] (03CR) 10QChris: [V:03+2 C:03+2] Allow β€œGerrit Managers” to import history [extensions/WandaScribe] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1204109 (owner: 10QChris) [06:30:59] (03PS1) 10QChris: Import done. Revoke import grants [extensions/WandaScribe] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1204110 [06:31:00] (03CR) 10QChris: [V:03+2 C:03+2] Import done. Revoke import grants [extensions/WandaScribe] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1204110 (owner: 10QChris) [06:32:06] spammy(?) phab user: https://phabricator.wikimedia.org/p/Cesar/ [07:38:07] If someone could disable them in id{p|m} as well would be great [08:10:22] 10Phabricator (2025-11-04), 06Release-Engineering-Team (Doing 😎), 07Essential-Work: Restrict project workboard access to logged in users only - https://phabricator.wikimedia.org/T403385#11365244 (10Bawolff) While I appreciate this probably wasn't done lightly, i do find it sad as before people become con... [08:38:40] 10Beta-Cluster-Infrastructure: Link to the alternative shortened URL is incorrect in Beta Cluster - https://phabricator.wikimedia.org/T357128#11365295 (10Ammarpad) [10:07:54] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/CategoryWatch] Add Echo phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1203210 (owner: 10Fomafix) [10:09:42] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/CategoryWatch] Add Echo phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1203210 (owner: 10Fomafix) [10:39:42] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T408273#11365717 (10Aklapper) [10:49:26] 10GitLab, 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services, 10wikimedia.biterg.io: Bitergia crawler bot runs into errors collection GitLab data - https://phabricator.wikimedia.org/T409903 (10Aklapper) 03NEW p:05Triageβ†’03Medium [11:04:32] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 13Patch-For-Review, 05Release, 05Train Deployments: 1.46.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T408272#11365839 (10Aklapper) [11:05:57] Train wreckage on wikifunctions, moving group1 back from wmf.2 to wmf.1 [11:10:27] 10Phabricator: Phabricator doesn't fetch/display a linked comment when too much activity has occurred in the task since it was made - https://phabricator.wikimedia.org/T409907 (10A_smart_kitten) 03NEW [11:22:09] 10Phabricator (Upstream), 07Upstream: Phabricator doesn't fetch/display a linked comment when too much activity has occurred in the task since it was made - https://phabricator.wikimedia.org/T409907#11365907 (10Aklapper) [11:31:14] 10GitLab, 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services, 10wikimedia.biterg.io: Bitergia crawler bot runs into errors collection GitLab data - https://phabricator.wikimedia.org/T409903#11365937 (10LSobanski) I get a 404 from https://gitlab.wikimedia.org/sonarbot/sonarqube.git wh... [11:39:11] 10GitLab, 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services, 10wikimedia.biterg.io: Bitergia crawler bot runs into errors collection GitLab data - https://phabricator.wikimedia.org/T409903#11365955 (10Peachey88) https://gitlab.wikimedia.org/sonarbot/ is a user, should it be offering... [11:42:06] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T408272#11365992 (10Aklapper) Rolled back group1 from 1.46.0-wmf.2 to 1.46.0-wmf.1 due to {T409876} making wikifunctions.org render an exce... [12:00:06] 10Phabricator (Upstream), 07Upstream: Phabricator doesn't fetch/display a linked comment when too much activity has occurred in the task since it was made - https://phabricator.wikimedia.org/T409907#11366057 (10A_smart_kitten) [12:51:16] 10Phabricator-Bot-Requests, 06Release-Engineering-Team (Doing 😎): Request for WMDE Board Statistics bot account - https://phabricator.wikimedia.org/T409809#11366315 (10ArthurTaylor) So I've tried that (at moderate expense to my sanity) - implementing the login process with python requests: https://gitlab.wikim... [12:54:17] 10GitLab, 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services, 10wikimedia.biterg.io: Bitergia crawler bot runs into errors collection GitLab data - https://phabricator.wikimedia.org/T409903#11366322 (10Izubiaurre) - https://gitlab.wikimedia.org/isaranto/readability-liftwing - htt... [13:41:13] 10Phabricator (Upstream), 07CSS, 07Upstream: When a workboard has a long sidebar, scrolling down that sidebar scrolls down the entire page (and scrolls away the workboard columns) - https://phabricator.wikimedia.org/T409705#11366547 (10Aklapper) [13:41:58] 10Phabricator (Upstream), 07CSS, 07Upstream: When a workboard has a long sidebar, scrolling down that sidebar scrolls down the entire page (and scrolls away the workboard columns) - https://phabricator.wikimedia.org/T409705#11366553 (10Aklapper) > I don't feel confident enough to formally propose this e.g. u... [13:57:49] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T408272#11366618 (10Jdforrester-WMF) >>! In T408272#11365992, @Aklapper wrote: > Rolled back group1 from 1.46.0-wmf.2 to 1.46.0-wmf.1 due t... [14:10:03] maintenance-disconnect-full-disks build 753981 integration-agent-docker-1053 (/: 36%, /srv: 95%, /var/lib/docker: 28%): OFFLINE due to disk space [14:15:03] maintenance-disconnect-full-disks build 753982 integration-agent-docker-1053 (/: 36%, /srv: 47%, /var/lib/docker: 26%): RECOVERY disk space OK [14:42:09] 10Phabricator (Upstream), 07Upstream, 07Voice & Tone: Ambiguous user-visible string "owner" mashes concepts of Assignee and Author - https://phabricator.wikimedia.org/T279827#11366832 (10Aklapper) [14:42:10] 10Phabricator, 06Release-Engineering-Team (Priority Backlog πŸ“₯): Update to Phorge upstream / Arcanist upstream (after 2025-09-08 done in T404134) - https://phabricator.wikimedia.org/T404375#11366833 (10Aklapper) [14:42:47] 10Phabricator (Upstream), 07Upstream, 07Voice & Tone: Ambiguous user-visible string "owner" mashes concepts of Assignee and Author - https://phabricator.wikimedia.org/T279827#11366836 (10Aklapper) 05Openβ†’03Stalled a:03Aklapper [15:11:18] 10GitLab, 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services, 10wikimedia.biterg.io: Bitergia crawler bot runs into errors collection GitLab data - https://phabricator.wikimedia.org/T409903#11367038 (10Dzahn) > Can we allowlist their IPs (or user agent which I believe is owlbot)? I... [15:13:08] 06Release-Engineering-Team, 06cloud-services-team, 10Toolforge: [toolforge_deploy_mr] Branches can only be tested for 24 hours, otherwise fail with CI too old. - https://phabricator.wikimedia.org/T408999#11367047 (10taavi) Dear #together, do you have opinions about raising reggie retention time? [15:16:40] (03open) 10jnuche: add GitLab CI component configuration [repos/releng/docpub] - 10https://gitlab.wikimedia.org/repos/releng/docpub/-/merge_requests/36 (https://phabricator.wikimedia.org/T409723) [15:21:37] (03merge) 10jnuche: add GitLab CI component configuration [repos/releng/docpub] - 10https://gitlab.wikimedia.org/repos/releng/docpub/-/merge_requests/36 (https://phabricator.wikimedia.org/T409723) [15:29:11] (03open) 10jnuche: fix component template file extension [repos/releng/docpub] - 10https://gitlab.wikimedia.org/repos/releng/docpub/-/merge_requests/37 (https://phabricator.wikimedia.org/T409723) [15:30:28] 10GitLab, 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services, 10wikimedia.biterg.io: Bitergia crawler bot runs into errors collection GitLab data - https://phabricator.wikimedia.org/T409903#11367180 (10Izubiaurre) //> private repos then we probably would not want to extract data to e... [15:31:09] (03merge) 10jnuche: fix component template file extension [repos/releng/docpub] - 10https://gitlab.wikimedia.org/repos/releng/docpub/-/merge_requests/37 (https://phabricator.wikimedia.org/T409723) [15:41:38] 10Phabricator-Bot-Requests, 06Release-Engineering-Team (Doing 😎): Request for WMDE Board Statistics bot account - https://phabricator.wikimedia.org/T409809#11367241 (10bd808) If I was going to build a Phabricator HTML scraper bot I think I would use a Developer account created via https://idm.wikimedia.org/ an... [15:45:04] p858snake|cloud: I blocked that https://ldap.toolforge.org/user/cesar developer account. Thanks for mentioning it. I'm glad I read backscroll today too. :) [15:49:12] 06Release-Engineering-Team (Doing 😎), 07Essential-Work, 13Patch-For-Review: Docpub: Add GitLab component - https://phabricator.wikimedia.org/T409723#11367301 (10Jdforrester-WMF) To check, we should now do the below? ` lang=diff - - project: repos/releng/docpub - file: includes/publish.yml + - component... [15:57:39] (03update) 10dancy: deploy_security.py: Copy patch with --no-preserve=mode [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/216 (https://phabricator.wikimedia.org/T409855) (owner: 10lucaswerkmeister-wmde) [16:08:51] 10GitLab (Administration, Settings & Policy), 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06cloud-services-team: gitlab: consider enabling docker container registry - https://phabricator.wikimedia.org/T304845#11367372 (10DPogorzelski-WMF) The ML team needs a place where to store large LLM docker contain... [16:09:43] (03merge) 10dancy: deploy_security.py: Copy patch with --no-preserve=mode [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/216 (https://phabricator.wikimedia.org/T409855) (owner: 10lucaswerkmeister-wmde) [16:10:08] 10Scap: scap canary check threshold warning is also shown at train rollback - https://phabricator.wikimedia.org/T409946 (10Aklapper) 03NEW [16:28:22] 10Phabricator, 06Release-Engineering-Team (Radar), 06collaboration-services, 06serviceops-radar, 07Sustainability (Incident Followup): Phabricator: Unable to view tasks in DB read-only mode - https://phabricator.wikimedia.org/T313879#11367463 (10Aklapper) To get closer to this in the long run, "Prevent w... [16:28:57] 10Phabricator (phabricator-next), 06Release-Engineering-Team: Merge upstream master from Phorge as of 2025-11-12 - https://phabricator.wikimedia.org/T409947 (10brennen) 03NEW [16:31:57] (03open) 10brennen: Merge upstream master from Phorge as of 2025-11-12 [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/108 (https://phabricator.wikimedia.org/T409947) [16:35:55] 10Phabricator, 06Release-Engineering-Team (Priority Backlog πŸ“₯): Update to Phorge upstream / Arcanist upstream (after 2025-09-08 done in T404134) - https://phabricator.wikimedia.org/T404375#11367517 (10Aklapper) [16:35:58] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 13Patch-For-Review: Merge upstream master from Phorge as of 2025-11-12 - https://phabricator.wikimedia.org/T409947#11367516 (10Aklapper) [16:36:51] (03open) 10brennen: Merge upstream master from Phorge as of 2025-11-12 (Arcanist) [repos/phabricator/arcanist] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/arcanist/-/merge_requests/8 (https://phabricator.wikimedia.org/T409947) [16:37:05] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 13Patch-For-Review: Merge upstream master from Phorge as of 2025-11-12 - https://phabricator.wikimedia.org/T409947#11367522 (10Aklapper) @brennen: Thanks! I am going to take a look at the diff in the next days. I hope there will be less merge conf... [16:38:12] (03open) 10brennen: update submodules for merge of 2025-11-12 phorge upstream [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/86 (https://phabricator.wikimedia.org/T409947) [16:52:18] !log deployed proposed phorge merge to https://phabricator.wmcloud.org/ (T409947) [16:52:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:52:21] T409947: Merge upstream master from Phorge as of 2025-11-12 - https://phabricator.wikimedia.org/T409947 [16:53:36] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 13Patch-For-Review: Merge upstream master from Phorge as of 2025-11-12 - https://phabricator.wikimedia.org/T409947#11367622 (10brennen) AFAICT, just the 2 minor ones in the `phabricator` repo. It's up for testing in devtools. [17:06:23] (03PS1) 10Hashar: utils: zuul-mw-jobs-runner add 'in-production' filter [integration/config] - 10https://gerrit.wikimedia.org/r/1204632 [17:07:17] (03CR) 10Hashar: [C:03+2] utils: zuul-mw-jobs-runner add 'in-production' filter [integration/config] - 10https://gerrit.wikimedia.org/r/1204632 (owner: 10Hashar) [17:08:03] (03CR) 10CI reject: [V:04-1] utils: zuul-mw-jobs-runner add 'in-production' filter [integration/config] - 10https://gerrit.wikimedia.org/r/1204632 (owner: 10Hashar) [17:08:09] pfff [17:09:08] (03PS2) 10Hashar: utils: zuul-mw-jobs-runner add 'in-production' filter [integration/config] - 10https://gerrit.wikimedia.org/r/1204632 [17:13:14] (03CR) 10Hashar: [C:03+2] utils: zuul-mw-jobs-runner add 'in-production' filter [integration/config] - 10https://gerrit.wikimedia.org/r/1204632 (owner: 10Hashar) [17:14:53] (03Merged) 10jenkins-bot: utils: zuul-mw-jobs-runner add 'in-production' filter [integration/config] - 10https://gerrit.wikimedia.org/r/1204632 (owner: 10Hashar) [17:17:02] 06Release-Engineering-Team (Doing 😎), 07Essential-Work: Docpub: Add GitLab component - https://phabricator.wikimedia.org/T409723#11367746 (10jnuche) >>! In T409723#11367301, @Jdforrester-WMF wrote: > To check, we should now do the below? > > ` lang=diff > - - project: repos/releng/docpub > - file: include... [17:28:19] 06Release-Engineering-Team (Doing 😎), 07Essential-Work: Docpub: Add GitLab component - https://phabricator.wikimedia.org/T409723#11367786 (10jnuche) 05Openβ†’03Resolved docpub can now be consumed using the usual, older include style, e.g.: ` include: - project: repos/releng/docpub file: includes/pu... [17:29:06] 06Release-Engineering-Team (Doing 😎), 07Essential-Work: Docpub: Add GitLab component - https://phabricator.wikimedia.org/T409723#11367788 (10jnuche) Oops, forgot to update the docs. MR coming soon [17:37:14] (03open) 10jnuche: add docs about the new GitLab CI component to README [repos/releng/docpub] - 10https://gitlab.wikimedia.org/repos/releng/docpub/-/merge_requests/38 (https://phabricator.wikimedia.org/T409723) [17:37:46] (03merge) 10jnuche: add docs about the new GitLab CI component to README [repos/releng/docpub] - 10https://gitlab.wikimedia.org/repos/releng/docpub/-/merge_requests/38 (https://phabricator.wikimedia.org/T409723) [17:59:42] 10GitLab (Administration, Settings & Policy), 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06cloud-services-team: gitlab: consider enabling docker container registry - https://phabricator.wikimedia.org/T304845#11367940 (10thcipriani) >>! In T304845#11367372, @DPogorzelski-WMF wrote: > The ML team needs a... [18:02:30] 10GitLab, 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services, 10wikimedia.biterg.io: Bitergia crawler bot runs into errors collection GitLab data - https://phabricator.wikimedia.org/T409903#11367949 (10Dzahn) @Izubiaurre Maybe let's take a step back for a second and let me ask this:... [18:27:21] 10GitLab (Administration, Settings & Policy), 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06cloud-services-team: gitlab: consider enabling docker container registry - https://phabricator.wikimedia.org/T304845#11368054 (10thcipriani) 05Stalledβ†’03Declined Declining this task to be clear we have no... [18:36:11] !log Updated expired release token for wmf-packages in integration Jenkins - T379203 [18:36:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:36:14] T379203: Add CI_RELEASE_TOKEN secret for {name}-maven-release jobs in jenkins - https://phabricator.wikimedia.org/T379203 [19:06:21] 06Release-Engineering-Team (Doing 😎), 07Essential-Work, 13Patch-For-Review, 05Release, 05Train Deployments: 1.46.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T408272#11368205 (10Aklapper) [21:40:58] !log Unblock 188.148.0.0/14 (T409968) [21:41:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:45:42] (03update) 10dancy: backport.py: Revise backport vote rule 3b [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1039 (https://phabricator.wikimedia.org/T362987) [21:56:42] 10Scap, 07Security: Document correct way to deploy security patches - https://phabricator.wikimedia.org/T409855#11368736 (10sbassett) Just noting again that `deploy_security.py` isn't really officially sanctioned or maintained by anyone on the #security-team or #product_safety_and_integrity. [22:23:26] 10Scap, 07Security: Document correct way to deploy security patches - https://phabricator.wikimedia.org/T409855#11368838 (10Dzahn) That sounds like a pretty good reason to edit the Wikitech page and remove parts like "Download the security deployment script" which links to it and calls it "the script". Maybe t... [22:27:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-docker-mobileapps02 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:27:40] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-docker-mobileapps02 on project deployment-prep - https://phabricator.wikimedia.org/T409979 (10wmcs-alerts) 03NEW [22:34:32] 10Scap, 07Security: Document correct way to deploy security patches - https://phabricator.wikimedia.org/T409855#11368857 (10A_smart_kitten) >>! In T409855#11368736, @sbassett wrote: > Just noting again that `deploy_security.py` isn't really officially sanctioned or maintained by anyone on the #security-team or... [22:48:34] 10Beta-Cluster-Infrastructure: Link to the alternative shortened URL is incorrect in Beta Cluster - https://phabricator.wikimedia.org/T357128#11368883 (10bd808) 05Openβ†’03Resolved I can't reproduce this. I would guess that the work in {T396012} where things were moved to the wmcloud.org parent domain fixe... [22:52:32] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-docker-mobileapps02 on project deployment-prep - https://phabricator.wikimedia.org/T409979#11368893 (10bd808) `lang=shell-session, COUNTEREXAMPLE bd808@deployment-docker-mobileapps02:~$ sudo -i puppet agent -tv Info: Using environmen... [23:22:29] (03update) 10dancy: backport.py: Revise backport vote rule 3b [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1039 (https://phabricator.wikimedia.org/T362987) [23:32:04] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-docker-mobileapps02 on project deployment-prep - https://phabricator.wikimedia.org/T409979#11368965 (10bd808) `lang=shell-session bd808@deployment-docker-mobileapps02:~$ df -h Filesystem Size Used Avail Use% Mounted on udev... [23:33:32] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-docker-mobileapps02 on project deployment-prep - https://phabricator.wikimedia.org/T409979#11368966 (10bd808) Let's start by truncating the log: `lang=shell-session root@deployment-docker-mobileapps02:/var/lib/docker/containers/9f525... [23:33:44] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-docker-mobileapps02 on project deployment-prep - https://phabricator.wikimedia.org/T409979#11368967 (10bd808) 05Openβ†’03In progress a:03bd808 [23:34:00] 10Continuous-Integration-Infrastructure, 06collaboration-services: allow mwmaint/cumin hosts to connect to http on contint - https://phabricator.wikimedia.org/T340788#11368970 (10Dzahn) 05Openβ†’03Stalled [23:36:54] 06Release-Engineering-Team, 06collaboration-services, 10mwcli, 06serviceops-radar: Set writeable HOME for nobody user in golang images instead of /nonexistent - https://phabricator.wikimedia.org/T331209#11368978 (10Dzahn) Based on previous comments I am boldly closing this as resolved. Finding out the "if... [23:39:42] 10Phabricator (2025-10-28), 06Release-Engineering-Team (Radar), 06collaboration-services, 10VPS-project-Phabricator: 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts fai... - https://phabricator.wikimedia.org/T403948#11368980 [23:42:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance deployment-docker-mobileapps02 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:42:36] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-docker-mobileapps02 on project deployment-prep - https://phabricator.wikimedia.org/T409979#11368986 (10bd808) The `profile::docker::engine::settings` hiera variable can be used to configure Docker to rotate logs among other things. D...