[00:01:41] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services, 10function-evaluator, and 3 others: Wikifunctions function orchestrator and evaluator test suites failing on GitLab CI with OOM errors - https://phabricator.wikimedia.org/T399348#11505923 (10vaughnwalters) @Jdforrester-W... [00:51:34] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T408280#11505984 (10ABreault-WMF) I backported [[ https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1223283 | a config change... [00:53:44] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T413802#11505992 (10SecurityPatchBot) [00:57:19] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T413802#11506007 (10Zabe) [06:50:36] 10Diffusion, 10Phabricator, 06Release-Engineering-Team, 07Performance Issue: Diffusion code view: Per-file change info is slow to load despite caching (diffusion.lastmodifiedquery) - https://phabricator.wikimedia.org/T403215#11506183 (10Pppery) Of course if you run a file from the command line then `php_sa... [07:24:54] 06Release-Engineering-Team, 10LDAP-Access-Requests, 06Security-Team, 06SRE, and 2 others: DannyS712 "offboarding" - https://phabricator.wikimedia.org/T413634#11506246 (10Dzahn) a:03DannyS712 [08:51:47] 10Diffusion, 10Phabricator, 06Release-Engineering-Team, 07Performance Issue: Diffusion code view: Per-file change info is slow to load despite caching (diffusion.lastmodifiedquery) - https://phabricator.wikimedia.org/T403215#11506382 (10Aklapper) Eh, thanks, makes sense. Guess I should have RTFM... [09:04:20] 10Phabricator (phabricator-next), 07Mobile: Gerrit patch status comments have/cause inconsistent spacing on mobile devices - https://phabricator.wikimedia.org/T414003#11506425 (10Aklapper) [09:05:06] 10Phabricator, 07dark-mode: Bot comment styles are not dark-mode compatible - https://phabricator.wikimedia.org/T414117#11506426 (10Aklapper) p:05Triage→03Low [09:08:03] 06Project-Admins, 06Editing-team, 10Section Editing (Mobile): Consider archiving the 'Section Editing (Mobile)' project - https://phabricator.wikimedia.org/T411044#11506437 (10Aklapper) Would love to see feedback from #Editing-team here [09:22:11] 10Beta-Cluster-Infrastructure, 06Traffic: CDN edge HAProxy config broken by missing `/usr/share/GeoIP/proxy.mmdb` data file - https://phabricator.wikimedia.org/T414111#11506475 (10SLyngshede-WMF) a:03SLyngshede-WMF [09:30:10] 10Beta-Cluster-Infrastructure, 06Traffic: CDN edge HAProxy config broken by missing `/usr/share/GeoIP/proxy.mmdb` data file - https://phabricator.wikimedia.org/T414111#11506506 (10SLyngshede-WMF) @ssingh The issue is in the Lua code that opens the database file, but fails. It's a little weird, because the code... [09:37:37] 06Release-Engineering-Team, 10LDAP-Access-Requests, 06Security-Team, 06SRE, and 2 others: DannyS712 "offboarding" - https://phabricator.wikimedia.org/T413634#11506526 (10JMeybohm) [09:40:44] 10Beta-Cluster-Infrastructure, 06Traffic: CDN edge HAProxy config broken by missing `/usr/share/GeoIP/proxy.mmdb` data file - https://phabricator.wikimedia.org/T414111#11506535 (10SLyngshede-WMF) Quick test revels that the error handling does NOTHING if the fail doesn't exist. [10:54:56] 06Release-Engineering-Team (Radar), 10Ceph, 06serviceops, 10SRE-swift-storage: Move the docker registry's /restricted prefix to Docker Distribution backed up by Ceph - https://phabricator.wikimedia.org/T412951#11506760 (10elukey) To keep archives happy - I am working in T394476 to properly onboard ceph apu... [11:09:17] 10Phabricator, 07dark-mode: Bot comment styles are not dark-mode compatible - https://phabricator.wikimedia.org/T414117#11506777 (10A_smart_kitten) Maybe this might just be a matter of (currently untested): `lang=diff,name=phui-timeline-view-wikimedia.css; L103-109 .phui-timeline-major-event a.phui-timeline-i... [11:10:01] 10Phabricator, 07dark-mode: Bot comment styles are not dark-mode compatible - https://phabricator.wikimedia.org/T414117#11506779 (10A_smart_kitten) [11:17:56] 06Release-Engineering-Team (Radar), 10Ceph, 06serviceops, 10SRE-swift-storage: Move the docker registry's /restricted prefix to Docker Distribution backed up by Ceph - https://phabricator.wikimedia.org/T412951#11506803 (10elukey) [13:45:05] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team (Seen), 10observability, 06SRE, 05Goal: Add Prometheus exporter to Jenkins instances - https://phabricator.wikimedia.org/T182759#11507488 (10WMDE-leszek) Bumping this ticket as I maybe might have an interest in seeing this progressing. I... [13:56:40] 10Phabricator (2026-01-06), 13Patch-For-Review: Deploy user style to reduce bot comments on Phabricator - https://phabricator.wikimedia.org/T393289#11507529 (10Urbanecm) >>! In T393289#11505391, @matmarex wrote: > {F71470893} This is working towards one of my concerns (I'm no longer able to see complete S... [14:32:34] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-mx03 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [15:20:21] 06Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops: mwdebug: people in the "deployment" group should be able to launch 'experimental' instances for testing purposes - https://phabricator.wikimedia.org/T324003#11507773 (10MLechvien-WMF) 05Open→03Resolved a:03MLechvien-WMF Clo... [15:23:05] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-mx03 in project deployment-prep - https://phabricator.wikimedia.org/T414206 (10wmcs-alerts) 03NEW [15:55:16] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06Abstract Wikipedia team, 06collaboration-services, and 4 others: Wikifunctions function orchestrator and evaluator test suites failing on GitLab CI with OOM errors - https://phabricator.wikimedia.org/T399348#11507969 (10dancy) After reading the... [16:17:38] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work, 13Patch-For-Review: Set up zuul web on zuul1001/zuul2001 - https://phabricator.wikimedia.org/T405119#11508049 (10Dzahn) You have a good point there, Tyler. Thank you! I have started looking and made a remind... [16:37:27] Could somebody please check the state of https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GlobalWatchlist/+/1221985 ? It should be merged, but looks like it stuck in the middle of jenkins-bot run. The manual rebase done, jenkins-bot copied +1, but somehow not +2. Thank you. [16:38:39] IKhitron: I'll try poking it [16:38:59] the +2 vote was added on PS85, the latest is PS86 [16:39:26] I gave it a +2 and now the gate-and-submit pipeline is running. [16:39:55] I see. Thank you both. [16:52:01] (03CR) 10Jforrester: zuul: Add php85 experimental to mediawiki/libs/less.php (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1218742 (https://phabricator.wikimedia.org/T411213) (owner: 10Krinkle) [16:59:00] (03PS1) 10Hashar: Zuul: [WikimediaEvents] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1224992 (https://phabricator.wikimedia.org/T389998) [16:59:30] (03CR) 10Hashar: [C:03+2] Zuul: [WikimediaEvents] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1224992 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [17:01:15] (03Merged) 10jenkins-bot: Zuul: [WikimediaEvents] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1224992 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [17:09:25] !log disabling beta update jobs to test bad extension-list T411516) [17:09:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:09:28] T411516: Add ability to ignore missing extensions in mergeMessageFileList's `--list-file` input - https://phabricator.wikimedia.org/T411516 [17:10:24] Now I get an error "Error: your composer.lock file is not up to date. Run "composer update --no-dev" to install newer dependencies". It's something I should know how to fix? I remind you I do not work from local git, if it's relevant. Thank you. [17:17:22] IKhitron: Looks like Tacsipacsi wants to participate. Hopefully they can help out. [17:20:58] IKhitron: Can you tell me more about what you mean when you say that you do not work from local git? [17:23:28] dancy Sure. From personal reasons I can't, at least for now, debug on my home computer. I use Patch Uploader for the first version, and edit on Gerrit, instead. So this composer, whatever it is, can't be mine, it's something on Gerrit, or on Mediawiki. [17:23:50] I see. [17:24:43] I'll see if I can help with the composer issue [17:33:07] !log re-enabling beta update jobs after test bad extension-list T411516 [17:33:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:33:12] T411516: Add ability to ignore missing extensions in mergeMessageFileList's `--list-file` input - https://phabricator.wikimedia.org/T411516 [17:37:16] 10Phabricator (2026-01-06), 13Patch-For-Review: Deploy user style to reduce bot comments on Phabricator - https://phabricator.wikimedia.org/T393289#11508313 (10matmarex) > Only one suggestion – could we make this on-click rather than on-hover? As long as the customizations use only CSS, that is not really... [17:38:09] 10Scap, 10MediaWiki-Internationalization: Add ability to ignore missing extensions in mergeMessageFileList's `--list-file` input - https://phabricator.wikimedia.org/T411516#11508316 (10thcipriani) Just tried on beta: ` jenkins-deploy@deployment-deploy04:/srv/mediawiki-staging$ git diff diff --git a/wmf-config... [17:44:12] 18:33:34 monolog/monolog: 2.11.0 installed, 2.9.3 required. [17:44:12] Someone broke the world [17:44:14] hi James_F :) [17:44:18] ah that is https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/1224842 [17:44:22] Update monolog/monolog from 2.9.3 to 2.11.0 [17:44:36] and the mediawiki/core one is https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1224843 [17:44:44] once that one merges, the problem will be fixed [17:44:45] success [17:45:24] hashar: That's the composer.lock issue? [17:45:36] Dear CI, when I tell you C+2 do your job. :-) [17:45:39] that is one I have encountered [17:45:42] I assume that is the same [17:45:47] Nod. Thanks. [17:46:05] I encountered the trouble while running a hundred or so of jobs which all failed :b [17:46:21] Mere hundreds [17:46:36] :-b [17:46:38] I'll run the batch later tonight [17:47:02] I am off! [17:49:37] Yeah, the VisualEditor repo is also blocked from merging anything by this. [17:51:09] Sounds like things will improve once https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1224843 merges [17:52:31] The most annoying kind of patch to have randomly decide it needs to take 30 minutes to merge. [18:17:03] 10Diffusion, 10Phabricator, 06Release-Engineering-Team, 07Performance Issue: Diffusion code view: Per-file change info is slow to load despite caching (diffusion.lastmodifiedquery) - https://phabricator.wikimedia.org/T403215#11508503 (10brennen) Yeah, I //think// the upstream is correct, although that cond... [18:25:12] That patch merged, and everything is fine now. [18:53:39] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06Abstract Wikipedia team, 06collaboration-services, and 4 others: Wikifunctions function orchestrator and evaluator test suites failing on GitLab CI with OOM errors - https://phabricator.wikimedia.org/T399348#11508642 (10vaughnwalters) >>! In T3... [19:10:51] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T408280#11508665 (10thcipriani) 05Open→03Resolved [19:33:38] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06Abstract Wikipedia team, 06collaboration-services, and 4 others: Wikifunctions function orchestrator and evaluator test suites failing on GitLab CI with OOM errors - https://phabricator.wikimedia.org/T399348#11508711 (10dancy) >>! In T399348#11... [21:15:00] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T413802#11508835 (10bd808) ##### Risky Patch! 🚂🔥 * **Change**: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1225023 *... [21:20:00] Project beta-update-databases-eqiad build #90477: 04FAILURE in 0.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/90477/ [21:21:23] Project beta-update-databases-eqiad build #90478: 04STILL FAILING in 1.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/90478/ [21:31:55] Yippee, build fixed! [21:31:55] Project beta-update-databases-eqiad build #90479: 09FIXED in 3 min 4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/90479/ [21:37:50] (03open) 10dancy: Revised resource watch implementation [repos/releng/k8s-pvc-cleaner] - 10https://gitlab.wikimedia.org/repos/releng/k8s-pvc-cleaner/-/merge_requests/35 [21:37:53] (03update) 10dancy: Revised resource watch implementation [repos/releng/k8s-pvc-cleaner] - 10https://gitlab.wikimedia.org/repos/releng/k8s-pvc-cleaner/-/merge_requests/35 [21:39:03] (03merge) 10dancy: Revised resource watch implementation [repos/releng/k8s-pvc-cleaner] - 10https://gitlab.wikimedia.org/repos/releng/k8s-pvc-cleaner/-/merge_requests/35 [21:40:42] (03open) 10dancy: Add initial .gitignore file [repos/releng/k8s-pvc-cleaner] - 10https://gitlab.wikimedia.org/repos/releng/k8s-pvc-cleaner/-/merge_requests/36 [21:40:44] (03update) 10dancy: Add initial .gitignore file [repos/releng/k8s-pvc-cleaner] - 10https://gitlab.wikimedia.org/repos/releng/k8s-pvc-cleaner/-/merge_requests/36 [21:41:28] (03merge) 10dancy: Add initial .gitignore file [repos/releng/k8s-pvc-cleaner] - 10https://gitlab.wikimedia.org/repos/releng/k8s-pvc-cleaner/-/merge_requests/36 [21:50:56] (03open) 10dancy: pvc-cleaner.py: Move a debug message [repos/releng/k8s-pvc-cleaner] - 10https://gitlab.wikimedia.org/repos/releng/k8s-pvc-cleaner/-/merge_requests/37 [21:50:59] (03update) 10dancy: pvc-cleaner.py: Move a debug message [repos/releng/k8s-pvc-cleaner] - 10https://gitlab.wikimedia.org/repos/releng/k8s-pvc-cleaner/-/merge_requests/37 [21:51:41] (03merge) 10dancy: pvc-cleaner.py: Move a debug message [repos/releng/k8s-pvc-cleaner] - 10https://gitlab.wikimedia.org/repos/releng/k8s-pvc-cleaner/-/merge_requests/37 [22:05:03] maintenance-disconnect-full-disks build 770780 integration-agent-docker-1062 (/: 25%, /srv: 97%, /var/lib/docker: 30%): OFFLINE due to disk space [22:10:03] maintenance-disconnect-full-disks build 770781 integration-agent-docker-1062 (/: 25%, /srv: 78%, /var/lib/docker: 29%): RECOVERY disk space OK [22:29:59] 10Scap, 10MediaWiki-Internationalization, 13Patch-For-Review: Add ability to ignore missing extensions in mergeMessageFileList's `--list-file` input - https://phabricator.wikimedia.org/T411516#11508924 (10bd808) @jeena will be shipping [[https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1225023|...