[01:15:32] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-webperf21 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [01:15:40] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-webperf21 in project deployment-prep - https://phabricator.wikimedia.org/T430606 (10wmcs-alerts) 03NEW [02:46:32] FIRING: InstanceDown: Project deployment-prep instance deployment-echostore02 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [02:46:38] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-echostore02 is down - https://phabricator.wikimedia.org/T430607 (10wmcs-alerts) 03NEW [04:01:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-echostore02 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [05:47:20] (03PS1) 10Hashar: zuul: playbooks for Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/1306484 [05:47:36] (03PS2) 10Hashar: zuul: playbooks for Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/1306484 [05:49:05] (03CR) 10CI reject: [V:04-1] zuul: playbooks for Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/1306484 (owner: 10Hashar) [06:01:30] (03PS3) 10Hashar: zuul: playbooks for Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/1306484 [06:03:01] (03CR) 10CI reject: [V:04-1] zuul: playbooks for Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/1306484 (owner: 10Hashar) [06:04:17] (03PS4) 10Hashar: zuul: playbooks for Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/1306484 [06:56:21] 06Release-Engineering-Team, 06Infrastructure-Foundations: Add tox-uv support to the tox-v{3,4} Docker images - https://phabricator.wikimedia.org/T421348#12068869 (10elukey) @thcipriani Hi! Do you have any preference/suggestion about the above? [07:22:57] 06Project-Admins: Create project tag for the MediaWiki Media Platform Team (new team within MediaWiki, starting FY26/27) - https://phabricator.wikimedia.org/T430544#12068917 (10SLopes-WMF) >>! In T430544#12066917, @Aklapper wrote: > Hi, where to find out what the "PP3 Responsible Reuse objective" is? Is there a... [07:44:01] 06Release-Engineering-Team (Doing 😎), 05Release, 05Train Deployments: 1.47.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T423918#12068959 (10Aklapper) [07:55:54] (03CR) 10Hashar: [C:03+2] zuul: playbooks for Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/1306484 (owner: 10Hashar) [08:30:22] (03CR) 10Hashar: [C:03+2] phan: Add Wikibase as EventBus phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1297767 (https://phabricator.wikimedia.org/T428176) (owner: 10Ottomata) [08:32:26] (03Merged) 10jenkins-bot: phan: Add Wikibase as EventBus phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1297767 (https://phabricator.wikimedia.org/T428176) (owner: 10Ottomata) [09:09:31] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team (Doing 😎), 06collaboration-services: Add more repositories to the CI instances gitcache mirrors - https://phabricator.wikimedia.org/T430627 (10hashar) 03NEW [09:10:57] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team (Doing 😎), 06collaboration-services: Add more repositories to the CI instances gitcache mirrors - https://phabricator.wikimedia.org/T430627#12069395 (10hashar) Copy pasting [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1302834/comm... [09:12:33] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team (Doing 😎), 06collaboration-services, 13Patch-For-Review: Add more repositories to the CI instances gitcache mirrors - https://phabricator.wikimedia.org/T430627#12069431 (10hashar) I will skip adding report of cumulative in Quibble (there i... [09:24:07] hi folks, i'm a member of the serviceops sre team, and am wondering if it might be possible to get +2 rights in the production-images repo? [09:27:31] 06Release-Engineering-Team (Doing 😎), 05Release, 05Train Deployments: 1.47.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T423918#12069506 (10Aklapper) Backported https://gerrit.wikimedia.org/r/c/mediawiki/skins/MinervaNeue/+/1306430 for T428220 into the wmf.9 branch via https://gerrit.wikim... [09:27:53] 06Release-Engineering-Team (Doing 😎), 05Release, 05Train Deployments: 1.47.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T423918#12069512 (10Aklapper) [09:28:51] bjensen: best to file a ticket in Phab [09:29:04] andre: will do, thanks! [09:31:02] 06Release-Engineering-Team: Deployment rights to production-images for Blake - https://phabricator.wikimedia.org/T430631 (10Blake) 03NEW [09:38:43] 10WikimediaDebug, 10API Platform, 06ServiceOps new, 10ServiceOps-Mediawiki, 10ServiceOps-SharedInfra: WikimediaDebug makes api.wikimedia.org always 301 redirect - https://phabricator.wikimedia.org/T430600#12069592 (10Clement_Goubert) p:05Triageβ†’03Medium The reason that happens is the `X-Wikimedia-Deb... [09:43:38] (03CR) 10Santiago Faci: "Hi James! Sorry for the delay! I missed this conversation." [integration/config] - 10https://gerrit.wikimedia.org/r/1248853 (owner: 10Jforrester) [09:44:04] 10WikimediaDebug, 10API Platform, 06ServiceOps new, 10ServiceOps-Mediawiki, 10ServiceOps-SharedInfra: WikimediaDebug makes api.wikimedia.org always 301 redirect - https://phabricator.wikimedia.org/T430600#12069655 (10Clement_Goubert) [10:00:46] 06Release-Engineering-Team: Deployment rights to production-images for Blake - https://phabricator.wikimedia.org/T430631#12069728 (10Blake) Incidentally, it looks like the repo might be locked down exclusively to gerrit admins (https://gerrit.wikimedia.org/r/admin/repos/operations/docker-images,access). I'm not... [10:04:05] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team (Doing 😎), 06collaboration-services, 13Patch-For-Review: Add more repositories to the CI instances gitcache mirrors - https://phabricator.wikimedia.org/T430627#12069753 (10hashar) Unpooled integration-agent-docker-1082 for the benchmarking... [10:38:28] 10Gerrit, 06Release-Engineering-Team: Deployment rights to production-images for Blake - https://phabricator.wikimedia.org/T430631#12069909 (10hashar) a:03hashar The sensible repositories are locked to Administrator of Gerrit to prevent unwanted changes. The `operations/docker-images`Β one is indeed locked in... [10:45:16] 10Gerrit, 06Release-Engineering-Team: Deployment rights to production-images for Blake - https://phabricator.wikimedia.org/T430631#12069934 (10Blake) 05Openβ†’03Invalid Hm, that's very strange - as far as I can tell, I'm using the user 'blake@wikimedia.org' on Gerrit. I don't think I've ever created or u... [10:49:05] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - gerrit2003 - https://phabricator.wikimedia.org/T430597#12069955 (10LSobanski) [11:29:35] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team (Doing 😎), 06collaboration-services, 13Patch-For-Review: Add more repositories to the CI instances gitcache mirrors - https://phabricator.wikimedia.org/T430627#12070075 (10ABran-WMF) 05Openβ†’03Resolved a:03ABran-WMF I ran puppet o... [12:01:44] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Gerrit backups are growing - https://phabricator.wikimedia.org/T411583#12070212 (10jcrespo) Sadly, I think it has not meaningfully affected the backup size: ` 724586 Incr 1,978 142.5 M OK 30-Jun-26 11:00 gerrit2002.wikimedia.org-Hourly... [12:24:02] 10WikimediaDebug, 10API Platform, 06ServiceOps new, 10ServiceOps-Mediawiki, 10ServiceOps-SharedInfra: WikimediaDebug makes api.wikimedia.org always 301 redirect - https://phabricator.wikimedia.org/T430600#12070283 (10Clement_Goubert) 05Openβ†’03Resolved a:03Clement_Goubert This should now be fixe... [12:30:05] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services: Update gitlab cookbooks and processes - https://phabricator.wikimedia.org/T430655 (10ABran-WMF) 03NEW [13:08:52] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - gerrit2003 - https://phabricator.wikimedia.org/T430597#12070517 (10ABran-WMF) 05Openβ†’03Resolved a:03ABran-WMF I updated the rule created in {T430421} to cast a wider net, it seem to have worked. [13:09:03] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - gerrit2003 - https://phabricator.wikimedia.org/T430597#12070524 (10ABran-WMF) [13:17:22] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Gerrit backups are growing - https://phabricator.wikimedia.org/T411583#12070551 (10ABran-WMF) After the cache exclusion the spare dropped to ~140 MB, roughly the "real delta" we'd want. So on a host that isn't actively computing diffs / updating the i... [13:20:24] 10Phabricator, 06Release-Engineering-Team (Doing 😎), 06ServiceOps new: Custom task form for Production Readiness Checklist process (SCROLL) - https://phabricator.wikimedia.org/T429692#12070558 (10MLechvien-WMF) Thanks a lot @Aklapper ! We would need the following 2 changes if possible: - In the banner at... [13:48:08] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Gerrit backups are growing - https://phabricator.wikimedia.org/T411583#12070708 (10jcrespo) I agree with your analysis and followup, and thanks again for taking this as seriously as I am. Let me share the results of the requested query with you, with ex... [14:59:08] beta slow/down "Error: 503, Backend fetch failed" [14:59:16] a few pages work from Varnish cache but thats it [14:59:28] https://en.wikipedia.beta.wmcloud.org/wiki/Special:Blankpage [15:00:10] pretty long timeout too, I guess it goes the full 60-90s for GET requests [15:00:27] which suggests it's not simply down but struggling/appearing to be reachable [15:13:29] !log Rebooting deployment-mwlog02.deployment-prep to clear stuck udp2log processes [15:13:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:15:50] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - gerrit2003 - https://phabricator.wikimedia.org/T430597#12071907 (10thcipriani) >>! In T430597#12070517, @ABran-WMF wrote: > I updated the rule created in {T430421} to cast a wider net, it seem to have worked. I noticed the analysis from tha... [16:19:39] 10MediaWiki-Releasing, 05MW-1.46-release: Write and send release announcement for 1.46.0 - https://phabricator.wikimedia.org/T423271#12071952 (10MSantos) 05Openβ†’03Resolved a:03MSantos [16:20:14] 10MediaWiki-Releasing, 07Documentation, 05MW-1.46-notes, 05MW-1.46-release: Finish up RELEASE-NOTES for 1.46.0 - https://phabricator.wikimedia.org/T423270#12071970 (10MSantos) 05Openβ†’03Resolved a:03Jgiannelos [16:21:36] 10MediaWiki-Releasing, 05MW-1.46-release, 05Release: Release MediaWiki 1.46.0 - https://phabricator.wikimedia.org/T423255#12071986 (10MSantos) 05Openβ†’03Resolved a:03MSantos 1.46.0 release is complete as of Today. [16:40:16] thcipriani: I am quite happy to still be brain dumping my sessions to Phabricator cause I had to rebuild the microk8s instance and thanksfully I had everything at https://phabricator.wikimedia.org/T395826 :) [17:04:36] hey James_F, sorry to be a clueless noob here. [17:04:48] apergos: <3 No worries at all. [17:05:07] apergos: Prod config is dark and full of terrors. [17:05:43] so we don't branch 1.47 for some time yet... as I understand it [17:06:13] we ust leave the line in there but commented out so that wen we get 1.47 branched, the next person just uncomments that bit? [17:06:33] (please excuse the typos) [17:06:47] Yeah. [17:06:59] The 1.47 branch will be in October. [17:07:42] I'll be in the mix for that one too. [17:08:13] Surely by then we'll have dumped it all on an LLM? Especially the prod config deploy bits. ;-) [17:09:16] please, I'd prefer not to have even more trouble sleeping than I already do :-P [17:11:52] * James_F grins. [17:49:51] 06Project-Admins: Create project tag for Wiki Script Scanner - https://phabricator.wikimedia.org/T430721 (10ASanford-WMF) 03NEW [17:50:25] James_F: one more question, should I schedule this for a backports/deployment window next? (I didn't do the last one, our dedicated volunteer did it iirc) [17:50:46] apergos: Yeah, whenever you can get it out. [17:50:59] okey dokey, thanks! [18:00:43] 06Project-Admins: Create project tag for Wiki Script Scanner - https://phabricator.wikimedia.org/T430721#12072675 (10Aklapper) > Should be viewable by users with https://phabricator.wikimedia.org/tag/acl_security/ privileges. I'd make this a public project; I do not see why this should be a hidden project. [18:02:56] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Create project tag for Wiki Script Scanner - https://phabricator.wikimedia.org/T430721#12072691 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Requested **public** project tag #Wiki-Script-Scanner has been created: https://phabricator.wikimedia.org... [18:04:23] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Create project tag for Wiki Script Scanner - https://phabricator.wikimedia.org/T430721#12072707 (10ASanford-WMF) Thanks! Yeah, public is fine. Some tasks are sensitive (they could reveal information that would assist attackers in bypassing the scanner... [18:05:13] 06Release-Engineering-Team, 10ChangeProp, 06Data-Engineering, 10EventStreams, and 15 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12072725 (10Sfaci) [18:06:05] 06Release-Engineering-Team, 10ChangeProp, 06Data-Engineering, 10EventStreams, and 13 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12072742 (10Sfaci) [18:09:39] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): PostBuild changing the status of successful builds to failure for no apparent reason - https://phabricator.wikimedia.org/T419488#12072769 (10Pppery) Happened twice in a row in h... [18:40:56] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 10Catalyst (Luka Ijo Pimeja Jan), 05MW-1.46-release, 13Patch-For-Review: Cannot select REL1_46 on PatchDemo - https://phabricator.wikimedia.org/T425165#12072954 (10neriah) a:05brennenβ†’03None [18:41:01] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 10Catalyst (Luka Ijo Pimeja Jan), 05MW-1.46-release, 13Patch-For-Review: Cannot select REL1_46 on PatchDemo - https://phabricator.wikimedia.org/T425165#12072956 (10neriah) a:03neriah [18:50:10] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 10Catalyst (Luka Ijo Pimeja Jan), 05MW-1.46-release, 13Patch-For-Review: Cannot select REL1_46 on PatchDemo - https://phabricator.wikimedia.org/T425165#12072975 (10neriah) 05Openβ†’03In progress p:05Triageβ†’03High [19:31:04] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-webperf21 in project deployment-prep - https://phabricator.wikimedia.org/T430606#12073223 (10dancy) ` dancy@deployment-webperf21:/$ sudo run-puppet-agent Error: No space left on device @ fptr_finalize_flush - /var/lib/puppet/ssl/... [19:34:21] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Create tag "Ukrainian-Sites" - https://phabricator.wikimedia.org/T428358#12073246 (10VadymTS1) ty! [20:02:56] 10MediaWiki-Releasing, 10ExtensionDistributor, 05MW-1.46-release, 13Patch-For-Review: Mark REL1_46 in ExtensionDistributor as a stable release - https://phabricator.wikimedia.org/T423272#12073397 (10MacFan4000) a:03ArielGlenn [20:32:53] 10MediaWiki-Releasing, 10ExtensionDistributor, 05MW-1.46-release: Mark REL1_46 in ExtensionDistributor as a stable release - https://phabricator.wikimedia.org/T423272#12073577 (10ArielGlenn) 05Openβ†’03Resolved [20:51:37] (03open) 10milazg: Rules: allow valid media types and handle polymorphic properties [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/14 (https://phabricator.wikimedia.org/T428149) [21:01:38] 10Scap: Scap / Spiderpig should not warn about i18n rebuild whenever extension.json is changed - https://phabricator.wikimedia.org/T430750 (10Tgr) 03NEW [21:03:47] (03update) 10milazg: Rules: allow valid media types and handle polymorphic properties [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/14 (https://phabricator.wikimedia.org/T428149) [22:38:31] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 10Catalyst (Luka Ijo Pimeja Jan), 05MW-1.46-release, 13Patch-For-Review: Cannot select REL1_46 on PatchDemo - https://phabricator.wikimedia.org/T425165#12073977 (10brennen) [23:22:16] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-webperf21 in project deployment-prep - https://phabricator.wikimedia.org/T430606#12074046 (10dancy) 05Openβ†’03Resolved a:03dancy Fixed by truncating /var/log/{syslog,messages,user.log}, run-puppet-agent, and systemctl re... [23:22:26] (03PS1) 10Eric Gardner: Zuul: Remove PageImages from MMV test dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1306794 [23:23:31] (03CR) 10Eric Gardner: "I posted a follow-up here: https://gerrit.wikimedia.org/r/c/integration/config/+/1306794" [integration/config] - 10https://gerrit.wikimedia.org/r/1305998 (owner: 10Eric Gardner) [23:25:05] (03CR) 10Eric Gardner: "See https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MultimediaViewer/+/1306490 for an example of the current broken CI status over t" [integration/config] - 10https://gerrit.wikimedia.org/r/1306794 (owner: 10Eric Gardner) [23:30:32] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance deployment-webperf21 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [23:33:26] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-echostore02 is down - https://phabricator.wikimedia.org/T430607#12074078 (10bd808) 05Openβ†’03Invalid `lang=shell-session bd808@deployment-echostore02:~$ w 23:30:48 up 476 days, 3:23, 2 users, load average: 0.43, 0.34, 0.36 USER... [23:40:50] 10Beta-Cluster-Infrastructure: deployment-webperf21 logspam from /usr/bin/confd: "WARNING Found no templates" - https://phabricator.wikimedia.org/T395945#12074083 (10bd808) a:03Peter `counterexample Jun 30 23:35:06 deployment-webperf21 confd[506]: 2026-06-30T23:35:06Z deployment-webperf21 /usr/bin/confd[506]:... [23:49:55] 10Beta-Cluster-Infrastructure: deployment-webperf21 logspam from /usr/bin/confd: "WARNING Found no templates" - https://phabricator.wikimedia.org/T395945#12074106 (10Dzahn) The "confd spam - WARNING Found no templates" has been a common issue across multiple systems. See: T356296, T417458 and I think more als... [23:53:25] 10Beta-Cluster-Infrastructure: deployment-webperf21 logspam from /usr/bin/confd: "WARNING Found no templates" - https://phabricator.wikimedia.org/T395945#12074112 (10Dzahn) You probably don't actually want confd to even try to start. Then it also won't spam about not finding templates. Or it can be configured s...