[08:36:04] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T423911#11920681 (10Aklapper) Things look calm apart from a spike of {T359823} on a single, closed wiki. [08:58:44] 10Phabricator (phabricator-next), 06Release-Engineering-Team (Doing 😎), 10Wikimedia-Phabricator-Extensions: Have Phabricator link Gerrit change-id to Gerrit search query - https://phabricator.wikimedia.org/T420363#11920741 (10A_smart_kitten) Noting for the task record -- IIUC, the merged patch differs from t... [11:15:26] (03PS1) 10Fomafix: Zuul: [mediawiki/extensions/Survey] Add AdminLinks Phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1287380 [11:56:41] (03update) 10addshore: Multiple features post hackathon 2026 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/649 [12:15:45] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T423911#11921383 (10Aklapper) 05Open→03Resolved Also commented in closed T419772#11920704 that we saw another occurrence. All in a... [12:59:41] (03merge) 10addshore: Multiple features post hackathon 2026 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/649 [13:08:58] (03update) 10addshore: Default XDEBUG_CONFIG to client_host=xdebug://gateway [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/644 (owner: 10hoo) [13:18:54] (03open) 10urbanecm: Switch to lowercase urbanecm [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/245 [13:20:01] no want wants an uppercase urbanecm [13:20:34] (03merge) 10reedy: Switch to lowercase urbanecm [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/245 (owner: 10urbanecm) [14:26:51] 06Project-Admins, 06Content-Transform-Team, 10MediaWiki-Language-converter: Where should tasks for mediawiki-libs-LangConv go? Should it have a dedicated tag? - https://phabricator.wikimedia.org/T388286#11921894 (10cscott) We are deprecating this library, it has been replaced with an implementation in core.... [14:27:33] 06Project-Admins, 06Content-Transform-Team, 10MediaWiki-Language-converter: Where should tasks for mediawiki-libs-LangConv go? Should it have a dedicated tag? - https://phabricator.wikimedia.org/T388286#11921899 (10OSleger-WMF) a:03MSantos [14:46:22] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-mx04 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [15:07:47] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-mx04 on project deployment-prep - https://phabricator.wikimedia.org/T425687#11922058 (10bd808) `lang=shell-session,counterexample bd808@deployment-mx04:~$ sudo -i puppet agent -tv Info: Using environment 'production' Info: Retrieving... [15:11:42] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-mx04 on project deployment-prep - https://phabricator.wikimedia.org/T425687#11922101 (10bd808) >>! In T425687#11908386, @dancy wrote: > Probably caused by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1283025 (T325394) @Morit... [15:18:14] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-mx04 on project deployment-prep - https://phabricator.wikimedia.org/T425687#11922152 (10bd808) One way to move forward would be to remove the now absent role from the instance and then get work started on building a replacement MX se... [15:35:09] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-mx04 on project deployment-prep - https://phabricator.wikimedia.org/T425687#11922237 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/580c6c098acbc7c67d5d2405048f36099480c39c%5E%21/#F0 ` diff --git a/d... [15:37:05] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-mx04 on project deployment-prep - https://phabricator.wikimedia.org/T425687#11922251 (10bd808) Puppet works again, just without management of the still installed MX service stack. `lang=shell-session bd808@deployment-mx04:~$ sudo -i... [15:37:43] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-mx04 on project deployment-prep - https://phabricator.wikimedia.org/T425687#11922257 (10bd808) [15:40:26] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Replace deployment-mx04 with a new OS and newer MX service stack - https://phabricator.wikimedia.org/T426326 (10bd808) 03NEW [15:40:56] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Replace deployment-mx04 with a new OS and newer MX service stack - https://phabricator.wikimedia.org/T426326#11922295 (10bd808) p:05Triage→03High [15:41:42] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Replace deployment-mx04 with a newer OS and MX service stack - https://phabricator.wikimedia.org/T426326#11922299 (10bd808) [15:42:33] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Replace deployment-mx04 with a newer OS and MX service stack - https://phabricator.wikimedia.org/T426326#11922305 (10bd808) [16:01:23] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance deployment-mx04 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [16:27:09] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-mx04 on project deployment-prep - https://phabricator.wikimedia.org/T425687#11922567 (10dancy) Thanks @bd808! [16:36:08] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-devtools, and 3 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11922645 (10Dzahn) I left a a bunch of comments I should have put here on... [16:42:40] (03update) 10addshore: Default XDEBUG_CONFIG to client_host=xdebug://gateway [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/644 (owner: 10hoo) [16:42:55] (03merge) 10addshore: Default XDEBUG_CONFIG to client_host=xdebug://gateway [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/644 (owner: 10hoo) [16:43:39] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Disaster recovery for k8s upgrade - https://phabricator.wikimedia.org/T419580#11922705 (10thcipriani) [17:26:56] (03CR) 10Vaughn Walters: "This is ready for review." [integration/config] - 10https://gerrit.wikimedia.org/r/1287020 (https://phabricator.wikimedia.org/T424441) (owner: 10Vaughn Walters) [17:49:12] 06Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10MW-on-K8s, 06ServiceOps new: Add helm rollback functionality to scap - https://phabricator.wikimedia.org/T371069#11922951 (10RLazarus) p:05Triage→03Medium [18:03:32] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-mx04 on project deployment-prep - https://phabricator.wikimedia.org/T425687#11922984 (10bd808) 05Open→03Resolved a:03bd808 The alert has cleared, so I'm going to resolve this. Things are in a sketchy state until {T426326} i... [18:09:07] 06Release-Engineering-Team (Radar), 10Scap, 10MediaWiki-Internationalization, 13Patch-For-Review, 07Wikimedia-Performance-recommendation: Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740#11922997 (10bvibber) Note that I can no longer push deploy... [18:40:57] (03CR) 10Jforrester: [C:03+2] jjb: [catalyst-daily-VisualEditor] Add daily run and ZUUL_PIPELINE [integration/config] - 10https://gerrit.wikimedia.org/r/1287020 (https://phabricator.wikimedia.org/T424441) (owner: 10Vaughn Walters) [18:41:13] 10MediaWiki-Releasing, 10MediaWiki-extensions-CodeMirror: Bundle Extension:CodeMirror with MediaWiki core - https://phabricator.wikimedia.org/T391926#11923060 (10Mr._Starfleet_Command) >>! In T391926#11439114, @Mr._Starfleet_Command wrote: > Agree that this should be bundled, but wait till v6 is out of beta.... [18:42:35] (03Merged) 10jenkins-bot: jjb: [catalyst-daily-VisualEditor] Add daily run and ZUUL_PIPELINE [integration/config] - 10https://gerrit.wikimedia.org/r/1287020 (https://phabricator.wikimedia.org/T424441) (owner: 10Vaughn Walters) [19:21:33] (03PS1) 10Wandji collins: Edit Repo Config [labs/tools/WdTmCollab] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1286275 [19:26:16] (03Abandoned) 10Wandji collins: Edit Repo Config [labs/tools/WdTmCollab] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1286275 (owner: 10Wandji collins) [19:33:26] 06Project-Admins: Create project tag for WikiLounge - https://phabricator.wikimedia.org/T426351 (10Jony) 03NEW [19:42:29] (03open) 10dancy: backport: Warn when a l10n-touching change is backported [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1187 (https://phabricator.wikimedia.org/T397089) [19:42:33] (03update) 10dancy: backport: Warn when a l10n-touching change is backported [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1187 (https://phabricator.wikimedia.org/T397089) [19:43:12] 10Scap, 13Patch-For-Review: scap backport should warn if it knows it will take a long time - https://phabricator.wikimedia.org/T397089#11923210 (10dancy) 05Open→03In progress p:05Triage→03High a:03dancy [19:45:48] (03update) 10dancy: backport: Warn when a l10n-touching change is backported [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1187 (https://phabricator.wikimedia.org/T397089) [19:50:48] (03update) 10kharlan: scap prep: Add foss_violations support [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1186 (https://phabricator.wikimedia.org/T403829) (owner: 10dancy) [19:59:33] 06Project-Admins: Create project tag for WikiLounge - https://phabricator.wikimedia.org/T426351#11923242 (10Aklapper) > Canonical code repository URL of the project: WikiLounge is an instance of the upstream open-source project https://github.com/thelounge/thelounge Hi, does that mean that this upstream codebas... [20:03:22] (03open) 10dancy: SpiderPig: Notify when job completes [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1188 [20:03:27] (03update) 10dancy: SpiderPig: Notify when job completes [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1188 [20:10:58] (03update) 10dancy: SpiderPig: Notify when job completes [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1188 [20:33:08] (03PS1) 10Vaughn Walters: jjb: [wikilambda-catalyst-end-to-end-daily] Add ZUUL_PIPELINE [integration/config] - 10https://gerrit.wikimedia.org/r/1287478 [20:35:06] 06Project-Admins: Create project tag for WikiLounge - https://phabricator.wikimedia.org/T426351#11923352 (10A_smart_kitten) (cross-referencing to {T424443}) [20:38:32] (03CR) 10Jforrester: [C:03+2] jjb: [wikilambda-catalyst-end-to-end-daily] Add ZUUL_PIPELINE [integration/config] - 10https://gerrit.wikimedia.org/r/1287478 (owner: 10Vaughn Walters) [20:40:12] (03Merged) 10jenkins-bot: jjb: [wikilambda-catalyst-end-to-end-daily] Add ZUUL_PIPELINE [integration/config] - 10https://gerrit.wikimedia.org/r/1287478 (owner: 10Vaughn Walters) [21:30:38] 06Release-Engineering-Team, 10Catalyst (PatchDemo): Cannot select REL1_46 on PatchDemo - https://phabricator.wikimedia.org/T425165#11923471 (10Dreamy_Jazz) Ran into this too just now [21:31:58] (03update) 10thcipriani: scap prep: Add foss_violations support [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1186 (https://phabricator.wikimedia.org/T403829) (owner: 10dancy) [21:32:00] (03approved) 10thcipriani: scap prep: Add foss_violations support [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1186 (https://phabricator.wikimedia.org/T403829) (owner: 10dancy) [22:02:22] (03merge) 10dancy: scap prep: Add foss_violations support [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1186 (https://phabricator.wikimedia.org/T403829) [22:04:15] (03open) 10dancy: Release 4.265.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1189 [22:06:37] (03merge) 10dancy: Release 4.265.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1189 [23:03:51] (03open) 10thcipriani: spiderpig train: fix deploy_promote "all" [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1190