[00:12:30] (03update) 10raymond-ndibe: [values.yaml] add support for webservice-defaults [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/23 (https://phabricator.wikimedia.org/T431403) [00:13:18] (03update) 10raymond-ndibe: [values.yaml] add support for webservice-defaults [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/23 (https://phabricator.wikimedia.org/T431403) [00:13:19] (03approved) 10raymond-ndibe: [values.yaml] add support for webservice-defaults [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/23 (https://phabricator.wikimedia.org/T431403) [00:13:25] (03merge) 10raymond-ndibe: [values.yaml] add support for webservice-defaults [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/23 (https://phabricator.wikimedia.org/T431403) [00:14:53] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: image-config: bump to 0.0.28-20260722001336-ea583302 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1346 (https://phabricator.wikimedia.org/T431403) [00:54:52] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component image-config [01:02:58] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config [01:12:33] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component image-config [01:21:20] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config [01:29:38] (03update) 10raymond-ndibe: image-config: bump to 0.0.28-20260722001336-ea583302 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1346 (https://phabricator.wikimedia.org/T431403) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:29:39] (03approved) 10raymond-ndibe: image-config: bump to 0.0.28-20260722001336-ea583302 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1346 (https://phabricator.wikimedia.org/T431403) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:29:44] (03merge) 10raymond-ndibe: image-config: bump to 0.0.28-20260722001336-ea583302 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1346 (https://phabricator.wikimedia.org/T431403) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:38:20] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [01:46:58] 10Tool-subtitler: Video is moved up while editing subtitles on Subtitler - https://phabricator.wikimedia.org/T432807 (10psubhashish1) 03NEW [02:24:56] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [02:25:29] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [02:30:25] RESOLVED: SystemdUnitFailed: opentofu-infra-diff.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:56:20] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T423410) [02:57:13] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T423410) [02:57:14] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T423410) [02:57:52] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [05:32:41] 06cloud-services-team, 10Data-Services, 06tools-platform-team: [wikireplicas] Create views for new wiki bolwiki - https://phabricator.wikimedia.org/T429954#12144296 (10Marostegui) [05:32:43] 06cloud-services-team, 10Data-Services, 06tools-platform-team: [wikireplicas] Create views for new wiki bolwiki - https://phabricator.wikimedia.org/T429954#12144299 (10Marostegui) Views can be created. [05:40:48] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 03Wikimania-Hackathon-2026: [Team Challenges #07E] Wikidata–Wikisource Lexicographical Data Visualization Tool - https://phabricator.wikimedia.org/T432095#12144312 (10HakanIST) [05:41:44] 10Tool-Pageviews, 06Data-Engineering, 06Data-Engineering-Icebox, 10Data-Engineering-Wikistats, and 2 others: Pageviews Analysis 3.0 (Vue + Codex) - https://phabricator.wikimedia.org/T378549#12144313 (10MusikAnimal) 05Open→03In progress a:03MusikAnimal With the help of Claude, we'll soon have Pageview... [05:43:29] 10Tool-Pageviews: Move Pageviews tool to Symfony - https://phabricator.wikimedia.org/T420834#12144320 (10MusikAnimal) [05:43:30] 10Tool-Pageviews, 06Data-Engineering, 06Data-Engineering-Icebox, 10Data-Engineering-Wikistats, and 2 others: Pageviews Analysis 3.0 (Vue + Codex) - https://phabricator.wikimedia.org/T378549#12144319 (10MusikAnimal) [05:55:10] 10Tool-Pageviews, 06Data-Engineering, 06Data-Engineering-Icebox, 10Data-Engineering-Wikistats, and 2 others: Pageviews Analysis 3.0 (Vue + Codex) - https://phabricator.wikimedia.org/T378549#12144322 (10MusikAnimal) Screenshot of a work-in-progress, using Symfony/Vue/Codex/Apache eCharts: {F95160168} [05:59:12] 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team: Investigate systemdunitfailed alerts for clouddb hosts - https://phabricator.wikimedia.org/T432745#12144336 (10Marostegui) All the ones like `prometheus-mysqld-exporter.service` simply need a `systemctl disable prometheus-mysqld-exporter.se... [06:28:32] (03open) 10waldir: Add universal language selector [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/11 [06:29:57] (03open) 10waldir: Draft: Fix sidebar loading bug [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/12 [06:34:47] (03close) 10waldir: Workaround for missing ISO3 code [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/10 (owner: 10jberkel) [06:36:19] (03update) 10waldir: Draft: Fix sidebar loading bug [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/12 [06:38:47] 10Toolforge, 06tools-platform-team, 10Data-Services (Quota-requests): Trove-only project request: Postgres database for Toolforge tool - omd - https://phabricator.wikimedia.org/T432818#12144431 (10Les4353) [06:50:17] (03open) 10waldir: Draft: Update geogson and fix mismatched countries [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/13 [06:57:42] (03approved) 10jforrester: Add universal language selector [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/11 (owner: 10waldir) [06:57:45] (03merge) 10jforrester: Add universal language selector [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/11 (owner: 10waldir) [06:58:52] (03update) 10jforrester: Fix sidebar loading bug [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/12 (owner: 10waldir) [06:58:59] (03update) 10jforrester: Fix sidebar loading bug [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/12 (owner: 10waldir) [06:59:06] (03approved) 10jforrester: Fix sidebar loading bug [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/12 (owner: 10waldir) [06:59:22] (03merge) 10jforrester: Fix sidebar loading bug [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/12 (owner: 10waldir) [07:00:00] (03update) 10jforrester: Draft: Update geogson and fix mismatched countries [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/13 (owner: 10waldir) [07:00:31] (03approved) 10jforrester: Draft: Update geogson and fix mismatched countries [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/13 (owner: 10waldir) [07:00:56] (03update) 10jforrester: Update geogson and fix mismatched countries [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/13 (owner: 10waldir) [07:01:04] (03merge) 10jforrester: Update geogson and fix mismatched countries [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/13 (owner: 10waldir) [07:07:50] (03open) 10waldir: Draft: Fix issues with loading demonym data [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/14 [07:13:04] (03open) 10waldir: Draft: UX papercuts [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/15 [07:13:22] (03open) 10jforrester: globe: Slowly auto-rotate the idle globe, with a toggle [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/16 [07:14:38] (03update) 10jforrester: Draft: Fix issues with loading demonym data [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/14 (owner: 10waldir) [07:15:36] (03update) 10jforrester: Fix issues with loading demonym data [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/14 (owner: 10waldir) [07:15:37] (03approved) 10jforrester: Fix issues with loading demonym data [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/14 (owner: 10waldir) [07:15:39] (03merge) 10jforrester: Fix issues with loading demonym data [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/14 (owner: 10waldir) [07:16:04] (03update) 10jforrester: Draft: UX papercuts [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/15 (owner: 10waldir) [07:17:40] (03approved) 10jforrester: Draft: UX papercuts [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/15 (owner: 10waldir) [07:17:41] (03update) 10jforrester: UX papercuts [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/15 (owner: 10waldir) [07:17:44] (03merge) 10jforrester: UX papercuts [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/15 (owner: 10waldir) [07:17:58] (03update) 10jforrester: globe: Slowly auto-rotate the idle globe, with a toggle [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/16 [07:27:51] (03open) 10waldir: Update source of country geoshapes in footer [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/17 [07:29:14] (03open) 10jforrester: A number of UX tweaks to shrink the view, to allow for more usability on mobile [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/18 [07:32:31] 10Cloud-VPS, 06tools-infrastructure-team: Migrate to dumps-nfs.w.o in Cloud VPS - https://phabricator.wikimedia.org/T432587#12144510 (10fgiunchedi) I audited the hosts above looking for running containers, and if any were mounting clouddumps nfs. None did which is good news since no action will be required for... [07:36:18] 10Cloud-VPS, 06tools-infrastructure-team: Migrate to dumps-nfs.w.o in Cloud VPS - https://phabricator.wikimedia.org/T432587#12144512 (10fgiunchedi) [07:40:26] (03update) 10jforrester: globe: Slowly auto-rotate the idle globe, with a toggle [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/16 [07:41:59] (03update) 10jforrester: globe: Slowly auto-rotate the idle globe, with a toggle [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/16 [07:42:11] 10Toolforge, 06tools-platform-team: [toolforge,prod] httproutes disappeared for 4 tools (at least) - https://phabricator.wikimedia.org/T432822 (10dcaro) 03NEW [07:42:24] 10Toolforge, 06tools-platform-team: [toolforge,prod] httproutes disappeared for 4 tools (at least) - https://phabricator.wikimedia.org/T432822#12144530 (10dcaro) 05Open→03In progress p:05Triage→03High a:03dcaro [07:42:57] (03merge) 10jforrester: Update source of country geoshapes in footer [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/17 (owner: 10waldir) [07:43:00] (03update) 10jforrester: A number of UX tweaks to shrink the view, to allow for more usability on mobile [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/18 [07:47:20] 10Toolforge, 06tools-platform-team: [toolforge,prod] httproutes disappeared for 4 tools (at least) - https://phabricator.wikimedia.org/T432822#12144551 (10dcaro) [07:47:30] 10Toolforge, 06tools-platform-team: [toolforge,prod] httproutes disappeared for 4 tools (at least) - https://phabricator.wikimedia.org/T432822#12144555 (10dcaro) [07:48:50] 10Toolforge, 06tools-platform-team: [toolforge,prod] httproutes disappeared for 4 tools (at least) - https://phabricator.wikimedia.org/T432822#12144556 (10dcaro) [07:48:53] 10Toolforge, 06tools-platform-team: [toolforge,prod] httproutes disappeared for 4 tools (at least) - https://phabricator.wikimedia.org/T432822#12144569 (10DamianZaremba) First alert received at 05:57 (cest) pods happy; ` tools.cluebotng@tools-bastion-15:~$ kubectl get pods NAME... [07:49:24] (03open) 10jforrester: footer: Shorten how we list things so the footer isn't so dominant [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/19 [07:49:29] (03approved) 10waldir: globe: Slowly auto-rotate the idle globe, with a toggle [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/16 (owner: 10jforrester) [07:49:55] 10Toolforge, 06tools-platform-team: [toolforge,prod] httproutes disappeared for 4 tools (at least) - https://phabricator.wikimedia.org/T432822#12144571 (10dcaro) [07:50:39] (03update) 10waldir: globe: Slowly auto-rotate the idle globe, with a toggle [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/16 (owner: 10jforrester) [07:51:01] (03merge) 10waldir: globe: Slowly auto-rotate the idle globe, with a toggle [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/16 (owner: 10jforrester) [07:51:12] (03update) 10jforrester: A number of UX tweaks to shrink the view, to allow for more usability on mobile [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/18 [07:51:13] (03update) 10jforrester: A number of UX tweaks to shrink the view, to allow for more usability on mobile [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/18 [07:51:53] (03update) 10jforrester: A number of UX tweaks to shrink the view, to allow for more usability on mobile [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/18 [07:54:46] 10Toolforge, 06tools-platform-team: [toolforge,prod] httproutes disappeared for 4 tools (at least) - https://phabricator.wikimedia.org/T432822#12144595 (10dcaro) [07:55:34] (03approved) 10waldir: A number of UX tweaks to shrink the view, to allow for more usability on mobile [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/18 (owner: 10jforrester) [07:55:44] (03merge) 10waldir: A number of UX tweaks to shrink the view, to allow for more usability on mobile [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/18 (owner: 10jforrester) [07:58:25] (03approved) 10waldir: footer: Shorten how we list things so the footer isn't so dominant [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/19 (owner: 10jforrester) [07:58:31] 10Toolforge, 06tools-platform-team: [toolforge,prod] httproutes disappeared for 4 tools (at least) - https://phabricator.wikimedia.org/T432822#12144605 (10dcaro) 05In progress→03Invalid It turns out that the routes were manually deleted, and as they are not created by deploy they were not recreated aut... [07:58:35] (03update) 10waldir: footer: Shorten how we list things so the footer isn't so dominant [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/19 (owner: 10jforrester) [07:59:02] (03merge) 10waldir: footer: Shorten how we list things so the footer isn't so dominant [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/19 (owner: 10jforrester) [08:00:31] (03approved) 10dcaro: server: default to HostToContainer propagated mounts [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/51 (https://phabricator.wikimedia.org/T432325) (owner: 10filippo) [08:03:08] 10Cloud-VPS, 06tools-infrastructure-team, 06tools-platform-team, 13Patch-For-Review: Make sure dumps-nfs mount/umount is propagated inside Toolforge containers - https://phabricator.wikimedia.org/T432325#12144615 (10fgiunchedi) [08:03:42] 10Toolforge, 06tools-platform-team: [toolforge,prod] httproutes disappeared for 4 tools (at least) - https://phabricator.wikimedia.org/T432822#12144617 (10DamianZaremba) Verified this was an oversight when running a clean (remove all objects) deployment for T403735 Original alert (actually went down); ` t... [08:07:58] 10PAWS, 06tools-infrastructure-team, 06tools-platform-team: Make sure dumps-nfs mounts are propagated inside PAWS containers - https://phabricator.wikimedia.org/T432583#12144621 (10fgiunchedi) [08:16:48] (03update) 10dcaro: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) (owner: 10raymond-ndibe) [08:17:29] 10Tool-leximap: Create the Design UI layout for the tool - https://phabricator.wikimedia.org/T432246#12144653 (10Dagmawi-M) 05Open→03Resolved a:03Dagmawi-M [08:20:34] 10Tool-leximap: Design and make the gadget UI component - https://phabricator.wikimedia.org/T432827 (10Dagmawi-M) 03NEW [08:22:29] (03open) 10jforrester: coverage: Add target-language pronunciation as a third dimension [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/20 [08:23:21] 10Tool-leximap: Add a rotating logo icon for the gadget when processing words. - https://phabricator.wikimedia.org/T432828 (10Dagmawi-M) 03NEW [08:27:22] (03update) 10countcount: Cache small wikis' /ipinfo phase-1 IP rows in a 10-minute in-memory TTL cache [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/59 [08:27:48] (03update) 10countcount: Cache small wikis' /ipinfo phase-1 IP rows in memory, rebuilt every 10 minutes by a background task [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/59 [08:39:27] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12144726 (10Marostegui) clouddb1030 has been pooled in s1 [08:39:33] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12144727 (10Marostegui) [08:47:19] (03approved) 10waldir: coverage: Add target-language pronunciation as a third dimension [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/20 (owner: 10jforrester) [08:47:41] (03merge) 10waldir: coverage: Add target-language pronunciation as a third dimension [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/20 (owner: 10jforrester) [09:03:07] (03open) 10xanonymusx: Improve capitalisation of displayed languages [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/21 [09:03:29] (03open) 10jforrester: README: Expand a bit [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/22 [09:04:20] (03open) 10waldir: Adopt the Codex design system [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/23 [09:10:07] (03approved) 10jforrester: Adopt the Codex design system [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/23 (owner: 10waldir) [09:14:22] FIRING: [2x] HAProxyBackendUnavailable: HAProxy service wikireplica-db-analytics-s2 backend clouddb1018.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [09:17:11] (03update) 10jforrester: Improve capitalisation of displayed languages [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/21 (owner: 10xanonymusx) [09:17:23] (03approved) 10jforrester: Improve capitalisation of displayed languages [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/21 (owner: 10xanonymusx) [09:17:33] (03merge) 10jforrester: Improve capitalisation of displayed languages [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/21 (owner: 10xanonymusx) [09:19:43] (03update) 10jforrester: README: Expand a bit [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/22 [09:21:46] (03update) 10jforrester: README: Expand a bit [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/22 [09:25:00] (03open) 10lucaswerkmeister: Move change tag into config [toolforge-repos/win-the-kiwi] - 10https://gitlab.wikimedia.org/toolforge-repos/win-the-kiwi/-/merge_requests/1 [09:26:41] (03approved) 10waldir: README: Expand a bit [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/22 (owner: 10jforrester) [09:27:55] (03merge) 10waldir: README: Expand a bit [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/22 (owner: 10jforrester) [09:29:04] 10Data-Services, 06tools-platform-team: [wikireplicas] Create views for new wiki bolwiki - https://phabricator.wikimedia.org/T429954#12144845 (10aputhin) p:05Triage→03Medium [09:30:45] 06tools-platform-team: [jobs-api] replace class mocks with magicmocks - https://phabricator.wikimedia.org/T432832 (10dcaro) 03NEW [09:32:38] (03update) 10jforrester: Adopt the Codex design system [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/23 (owner: 10waldir) [09:33:54] 06tools-platform-team: [jobs-api] replace class mocks with magicmocks - https://phabricator.wikimedia.org/T432832#12144874 (10dcaro) [09:34:10] 10Toolforge, 06tools-platform-team: [jobs-api] replace class mocks with magicmocks - https://phabricator.wikimedia.org/T432832#12144877 (10dcaro) [09:34:53] (03update) 10countcount: Cache small wikis' /ipinfo phase-1 IP rows in memory, rebuilt every 10 minutes by a background task [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/59 [09:34:57] 10Toolforge, 06tools-platform-team, 10Data-Services (Quota-requests): Trove-only project request: Postgres database for Toolforge tool - omd - https://phabricator.wikimedia.org/T432818#12144880 (10dcaro) +1 [09:35:22] (03update) 10countcount: Cache small wikis' /ipinfo phase-1 IP rows in memory, rebuilt every 10 minutes by a background task [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/59 [09:40:20] (03update) 10jforrester: Adopt the Codex design system [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/23 (owner: 10waldir) [09:40:28] (03update) 10jforrester: Adopt the Codex design system [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/23 (owner: 10waldir) [09:40:44] (03merge) 10jforrester: Adopt the Codex design system [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/23 (owner: 10waldir) [09:51:46] (03open) 10jforrester: footer: Move detailed credits into a pop-up dialog [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/24 [09:58:20] (03merge) 10lucaswerkmeister: Move change tag into config [toolforge-repos/win-the-kiwi] - 10https://gitlab.wikimedia.org/toolforge-repos/win-the-kiwi/-/merge_requests/1 [10:01:28] (03update) 10jforrester: footer: Move detailed credits into a pop-up dialog [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/24 [10:01:59] (03update) 10jforrester: Various UX tweaks to save more space now we're on Codex [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/24 [10:14:15] (03update) 10tlepage: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 [10:15:00] (03update) 10renovatebot: Update pnpm to v11.15.1 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/9 [10:15:09] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/8 [10:16:10] (03update) 10renovatebot: Update pnpm to v11.15.1 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/44 [10:16:15] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/43 [10:17:21] 10Tool-lexeme-forms, 06translatewiki.net: l10n-bot pushed a twn branch for Wikidata Lexeme Forms but did not create a merge request - https://phabricator.wikimedia.org/T432838 (10LucasWerkmeister) 03NEW [10:18:33] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [10:18:39] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [10:20:14] (03update) 10jforrester: Various UX tweaks to save more space now we're on Codex [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/24 [10:20:18] (03update) 10jforrester: Various UX tweaks to save more space now we're on Codex [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/24 [10:20:37] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 03Wikimania-Hackathon-2026: [Team Challenges #07E] Wikidata–Wikisource Lexicographical Data Visualization Tool - https://phabricator.wikimedia.org/T432095#12145045 (10Dagmawi-M) hi @Kengkong1 @HakanIST Can you send us your photo as... [10:24:46] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 03Wikimania-Hackathon-2026: [Team Challenges #07E] Wikidata–Wikisource Lexicographical Data Visualization Tool - https://phabricator.wikimedia.org/T432095#12145056 (10Kengkong1) File:Kengkong (Kisenge Anthony Mbaga) Photo.jpg|Kengkon... [10:25:07] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 03Wikimania-Hackathon-2026: [Team Challenges #07E] Wikidata–Wikisource Lexicographical Data Visualization Tool - https://phabricator.wikimedia.org/T432095#12145057 (10Kengkong1) hi @Dagmawi-M, here thanks File:Kengkong (Kisenge Anth... [10:27:02] 10Tool-lexeme-forms, 06translatewiki.net: l10n-bot pushed a twn branch for Wikidata Lexeme Forms but did not create a merge request - https://phabricator.wikimedia.org/T432838#12145060 (10Nikerabbit) I am expecting this to be work on Thursday. There was an issue related to {T430530}. [10:29:07] (03approved) 10waldir: Various UX tweaks to save more space now we're on Codex [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/24 (owner: 10jforrester) [10:29:12] (03merge) 10waldir: Various UX tweaks to save more space now we're on Codex [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/24 (owner: 10jforrester) [10:29:40] (03open) 10jforrester: Draft: panel: Deep-link each missing field to where it can be fixed [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/25 [10:29:45] (03update) 10jforrester: Draft: panel: Deep-link each missing field to where it can be fixed [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/25 [10:29:49] (03update) 10jforrester: panel: Deep-link each missing field to where it can be fixed [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/25 [10:36:09] (03approved) 10waldir: panel: Deep-link each missing field to where it can be fixed [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/25 (owner: 10jforrester) [10:36:12] (03merge) 10waldir: panel: Deep-link each missing field to where it can be fixed [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/25 (owner: 10jforrester) [10:37:39] 10Tool-lexeme-forms, 06translatewiki.net: l10n-bot pushed a twn branch for Wikidata Lexeme Forms but did not create a merge request - https://phabricator.wikimedia.org/T432838#12145121 (10LucasWerkmeister) I see, thanks! (I just noticed that Wikidata Image Positions has the same issue, but you also mentioned t... [10:40:31] (03open) 10jforrester: Add a .mailmap so git history is slightly cleaner [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/26 [10:40:48] (03merge) 10jforrester: Add a .mailmap so git history is slightly cleaner [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/26 [10:44:37] (03open) 10lucaswerkmeister: Fix CSRF handling [toolforge-repos/win-the-kiwi] - 10https://gitlab.wikimedia.org/toolforge-repos/win-the-kiwi/-/merge_requests/2 [10:48:52] (03open) 10jforrester: build: Add Toolforge build service config for static hosting [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/27 [10:49:12] (03merge) 10seb35: Fix CSRF handling [toolforge-repos/win-the-kiwi] - 10https://gitlab.wikimedia.org/toolforge-repos/win-the-kiwi/-/merge_requests/2 (owner: 10lucaswerkmeister) [10:49:18] (03merge) 10jforrester: build: Add Toolforge build service config for static hosting [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/27 [10:58:15] (03open) 10jforrester: README: Document Toolforge deployment and refresh moved paths [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/28 [11:11:48] FIRING: PuppetFailure: Puppet has failed on clouddb1031:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [11:13:58] 06tools-platform-team, 10Toolhub, 13Patch-For-Review: Replace Bullseye base image with Trixie before August 2026 EOL deadline - https://phabricator.wikimedia.org/T425303#12145233 (10fnegri) 05In progress→03Resolved Deployed the new version to staging, codfw and eqiad following the instructions at htt... [11:14:12] (03update) 10renovatebot: Update pnpm to v11.15.1 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/9 [11:14:12] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/8 [11:15:02] (03update) 10renovatebot: Update pnpm to v11.15.1 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/44 [11:15:03] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/43 [11:20:14] (03open) 10waldir: Improve legend and tab navigation [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/29 [11:21:39] (03approved) 10jforrester: Improve legend and tab navigation [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/29 (owner: 10waldir) [11:21:42] (03merge) 10jforrester: Improve legend and tab navigation [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/29 (owner: 10waldir) [11:21:58] (03update) 10jforrester: README: Document Toolforge deployment and refresh moved paths [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/28 [11:31:29] (03update) 10jforrester: README: Document Toolforge deployment and refresh moved paths [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/28 [11:31:44] (03update) 10jforrester: README: Document Toolforge deployment and refresh moved paths [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/28 [11:46:48] RESOLVED: PuppetFailure: Puppet has failed on clouddb1031:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [11:48:16] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12145436 (10Marostegui) clouddb1031 replicating on s2 and s7 [11:49:22] RESOLVED: [2x] HAProxyBackendUnavailable: HAProxy service wikireplica-db-analytics-s2 backend clouddb1018.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [11:51:59] (03open) 10jberkel: Display demonyms by gender/plural [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/30 [11:56:02] (03open) 10tlepage: Draft: Support logs-api pagination [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/163 (https://phabricator.wikimedia.org/T428253) [12:00:19] (03update) 10tlepage: Draft: Support logs-api pagination [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/163 (https://phabricator.wikimedia.org/T428253) [12:04:15] (03update) 10jforrester: README: Document Toolforge deployment and refresh moved paths [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/28 [12:04:16] (03update) 10jforrester: README: Document Toolforge deployment and refresh moved paths [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/28 [12:04:21] (03update) 10jforrester: README: Document Toolforge deployment and refresh moved paths [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/28 [12:05:46] (03update) 10tlepage: Draft: Support logs-api pagination [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/163 (https://phabricator.wikimedia.org/T428253) [12:09:23] (03approved) 10jforrester: Display demonyms by gender/plural [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/30 (owner: 10jberkel) [12:14:55] (03update) 10waldir: Display demonyms by gender/plural [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/30 (owner: 10jberkel) [12:16:58] (03update) 10jberkel: Display demonyms by gender/plural [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/30 [12:17:28] (03update) 10waldir: Display demonyms by gender [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/30 (owner: 10jberkel) [12:18:58] (03update) 10waldir: Display demonyms by gender/plural [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/30 (owner: 10jberkel) [12:19:16] (03merge) 10waldir: Display demonyms by gender/plural [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/30 (owner: 10jberkel) [12:22:15] 06tools-infrastructure-team, 06Data-Platform-SRE: Migrate to dumps-nfs.w.o in production - https://phabricator.wikimedia.org/T432212#12145558 (10fgiunchedi) [12:23:51] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: image-config: add webservice-defaults to standard images - https://phabricator.wikimedia.org/T431403#12145573 (10dcaro) @Raymond_Ndibe this is done right? [12:24:02] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: image-config: add webservice-defaults to standard images - https://phabricator.wikimedia.org/T431403#12145574 (10dcaro) (we can do the cleanup too though as part of this) [12:24:40] (03approved) 10waldir: README: Document Toolforge deployment and refresh moved paths [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/28 (owner: 10jforrester) [12:25:02] (03update) 10waldir: README: Document Toolforge deployment and refresh moved paths [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/28 (owner: 10jforrester) [12:25:46] (03merge) 10waldir: README: Document Toolforge deployment and refresh moved paths [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/28 (owner: 10jforrester) [12:27:50] 10Toolforge, 06tools-platform-team: [jobs-api] separate jobs-framework k8s object templates from code - https://phabricator.wikimedia.org/T358815#12145585 (10dcaro) 05Open→03Declined I think this can be closed, will reopen with more info later if needed. [12:40:27] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 - https://phabricator.wikimedia.org/T431682#12145603 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: `cloudvirt1048.eqiad.wmnet... [12:43:56] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 - https://phabricator.wikimedia.org/T431682#12145638 (10fgiunchedi) @VRiley-WMF I realized I should be doing the decommission cloudvirt1048 myself, host is now decom'd (mgmt dn... [12:44:44] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12145639 (10fgiunchedi) [12:51:55] (03update) 10filippo: Revert "bandaid high RLIMIT_NOFILE in gitlab runners" [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/90 (https://phabricator.wikimedia.org/T426827) [12:52:07] (03approved) 10filippo: Revert "bandaid high RLIMIT_NOFILE in gitlab runners" [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/90 (https://phabricator.wikimedia.org/T426827) [12:52:13] (03merge) 10filippo: Revert "bandaid high RLIMIT_NOFILE in gitlab runners" [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/90 (https://phabricator.wikimedia.org/T426827) [12:59:03] (03update) 10filippo: utils: add build_images.py [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/88 [13:00:27] (03merge) 10filippo: playbooks: add /mnt/nfs/dumps [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/329 (https://phabricator.wikimedia.org/T411248) [13:06:06] 10PAWS, 06tools-platform-team: [bug] Notebook container not spawing for user Soumendrapatra21 - https://phabricator.wikimedia.org/T431744#12145739 (10dcaro) 05Open→03Declined a:03dcaro It's been already some time without reply, I'll close but please reopen if/when you provide the missing information. [13:06:17] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: TLS Certificate Provisioning for wikiba.se Subdomain - https://phabricator.wikimedia.org/T430057#12145743 (10Jacob_WMDE) Hi @taavi, that was a mistake when setting the record. It's now been corrected. [13:09:53] (03open) 10filippo: debian-builder: update to latest images [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/92 (https://phabricator.wikimedia.org/T426827) [13:10:34] (03update) 10dcaro: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) (owner: 10raymond-ndibe) [13:11:55] 10Tool-CiteFix: Promote Citefix from Userscript to Gadget - https://phabricator.wikimedia.org/T432858 (10Gopavasanth) 03NEW [13:11:59] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: TLS Certificate Provisioning for wikiba.se Subdomain - https://phabricator.wikimedia.org/T430057#12145783 (10taavi) 05Open→03Resolved a:03taavi This should be all done from our end. [13:15:39] 10Toolforge, 06tools-platform-team: [jobs-api] cleanup resources when failing to create a job - https://phabricator.wikimedia.org/T432687#12145828 (10dcaro) p:05Triage→03High a:03dcaro [13:15:43] 10Toolforge, 06tools-platform-team: [jobs-api] cleanup resources when failing to create a job - https://phabricator.wikimedia.org/T432687#12145830 (10dcaro) p:05High→03Triage a:05dcaro→03None [13:19:45] 10Toolforge, 06tools-platform-team: [jobs-api] cleanup resources when failing to create a job - https://phabricator.wikimedia.org/T432687#12145850 (10dcaro) p:05Triage→03High a:03dcaro [13:19:58] (03open) 10dcaro: core: cleanup after failing to create job in runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/358 [13:20:02] 10Tool-CiteFix: Promote Citefix from Userscript to Gadget - https://phabricator.wikimedia.org/T432858#12145855 (10Gopavasanth) ++ Adding @Johannes_Richter_WMDE , since we spoke on about this project during the Wikimania 2026 Team challenges. [13:22:33] 10Tool-wikinewsie: Wikinewsie Clickable Blue Links - https://phabricator.wikimedia.org/T432859 (10ZhaoFJx) 03NEW [13:22:34] 10Toolforge, 06tools-platform-team: [jobs-api] cleanup resources when failing to create a job - https://phabricator.wikimedia.org/T432687#12145873 (10dcaro) 05Open→03In progress [13:22:40] 10Tool-wikinewsie: Wikinewsie Clickable Blue Links - https://phabricator.wikimedia.org/T432859#12145878 (10ZhaoFJx) a:03ZhaoFJx [13:22:50] 10Tool-wikinewsie: Wikinewsie Clickable Blue Links - https://phabricator.wikimedia.org/T432859#12145880 (10ZhaoFJx) 05Open→03In progress [13:25:18] (03update) 10dcaro: core: cleanup after failing to create job in runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/358 [13:30:14] (03close) 10filippo: Export dumps-nfs directory [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/50 (https://phabricator.wikimedia.org/T411248) [13:38:39] (03update) 10dcaro: core: cleanup after failing to create job in runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/358 [13:41:24] 10Cloud-VPS, 06tools-infrastructure-team: Network unavailable on a few VMs - https://phabricator.wikimedia.org/T432426#12146025 (10Andrew) I asked @Jelto about one of his affected hosts and he says "But occasionally some of our WMCS instance lose network connectivity and I just "soft reboot" them from the hori... [13:48:31] (03update) 10dcaro: core: cleanup after failing to create job in runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/358 [13:51:59] 10Toolforge, 06tools-platform-team, 07Epic: ST5.4.2 Toolforge Alerting System - https://phabricator.wikimedia.org/T432860 (10fnegri) 03NEW [13:55:10] 10Toolforge, 06tools-platform-team: Verify if we can define alert conditions in Prometheus - https://phabricator.wikimedia.org/T432863 (10fnegri) 03NEW [13:55:59] 10Toolforge, 06tools-platform-team: Verify if we can define alert conditions in Prometheus - https://phabricator.wikimedia.org/T432863#12146137 (10fnegri) [13:57:04] 10Toolforge, 06tools-platform-team, 07Epic: Provide a way for users to enable and disable alerts - https://phabricator.wikimedia.org/T432864 (10fnegri) 03NEW [13:58:25] 10Toolforge, 06tools-platform-team: Set up a Toolforge “alerting service” that emails tool maintainers when an alert condition is detected - https://phabricator.wikimedia.org/T432865 (10fnegri) 03NEW [13:58:33] 10Toolforge, 06tools-platform-team: Provide a way for users to enable and disable alerts - https://phabricator.wikimedia.org/T432864#12146177 (10fnegri) [14:02:34] (03update) 10dcaro: core: cleanup after failing to create job in runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/358 [14:08:12] 10Toolforge, 06tools-platform-team: [logs-api,loki] Implement a system-level logging endpoint (write) - https://phabricator.wikimedia.org/T432565#12146211 (10aputhin) [14:10:40] 10Toolforge, 06tools-platform-team: [logs-cli,toolforge-cli] Add subcommand toolforge logs to query and retrieve the relevant logs, will have to design a minimal interface to filter by event/subsystem/time/etc. - https://phabricator.wikimedia.org/T432572#12146222 (10aputhin) a:05dcaro→03CCiufo-WMF Please r... [14:12:46] (03open) 10waldir: Improve language selector [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/31 [14:12:56] (03update) 10waldir: Improve language selector [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/31 [14:20:07] FIRING: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [14:28:16] 10Toolforge, 06tools-platform-team: [logs-cli,toolforge-cli] Add subcommand toolforge logs to query and retrieve the relevant logs, will have to design a minimal interface to filter by event/subsystem/time/etc. - https://phabricator.wikimedia.org/T432572#12146313 (10aputhin) [14:39:36] FIRING: InstanceDown: Project metricsinfra instance metricsinfra-prometheus-2 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:39:36] FIRING: [3x] TargetDown: Job blackbox-exporter is unreachable in project metricsinfra instance metricsinfra-prometheus-2 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [14:40:10] !log taavi@cloudcumin1001 metricsinfra START - Cookbook wmcs.openstack.quota_increase by 8 cores, 16384 ram [14:40:18] !log taavi@cloudcumin1001 metricsinfra END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 8 cores, 16384 ram [14:44:19] 10Toolforge, 06tools-platform-team: Verify if we can define alert conditions in Prometheus - https://phabricator.wikimedia.org/T432863#12146459 (10aputhin) Proof of concept: fire alerts with a test tool and see if it appears in Prometheus [14:44:36] RESOLVED: InstanceDown: Project metricsinfra instance metricsinfra-prometheus-2 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:44:36] RESOLVED: [3x] TargetDown: Job blackbox-exporter is unreachable in project metricsinfra instance metricsinfra-prometheus-2 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [14:45:02] (03merge) 10jforrester: Improve language selector [toolforge-repos/map-the-gap] - 10https://gitlab.wikimedia.org/toolforge-repos/map-the-gap/-/merge_requests/31 (owner: 10waldir) [14:46:03] !log taavi@cloudcumin1001 metricsinfra START - Cookbook wmcs.vps.refresh_puppet_certs on metricsinfra-prometheus-4.metricsinfra.eqiad1.wikimedia.cloud [14:47:28] !log taavi@cloudcumin1001 metricsinfra END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on metricsinfra-prometheus-4.metricsinfra.eqiad1.wikimedia.cloud [14:49:36] FIRING: [2x] TargetDown: Job blackbox-exporter is unreachable in project metricsinfra instance metricsinfra-prometheus-4 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [14:54:36] FIRING: [6x] TargetDown: Job blackbox-exporter is unreachable in project metricsinfra instance metricsinfra-prometheus-2 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [14:54:36] 06cloud-services-team, 10Tool-paulina, 10Toolforge, 06tools-platform-team, 07Kubernetes: Automated traffic affects tool performance - https://phabricator.wikimedia.org/T432878 (10Pepe_piton) 03NEW [14:55:10] (03PS1) 10Majavah: api: Fix joinedload() for current SQLAlchemy [cloud/metricsinfra/prometheus-manager] - 10https://gerrit.wikimedia.org/r/1313976 [15:05:39] (03CR) 10Majavah: [C:03+2] api: Fix joinedload() for current SQLAlchemy [cloud/metricsinfra/prometheus-manager] - 10https://gerrit.wikimedia.org/r/1313976 (owner: 10Majavah) [15:06:16] (03Merged) 10jenkins-bot: api: Fix joinedload() for current SQLAlchemy [cloud/metricsinfra/prometheus-manager] - 10https://gerrit.wikimedia.org/r/1313976 (owner: 10Majavah) [15:09:28] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [envars] Provide a $TOOL or $TOOL_NAME default envvar - https://phabricator.wikimedia.org/T419601#12146614 (10dcaro) [15:18:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [15:24:13] 10Tool-glam-tool-hospital: Wikimedia Hackathon 2026: GLAM Tool Hospital - https://phabricator.wikimedia.org/T424464#12146686 (10Aklapper) 05Open→03Resolved a:03Dactylantha Assuming this is resolved. [15:26:56] FIRING: MetricsinfraPrometheusLowRetention: Prometheus is storing less than 20 days of data on metricsinfra-prometheus-4. - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMetricsinfraPrometheusLowRetention [15:28:03] FIRING: [2x] TargetDown: Job toolsdb-mariadb is unreachable in project tools instance tools-db-6 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [15:28:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [15:29:36] RESOLVED: [3x] TargetDown: Job blackbox-exporter is unreachable in project metricsinfra instance metricsinfra-prometheus-4 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [15:33:03] RESOLVED: [2x] TargetDown: Job toolsdb-mariadb is unreachable in project tools instance tools-db-6 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [15:42:30] (03merge) 10taavi: tools: Sort .svc. DNS entries [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/122 [15:42:33] (03update) 10taavi: tools: Add Opensearch svc DNS entries [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/123 (https://phabricator.wikimedia.org/T401818) [15:43:42] (03merge) 10taavi: tools: Add Opensearch svc DNS entries [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/123 (https://phabricator.wikimedia.org/T401818) [15:43:43] (03update) 10taavi: tools: Add volumes to Opensearch VMs [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/124 (https://phabricator.wikimedia.org/T401818) [15:45:09] (03merge) 10taavi: tools: Add volumes to Opensearch VMs [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/124 (https://phabricator.wikimedia.org/T401818) [15:48:40] (03update) 10dcaro: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 (owner: 10tlepage) [15:50:15] 10Toolforge, 06tools-platform-team: Set up a Toolforge “alerting service” that emails tool maintainers when an alert condition is detected - https://phabricator.wikimedia.org/T432865#12146829 (10komla) [16:02:49] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: [hypothesis] ST5.4.1 Extend logging capabilities - https://phabricator.wikimedia.org/T432564#12146875 (10aputhin) [16:03:20] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-api,loki] Implement a system-level logging endpoint (write) - https://phabricator.wikimedia.org/T432565#12146877 (10aputhin) [16:03:32] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-api] Implement an endpoint to retrieve system-level event logs - https://phabricator.wikimedia.org/T432566#12146880 (10aputhin) [16:04:03] 10Toolforge, 06tools-platform-team, 07OKR-Work: [jobs-api] Send webservice start/stop/restart events to the new logs-api event log endpoint - https://phabricator.wikimedia.org/T432570#12146881 (10aputhin) [16:04:24] 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] send deployment events to the logs-api system-event log endpoint - https://phabricator.wikimedia.org/T432571#12146882 (10aputhin) [16:04:51] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-cli,toolforge-cli] Add subcommand toolforge logs to query and retrieve the relevant logs, will have to design a minimal interface to filter by event/subsystem/time/etc. - https://phabricator.wikimedia.org/T432572#12146883 (10aputhin) [16:07:52] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.2 Toolforge Alerting System - https://phabricator.wikimedia.org/T432860#12146892 (10aputhin) [16:09:00] 10Toolforge, 06tools-platform-team, 07OKR-Work: Verify if we can define alert conditions in Prometheus - https://phabricator.wikimedia.org/T432863#12146894 (10aputhin) [16:09:17] 10Toolforge, 06tools-platform-team, 07OKR-Work: Provide a way for users to enable and disable alerts - https://phabricator.wikimedia.org/T432864#12146897 (10aputhin) [16:13:12] 10Toolforge, 06tools-platform-team, 07OKR-Work: Set up a Toolforge “alerting service” that emails tool maintainers when an alert condition is detected - https://phabricator.wikimedia.org/T432865#12146904 (10aputhin) [16:21:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:25:03] (03update) 10tlepage: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 [16:26:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:28:13] (03update) 10tlepage: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 [16:32:40] (03PS5) 10Jsn.sherman: Repository: config-driven dblist assembly [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309801 (https://phabricator.wikimedia.org/T344090) [16:32:40] (03PS8) 10Jsn.sherman: Repository: cover isWMF branches [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309830 (https://phabricator.wikimedia.org/T344090) [16:32:41] (03PS4) 10Jsn.sherman: EditCounterRepository: unit test coverage [labs/xtools] - 10https://gerrit.wikimedia.org/r/1312259 (https://phabricator.wikimedia.org/T384711) [16:32:41] (03PS4) 10Jsn.sherman: Repository: unbound params in untested branches [labs/xtools] - 10https://gerrit.wikimedia.org/r/1312267 (https://phabricator.wikimedia.org/T384711) [16:32:42] (03PS4) 10Jsn.sherman: Repository: unit test coverage [labs/xtools] - 10https://gerrit.wikimedia.org/r/1312268 (https://phabricator.wikimedia.org/T384711) [17:08:38] (03PS1) 10Majavah: add fake opensearch keepalived password [labs/private] - 10https://gerrit.wikimedia.org/r/1314054 [17:10:09] (03CR) 10Majavah: [V:03+2 C:03+2] add fake opensearch keepalived password [labs/private] - 10https://gerrit.wikimedia.org/r/1314054 (owner: 10Majavah) [17:12:14] (03update) 10tlepage: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 [17:58:20] PROBLEM - Host cloudvirt1071 is DOWN: PING CRITICAL - Packet loss = 100% [17:58:33] FIRING: [7x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:02:10] FIRING: CloudVirtDown: Cloudvirt node cloudvirt1071 is down. #page - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CloudVirtDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1071 - https://alerts.wikimedia.org/?q=alertname%3DCloudVirtDown [18:02:33] FIRING: [2x] InstanceDown: Project tools instance tools-k8s-haproxy-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:03:33] FIRING: [8x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:03:50] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1071 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [18:03:59] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.set_maintenance (T431374) [18:04:02] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) (T431374) [18:04:05] T431374: cloudvirt1071 crash, again - https://phabricator.wikimedia.org/T431374 [18:04:47] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops: cloudvirt1071 crash, again - https://phabricator.wikimedia.org/T431374#12147464 (10Andrew) Crashed again just now [18:09:48] RECOVERY - Host cloudvirt1071 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms [18:11:19] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1071.eqiad.wmnet' (T431374) [18:11:27] T431374: cloudvirt1071 crash, again - https://phabricator.wikimedia.org/T431374 [18:12:33] FIRING: [2x] InstanceDown: Project tools instance tools-k8s-haproxy-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:13:33] RESOLVED: [8x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:13:50] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1071 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [18:16:17] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1071.eqiad.wmnet' (T431374) [18:17:33] RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-haproxy-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:17:46] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops: cloudvirt1071 crash, again - https://phabricator.wikimedia.org/T431374#12147518 (10Andrew) Host is now drained and in maintenance mode. I don't have great ideas about what to look for next. Syslog shows a 15-minute gap. ` 2026-07... [18:20:40] RESOLVED: CloudVirtDown: Cloudvirt node cloudvirt1071 is down. #page - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CloudVirtDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1071 - https://alerts.wikimedia.org/?q=alertname%3DCloudVirtDown [18:52:01] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 03Wikimania-Hackathon-2026: [Team Challenges #07E] Wikidata–Wikisource Lexicographical Data Visualization Tool - https://phabricator.wikimedia.org/T432095#12147603 (10Ninovolador) Hello! I tested the tool and it seems very promising... [19:15:07] RESOLVED: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [19:21:07] FIRING: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [20:03:46] 06cloud-services-team, 10Tool-paulina, 10Toolforge, 06tools-platform-team, 07Kubernetes: Automated traffic affects tool performance - https://phabricator.wikimedia.org/T432878#12147778 (10BLiviero-WMF) If it's visible to you, this phab ticket on Xtools contains a dialog about what other developers were t... [20:33:38] 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.33 - https://phabricator.wikimedia.org/T408785#12147833 (10Andrew) https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Upgrading_Kubernetes [20:35:52] 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.33 - https://phabricator.wikimedia.org/T408785#12147835 (10Andrew) [20:37:28] 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.33 - https://phabricator.wikimedia.org/T408785#12147836 (10Andrew) [20:39:00] 10Cloud-VPS (Quota-requests), 10Catalyst (Luka Ijo Pimeja Jan): Quota increase request for project catalyst - https://phabricator.wikimedia.org/T432617#12147849 (10jeena) [20:41:19] 10PAWS, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T431055#12147877 (10LibUp-bot) A new upstream version of Pywikibot is now available: 11.6.0. * https://wikitech.wikimedia.org/wiki/PAWS/Admin#Update_the_pywikibot_version * https://gerrit.wikimedia.org/g/pywik... [20:41:21] 10Toolforge, 06tools-platform-team: [pywikibot-buildservice] New upstream release for Pywikibot - https://phabricator.wikimedia.org/T431056#12147878 (10LibUp-bot) A new upstream version of Pywikibot is now available: 11.6.0. * https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Pywikibot_image * https:/... [20:48:10] 06cloud-services-team, 10Tool-paulina, 10Toolforge, 06tools-platform-team, 07Kubernetes: Automated traffic affects tool performance - https://phabricator.wikimedia.org/T432878#12147898 (10bd808) The quickest protection that a web tool can add is Wikimedia OAuth identity-only login before doing any backen... [20:59:45] 10Tool-wiki-gender-stats: Add a button to download table data - https://phabricator.wikimedia.org/T432919 (10Danya) 03NEW [21:00:15] 10Tool-wiki-gender-stats: Add a button to download table data - https://phabricator.wikimedia.org/T432919#12147976 (10Danya) [21:00:16] 10Tool-wiki-gender-stats: Add a button to download table data - https://phabricator.wikimedia.org/T432919#12147978 (10Danya) p:05Triage→03Medium [21:00:46] 10Tool-wiki-gender-stats: Add a button to download tables’ data - https://phabricator.wikimedia.org/T432919#12147981 (10Danya) [21:11:07] RESOLVED: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [21:13:57] (03update) 10renovatebot: Update pnpm to v11.15.1 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/9 [21:14:25] (03update) 10renovatebot: Update pnpm to v11.16.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/9 [21:15:27] (03update) 10renovatebot: Update pnpm to v11.16.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/44 [21:15:27] (03update) 10renovatebot: Update pnpm to v11.16.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/44 [21:16:25] (03update) 10renovatebot: Update pnpm to v11.15.1 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/24 [21:18:02] 10Tool-wiki-gender-stats: Migrate to ToolsDB - https://phabricator.wikimedia.org/T432053#12148023 (10Danya) [21:18:03] 10Tool-wiki-gender-stats: Try to integrate data from Humaniki - https://phabricator.wikimedia.org/T432516#12148022 (10Danya) [21:18:34] 10Tool-wiki-gender-stats: Try to integrate data from Humaniki - https://phabricator.wikimedia.org/T432516#12148024 (10Danya) 05Open→03Stalled Stalled until T432053 is resolved. [21:21:08] 10Tool-wiki-gender-stats: Add a button to download tables’ data - https://phabricator.wikimedia.org/T432919#12148044 (10Danya) [21:21:54] 10Tool-wiki-gender-stats: [VERSION] 0.5.0 - https://phabricator.wikimedia.org/T432922 (10Danya) 03NEW [21:21:56] 10Tool-wiki-gender-stats: [VERSION] 0.5.0 - https://phabricator.wikimedia.org/T432922#12148055 (10Danya) [21:22:01] 10Tool-wiki-gender-stats: [VERSION] 0.5.0 - https://phabricator.wikimedia.org/T432922#12148057 (10Danya) p:05Triage→03Medium [21:22:06] 10Tool-wiki-gender-stats: [VERSION] 0.5.0 - https://phabricator.wikimedia.org/T432922#12148059 (10Danya) p:05Medium→03High [21:23:51] 10Tool-wiki-gender-stats: [VERSION] 0.5.0 - https://phabricator.wikimedia.org/T432922#12148061 (10Danya) [21:23:52] 10Tool-wiki-gender-stats: Link to wikis and Wikidata elements in languages, countries, and professions tables - https://phabricator.wikimedia.org/T425089#12148060 (10Danya) [21:23:54] 10Tool-wiki-gender-stats: [VERSION] 0.5.0 - https://phabricator.wikimedia.org/T432922#12148063 (10Danya) [21:23:55] 10Tool-wiki-gender-stats: Add a button to download tables’ data - https://phabricator.wikimedia.org/T432919#12148062 (10Danya) [21:23:57] 10Tool-wiki-gender-stats: [VERSION] 0.5.0 - https://phabricator.wikimedia.org/T432922#12148065 (10Danya) [21:23:58] 10Tool-wiki-gender-stats: Add button to download charts as PNGs - https://phabricator.wikimedia.org/T432545#12148064 (10Danya) [21:24:01] 10Tool-wiki-gender-stats: Translate language names in genders-by-language page - https://phabricator.wikimedia.org/T426057#12148066 (10Danya) [21:24:05] 10Tool-wiki-gender-stats: [VERSION] 0.5.0 - https://phabricator.wikimedia.org/T432922#12148067 (10Danya) [21:24:09] 10Tool-wiki-gender-stats: Add a Privacy Statement - https://phabricator.wikimedia.org/T426262#12148068 (10Danya) [21:24:13] 10Tool-wiki-gender-stats: Add a search bar in the aggregated regions dialog - https://phabricator.wikimedia.org/T432534#12148070 (10Danya) [21:24:17] 10Tool-wiki-gender-stats: [VERSION] 0.5.0 - https://phabricator.wikimedia.org/T432922#12148071 (10Danya) [21:25:47] 10Tool-wiki-gender-stats: Use local storage instead of cookies for FilterWidget - https://phabricator.wikimedia.org/T432533#12148073 (10Danya) [21:25:48] 10Tool-wiki-gender-stats: [VERSION] 0.5.0 - https://phabricator.wikimedia.org/T432922#12148074 (10Danya) [21:25:59] 10Tool-wiki-gender-stats: [VERSION] 0.5.0 - https://phabricator.wikimedia.org/T432922#12148076 (10Danya) [21:26:00] 10Tool-wiki-gender-stats: Change link to contribution guide - https://phabricator.wikimedia.org/T432532#12148075 (10Danya) [21:27:56] 10Tool-wiki-gender-stats: Content-Security-Policy when loading OSM map tiles - https://phabricator.wikimedia.org/T432531#12148080 (10Danya) [22:14:09] (03update) 10renovatebot: Update pnpm to v11.16.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/9 [22:14:56] (03update) 10renovatebot: Update pnpm to v11.16.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/44