[01:54:07] 10Toolforge, 06tools-platform-team, 07good first task: [builds-cli] does not output valid json when there's no builds - https://phabricator.wikimedia.org/T429229#12052371 (10Buxtang) Hello, I have local changes to propose for this issue, essentially just adding err=True to the click.echo that's causing this... [03:04:27] 10Tool-bash: Bash: Slime application error on login attempt - https://phabricator.wikimedia.org/T430088#12052417 (10Peachey88) 05Open→03Invalid Caused by {T430092}. [03:07:10] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06DC-Ops, 10ops-eqiad: Ensure cloudvirt capacity is more evenly spread out among racks - https://phabricator.wikimedia.org/T424658#12052421 (10Jclark-ctr) [03:50:10] 10Tool-glam-tool-hospital: Other / I am not sure - Feature request - This task requests a way for `values from wikidata` to store the underlying QID (while the user still sees and selects a human-readable label) — - https://phabricator.wikimedia.org/T430098 (10GLAM-Tool-Hospital-Bot) 03NEW p:05Triage→03High [04:16:49] 10Tool-Pageviews, 06Data-Engineering, 06Data-Engineering-Icebox, 10Data-Engineering-Wikistats, and 2 others: Pageviews Analysis 3.0 (Vue + Codex) - https://phabricator.wikimedia.org/T378549#12052453 (10MusikAnimal) [04:47:48] 10Tool-subtitler: Localisation - https://phabricator.wikimedia.org/T430099 (10psubhashish1) 03NEW [04:48:35] 10Tool-subtitler: How to see the number of languages already subtitled for a particular video/audio file on Commons - https://phabricator.wikimedia.org/T430100 (10psubhashish1) 03NEW [04:54:10] 10Tool-subtitler: User panel, admin panel and event coordination for Subtitler - https://phabricator.wikimedia.org/T430101 (10psubhashish1) 03NEW [05:22:10] 10Tool-subtitler: In-subtitle line break removes characters from preceding word - https://phabricator.wikimedia.org/T430105 (10psubhashish1) 03NEW [05:34:46] 10Tool-subtitler: Localisation feature needs to be added - https://phabricator.wikimedia.org/T430107 (10psubhashish1) 03NEW [05:35:23] 10Tool-subtitler: Localisation - https://phabricator.wikimedia.org/T430099#12052595 (10psubhashish1) →14Duplicate dup:03T430107 [05:35:26] 10Tool-subtitler: Localisation feature needs to be added - https://phabricator.wikimedia.org/T430107#12052597 (10psubhashish1) [05:40:49] 10Tool-subtitler: In-subtitle line break removes characters from preceding word and adds an extra line - https://phabricator.wikimedia.org/T430105#12052612 (10psubhashish1) [05:43:12] 10Tool-subtitler: Update/upload TimedText on Commons - https://phabricator.wikimedia.org/T429917#12052617 (10psubhashish1) 05Open→03Resolved [05:44:24] 10Tool-subtitler: In-subtitle line break removes characters from preceding word and adds an extra line - https://phabricator.wikimedia.org/T430105#12052618 (10psubhashish1) [06:07:02] 10Cloud-VPS (Debian Bullseye Deprecation), 06collaboration-services: shutdown sourcebot1 instance in cloud VPS - https://phabricator.wikimedia.org/T429496#12052630 (10komla) [06:07:33] 10Cloud-VPS (Debian Bullseye Deprecation), 06collaboration-services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12052631 (10komla) [06:10:19] 10Cloud-VPS (Debian Bullseye Deprecation), 10VPS-Projects, 10VisualEditor, 10Catalyst (PatchDemo): Clean up the patchdemo3.visualeditor.eqiad1.wikimedia.cloud instance - https://phabricator.wikimedia.org/T429836#12052632 (10komla) [06:26:34] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12052640 (10fgiunchedi) Good find @ayounsi ! Also with respect to racking these hosts, my preference would be to have one host per rack once we can do 25G... [06:42:55] 06cloud-services-team, 10Cloud-VPS: Monitoring/metrics for trove instances - https://phabricator.wikimedia.org/T402738#12052668 (10fgiunchedi) >>! In T402738#12040784, @Andrew wrote: >>>! In T402738#12040229, @fgiunchedi wrote: >>>>! In T402738#12033703, @Andrew wrote: >>>>>! In T402738#12033646, @fgiunchedi w... [06:47:30] 06cloud-services-team, 10Toolforge: [toolforge] Rebuild Toolforge workers for bigger node sizes and in new VXLAN network - https://phabricator.wikimedia.org/T390228#12052674 (10fgiunchedi) Work in {T420565} has been completed with new defaults for memory requests, I'm in favor of defaulting to bigger workers. [06:51:07] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T424802) [06:51:14] T424802: cloudvirt1075 in 'maintenance' aggregate - https://phabricator.wikimedia.org/T424802 [06:51:16] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T424802) [06:55:27] 06cloud-services-team, 10Cloud-VPS: cloudvirt1075 in 'maintenance' aggregate - https://phabricator.wikimedia.org/T424802#12052680 (10fgiunchedi) I ran the cookbook, which looked like it worked and doesn't seem like it did? ` root@cloudcumin1001:~# cookbook wmcs.openstack.cloudvirt.unset_maintenance --task-id... [06:56:13] 06cloud-services-team, 10Cloud-VPS: cloudvirt1075 in 'maintenance' aggregate - https://phabricator.wikimedia.org/T424802#12052681 (10fgiunchedi) @Andrew does this behavior ring a bell? And similarly, I'm not sure the other three hosts are expected to be in maintenance ? [06:58:53] 10Cloud-VPS (Debian Bullseye Deprecation), 10WMIT-Infrastructure: Cloud VPS wlmitvisual01: upgrade Debian bullseye -> trixie - https://phabricator.wikimedia.org/T429723#12052683 (10Ysogo) Following discussion with WMI boarding, we are now planning to move the application now running on wlm-it-visual to wlm-app... [08:03:55] 06cloud-services-team, 10Toolforge, 06tools-platform-team: Audit tools memory requests vs actual usage - https://phabricator.wikimedia.org/T420565#12052909 (10dcaro) > The longer term plan/actionable is to expose resource utilization numbers directly to tool maintainers and they can take action as needed... [08:04:22] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12052913 (10Marostegui) I've pooled clouddb1026 with weight (10/100) in s1, and if it goes well for a few hours I will fully pool. [08:05:22] FIRING: HAProxyBackendUnavailable: HAProxy service wikireplica-db-web-s1 backend clouddb1026.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [08:06:53] (03update) 10dcaro: core: don't use runtime jobs as source of truth [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/314 [08:07:05] (03update) 10dcaro: core: don't use runtime jobs as source of truth [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/314 [08:11:40] 06cloud-services-team, 10Striker: [striker] Unable to create phabricator project for tool odivoir – HTTP 500 Internal Server Error on toolsadmin - https://phabricator.wikimedia.org/T429857#12052948 (10dcaro) [08:18:48] 06cloud-services-team, 10Striker: [striker] Failing to create new phabricator projects - https://phabricator.wikimedia.org/T430120 (10dcaro) 03NEW [08:26:25] 06cloud-services-team, 10Cloud-VPS: cloudvirt1075 in 'maintenance' aggregate - https://phabricator.wikimedia.org/T424802#12052986 (10fgiunchedi) cc {T284747} [08:28:29] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06tools-infrastructure-team: Put cloudvirt10[77-80] in service - https://phabricator.wikimedia.org/T429563#12052997 (10fgiunchedi) Following up from the network syncup from yesterday: I'll be trying to provision cloudvirt1077 end to end and see what is... [08:31:39] 06cloud-services-team, 10Striker, 06tools-platform-team: [striker] Failing to create new phabricator projects - https://phabricator.wikimedia.org/T430120#12053011 (10dcaro) [08:32:01] 06cloud-services-team, 10Striker, 06tools-platform-team: [striker] Unable to create phabricator project for tool odivoir – HTTP 500 Internal Server Error on toolsadmin - https://phabricator.wikimedia.org/T429857#12053013 (10dcaro) [08:40:46] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T429907) [08:40:51] T429907: [jobs-api] using "--filelog" without "--mount" saves invalid jobs into storage - https://phabricator.wikimedia.org/T429907 [08:50:18] 06cloud-services-team, 10Toolforge, 06tools-platform-team: Audit tools memory requests vs actual usage - https://phabricator.wikimedia.org/T420565#12053064 (10fgiunchedi) >>! In T420565#12052909, @dcaro wrote: >> The longer term plan/actionable is to expose resource utilization numbers directly to tool m... [08:54:12] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api (T429907) [08:54:17] T429907: [jobs-api] using "--filelog" without "--mount" saves invalid jobs into storage - https://phabricator.wikimedia.org/T429907 [08:57:06] (03update) 10fnegri: jobs-api: bump to 0.0.515-20260624163159-bc15c172 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1305 (https://phabricator.wikimedia.org/T428354 https://phabricator.wikimedia.org/T429907) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [08:57:30] (03merge) 10fnegri: jobs-api: bump to 0.0.515-20260624163159-bc15c172 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1305 (https://phabricator.wikimedia.org/T428354 https://phabricator.wikimedia.org/T429907) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [08:58:58] 06tools-platform-team, 13Patch-For-Review: [jobs-api] using "--filelog" without "--mount" saves invalid jobs into storage - https://phabricator.wikimedia.org/T429907#12053108 (10fnegri) 05Open→03Resolved Deployed to tools and verified it works. [09:08:56] FIRING: [6x] SystemdUnitDown: The service unit libvirtd-admin.socket is in failed status on host cloudvirt1077. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1077 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [09:18:24] (03open) 10countcount: Update lockfile to included updated quinn [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/25 [09:20:08] (03update) 10countcount: Update lockfile to included updated quinn [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/25 [09:20:50] PROBLEM - SSH on cloudvirt1077 is CRITICAL: connect to address 10.64.151.9 and port 22: Connection refused https://wikitech.wikimedia.org/wiki/SSH/monitoring [09:21:50] RECOVERY - SSH on cloudvirt1077 is OK: SSH OK - OpenSSH_10.0p2 Debian-7+deb13u4 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [09:23:56] RESOLVED: [6x] SystemdUnitDown: The service unit libvirtd-admin.socket is in failed status on host cloudvirt1077. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1077 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [09:35:28] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Deploy the OpenAPI linter in CI - https://phabricator.wikimedia.org/T422920#12053207 (10KBach) [09:35:50] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Deploy the OpenAPI linter in CI - https://phabricator.wikimedia.org/T422920#12053217 (10KBach) p:05Triage→03Medium [09:38:41] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:49:06] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06tools-infrastructure-team: Deprecated live_migration_uri nova-compute config option - https://phabricator.wikimedia.org/T430129 (10fgiunchedi) 03NEW [10:00:34] (03merge) 10countcount: Update lockfile to included updated quinn [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/25 [10:17:40] (03update) 10renovatebot: Draft: Update Rust crate tower-http to 0.7.0 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/23 [10:19:29] (03close) 10dcaro: k8s: use job_type instead of custom string for labels [repos/cloud/toolforge/jobs-api] (dont_use_kind) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/321 [10:20:22] RESOLVED: HAProxyBackendUnavailable: HAProxy service wikireplica-db-web-s1 backend clouddb1026.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [10:21:33] (03update) 10dcaro: get_job_from_k8s: stop using kind and use job_type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/320 [10:22:37] (03update) 10dcaro: get_job_from_k8s: stop using kind and use job_type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/320 [10:22:41] (03update) 10dcaro: get_job_from_k8s: stop using kind and use job_type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/320 [10:22:42] 06tools-infrastructure-team, 06Data-Persistence: cloudlb100X cannot talk to new host clouddb1026 - https://phabricator.wikimedia.org/T430132 (10fnegri) 03NEW [10:22:59] (03update) 10dcaro: get_job_from_k8s: stop using kind and use job_type [repos/cloud/toolforge/jobs-api] (ignore_runtime_jobs_not_in_storage) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/320 [10:46:33] (03open) 10fnegri: d/changelog: bump to 16.1.30 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/159 (https://phabricator.wikimedia.org/T428354) [11:23:39] 06tools-infrastructure-team, 06Data-Persistence: cloudlb100X cannot talk to new host clouddb1026 - https://phabricator.wikimedia.org/T430132#12053692 (10Marostegui) For what is worth a host which is in the same subnet also works: ` root@cloudlb1001:~# telnet clouddb1015.eqiad.wmnet 3316 Trying 10.64.16.141...... [11:54:21] (03update) 10dcaro: get_job_from_k8s: stop using kind and use job_type [repos/cloud/toolforge/jobs-api] (ignore_runtime_jobs_not_in_storage) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/320 [12:00:57] (03update) 10dcaro: get_job_from_k8s: stop using kind and use job_type [repos/cloud/toolforge/jobs-api] (ignore_runtime_jobs_not_in_storage) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/320 [12:01:49] (03open) 10dcaro: runtime, core: only fetch oneoffs from runtime [repos/cloud/toolforge/jobs-api] (dont_use_kind) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 [12:02:11] (03close) 10dcaro: k8s: when deleting a job, use it's own function [repos/cloud/toolforge/jobs-api] (use_job_type_labels) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/322 [12:02:32] (03close) 10dcaro: k8s: use per-job-type function to delete all jobs [repos/cloud/toolforge/jobs-api] (delete_per_job_type) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/325 [12:16:04] (03update) 10dcaro: runtime, core: only fetch oneoffs from runtime [repos/cloud/toolforge/jobs-api] (dont_use_kind) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 [12:23:54] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/73 [12:24:17] 06cloud-services-team, 10Data-Services, 10API-Portal, 06DBA, and 4 others: Delete apiportalwiki from wikireplicas - https://phabricator.wikimedia.org/T430102#12053923 (10Marostegui) [12:41:20] 10Toolforge, 06tools-platform-team, 07good first task: [builds-cli] does not output valid json when there's no builds - https://phabricator.wikimedia.org/T429229#12053979 (10DamianZaremba) >>! In T429229#12052371, @Buxtang wrote: > However I don't seem to have a contributor role to the build-cli repo so I ca... [12:54:55] 10Data-Services, 06tools-infrastructure-team, 06Data-Persistence: cloudlb100X cannot talk to new host clouddb1026 - https://phabricator.wikimedia.org/T430132#12054039 (10Marostegui) [12:55:12] 10Data-Services, 06tools-infrastructure-team, 06Data-Persistence: cloudlb100X cannot talk to new host clouddb1026 - https://phabricator.wikimedia.org/T430132#12054041 (10Marostegui) [12:55:16] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12054040 (10Marostegui) [13:00:31] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Put cloudvirt10[77-80] in service - https://phabricator.wikimedia.org/T429563#12054050 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1003 for host cloudvirt1077... [13:11:09] 10Cloud-VPS (Quota-requests), 10Catalyst: Volume number quota increase for catalyst-dev - https://phabricator.wikimedia.org/T430157 (10jnuche) 03NEW [13:15:39] (03open) 10dcaro: k8s: use job_type for label generating functions [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/328 [13:19:26] (03open) 10dcaro: refactor: rename user to tool_account where needed [repos/cloud/toolforge/jobs-api] (use_job_kind_for_labels) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/329 [13:27:45] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12054195 (10Marostegui) After pooling clouddb1026 with the same weight as clouddb1013, there was no traffic arriving, after discussing with @fne... [13:28:24] 10Cloud-VPS (Quota-requests), 10Catalyst: Volume number quota increase for catalyst-dev - https://phabricator.wikimedia.org/T430157#12054198 (10Andrew) +1 [13:28:59] 10Cloud-VPS (Quota-requests), 10Catalyst: Volume number quota increase for catalyst-dev - https://phabricator.wikimedia.org/T430157#12054203 (10fgiunchedi) LGTM, I'll seek two +1s for this request When the time comes, this is the command: ` cookbook wmcs.openstack.quota_increase --cluster-name eqiad1 --proje... [13:30:27] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-36 (2026-06-16 to 2026-06-30)), 07OKR-Work, 13Patch-For-Review: Document linter rules - https://phabricator.wikimedia.org/T422930#12054220 (10KBach) [13:31:37] (03update) 10kbach: Draft: Link to rule reference [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/25 (https://phabricator.wikimedia.org/T422930) [13:32:34] (03update) 10kbach: Link to rule reference [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/25 (https://phabricator.wikimedia.org/T422930) [13:34:28] (03update) 10kbach: Link to rule reference [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/25 (https://phabricator.wikimedia.org/T422930) [13:35:48] (03update) 10kbach: Link to rule reference [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/25 (https://phabricator.wikimedia.org/T422930) [13:36:56] 06cloud-services-team, 10Data-Services, 10API-Portal, 06DBA, and 4 others: Delete apiportalwiki from wikireplicas - https://phabricator.wikimedia.org/T430102#12054242 (10apaskulin) Thanks for opening this, @Marostegui! Yes, they can be deleted entirely from wikireplicas [13:38:56] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:45:26] (03open) 10dcaro: runtime: split create_job per-job type [repos/cloud/toolforge/jobs-api] (rename_user_to_tool_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/330 [13:45:57] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-36 (2026-06-16 to 2026-06-30)): Generate a valid semver info.version for the default MediaWiki REST module - https://phabricator.wikimedia.org/T427359#12054267 (10MGoncalves-WMF) a:03MGoncalves-WMF [13:47:23] (03update) 10dcaro: runtime: split create_job per-job type [repos/cloud/toolforge/jobs-api] (rename_user_to_tool_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/330 [13:47:29] (03approved) 10aghirelli: Link to rule reference [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/25 (https://phabricator.wikimedia.org/T422930) (owner: 10kbach) [13:47:41] (03merge) 10aghirelli: Link to rule reference [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/25 (https://phabricator.wikimedia.org/T422930) (owner: 10kbach) [13:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:49:45] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-36 (2026-06-16 to 2026-06-30)), 07OKR-Work, 13Patch-For-Review: Document linter rules - https://phabricator.wikimedia.org/T422930#12054293 (10KBach) [13:49:57] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-36 (2026-06-16 to 2026-06-30)), 07OKR-Work, 13Patch-For-Review: Document linter rules - https://phabricator.wikimedia.org/T422930#12054294 (10KBach) 05In progress→03Resolved [13:54:06] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12054316 (10aputhin) p:05Triage→03High [14:00:04] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 10API-Portal, and 5 others: Delete apiportalwiki from wikireplicas - https://phabricator.wikimedia.org/T430102#12054334 (10fnegri) 05Open→03In progress a:03fnegri [14:03:11] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-api] support exposing continuous jobs to the internet - https://phabricator.wikimedia.org/T423408#12054358 (10aputhin) p:05Triage→03High [14:03:24] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-cli] support publishing continuous job to the internet - https://phabricator.wikimedia.org/T423410#12054359 (10aputhin) p:05Triage→03High [14:04:12] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06tools-infrastructure-team: Put cloudvirt10[77-80] in service - https://phabricator.wikimedia.org/T429563#12054362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1003 for host cloudvirt1077.eqiad.wmnet with OS trixi... [14:15:15] 06cloud-services-team: Rebuild Horizon with new blubber version - https://phabricator.wikimedia.org/T430164 (10Andrew) 03NEW [14:28:45] 06cloud-services-team, 10Cloud-VPS: cloudvirt1075 in 'maintenance' aggregate - https://phabricator.wikimedia.org/T424802#12054459 (10Andrew) I'm not sure why this is happening. Most likely it's due to a different cookbook (maybe wmcs.openstack.cloudvirt.safe_reboot) failing and leaving things in an inconsisten... [14:30:20] 06cloud-services-team, 10Data-Services, 10API-Portal, 06DBA, and 4 others: Delete apiportalwiki from wikireplicas - https://phabricator.wikimedia.org/T430102#12054469 (10fnegri) a:05fnegri→03None [14:30:26] 06cloud-services-team, 10Data-Services, 10API-Portal, 06DBA, and 4 others: Delete apiportalwiki from wikireplicas - https://phabricator.wikimedia.org/T430102#12054461 (10fnegri) Tried using `maintain-views --drop --database apiportalwiki`, but that doesn't work because apiportalwiki is no longer present in... [14:36:26] 06cloud-services-team, 10Data-Services, 10API-Portal, 06DBA, and 4 others: Delete apiportalwiki from wikireplicas - https://phabricator.wikimedia.org/T430102#12054488 (10Marostegui) So the database in production still has data - @apaskulin is this database going to be dropped soon in production databases?... [14:37:53] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Deploy the OpenAPI linter in CI - https://phabricator.wikimedia.org/T422920#12054496 (10BPirkle) This is a type of work we don't usually do, and there is therefore some uncertainty regarding effort level. If it turns out that a portion of the work... [14:50:18] 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 07Essential-Work, 06Moderator-Tools-Team (Kanban): twl: Replace deprecated Bullseye VMs in Cloud VPS - https://phabricator.wikimedia.org/T402054#12054566 (10TheresNoTime) Notes for me: - https://www.mediawiki.org/wiki/Moderator_Tools/Deve... [14:50:24] 10Tool-wikimedia-attribution, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07OKR-Work: Include contributor_count in Attribution API responses - https://phabricator.wikimedia.org/T429834#12054568 (10HCoplin-WMF) [14:51:37] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/73 (owner: 10l10n-bot) [14:51:40] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/73 (owner: 10l10n-bot) [14:52:23] 10Tool-wikimedia-attribution, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07OKR-Work: Include contributor_count in Attribution API responses - https://phabricator.wikimedia.org/T429834#12054602 (10HCoplin-WMF) [14:53:31] 06cloud-services-team, 10Cloud-VPS: Can we stop using NFS in wikidumpparse? - https://phabricator.wikimedia.org/T430167 (10Andrew) 03NEW [14:54:26] 06cloud-services-team: Update all cloud-vps Bullseye NFS servers - https://phabricator.wikimedia.org/T429793#12054628 (10Andrew) [15:22:35] 06cloud-services-team, 10Data-Services, 10API-Portal, 06DBA, and 4 others: Delete apiportalwiki from wikireplicas - https://phabricator.wikimedia.org/T430102#12054829 (10apaskulin) @Marostegui Yes, I believe the plan is to drop the production databases, but I'm not sure when that is planned to happen. @Cle... [15:23:01] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Deploy the OpenAPI linter in CI - https://phabricator.wikimedia.org/T422920#12054843 (10AGhirelli-WMF) The repo url where to start working on is: https://gerrit.wikimedia.org/r/q/project:mediawiki/tools/api-testing [15:23:46] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Deploy the OpenAPI linter in CI - https://phabricator.wikimedia.org/T422920#12054845 (10AGhirelli-WMF) [15:25:11] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-36 (2026-06-16 to 2026-06-30)): Generate a valid semver info.version for the default MediaWiki REST module - https://phabricator.wikimedia.org/T427359#12054861 (10HCoplin-WMF) Hi friends, My preference is that we don't use the... [15:28:27] 06cloud-services-team, 10PAWS, 06tools-platform-team: PAWS disk space is running out - https://phabricator.wikimedia.org/T429578#12054881 (10dcaro) [15:33:22] 10Tool-wikimedia-attribution, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07OKR-Work: Include contributor_count in Attribution API responses - https://phabricator.wikimedia.org/T429834#12054942 (10AGhirelli-WMF) We need to add an abstraction layer PHP class to read this info from the DB (probably put it in... [15:37:36] 10Tool-wikimedia-attribution, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07OKR-Work: Include contributor_count in Attribution API responses - https://phabricator.wikimedia.org/T429834#12054961 (10AGhirelli-WMF) [15:38:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:50:49] 06tools-platform-team: Drive community adoption of Toolforge Push-to-Deploy - https://phabricator.wikimedia.org/T430181 (10komla) 03NEW [15:53:00] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12055099 (10fgiunchedi) a:05Andrew→03fgiunchedi Taking this on, I'll re-assign as needed once we have a path forward [16:10:13] 10Cloud-VPS (Quota-requests), 10Catalyst: Volume number quota increase for catalyst-dev - https://phabricator.wikimedia.org/T430157#12055185 (10fgiunchedi) +1 too btw [16:10:51] (03approved) 10dcaro: d/changelog: bump to 16.1.30 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/159 (https://phabricator.wikimedia.org/T428354) (owner: 10fnegri) [16:11:02] 10Cloud-VPS (Project-requests): Request creation of wikimedia_commons_uploader_telegram_bot VPS project - https://phabricator.wikimedia.org/T429729#12055188 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi I am tentatively resolving this task since: * @vitaly-zdanevich is going to try Toolforge * it is... [16:12:43] (03open) 10buxtang: Move "no builds" message to stderr in order to avoid violating --json flag [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/139 [16:12:52] 10Cloud-VPS (Project-requests): Request creation of wikimedia_commons_uploader_telegram_bot VPS project - https://phabricator.wikimedia.org/T429729#12055194 (10JJMC89) 05Resolved→03Declined [16:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:19:45] 10Data-Services, 06tools-infrastructure-team, 06Data-Persistence: cloudlb100X cannot talk to new host clouddb1026 - https://phabricator.wikimedia.org/T430132#12055283 (10Volans) I've quickly checked firewall and dns and they seem fine (no IPv6 AAAA record on the DB, `cloudlb1001` listed in iptables for port... [16:21:42] 06cloud-services-team, 10Toolforge: lima-kilo: container image caching - https://phabricator.wikimedia.org/T362967#12055305 (10dcaro) 05Open→03Resolved a:03dcaro This was done already adding the cache drive [16:22:39] 06cloud-services-team, 10Toolforge: [lima-kilo] Ldap gets reset when restarting the VM, so existing user SSL certificates are not valid anymore - https://phabricator.wikimedia.org/T427801#12055320 (10dcaro) [16:23:23] (03update) 10fnegri: d/changelog: bump to 16.1.30 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/159 (https://phabricator.wikimedia.org/T428354) [16:23:36] (03merge) 10fnegri: d/changelog: bump to 16.1.30 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/159 (https://phabricator.wikimedia.org/T428354) [16:28:20] !log fnegri@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [16:28:28] !log fnegri@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli [16:28:29] !log andrew@cloudcumin1001 puppet-diffs START - Cookbook wmcs.nfs.add_server [16:28:40] (03close) 10dcaro: deploy components - don't report as changed [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/266 (owner: 10damian) [16:28:46] !log andrew@cloudcumin1001 puppet-diffs END (ERROR) - Cookbook wmcs.nfs.add_server (exit_code=97) [16:28:49] !log andrew@cloudcumin1001 puppet-diffs START - Cookbook wmcs.nfs.add_server [16:29:00] !log fnegri@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [16:29:06] !log andrew@cloudcumin1001 puppet-diffs END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99) [16:29:23] !log andrew@cloudcumin1001 puppet-diffs START - Cookbook wmcs.nfs.add_server [16:29:40] !log andrew@cloudcumin1001 puppet-diffs END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99) [16:30:48] !log andrew@cloudcumin1001 puppet-diffs START - Cookbook wmcs.nfs.add_server [16:34:33] 10Data-Services, 06tools-infrastructure-team, 06Data-Persistence: cloudlb100X cannot talk to new host clouddb1026 - https://phabricator.wikimedia.org/T430132#12055367 (10cmooney) Folks, The packets here were getting blocked on the CR router as traffic left the cloud realm going to the new clouddb host. Pro... [16:35:04] 06cloud-services-team, 10Toolforge: [lima-kilo] toolforge components are reported as changed on every ansible run - https://phabricator.wikimedia.org/T402689#12055374 (10dcaro) 05Open→03Invalid Copying the comment from the MR https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests... [16:36:24] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: DRAFT Decision request - Focus for improving lima-kilo developer experience - https://phabricator.wikimedia.org/T403051#12055390 (10dcaro) [16:37:11] !log fnegri@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [16:38:50] !log andrew@cloudcumin1001 puppet-diffs END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) [16:43:30] 06cloud-services-team, 10Striker, 06tools-platform-team: [striker] Failing to create new phabricator projects - https://phabricator.wikimedia.org/T430120#12055564 (10bd808) https://phabricator.wikimedia.org/conduit/method/project.search/ changed somehow. Failing code: `lang=python def get_project(self,... [16:57:36] (03CR) 10Andrew Bogott: nfs: do DNS flip on network migration (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189868 (owner: 10Filippo Giunchedi) [17:00:13] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [17:04:50] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Specifying --filelog-stdout or --filelog-stderr requires --filelog - https://phabricator.wikimedia.org/T428354#12055737 (10fnegri) 05In progress→03Resolved I've just deployed jobs-cli 16.1.30 where the extra `--filelog` is now no longer required... [17:08:14] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [17:12:40] 06cloud-services-team, 10Striker, 06tools-platform-team: [striker] Failing to create new phabricator projects - https://phabricator.wikimedia.org/T430120#12055746 (10dcaro) @bd808 thanks! Will have a look [17:16:48] 06cloud-services-team, 06tools-infrastructure-team: Patch demo resource check - https://phabricator.wikimedia.org/T429990#12055750 (10BLiviero-WMF) More info supporting documentation: https://docs.google.com/document/d/1rrAZ60ljicZ0aFE7avLRtiLI_9Km0rGZzpTazLJFTOA/edit?tab=t.0#heading=h.ewnklj693zaf Resource w... [17:25:37] (03update) 10dcaro: install-binary-from-url - add checksums for dest [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/263 (https://phabricator.wikimedia.org/T402684) (owner: 10damian) [17:27:11] (03update) 10dcaro: install-binary-from-url - add checksums for dest [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/263 (https://phabricator.wikimedia.org/T402684) (owner: 10damian) [17:41:42] (03update) 10dcaro: install-binary-from-url - add checksums for dest [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/263 (https://phabricator.wikimedia.org/T402684) (owner: 10damian) [17:43:16] (03update) 10dcaro: install-binary-from-url - add checksums for dest [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/263 (https://phabricator.wikimedia.org/T402684) (owner: 10damian) [17:46:15] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [lima-kilo] Only download artefacts if target binary checksum does not match - https://phabricator.wikimedia.org/T402684#12055880 (10dcaro) a:03dcaro I'm pushing this as it's easy to finish up. [17:47:22] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [lima-kilo] Improve convergence - https://phabricator.wikimedia.org/T402672#12055897 (10dcaro) a:03dcaro I'll close this once the subtask is done, I think that we can reopen a new one once we finish deciding the decision request. [17:52:56] 10Toolforge, 06tools-platform-team: ToolforgeWebHighErrorRate should not page if a single tool is down - https://phabricator.wikimedia.org/T429738#12055925 (10aputhin) [18:00:35] (03CR) 10David Caro: nfs: do DNS flip on network migration (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189868 (owner: 10Filippo Giunchedi) [18:03:39] (03update) 10dcaro: install-binary-from-url - add checksums for dest [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/263 (https://phabricator.wikimedia.org/T402684) (owner: 10damian) [18:51:59] 10Cloud-VPS, 06tools-infrastructure-team: Rebalance WMCS instance backups between the two hosts - https://phabricator.wikimedia.org/T430018#12056224 (10Andrew) a:03Andrew [19:51:10] 06cloud-services-team, 10Striker, 06tools-platform-team: [striker] Failing to create new phabricator projects - https://phabricator.wikimedia.org/T430120#12056377 (10bd808) [20:01:12] 06cloud-services-team, 10Striker, 06tools-platform-team: [striker] Failing to create new phabricator projects - https://phabricator.wikimedia.org/T430120#12056420 (10bd808) It is possible that the `attachments.uris: True` value has been junk for a long time and {rPHABc39bb3e1e397b2eec1ce28883f11907a71d18d31}... [21:05:50] (03update) 10buxtang: Move "no builds" message to stderr in order to avoid violating --json flag [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/139 [21:14:42] (03update) 10buxtang: Move "no builds" message to stderr in order to avoid violating --json flag [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/139 [22:39:09] (03approved) 10bd808: README: Replace repo mirror URI with canonical URI [toolforge-repos/my-first-flask-oauth-tool] - 10https://gitlab.wikimedia.org/toolforge-repos/my-first-flask-oauth-tool/-/merge_requests/1 (https://phabricator.wikimedia.org/T405596) (owner: 10aklapper) [22:39:15] (03merge) 10bd808: README: Replace repo mirror URI with canonical URI [toolforge-repos/my-first-flask-oauth-tool] - 10https://gitlab.wikimedia.org/toolforge-repos/my-first-flask-oauth-tool/-/merge_requests/1 (https://phabricator.wikimedia.org/T405596) (owner: 10aklapper)