[00:31:57] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Network unavailable on a few VMs - https://phabricator.wikimedia.org/T432426#12165531 (10Andrew) The latest: - I just now found a Trixie host without an IP. I don't know for certain that it broke post-patch but I fear the worst. We should keep an... [00:38:46] 10Cloud-VPS, 06tools-infrastructure-team: many false positive alertname=InstanceDown - https://phabricator.wikimedia.org/T433173#12165545 (10Andrew) A few spot checks: accounts-appserver9.account-creation-assistance.eqiad1.wikimedia.cloud: Use the central puppetserver (puppetmaster.cloudinfra.wmflabs.org) and... [03:08:16] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-platform-team: Send follow up messages to Cloud VPS project maintainers about Debian Bullseye deprecation - https://phabricator.wikimedia.org/T433334#12165734 (10komla) [03:37:22] (03update) 10countcount: Cache small wikis' /ipinfo phase-1 IP rows in memory, rebuilt every 10 minutes by a background task [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/59 [03:55:28] (03update) 10countcount: Cache small wikis' /ipinfo phase-1 IP rows in memory, rebuilt every 10 minutes by a background task [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/59 [03:55:35] (03update) 10countcount: Cache small wikis' /ipinfo phase-1 IP rows in memory, rebuilt every 10 minutes by a background task [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/59 [03:58:47] (03merge) 10countcount: Cache small wikis' /ipinfo phase-1 IP rows in memory, rebuilt every 10 minutes by a background task [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/59 [04:05:24] (03open) 10countcount: Log row count and elapsed time on each probe cache refresh [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/66 [04:08:53] (03merge) 10countcount: Log row count and elapsed time on each probe cache refresh [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/66 [04:16:22] (03update) 10renovatebot: Update Rust crate serial_test to v4 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/61 [05:16:17] (03update) 10renovatebot: Update dependency jsdom to v30 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/27 [05:43:49] (03update) 10countcount: List global block exceptions [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/64 [05:47:05] 10Tool-wmf-openapi-linter, 06Tech-Docs-Team, 06MediaWiki-API-Platform-Team (MediaWiki-Interfaces-Sprints), 05MW-1.47-notes (1.47.0-wmf.13; 2026-07-28), 07OKR-Work: Fix description and example issues for page, revision, search, and file paths in the Media... - https://phabricator.wikimedia.org/T428153#12165879 [05:47:07] 10Tool-wmf-openapi-linter, 06Tech-Docs-Team, 03[MWI] FY2025-26 Q4, 07OKR-Work: [Hypothesis] 5.2.5b: Productionalize API spec linting - https://phabricator.wikimedia.org/T422476#12165881 (10KBach) [06:11:02] (03update) 10countcount: List global block exceptions [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/64 [06:11:20] (03update) 10countcount: Serve global block exceptions from an in-memory snapshot [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/64 [06:15:44] (03update) 10renovatebot: Update dependency jsdom to v30 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/27 [06:51:25] (03update) 10countcount: Serve global block exceptions from an in-memory snapshot [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/64 [07:24:56] 10Tool-wmf-openapi-linter, 06Tech-Docs-Team, 03[MWI] FY2025-26 Q4, 07OKR-Work: [Hypothesis] 5.2.5b: Productionalize API spec linting - https://phabricator.wikimedia.org/T422476#12166026 (10KBach) [07:43:18] 10Tool-wmf-openapi-linter, 06Tech-Docs-Team, 03[MWI] FY2025-26 Q4, 07OKR-Work: [Hypothesis] 5.2.5b: Productionalize API spec linting - https://phabricator.wikimedia.org/T422476#12166067 (10KBach) [07:47:36] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: [infra] Allow some opt-in scraping solution for tools - https://phabricator.wikimedia.org/T433395#12166073 (10dcaro) >>! In T433395#12164572, @taavi wrote: > In the end it'll need to be a HAProxy map file on the proxy hosts, with an entry per d... [07:47:59] 10Tool-wmf-openapi-linter, 06Tech-Docs-Team, 03[MWI] FY2025-26 Q4, 07OKR-Work: [Hypothesis] 5.2.5b: Productionalize API spec linting - https://phabricator.wikimedia.org/T422476#12166074 (10KBach) 05Open→03Resolved This hypothesis is complete so I'm resolving this task. Any open sub-tasks are either... [07:48:44] 10Toolforge, 06tools-platform-team: [components-api] Allow setting opt-in flags for special scraping control measures - https://phabricator.wikimedia.org/T433397#12166081 (10dcaro) [07:51:34] 10Tool-wmf-openapi-linter, 06Tech-Docs-Team, 03[MWI] FY2025-26 Q4, 07OKR-Work: [Hypothesis] 5.2.5b: Productionalize API spec linting - https://phabricator.wikimedia.org/T422476#12166083 (10KBach) [07:55:03] 10Tool-wikicontest, 07Privacy: WikiEval Tool / WikiContest violates Toolforge Privacy Policy - https://phabricator.wikimedia.org/T427744#12166085 (10Aklapper) 05Open→03Resolved Thank you, this seems to have been fixed in the meantime (please resolve tickets once they are fixed). [07:58:14] (03update) 10dcaro: Support logs-api pagination [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/163 (https://phabricator.wikimedia.org/T428253) (owner: 10tlepage) [07:59:02] (03update) 10dcaro: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 (owner: 10tlepage) [07:59:48] (03update) 10dcaro: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 (https://phabricator.wikimedia.org/T428253) (owner: 10tlepage) [07:59:56] (03update) 10dcaro: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 (https://phabricator.wikimedia.org/T428253) (owner: 10tlepage) [08:13:14] (03update) 10countcount: Serve global block exceptions from an in-memory snapshot [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/64 [08:13:29] (03update) 10countcount: Report locally disabled global blocks from an in-memory snapshot [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/64 [08:55:23] (03close) 10countcount: Replace the OAuth login with an optional SOCKS5 proxy to Toolforge [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/25 [09:14:24] (03update) 10countcount: Report locally disabled global blocks from an in-memory snapshot [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/64 [09:17:42] !log dcaro@cloudcumin1001 pageviews START - Cookbook wmcs.vps.create_project for project pageviews in eqiad1 (T433305) [09:17:44] dcaro@cloudcumin1001: Unknown project "pageviews" [09:17:44] dcaro@cloudcumin1001: Did you mean to say "tools.pageviews" instead? [09:17:44] T433305: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305 [09:17:46] !log dcaro@cloudcumin1001 pageviews END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project pageviews in eqiad1 (T433305) [09:17:46] dcaro@cloudcumin1001: Unknown project "pageviews" [09:17:47] dcaro@cloudcumin1001: Did you mean to say "tools.pageviews" instead? [09:20:00] 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12166397 (10dcaro) Hi @MusikAnimal, @Ottomata, the name `pageviews` is already taken by the web proxy `pageviews.wmcloud.org`, we would need either changing the webproxy, or t... [09:26:42] 10Toolforge (Quota-requests): Request increased quota for statanalyser Toolforge tool - https://phabricator.wikimedia.org/T433184#12166415 (10dcaro) a:03dcaro [09:28:19] 10Toolforge (Quota-requests), 06tools-platform-team: Request increased quota for statanalyser Toolforge tool - https://phabricator.wikimedia.org/T433184#12166416 (10dcaro) [09:28:57] 10Toolforge (Quota-requests), 06tools-platform-team: Request increased quota for statanalyser Toolforge tool - https://phabricator.wikimedia.org/T433184#12166418 (10dcaro) [09:29:00] 10Toolforge (Quota-requests), 06tools-platform-team: Request increased quota for statanalyser Toolforge tool - https://phabricator.wikimedia.org/T433184#12166419 (10dcaro) 05Open→03In progress [09:34:35] (03open) 10dcaro: statanalizer: increase quota [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1350 [09:35:52] (03approved) 10tlepage: statanalizer: increase quota [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1350 (owner: 10dcaro) [09:40:06] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers (T433184) [09:40:11] T433184: Request increased quota for statanalyser Toolforge tool - https://phabricator.wikimedia.org/T433184 [09:40:19] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers (T433184) [09:44:24] (03update) 10dcaro: statanalizer: increase quota [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1350 [09:45:23] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers (T433184) [09:45:28] T433184: Request increased quota for statanalyser Toolforge tool - https://phabricator.wikimedia.org/T433184 [09:45:36] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers (T433184) [10:00:02] 10Cloud-VPS, 06tools-infrastructure-team, 06tools-platform-team: Make sure dumps-nfs mount/umount is propagated inside Toolforge containers - https://phabricator.wikimedia.org/T432325#12166549 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi This is done -- toolforge now propagates `/mnt/nfs` mounts... [10:00:09] (03approved) 10dcaro: statanalizer: increase quota [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1350 [10:00:23] (03merge) 10dcaro: statanalizer: increase quota [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1350 [10:00:34] 10Toolforge (Quota-requests), 06tools-platform-team: Request increased quota for statanalyser Toolforge tool - https://phabricator.wikimedia.org/T433184#12166554 (10dcaro) 05In progress→03Resolved [10:20:13] 10Tool-wmf-openapi-linter, 06Tech-Docs-Team, 03[MWI] FY2025-26 Q4, 07OKR-Work: [Hypothesis] 5.2.5b: Productionalize API spec linting - https://phabricator.wikimedia.org/T422476#12166609 (10KBach) [10:23:38] (03merge) 10countcount: Report locally disabled global blocks from an in-memory snapshot [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/64 [11:17:04] 06tools-platform-team: [components-api] ugrade pre-commit (and ruff) - https://phabricator.wikimedia.org/T433492 (10dcaro) 03NEW [11:17:05] 06tools-platform-team: [components-api] ugrade pre-commit (and ruff) - https://phabricator.wikimedia.org/T433492#12166763 (10dcaro) 05Open→03In progress [11:17:16] 06tools-platform-team: [components-api] ugrade pre-commit (and ruff) - https://phabricator.wikimedia.org/T433492#12166764 (10dcaro) p:05Triage→03Medium [11:17:54] (03open) 10dcaro: global: upgrade pre-commits [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/177 [11:19:15] (03update) 10renovatebot: Update Rust crate serial_test to v4 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/61 [11:23:07] (03open) 10countcount: Parse block durations with timelib for PHP strtotime parity [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/67 [11:23:14] (03update) 10countcount: Parse block durations with timelib for PHP strtotime parity [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/67 [11:26:25] 10Cloud-VPS (Debian Bullseye Deprecation), 10CFSSL-PKI, 06Infrastructure-Foundations: Rebuild VMs in PKI cloud-vps project - https://phabricator.wikimedia.org/T405017#12166774 (10elukey) 05Open→03Resolved a:03elukey Old VMs deleted! [11:33:08] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team: Upgrade IDP server in cloudinfra project off of Bullseye - https://phabricator.wikimedia.org/T402006#12166790 (10SLyngshede-WMF) We do still add the client support, but apparently we lost server support, at least that's my current theory.... [11:49:29] (03update) 10countcount: Parse block durations with timelib for PHP strtotime parity [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/67 [11:49:36] (03update) 10countcount: Parse block durations with timelib for PHP strtotime parity [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/67 [12:08:41] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:25:21] (03merge) 10countcount: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 (owner: 10renovatebot) [12:28:50] (03merge) 10countcount: Parse block durations with timelib for PHP strtotime parity [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/67 [12:29:45] 10Cloud-VPS (Debian Bullseye Deprecation), 10Toolforge, 06tools-infrastructure-team: [infra] Toolforge: migrate to Debian Bookworm or later - https://phabricator.wikimedia.org/T387005#12166931 (10aputhin) [12:31:05] 10Toolforge, 06tools-platform-team, 07OKR-Work: [loki,alloy,logs-api] reconfigure the labels and add metadata - https://phabricator.wikimedia.org/T432969#12166932 (10aputhin) p:05Triage→03High [12:31:21] 10Toolforge, 06tools-platform-team, 07OKR-Work: Set up a Toolforge “alerting service” that emails tool maintainers when an alert condition is detected - https://phabricator.wikimedia.org/T432865#12166934 (10aputhin) p:05Triage→03High [12:31:43] 10Toolforge, 06tools-platform-team, 07OKR-Work: Provide a way for users to enable and disable alerts - https://phabricator.wikimedia.org/T432864#12166940 (10aputhin) p:05Triage→03High [12:33:08] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-api,loki] Implement a system-level logging endpoint (write) - https://phabricator.wikimedia.org/T432565#12166943 (10aputhin) p:05Triage→03High a:05dcaro→03None [12:35:02] 10Toolforge, 06tools-platform-team: [components-api] build quota failures leave state unclear - https://phabricator.wikimedia.org/T422951#12166960 (10aputhin) Moving back to backlog until we get more info (i.e. answer to question above) [12:36:09] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-cli,toolforge-cli] Add subcommand toolforge logs to query and retrieve the relevant logs, will have to design a minimal interface to filter by event/subsystem/time/etc. - https://phabricator.wikimedia.org/T432572#12166962 (10aputhin) p:05Triage→03High [12:36:44] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [o11y] Automate daily count of successful SSH logins as a measure of user activity - https://phabricator.wikimedia.org/T423549#12166964 (10aputhin) p:05Triage→03Medium [12:38:48] (03update) 10dcaro: global: upgrade pre-commits [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/177 [12:38:51] (03update) 10dcaro: global: upgrade pre-commits [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/177 [12:40:47] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12166983 (10ops-monitoring-bot) Rebooting db-test2001.codfw.wmnet [12:49:09] (03update) 10dcaro: global: upgrade pre-commits [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/177 [12:49:34] (03update) 10dcaro: global: upgrade pre-commits [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/177 [12:55:48] (03update) 10dcaro: global: upgrade pre-commits [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/177 [13:08:10] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12167095 (10Marostegui) [13:08:23] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12167096 (10Marostegui) clouddb1033 pooled in s5 and s8