[00:41:30] (03update) 10raymond-ndibe: status: add StatusShort enum and basic job status detection [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/299 (https://phabricator.wikimedia.org/T401172) [00:44:55] (03merge) 10raymond-ndibe: status: add StatusShort enum and basic job status detection [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/299 (https://phabricator.wikimedia.org/T401172) [00:45:01] (03update) 10raymond-ndibe: status: add duration field [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/302 (https://phabricator.wikimedia.org/T401172) [00:48:26] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.500-20260602004512-d7c843df [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1279 (https://phabricator.wikimedia.org/T401172) [00:48:36] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.500-20260602004512-d7c843df [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1279 (https://phabricator.wikimedia.org/T401172) [00:53:46] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [00:54:31] (03update) 10raymond-ndibe: status: add duration field [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/302 (https://phabricator.wikimedia.org/T401172) [00:54:43] (03update) 10raymond-ndibe: status: add up_to_date field and API warnings [repos/cloud/toolforge/jobs-api] (status_add_duration) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/303 (https://phabricator.wikimedia.org/T401172) [00:55:09] (03update) 10raymond-ndibe: status: add messages to status [repos/cloud/toolforge/jobs-api] (status_add_up_to_date) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/304 (https://phabricator.wikimedia.org/T401172) [00:55:29] (03update) 10raymond-ndibe: status: add out-of-quota detection [repos/cloud/toolforge/jobs-api] (status_add_messages) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/305 (https://phabricator.wikimedia.org/T401172) [00:55:47] (03update) 10raymond-ndibe: status: handle restart messaging and count [repos/cloud/toolforge/jobs-api] (status_add_quota_detection) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/306 (https://phabricator.wikimedia.org/T401172) [00:56:06] (03update) 10raymond-ndibe: status: handle edge cases [repos/cloud/toolforge/jobs-api] (status_add_restarting_detection) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/307 (https://phabricator.wikimedia.org/T401172) [01:05:12] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [01:33:47] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [01:46:12] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [01:46:33] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.500-20260602004512-d7c843df [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1279 (https://phabricator.wikimedia.org/T401172) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:46:35] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.500-20260602004512-d7c843df [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1279 (https://phabricator.wikimedia.org/T401172) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:46:46] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.500-20260602004512-d7c843df [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1279 (https://phabricator.wikimedia.org/T401172) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [05:06:09] FIRING: ToolsDBAlmostFull: ToolsDB is almost full - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBAlmostFull - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBAlmostFull [05:12:06] (03open) 10countcount: Add GitLab CI pipeline [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/12 [05:13:52] (03update) 10countcount: Add GitLab CI pipeline [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/12 [05:15:26] (03merge) 10countcount: Add GitLab CI pipeline [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/12 [05:19:32] (03open) 10countcount: Update non-breaking dependencies [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/13 [05:21:18] (03update) 10countcount: Update non-breaking dependencies [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/13 [05:26:55] (03merge) 10countcount: Update non-breaking dependencies [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/13 [05:28:13] (03open) 10countcount: Apply security dependency updates (urllib3, requests, python-dotenv) [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/14 [05:29:46] (03update) 10countcount: Apply security dependency updates (urllib3, requests, python-dotenv) [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/14 [05:29:55] (03update) 10countcount: Security dependency updates + raise requires-python to >=3.10 [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/14 [05:32:37] (03update) 10countcount: Security dependency updates + raise requires-python to >=3.10 [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/14 [05:33:27] (03merge) 10countcount: Security dependency updates + raise requires-python to >=3.10 [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/14 [05:34:31] (03open) 10countcount: Add AGENTS.md symlink to CLAUDE.md [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/15 [05:35:17] (03update) 10countcount: Add AGENTS.md symlink to CLAUDE.md [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/15 [05:35:25] (03update) 10countcount: Add AGENTS.md symlink + document buildpack Python selection [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/15 [05:37:15] (03merge) 10countcount: Add AGENTS.md symlink + document buildpack Python selection [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/15 [05:38:23] (03open) 10countcount: Fix stale "24 hours" lookback wording in docstring and --help [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/16 [05:38:30] (03update) 10renovatebot: Update python Docker tag to v3.14 [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/17 [05:38:34] (03update) 10renovatebot: Update dependency packaging to v26 [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/18 [05:38:36] (03open) 10renovatebot: Update python Docker tag to v3.14 [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/17 [05:38:39] (03update) 10renovatebot: Update dependency pywikibot to v11 [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/19 [05:38:40] (03close) 10renovatebot: Update dependency certifi to v2025.11.12 [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/7 [05:38:41] (03update) 10renovatebot: Update dependency certifi to v2025.11.12 - autoclosed [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/7 [05:38:43] (03close) 10renovatebot: Update dependency certifi to v2026 [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/11 [05:38:43] (03update) 10renovatebot: Update dependency certifi to v2026 - autoclosed [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/11 [05:38:46] (03close) 10renovatebot: Update dependency charset-normalizer to v3.4.7 [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/6 [05:38:50] (03update) 10renovatebot: Update dependency charset-normalizer to v3.4.7 - autoclosed [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/6 [05:38:54] (03open) 10renovatebot: Update dependency packaging to v26 [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/18 [05:38:58] (03close) 10renovatebot: Update dependency mwparserfromhell to v0.7.2 [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/8 [05:39:02] (03update) 10renovatebot: Update dependency mwparserfromhell to v0.7.2 - autoclosed [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/8 [05:39:06] (03close) 10renovatebot: Update dependency pymysql to v1.2.0 [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/9 [05:39:10] (03update) 10renovatebot: Update dependency pymysql to v1.2.0 - autoclosed [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/9 [05:39:14] (03close) 10renovatebot: Update dependency idna to v3.15 [SECURITY] [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/2 [05:39:18] (03update) 10renovatebot: Update dependency idna to v3.15 [SECURITY] - autoclosed [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/2 [05:39:22] (03close) 10renovatebot: Update dependency python-dotenv to v1.2.2 [SECURITY] [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/3 [05:39:27] (03update) 10renovatebot: Update dependency python-dotenv to v1.2.2 [SECURITY] - autoclosed [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/3 [05:39:31] (03close) 10renovatebot: Update dependency requests to v2.33.0 [SECURITY] [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/4 [05:39:35] (03update) 10renovatebot: Update dependency requests to v2.33.0 [SECURITY] - autoclosed [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/4 [05:39:39] (03close) 10renovatebot: Update dependency urllib3 to v2.7.0 [SECURITY] [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/5 [05:39:43] (03update) 10renovatebot: Update dependency urllib3 to v2.7.0 [SECURITY] - autoclosed [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/5 [05:39:47] (03open) 10renovatebot: Update dependency pywikibot to v11 [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/19 [05:39:51] (03close) 10renovatebot: Update dependency pywikibot to v10.7.6 [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/10 [05:39:55] (03update) 10renovatebot: Update dependency pywikibot to v10.7.6 - autoclosed [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/10 [05:39:59] (03merge) 10countcount: Fix stale "24 hours" lookback wording in docstring and --help [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/16 [05:46:36] (03open) 10countcount: Bump all dependencies to latest versions [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/20 [05:47:44] (03merge) 10countcount: Bump all dependencies to latest versions [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/20 [05:51:23] (03open) 10countcount: Raise requires-python to >=3.12 to match runtime [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/21 [05:52:26] (03merge) 10countcount: Raise requires-python to >=3.12 to match runtime [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/21 [06:02:42] (03open) 10countcount: Remove requirements.txt (deps via pyproject.toml + uv.lock) [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/22 [06:03:20] (03merge) 10countcount: Remove requirements.txt (deps via pyproject.toml + uv.lock) [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/22 [06:06:20] (03close) 10renovatebot: Update dependency packaging to v26 [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/18 [06:06:21] (03update) 10renovatebot: Update dependency packaging to v26 - autoclosed [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/18 [06:06:22] (03close) 10renovatebot: Update dependency pywikibot to v11 [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/19 [06:06:23] (03update) 10renovatebot: Update dependency pywikibot to v11 - autoclosed [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/19 [06:06:29] (03update) 10renovatebot: Update python Docker tag to v3.14 [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/17 [06:06:45] (03open) 10countcount: Simplify and de-duplicate the codebase [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/23 [06:22:46] (03open) 10countcount: Add .worktreeinclude to copy local-dev secrets into worktrees [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/24 [06:28:53] (03update) 10countcount: Add .worktreeinclude to copy local-dev secrets into worktrees [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/24 [06:29:06] (03update) 10countcount: Add .worktreeinclude to copy local-dev secrets into worktrees [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/24 [06:39:47] (03update) 10countcount: Add .worktreeinclude to copy local-dev secrets into worktrees [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/24 [06:39:59] (03update) 10countcount: Copy gitignored local-dev files into new worktrees (.worktreeinclude + Zed/Codex wiring) [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/24 [06:43:01] (03merge) 10countcount: Copy gitignored local-dev files into new worktrees (.worktreeinclude + Zed/Codex wiring) [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/24 [06:49:14] (03close) 10countcount: Update python Docker tag to v3.14 [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/17 (owner: 10renovatebot) [06:52:42] (03update) 10countcount: Simplify and de-duplicate the codebase [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/23 [06:53:28] (03merge) 10countcount: Simplify and de-duplicate the codebase [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/23 [06:56:32] (03open) 10countcount: Reconnect dropped DB connections and harden fail_run [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/25 [07:04:14] (03merge) 10countcount: Reconnect dropped DB connections and harden fail_run [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/25 [07:08:42] (03open) 10countcount: Type-check the whole codebase with pyright, not just the entry point [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/26 [07:09:22] (03update) 10countcount: Type-check the whole codebase with pyright, not just the entry point [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/26 [07:09:26] (03merge) 10countcount: Type-check the whole codebase with pyright, not just the entry point [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/26 [07:16:10] RESOLVED: ToolsDBAlmostFull: ToolsDB is almost full - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBAlmostFull - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBAlmostFull [07:17:28] (03open) 10countcount: Tighten type annotations [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/27 [07:19:13] (03update) 10countcount: Tighten type annotations [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/27 [07:24:13] (03update) 10countcount: Tighten type annotations [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/27 [07:25:13] (03update) 10countcount: Tighten type annotations [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/27 [07:27:55] (03merge) 10countcount: Tighten type annotations [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/27 [07:29:19] (03open) 10countcount: Make VotingRightsResult a dataclass [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/28 [07:29:53] (03merge) 10countcount: Make VotingRightsResult a dataclass [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/28 [08:34:11] 10Tool-wikiloves: wikiloves updates fail with "Unknown column 'cl_to' in 'SELECT'" - https://phabricator.wikimedia.org/T420408#11975400 (10JeanFred) 05Open→03Resolved a:03JeanFred [08:35:41] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Support requestBody.description in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427355#11975403 (10KBach) p:05High→03Medium [08:38:06] 10Tool-wikiloves: WL Sciences : even years are missing - https://phabricator.wikimedia.org/T379626#11975407 (10JeanFred) Looks like only odd years are defined in https://commons.wikimedia.org/wiki/Module:WL_data − feel free to add the missing data! [08:38:52] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Improve linting - detect examples in nested schema.properties - https://phabricator.wikimedia.org/T424002#11975408 (10KBach) p:05Medium→03High [08:38:58] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Publish the Wikimedia Spectral ruleset to NPM - https://phabricator.wikimedia.org/T425627#11975409 (10KBach) p:05Medium→03High [08:39:48] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Generate a valid semver info.version for the default MediaWiki REST module - https://phabricator.wikimedia.org/T427359#11975412 (10KBach) p:05High→03Medium [09:31:56] FIRING: SystemdUnitDown: The service unit nginx.service is in failed status on host clouddumps1002. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [09:33:14] RESOLVED: [2x] ProbeDown: Service clouddumps1002:443 has failed probes (http_dumps_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Dumps/SQL-XML_Dumps#NFS_share_and/or_web_server_issues - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [09:36:56] RESOLVED: SystemdUnitDown: The service unit nginx.service is in failed status on host clouddumps1002. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [09:40:56] (03PS1) 10Jelto: update blubber buildkit image to v1.8.0 [labs/striker] - 10https://gerrit.wikimedia.org/r/1296518 (https://phabricator.wikimedia.org/T321316) [09:42:24] (03CR) 10CI reject: [V:04-1] update blubber buildkit image to v1.8.0 [labs/striker] - 10https://gerrit.wikimedia.org/r/1296518 (https://phabricator.wikimedia.org/T321316) (owner: 10Jelto) [10:26:10] FIRING: ToolsDBAlmostFull: ToolsDB is almost full - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBAlmostFull - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBAlmostFull [10:52:48] FIRING: PuppetFailure: Puppet has failed on clouddumps1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [11:21:42] 10Cloud-VPS, 06tools-infrastructure-team: Renew metricsinfra CA certificate - https://phabricator.wikimedia.org/T427905#11975925 (10taavi) 05Open→03Resolved [11:25:39] RESOLVED: PuppetCertificateAboutToExpire: Puppet CA certificate Puppet CA: metricsinfra-puppetmaster-1.metricsinfra.eqiad1.wikimedia.cloud is about to expire in 12d 5h 16m 10s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [11:27:14] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#11975964 (10BTullis) The first part seems to be working. We have nginx emitting ECS formatted log ent... [11:32:48] RESOLVED: PuppetFailure: Puppet has failed on clouddumps1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [11:49:16] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Publish the Wikimedia Spectral ruleset to NPM - https://phabricator.wikimedia.org/T425627#11976017 (10KBach) [11:49:45] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#11976018 (10BTullis) I'm aiming to roll out the rsyslog change (https://gerrit.wikimedia.org/r/c/oper... [12:33:58] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914 (10aputhin) 03NEW [12:55:41] (03open) 10taavi: kind: Upgrade Kubernetes to 1.32 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/325 (https://phabricator.wikimedia.org/T379047) [12:55:46] (03update) 10taavi: kind: Upgrade Kubernetes to 1.32 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/325 (https://phabricator.wikimedia.org/T379047) [12:57:57] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge, 13Patch-For-Review: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.32 - https://phabricator.wikimedia.org/T379047#11976285 (10taavi) [12:58:29] 10Toolforge, 06tools-infrastructure-team: Upgrade toolsbeta cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427918 (10taavi) 03NEW [12:59:07] 10Toolforge, 06tools-infrastructure-team: Upgrade tools cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427919 (10taavi) 03NEW [13:20:13] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-35 (2026-06-02 to 2026-06-16)): Support root-level externalDocs in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427356#11976411 (10HCoplin-WMF) [13:27:13] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-35 (2026-06-02 to 2026-06-16)), 07OKR-Work: Improve linting - detect examples in nested schema.properties - https://phabricator.wikimedia.org/T424002#11976436 (10HCoplin-WMF) [13:28:39] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-35 (2026-06-02 to 2026-06-16)): Support requestBody.description in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427355#11976456 (10HCoplin-WMF) [13:28:45] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-35 (2026-06-02 to 2026-06-16)): Support response body examples in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427361#11976458 (10HCoplin-WMF) [13:28:56] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-35 (2026-06-02 to 2026-06-16)): Support requestBody content examples in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427360#11976460 (10HCoplin-WMF) [13:36:32] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-35 (2026-06-02 to 2026-06-16)), 07OKR-Work: MediaWiki REST API appropriately defines array types for parameters - https://phabricator.wikimedia.org/T409113#11976485 (10HCoplin-WMF) [13:59:49] 10Toolforge, 06tools-infrastructure-team: Upgrade toolsbeta cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427918#11976576 (10taavi) p:05Triage→03High [13:59:56] 10Toolforge, 06tools-infrastructure-team: Upgrade tools cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427919#11976577 (10taavi) p:05Triage→03High [14:00:20] (03update) 10taavi: kind: Upgrade Kubernetes to 1.32 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/325 (https://phabricator.wikimedia.org/T379047) [14:04:15] 10Toolforge, 06tools-infrastructure-team: Upgrade Kyverno to 1.16 series - https://phabricator.wikimedia.org/T427928 (10taavi) 03NEW [14:06:54] 10Toolforge, 06tools-infrastructure-team: Upgrade Kyverno to 1.16 series - https://phabricator.wikimedia.org/T427928#11976633 (10taavi) [14:07:13] 06cloud-services-team, 10Quarry: replace ingress-nginx in Quarry - https://phabricator.wikimedia.org/T427849#11976639 (10Andrew) taavi says: "in quarry we can probably get away with just exposing the service directly as a NodePort and not having anything in the middle for simplicity. maybe in paws as well, if... [14:17:19] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-34 (2026-05-19 to 2026-06-02)): Exclude link objects from wikimedia-paths-parameter-example-exists - https://phabricator.wikimedia.org/T425920#11976665 (10MGoncalves-WMF) 05In progress→03Resolved [14:18:20] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-34 (2026-05-19 to 2026-06-02)): Fix or disable oas3-valid-media-example date validation - https://phabricator.wikimedia.org/T425952#11976671 (10MGoncalves-WMF) 05Open→03Resolved [14:20:12] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-34 (2026-05-19 to 2026-06-02)), 07OKR-Work: Fix schema property example validation - https://phabricator.wikimedia.org/T426836#11976675 (10MGoncalves-WMF) 05Open→03In progress [14:35:11] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [14:39:49] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-35 (2026-06-02 to 2026-06-16)), 07OKR-Work, 13Patch-For-Review: Improve linting - enum descriptions - https://phabricator.wikimedia.org/T424210#11976740 (10HCoplin-WMF) [14:40:33] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-35 (2026-06-02 to 2026-06-16)), 07OKR-Work: Fix schema property example validation - https://phabricator.wikimedia.org/T426836#11976757 (10HCoplin-WMF) [14:41:47] (03update) 10raymond-ndibe: support exposing continuous jobs to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T388092) [14:42:34] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 07OKR-Work: [SPIKE] Identify OAD properties not supported by MediaWiki REST Framework - https://phabricator.wikimedia.org/T425942#11976781 (10HCoplin-WMF) 05Open→03Resolved [14:49:17] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T388092) [14:51:14] (03update) 10raymond-ndibe: support exposing continuous jobs to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T388092) [15:07:13] (03update) 10raymond-ndibe: add jobs-api backend to webservice-cli [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T348755) [15:14:10] (03update) 10raymond-ndibe: Draft: add jobs-api backend to webservice-cli [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T348755) [15:15:18] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Puppet fail to create volume group for ephemeral disk space when it is sda (instead of sdb) - https://phabricator.wikimedia.org/T422258#11977056 (10hashar) Finally I have managed to test it! I cherry picked yo... [15:16:13] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Puppet fail to create volume group for ephemeral disk space when it is sda (instead of sdb) - https://phabricator.wikimedia.org/T422258#11977087 (10hashar) I also cherry-picked https://gerrit.wikimedia.org/r/c/... [15:22:30] (03approved) 10raymond-ndibe: toolforge_weld/logs/kubernetes.py: Handle continuation log records [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/92 (https://phabricator.wikimedia.org/T427449) (owner: 10dancy) [15:23:48] (03merge) 10raymond-ndibe: toolforge_weld/logs/kubernetes.py: Handle continuation log records [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/92 (https://phabricator.wikimedia.org/T427449) (owner: 10dancy) [15:26:08] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Puppet fail to create volume group for ephemeral disk space when it is sda (instead of sdb) - https://phabricator.wikimedia.org/T422258#11977210 (10Andrew) 05Open→03Resolved [15:29:35] (03open) 10raymond-ndibe: d/changelog: bump to 1.6.14 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/93 (https://phabricator.wikimedia.org/T422184 https://phabricator.wikimedia.org/T427449) [15:29:52] (03update) 10raymond-ndibe: d/changelog: bump to 1.6.14 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/93 (https://phabricator.wikimedia.org/T422184 https://phabricator.wikimedia.org/T427449) [15:31:28] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [15:35:50] 10Toolforge, 06tools-platform-team, 07Epic, 13Patch-For-Review: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#11977346 (10Raymond_Ndibe) 05Stalled→03In progress [15:36:54] 10Toolforge, 06tools-platform-team, 07Epic, 13Patch-For-Review: [jobs-api] allow exposing continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#11977366 (10Raymond_Ndibe) 05Stalled→03In progress [15:44:50] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld [15:45:26] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [15:59:54] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld [16:10:21] (03update) 10raymond-ndibe: d/changelog: bump to 1.6.14 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/93 (https://phabricator.wikimedia.org/T422184 https://phabricator.wikimedia.org/T427449) [16:10:25] (03approved) 10raymond-ndibe: d/changelog: bump to 1.6.14 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/93 (https://phabricator.wikimedia.org/T422184 https://phabricator.wikimedia.org/T427449) [16:10:34] (03merge) 10raymond-ndibe: d/changelog: bump to 1.6.14 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/93 (https://phabricator.wikimedia.org/T422184 https://phabricator.wikimedia.org/T427449) [16:22:22] 06cloud-services-team, 06Commons, 10SRE-swift-storage: Uncompressed TIFFs on commons - https://phabricator.wikimedia.org/T427949 (10Andrew) 03NEW [16:23:13] 06cloud-services-team, 06Commons, 10SRE-swift-storage: Uncompressed TIFFs on commons - https://phabricator.wikimedia.org/T427949#11977818 (10Andrew) [16:24:06] 06cloud-services-team, 06Commons, 10SRE-swift-storage: Uncompressed TIFFs on commons - https://phabricator.wikimedia.org/T427949#11977825 (10Andrew) [16:26:51] 06cloud-services-team, 06Commons, 10SRE-swift-storage: Uncompressed TIFFs on commons - https://phabricator.wikimedia.org/T427949#11977884 (10MatthewVernon) TIFF compression is fairly easy via [[ https://manpages.debian.org/trixie/libtiff-tools/tiffcp.1.en.html | tiffcp ]] (I'm not a compression specialist, b... [16:27:44] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: gitlab-webhooks: toolforge webservice logs -f choking on a log message - https://phabricator.wikimedia.org/T427449#11977901 (10dancy) 05Open→03Resolved a:03dancy Looks good now. [16:33:14] 10Toolforge, 06tools-platform-team: [Toolforge Sustainability Framework] Create an inventory of Toolforge actions - https://phabricator.wikimedia.org/T420559#11977978 (10komla) The inventory list was created and was used to assign scores for Web and API coverage subcategories. The framework now has all subcat... [16:38:07] 10Tools, 07Performance Issue: Stalktoy is often slow or not responding - https://phabricator.wikimedia.org/T427941#11977984 (10Reedy) [16:50:56] FIRING: [2x] SystemdUnitDown: The service unit kiwix-mirror-update.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [16:56:36] 06tools-platform-team: How to Deploy on Toolforge Workshop - https://phabricator.wikimedia.org/T427961 (10komla) 03NEW [16:57:37] 06tools-platform-team: How to Deploy on Toolforge Workshop - https://phabricator.wikimedia.org/T427961#11978132 (10komla) Some of these members used Toolforge many years ago and are looking for a refresher on the deployment process [16:58:32] 10Toolforge, 06tools-platform-team: [Toolforge]: Automate daily count of successful SSH logins as a measure of user activity - https://phabricator.wikimedia.org/T423549#11978133 (10komla) The systemd timer has been deployed via puppet to Gerrit and is going through [[ https://gerrit.wikimedia.org/r/c/operation... [17:55:47] 10Tool-wikiloves: Add Wiki Loves Ramadan to Wikiloves Statistics Tool - https://phabricator.wikimedia.org/T427971 (10Jony) 03NEW [18:23:34] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.98% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [18:28:34] RESOLVED: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.977% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [18:45:56] FIRING: [2x] SystemdUnitDown: The systemd unit kiwix-mirror-update.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [18:46:51] 10Tool-humaniki-2: [VERSION] 0.3.0 - https://phabricator.wikimedia.org/T427979 (10Danya) 03NEW [18:46:53] 10Tool-humaniki-2: [VERSION] 0.3.0 - https://phabricator.wikimedia.org/T427979#11978712 (10Danya) p:05Triage→03High [18:56:48] 10Tool-wikicontest, 07Privacy: WikiEval Tool / WikiContest violates Toolforge Privacy Policy - https://phabricator.wikimedia.org/T427744#11978732 (10GauriGupta) a:03GauriGupta [19:08:20] 10Tool-wikicontest, 07Privacy: WikiEval Tool / WikiContest violates Toolforge Privacy Policy - https://phabricator.wikimedia.org/T427744#11978754 (10GauriGupta) This is the PR associated with this task and is ready for review:https://github.com/indictechcom/wikicontest/pull/204/changes/e7583b760cf0ecbfccdbf052... [20:15:56] RESOLVED: SystemdUnitDown: The service unit kiwix-mirror-update.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [20:15:56] FIRING: [2x] SystemdUnitDown: The systemd unit kiwix-mirror-update.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [22:06:10] FIRING: [2x] ToolsDBAlmostFull: ToolsDB is almost full - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBAlmostFull - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBAlmostFull [23:56:01] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-35 (2026-06-02 to 2026-06-16)), 07OKR-Work: Improve linting - detect examples in nested schema.properties - https://phabricator.wikimedia.org/T424002#11979691 (10KineticPelagic) 05Open→03In progress a:03KineticPelagic