[02:23:07] 10wikitech.wikimedia.org, 10Wikimedia-Site-requests: Disable or restrict upload in Wikitech - https://phabricator.wikimedia.org/T431648 (10Bugreporter) 03NEW [03:10:18] (03PS1) 10MusikAnimal: twig: remove reference to release date on error page [labs/xtools] - 10https://gerrit.wikimedia.org/r/1308806 [03:11:12] (03CR) 10MusikAnimal: [C:03+2] twig: remove reference to release date on error page [labs/xtools] - 10https://gerrit.wikimedia.org/r/1308806 (owner: 10MusikAnimal) [03:11:54] (03Merged) 10jenkins-bot: twig: remove reference to release date on error page [labs/xtools] - 10https://gerrit.wikimedia.org/r/1308806 (owner: 10MusikAnimal) [03:54:14] (03PS7) 10MusikAnimal: RateLimitSubscriber: anon db traffic + rate limits [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T384711) [04:06:08] (03PS1) 10Jsn.sherman: migrations: add explicit length to string columns [labs/xtools] - 10https://gerrit.wikimedia.org/r/1308808 (https://phabricator.wikimedia.org/T280734) [04:06:11] (03PS1) 10Jsn.sherman: docker compose: containerize XTools for local dev [labs/xtools] - 10https://gerrit.wikimedia.org/r/1308809 (https://phabricator.wikimedia.org/T280734) [04:32:40] (03PS1) 10MusikAnimal: RateLimitSubscriber: allow subrequests with uselang query string param [labs/xtools] - 10https://gerrit.wikimedia.org/r/1308811 [04:36:52] (03CR) 10MusikAnimal: [C:03+2] "Self-merging as is the norm in XTools-land :) I wanted to get this fixed apart from Ifbf7ba5466f which is largely about a separate issue." [labs/xtools] - 10https://gerrit.wikimedia.org/r/1308811 (owner: 10MusikAnimal) [04:37:33] (03Merged) 10jenkins-bot: RateLimitSubscriber: allow subrequests with uselang query string param [labs/xtools] - 10https://gerrit.wikimedia.org/r/1308811 (owner: 10MusikAnimal) [04:37:53] (03PS8) 10MusikAnimal: RateLimitSubscriber: anon db traffic + rate limits [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T384711) [04:53:54] (03CR) 10MusikAnimal: RateLimitSubscriber: anon db traffic + rate limits (033 comments) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T384711) (owner: 10MusikAnimal) [04:57:13] (03CR) 10MusikAnimal: [C:03+2] migrations: add explicit length to string columns [labs/xtools] - 10https://gerrit.wikimedia.org/r/1308808 (https://phabricator.wikimedia.org/T280734) (owner: 10Jsn.sherman) [04:57:55] (03Merged) 10jenkins-bot: migrations: add explicit length to string columns [labs/xtools] - 10https://gerrit.wikimedia.org/r/1308808 (https://phabricator.wikimedia.org/T280734) (owner: 10Jsn.sherman) [05:33:52] (03CR) 10MusikAnimal: "(first off, thank you for your *extreme* patience in getting this reviewed)" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276371 (https://phabricator.wikimedia.org/T198872) (owner: 10Alien4444) [05:36:06] (03CR) 10MusikAnimal: "Sorry, I meant to mention the example I used for testing: `/sc/enwiki/MusikAnimal`" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276371 (https://phabricator.wikimedia.org/T198872) (owner: 10Alien4444) [05:53:43] 06cloud-services-team, 10Data-Services, 10BetaFeatures, 06Data-Engineering, and 5 others: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#12102995 (10Marostegui) [06:07:33] (03open) 10countcount: Forbid unsafe code [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/22 [06:07:33] (03open) 10countcount: Forbid unsafe code [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/40 [06:12:47] (03merge) 10countcount: Forbid unsafe code [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/40 [06:14:58] (03merge) 10countcount: Forbid unsafe code [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/22 [06:15:49] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/28 [06:16:54] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/20 [07:09:34] (03update) 10samwilson: Improve CSS complexity test [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/14 (https://phabricator.wikimedia.org/T271595 https://phabricator.wikimedia.org/T420835) [07:11:42] (03update) 10samwilson: Use library for CSS parsing when checking for ID selectors [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/14 (https://phabricator.wikimedia.org/T271595 https://phabricator.wikimedia.org/T420835) [07:12:06] (03update) 10samwilson: Use library for CSS parsing when checking for ID selectors [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/14 (https://phabricator.wikimedia.org/T271595 https://phabricator.wikimedia.org/T420835) [07:23:42] (03open) 10countcount: Show when a not-currently-eligible user was last generally eligible [toolforge-repos/stimmberechtigung] - 10https://gitlab.wikimedia.org/toolforge-repos/stimmberechtigung/-/merge_requests/7 [07:32:03] (03update) 10countcount: Show when a not-currently-eligible user was last generally eligible [toolforge-repos/stimmberechtigung] - 10https://gitlab.wikimedia.org/toolforge-repos/stimmberechtigung/-/merge_requests/7 [07:36:01] (03update) 10countcount: Show when a not-currently-eligible user was last generally eligible [toolforge-repos/stimmberechtigung] - 10https://gitlab.wikimedia.org/toolforge-repos/stimmberechtigung/-/merge_requests/7 [07:52:30] (03merge) 10countcount: Show when a not-currently-eligible user was last generally eligible [toolforge-repos/stimmberechtigung] - 10https://gitlab.wikimedia.org/toolforge-repos/stimmberechtigung/-/merge_requests/7 [08:05:20] (03update) 10raymond-ndibe: bump_version: improve the message when bumping the version [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/142 (owner: 10dcaro) [08:07:34] (03approved) 10dcaro: k8s: use job_type for label generating functions [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/328 [08:07:39] (03merge) 10dcaro: k8s: use job_type for label generating functions [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/328 [08:07:41] (03update) 10dcaro: runtime: remove diff_with_running_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/336 [08:07:44] (03update) 10dcaro: stop passing tool name [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/337 [08:09:15] (03update) 10dcaro: bump_version: improve the message when bumping the version [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/142 [08:10:15] (03update) 10dcaro: bump_version: improve the message when bumping the version [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/142 [08:10:59] FIRING: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [08:11:22] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.531-20260709080751-46dcb017 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1324 (https://phabricator.wikimedia.org/T430071) [08:11:23] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.531-20260709080751-46dcb017 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1324 (https://phabricator.wikimedia.org/T430071) [08:11:36] FIRING: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [08:15:59] RESOLVED: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [08:16:36] RESOLVED: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [08:17:51] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [08:27:33] (03update) 10raymond-ndibe: k8s/jobs.py: set PORT env whenever a job exposes a port [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/341 (https://phabricator.wikimedia.org/T431274) [08:29:10] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [08:29:19] (03open) 10tlepage: Draft: move deps: tox.ini → pyproject.toml [repos/cloud/toolforge/jobs-api] (fix-test-get-one-off-job-status) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/345 [08:29:21] (03update) 10raymond-ndibe: [values.yaml] add support for webservice-defaults [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/23 (https://phabricator.wikimedia.org/T431403) [08:30:02] (03update) 10tlepage: Draft: move deps: tox.ini → pyproject.toml [repos/cloud/toolforge/jobs-api] (fix-test-get-one-off-job-status) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/345 [08:41:24] 10wikitech.wikimedia.org, 10Wikimedia-Site-requests, 07Community-consensus-needed: Disable or restrict upload in Wikitech - https://phabricator.wikimedia.org/T431648#12103457 (10neriah) 05Open→03Stalled [08:51:15] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12103481 (10ayounsi) 05Open→03Resolved Maintenance done, last repool in progress. [09:00:49] (03update) 10tlepage: Draft: move deps: tox.ini → pyproject.toml [repos/cloud/toolforge/jobs-api] (fix-test-get-one-off-job-status) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/345 [09:00:50] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [09:07:00] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work, 13Patch-For-Review: Define a process for OAD resource synchronization - https://phabricator.wikimedia.org/T422927#12103551 (10KBach) [09:09:04] 10Tool-translatetagger, 06Indic-TechCom, 10LPL sprints, 10MediaWiki-extensions-Translate, and 3 others: Integrate TranslateTagger functions/workflow into Special:PagePreparation of Translate Extension - https://phabricator.wikimedia.org/T430257#12103570 (10Nikerabbit) [09:14:16] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [09:15:46] (03update) 10raymond-ndibe: kubernetes.py: detect conflict with httproutes created by jobs-api [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/114 (https://phabricator.wikimedia.org/T431417) [09:18:43] (03approved) 10dcaro: jobs-api: bump to 0.0.531-20260709080751-46dcb017 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1324 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [09:18:52] (03merge) 10dcaro: jobs-api: bump to 0.0.531-20260709080751-46dcb017 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1324 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [09:18:53] (03approved) 10dcaro: runtime: remove diff_with_running_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/336 [09:18:59] (03merge) 10dcaro: runtime: remove diff_with_running_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/336 [09:19:19] (03update) 10tlepage: Draft: move deps: tox.ini → pyproject.toml [repos/cloud/toolforge/jobs-api] (fix-test-get-one-off-job-status) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/345 [09:22:35] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.532-20260709091928-84bbee5f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1325 (https://phabricator.wikimedia.org/T430071) [09:22:35] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.532-20260709091928-84bbee5f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1325 (https://phabricator.wikimedia.org/T430071) [09:25:58] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [09:28:00] (03update) 10dcaro: stop passing tool name [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/337 [09:34:23] (03update) 10dcaro: k8s/jobs.py: set PORT env whenever a job exposes a port [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/341 (https://phabricator.wikimedia.org/T431274) (owner: 10raymond-ndibe) [09:35:43] (03update) 10dcaro: core,runtime: don't pass tool_name if not needed [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/337 [09:37:16] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [09:42:20] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Make webservice-cli use jobs-api - https://phabricator.wikimedia.org/T431421#12103773 (10Raymond_Ndibe) >>! In T431421#12097427, @bd808 wrote: >> `webservice shell` stays as it is, unless we add support for shelling into containers in jobs-api too. > {T43127... [09:43:08] 10Toolforge (Push-to-Deploy), 06tools-platform-team: jobs-api: Expose state and webservice_defaults on /images endpoint - https://phabricator.wikimedia.org/T431679 (10Raymond_Ndibe) 03NEW [09:45:31] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'cloudinfra-db' [09:46:11] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [09:47:32] FIRING: TargetDown: Job mariadb is unreachable in project cloudinfra instance cloudinfra-db03 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [09:49:15] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'cloudinfra-db' [09:49:45] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12103823 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jmm@cumin2003 for hosts: `testvm2007.co... [09:52:32] RESOLVED: TargetDown: Job mariadb is unreachable in project cloudinfra instance cloudinfra-db03 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [09:53:02] FIRING: TargetDown: Job mariadb is unreachable in project cloudinfra instance cloudinfra-db03 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [09:57:47] RESOLVED: TargetDown: Job mariadb is unreachable in project cloudinfra instance cloudinfra-db03 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [09:58:47] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.refresh_puppet_certs on cloudinfra-db-6.cloudinfra.eqiad1.wikimedia.cloud [10:00:47] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12103893 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jmm@cumin2003 for hosts: `testvm2008.wi... [10:00:48] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [10:01:16] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: Rebalance cloudvirts out of E4 - https://phabricator.wikimedia.org/T431682 (10fgiunchedi) 03NEW [10:01:34] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on cloudinfra-db-6.cloudinfra.eqiad1.wikimedia.cloud [10:03:05] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: Ensure cloudvirt capacity is more evenly spread out among racks - https://phabricator.wikimedia.org/T424658#12103917 (10fgiunchedi) >>! In T424658#12101239, @VRiley-WMF wrote: > @fgiunchedi I was curious to know if ther... [10:05:07] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: Rebalance cloudvirts out of E4 - https://phabricator.wikimedia.org/T431682#12103927 (10fgiunchedi) [10:06:47] (03CR) 10Filippo Giunchedi: [C:03+2] cloudvirt: use 'compute service set' to handle maintenance [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1307689 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [10:07:37] (03approved) 10dcaro: jobs-api: bump to 0.0.532-20260709091928-84bbee5f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1325 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:07:40] (03CR) 10Filippo Giunchedi: [C:03+2] "I'm for not hoarding code unless we have to, if/when we'll get to handling aggregates in cookbooks again we can restore this code" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1307690 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [10:07:42] (03merge) 10dcaro: jobs-api: bump to 0.0.532-20260709091928-84bbee5f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1325 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:07:54] (03CR) 10Filippo Giunchedi: [V:03+2 C:03+2] wmcs_libs: deprecate openstack aggregates handling [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1307690 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [10:10:15] (03update) 10dcaro: runtime: split create_job per-job type [repos/cloud/toolforge/jobs-api] (stop_passing_tool_name) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/330 [10:10:33] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.set_maintenance [10:11:00] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) [10:11:31] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.set_maintenance [10:11:33] !log filippo@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99) [10:12:45] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.remove_instance for instance syslog-server-audit01 [10:13:01] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance syslog-server-audit01 [10:13:07] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.remove_instance for instance syslog-server-audit02 [10:13:53] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance syslog-server-audit02 [10:20:17] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Upgrade cloudinfra database hosts off of Bullseye - https://phabricator.wikimedia.org/T402005#12104013 (10taavi) Done except I need to clean up the VMs in a couple of days. [10:20:34] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation), 07IPv6: Refresh cloudinfra central syslog audit servers to run on Trixie and enable IPv6 - https://phabricator.wikimedia.org/T402007#12104014 (10taavi) 05Open→03Resolved [10:22:31] (03PS1) 10Filippo Giunchedi: wmcs_libs: quote compute_service_disable reason [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1309119 (https://phabricator.wikimedia.org/T424802) [10:24:09] (03update) 10raymond-ndibe: support exposing continuous jobs to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [10:25:45] (03CR) 10CI reject: [V:04-1] wmcs_libs: quote compute_service_disable reason [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1309119 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [10:26:11] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12104039 (10ayounsi) 05Resolved→03Open [10:28:21] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T431410) [10:31:24] 06cloud-services-team, 10Cloud-VPS: openstack: alert for cloudvirts without aggregate or with unexpected set of them - https://phabricator.wikimedia.org/T284747#12104060 (10fgiunchedi) [10:31:57] (03update) 10taavi: Add and import initial Redis service module [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/106 (https://phabricator.wikimedia.org/T360704) [10:32:03] (03merge) 10taavi: Add and import initial Redis service module [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/106 (https://phabricator.wikimedia.org/T360704) [10:32:08] (03PS2) 10Filippo Giunchedi: wmcs_libs: quote compute_service_disable reason [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1309119 (https://phabricator.wikimedia.org/T424802) [10:35:20] 10Cloud-VPS, 06tools-infrastructure-team: openstack: alert for cloudvirts without aggregate or with unexpected set of them - https://phabricator.wikimedia.org/T284747#12104079 (10fgiunchedi) [10:35:26] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12104080 (10ops-monitoring-bot) Draining ganeti2033.codfw.wmnet of running VMs [10:46:19] 10Cloud-VPS, 06tools-infrastructure-team: openstack: alert for cloudvirts without aggregate or with unexpected set of them - https://phabricator.wikimedia.org/T284747#12104153 (10fgiunchedi) p:05Low→03Medium [10:47:59] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12104158 (10MoritzMuehlenhoff) [10:52:52] 06cloud-services-team, 10Toolforge: [lima-kilo] fails to inject ldap accounts - https://phabricator.wikimedia.org/T431686 (10DamianZaremba) 03NEW [11:01:04] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12104219 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jmm@cumin2003 for hosts: `testvm2005.codfw.... [11:02:53] (03update) 10raymond-ndibe: bump_version: improve the message when bumping the version [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/142 (owner: 10dcaro) [11:02:54] (03update) 10raymond-ndibe: bump_version: improve the message when bumping the version [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/142 (owner: 10dcaro) [11:03:01] (03update) 10raymond-ndibe: bump_version: improve the message when bumping the version [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/142 (owner: 10dcaro) [11:04:05] 06cloud-services-team, 10Toolforge: [lima-kilo] fails to inject ldap accounts - https://phabricator.wikimedia.org/T431686#12104228 (10DamianZaremba) [11:04:51] 06cloud-services-team, 10Toolforge: [lima-kilo] fails to inject ldap accounts - https://phabricator.wikimedia.org/T431686#12104243 (10DamianZaremba) Kubernetes is not ready; ` damian@lima-kilo:~$ kubectl get pods -A NAMESPACE NAME READY STATUS... [11:05:13] (03update) 10taavi: service: Allow managing VM port access to a VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/107 (https://phabricator.wikimedia.org/T360704) [11:05:14] (03update) 10taavi: redis: Add Keepalived VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/108 (https://phabricator.wikimedia.org/T360704) [11:05:16] (03update) 10taavi: toolsbeta: Add new Toolsbeta Redis hosts [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/109 (https://phabricator.wikimedia.org/T360704) [11:05:35] (03open) 10taavi: service: Allow managing VM port access to a VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/107 (https://phabricator.wikimedia.org/T360704) [11:05:36] (03open) 10taavi: toolsbeta: Add new Toolsbeta Redis hosts [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/109 (https://phabricator.wikimedia.org/T360704) [11:05:37] (03open) 10taavi: redis: Add Keepalived VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/108 (https://phabricator.wikimedia.org/T360704) [11:06:37] (03update) 10taavi: toolsbeta: Add new Toolsbeta Redis hosts [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/109 (https://phabricator.wikimedia.org/T360704) [11:06:42] (03update) 10taavi: redis: Add Keepalived VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/108 (https://phabricator.wikimedia.org/T360704) [11:06:44] (03update) 10taavi: service: Allow managing VM port access to a VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/107 (https://phabricator.wikimedia.org/T360704) [11:06:54] 06cloud-services-team, 10Toolforge: [lima-kilo] fails to inject ldap accounts - https://phabricator.wikimedia.org/T431686#12104254 (10DamianZaremba) ` damian@lima-kilo:~$ kubectl -n kube-system delete pod kindnet-k89vf pod "kindnet-k89vf" deleted ` ` damian@lima-kilo:~$ kubectl get pods -A NAMESPACE... [11:08:41] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:09:27] (03update) 10taavi: toolsbeta: Add new Toolsbeta Redis hosts [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/109 (https://phabricator.wikimedia.org/T360704) [11:09:32] (03update) 10taavi: redis: Add Keepalived VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/108 (https://phabricator.wikimedia.org/T360704) [11:09:36] (03update) 10taavi: service: Allow managing VM port access to a VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/107 (https://phabricator.wikimedia.org/T360704) [11:09:46] 06cloud-services-team, 10Toolforge: [lima-kilo] fails to inject ldap accounts - https://phabricator.wikimedia.org/T431686#12104292 (10DamianZaremba) It would be useful if the kind/k8s setup role checked the cluster is healthy before finishing. That would catch any transient issues due to e.g. slow/broken inte... [11:10:37] (03update) 10taavi: toolsbeta: Add new Toolsbeta Redis hosts [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/109 (https://phabricator.wikimedia.org/T360704) [11:10:42] (03update) 10taavi: redis: Add Keepalived VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/108 (https://phabricator.wikimedia.org/T360704) [11:10:44] (03update) 10taavi: service: Allow managing VM port access to a VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/107 (https://phabricator.wikimedia.org/T360704) [11:11:51] 06cloud-services-team, 10Toolforge: [lima-kilo] fails to inject ldap accounts - https://phabricator.wikimedia.org/T431686#12104327 (10DamianZaremba) Eventually; ` damian@lima-kilo:~$ kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE foxt... [11:12:36] (03update) 10taavi: toolsbeta: Add new Toolsbeta Redis hosts [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/109 (https://phabricator.wikimedia.org/T360704) [11:12:46] (03update) 10taavi: redis: Add Keepalived VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/108 (https://phabricator.wikimedia.org/T360704) [11:12:48] (03update) 10taavi: service: Allow managing VM port access to a VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/107 (https://phabricator.wikimedia.org/T360704) [11:13:26] 06cloud-services-team, 10Toolforge: [lima-kilo] check for k8s health (services running) / ldap service container running prior to executing account injection - https://phabricator.wikimedia.org/T431686#12104330 (10DamianZaremba) [11:14:46] (03update) 10taavi: toolsbeta: Add new Toolsbeta Redis hosts [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/109 (https://phabricator.wikimedia.org/T360704) [11:14:51] (03update) 10taavi: redis: Add Keepalived VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/108 (https://phabricator.wikimedia.org/T360704) [11:18:44] (03update) 10taavi: service: Allow managing VM port access to a VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/107 (https://phabricator.wikimedia.org/T360704) [11:18:46] (03update) 10taavi: redis: Add Keepalived VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/108 (https://phabricator.wikimedia.org/T360704) [11:18:46] (03update) 10taavi: toolsbeta: Add new Toolsbeta Redis hosts [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/109 (https://phabricator.wikimedia.org/T360704) [11:18:48] (03update) 10taavi: toolsbeta: Flip Redis DNS entry to new cluster [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/110 (https://phabricator.wikimedia.org/T360704) [11:18:58] (03update) 10taavi: toolsbeta: Add new Toolsbeta Redis hosts [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/109 (https://phabricator.wikimedia.org/T360704) [11:19:07] (03update) 10taavi: toolsbeta: Flip Redis DNS entry to new cluster [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/110 (https://phabricator.wikimedia.org/T360704) [11:19:08] (03open) 10taavi: toolsbeta: Flip Redis DNS entry to new cluster [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/110 (https://phabricator.wikimedia.org/T360704) [11:19:09] (03update) 10taavi: redis: Add Keepalived VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/108 (https://phabricator.wikimedia.org/T360704) [11:19:11] (03update) 10taavi: service: Allow managing VM port access to a VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/107 (https://phabricator.wikimedia.org/T360704) [11:20:50] (03update) 10taavi: toolsbeta: Flip Redis DNS entry to new cluster [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/110 (https://phabricator.wikimedia.org/T360704) [11:21:00] (03update) 10taavi: toolsbeta: Add new Toolsbeta Redis hosts [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/109 (https://phabricator.wikimedia.org/T360704) [11:21:09] (03update) 10taavi: redis: Add Keepalived VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/108 (https://phabricator.wikimedia.org/T360704) [11:21:15] (03update) 10taavi: service: Allow managing VM port access to a VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/107 (https://phabricator.wikimedia.org/T360704) [11:24:53] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.reboot for all nodes [11:24:57] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for all nodes [11:36:11] (03update) 10damian: Draft: command - selectively wrap command [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/333 (https://phabricator.wikimedia.org/T430578) [11:39:30] 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 07Essential-Work, 06Moderator-Tools-Team (Kanban): twl: Replace deprecated Bullseye VMs in Cloud VPS - https://phabricator.wikimedia.org/T402054#12104493 (10taavi) Unprivileged user namespaces are disabled on Wikimedia servers by default s... [11:46:15] FIRING: [8x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-gateway-3.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [11:46:48] (03merge) 10dcaro: core,runtime: don't pass tool_name if not needed [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/337 [11:46:51] (03update) 10dcaro: runtime: split create_job per-job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/330 [11:48:24] (03update) 10damian: Draft: command - selectively wrap command [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/333 (https://phabricator.wikimedia.org/T430578) [11:50:18] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.533-20260709114700-54de0675 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1326 (https://phabricator.wikimedia.org/T430071) [11:50:29] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.533-20260709114700-54de0675 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1326 (https://phabricator.wikimedia.org/T430071) [11:51:14] FIRING: [18x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-control-11.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDow [11:52:16] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes [11:56:14] RESOLVED: [20x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-control-10.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerD [11:59:19] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [12:00:22] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T431410) [12:00:45] (03update) 10dcaro: runtime: split create_job per-job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/330 [12:01:03] (03update) 10damian: Draft: command - selectively wrap command [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/333 (https://phabricator.wikimedia.org/T430578) [12:01:43] (03open) 10dcaro: status_deprecated: delete duplicated pod fetching [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/346 [12:01:57] (03approved) 10dcaro: status_deprecated: delete duplicated pod fetching [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/346 [12:02:01] (03merge) 10dcaro: status_deprecated: delete duplicated pod fetching [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/346 [12:02:05] (03update) 10dcaro: runtime: split create_job per-job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/330 [12:02:35] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.534-20260709115912-e38d1b70 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1326 (https://phabricator.wikimedia.org/T430071) [12:02:43] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.534-20260709115912-e38d1b70 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1326 (https://phabricator.wikimedia.org/T430071) [12:08:18] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1309150 (owner: 10L10n-bot) [12:10:08] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12104734 (10TLepage-WMF) [12:11:11] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [12:12:59] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T431410) [12:14:27] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [12:15:27] 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 07Essential-Work, 06Moderator-Tools-Team (Kanban): twl: Replace deprecated Bullseye VMs in Cloud VPS - https://phabricator.wikimedia.org/T402054#12104794 (10jsn.sherman) >>! In T402054#12104493, @taavi wrote: > Unprivileged user namespaces... [12:24:00] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T431410) [12:25:32] (03open) 10raymond-ndibe: models.py: expose state and webservice_defaults in images [repos/cloud/toolforge/jobs-api] (add_webservice_endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/347 (https://phabricator.wikimedia.org/T431679) [12:25:33] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: Rebalance cloudvirts out of E4 - https://phabricator.wikimedia.org/T431682#12104867 (10fgiunchedi) [12:25:34] (03update) 10raymond-ndibe: models.py: expose state and webservice_defaults in images [repos/cloud/toolforge/jobs-api] (add_webservice_endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/347 (https://phabricator.wikimedia.org/T431679) [12:26:38] (03CR) 10Filippo Giunchedi: [C:03+2] wmcs_libs: quote compute_service_disable reason [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1309119 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [12:27:15] (03open) 10dcaro: k8s: use K8sKind instead of directly the k8s kinds [repos/cloud/toolforge/jobs-api] (split_create_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/348 [12:31:16] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [12:33:28] (03update) 10dcaro: runtime: split create_job per-job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/330 [12:34:46] (03approved) 10dcaro: jobs-api: bump to 0.0.534-20260709115912-e38d1b70 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1326 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:34:55] (03merge) 10dcaro: jobs-api: bump to 0.0.534-20260709115912-e38d1b70 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1326 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:37:24] 06tools-platform-team: [ci,toolforge-deploy] Deduplicate bugs in the deployment branches created - https://phabricator.wikimedia.org/T431704 (10dcaro) 03NEW [12:38:39] (03Abandoned) 10Nikerabbit: Localisation updates from https://translatewiki.net. [labs/tools/wikiinfo] - 10https://gerrit.wikimedia.org/r/1307785 (owner: 10L10n-bot) [12:39:17] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12105017 (10TLepage-WMF) [12:40:06] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team: Migrate metricsinfra project off of Bullseye - https://phabricator.wikimedia.org/T401813#12105022 (10taavi) a:03taavi [12:40:22] (03CR) 10Nikerabbit: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/wikiinfo] - 10https://gerrit.wikimedia.org/r/1309169 (owner: 10L10n-bot) [12:40:32] (03open) 10countcount: Give the ToolsDB pool the same slow-statement logging and idle timeout as the replica pools [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/41 [12:44:39] (03CR) 10Nikerabbit: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1309150 (owner: 10L10n-bot) [12:45:29] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team: Migrate metricsinfra project off of Bullseye - https://phabricator.wikimedia.org/T401813#12105037 (10taavi) We'll also need to move the Prometheus instances and their special reserved IPs to the VXLAN network. New IPs have been allocated:... [12:46:17] FIRING: JobUnavailable: Reduced availability for job blackbox_http in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [12:51:17] RESOLVED: JobUnavailable: Reduced availability for job blackbox_http in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [12:51:59] (03update) 10dcaro: runtime: split create_job per-job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/330 [12:52:43] (03open) 10taavi: Update security groups for metricsinfra VXLAN/Dualstack IPs [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/332 (https://phabricator.wikimedia.org/T401813) [12:53:40] !log taavi@cloudcumin1001 tools END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes [12:53:45] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for all nodes [12:53:53] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/332 [12:54:18] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/332 [13:00:44] (03PS2) 10Jsn.sherman: docker compose: containerize XTools for local dev [labs/xtools] - 10https://gerrit.wikimedia.org/r/1308809 (https://phabricator.wikimedia.org/T280734) [13:13:54] (03update) 10raymond-ndibe: runtime: split create_job per-job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/330 (owner: 10dcaro) [13:13:57] (03approved) 10raymond-ndibe: runtime: split create_job per-job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/330 (owner: 10dcaro) [13:20:53] 10VPS-project-devtools, 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#12105202 (10Arnoldokoth) I've setup `phabricato-trixie` and the lats merge... [13:21:15] 10Tool-translatetagger, 06Indic-TechCom, 10LPL sprints, 10MediaWiki-extensions-Translate, and 3 others: Integrate TranslateTagger functions/workflow into Special:PagePreparation of Translate Extension - https://phabricator.wikimedia.org/T430257#12105205 (10abi_) [13:23:19] 10Tool-translatetagger, 06Indic-TechCom, 10LPL sprints, 10MediaWiki-extensions-Translate, and 3 others: Integrate TranslateTagger functions/workflow into Special:PagePreparation of Translate Extension - https://phabricator.wikimedia.org/T430257#12105216 (10abi_) [13:24:28] 10Tool-translatetagger, 06Indic-TechCom, 10MediaWiki-extensions-Translate, 07Community-collaboration, and 3 others: Integrate TranslateTagger functions/workflow into Special:PagePreparation of Translate Extension - https://phabricator.wikimedia.org/T430257#12105222 (10Nikerabbit) [13:29:57] !log taavi@cloudcumin1001 metricsinfra START - Cookbook wmcs.vps.refresh_puppet_certs on metricsinfra-controller-3.metricsinfra.eqiad1.wikimedia.cloud [13:30:07] !log taavi@cloudcumin1001 metricsinfra END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on metricsinfra-controller-3.metricsinfra.eqiad1.wikimedia.cloud [13:34:36] FIRING: TargetDown: Job manager is unreachable in project metricsinfra instance metricsinfra-controller-3 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [13:40:26] !log taavi@cloudcumin1001 metricsinfra START - Cookbook wmcs.vps.refresh_puppet_certs on metricsinfra-controller-3.metricsinfra.eqiad1.wikimedia.cloud [13:42:05] !log taavi@cloudcumin1001 metricsinfra END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on metricsinfra-controller-3.metricsinfra.eqiad1.wikimedia.cloud [13:44:32] FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-75 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:45:32] (03PS1) 10Majavah: build: Update PyYAML [cloud/metricsinfra/prometheus-manager] - 10https://gerrit.wikimedia.org/r/1309203 [13:46:16] (03CR) 10Majavah: [C:03+2] build: Update PyYAML [cloud/metricsinfra/prometheus-manager] - 10https://gerrit.wikimedia.org/r/1309203 (owner: 10Majavah) [13:46:51] (03Merged) 10jenkins-bot: build: Update PyYAML [cloud/metricsinfra/prometheus-manager] - 10https://gerrit.wikimedia.org/r/1309203 (owner: 10Majavah) [13:49:32] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-75 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:51:02] FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-74 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:52:38] (03PS1) 10Majavah: build: Update Flask packages [cloud/metricsinfra/prometheus-manager] - 10https://gerrit.wikimedia.org/r/1309205 [13:53:52] (03CR) 10Majavah: [C:03+2] build: Update Flask packages [cloud/metricsinfra/prometheus-manager] - 10https://gerrit.wikimedia.org/r/1309205 (owner: 10Majavah) [13:54:29] (03Merged) 10jenkins-bot: build: Update Flask packages [cloud/metricsinfra/prometheus-manager] - 10https://gerrit.wikimedia.org/r/1309205 (owner: 10Majavah) [13:54:36] RESOLVED: TargetDown: Job manager is unreachable in project metricsinfra instance metricsinfra-controller-3 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [13:56:02] RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-74 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:02:06] FIRING: [2x] TargetDown: Job manager is unreachable in project metricsinfra instance metricsinfra-controller-2 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [14:02:32] FIRING: InstanceDown: Project metricsinfra instance metricsinfra-controller-2 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:03:18] FIRING: MetricsinfraMaintainProjectsBroken: metricsinfra failing to update projects - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MetricsinfraMaintainProjectsBroken - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMetricsinfraMaintainProjectsBroken [14:07:06] RESOLVED: [2x] TargetDown: Job manager is unreachable in project metricsinfra instance metricsinfra-controller-2 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [14:07:32] RESOLVED: InstanceDown: Project metricsinfra instance metricsinfra-controller-2 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:08:18] RESOLVED: MetricsinfraMaintainProjectsBroken: metricsinfra failing to update projects - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MetricsinfraMaintainProjectsBroken - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMetricsinfraMaintainProjectsBroken [14:08:32] FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-70 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:11:36] FIRING: NfsAlmostFull: The NFS drive is over 85% capacity (currently 98.27%) at host paws-nfs-2 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull [14:13:32] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-70 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:29:17] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 07Epic: [cicd] Streamline toolforge cli deployment and external contributor ci flows - https://phabricator.wikimedia.org/T392524#12105565 (10dcaro) [14:31:21] 06cloud-services-team, 10Cloud-VPS: Update all cloud-vps Bullseye NFS servers - https://phabricator.wikimedia.org/T429793#12105584 (10taavi) [14:38:32] FIRING: InstanceDown: Project gitlab-runners instance runner-1035 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:40:26] 06cloud-services-team, 10Cloud-VPS: Consider splitting tf-infra-test into multiple "test suites" - https://phabricator.wikimedia.org/T423560#12105669 (10taavi) [14:40:43] 06cloud-services-team, 10VPS-Projects: Dumps: replace Bullseye nfs server with Trixie - https://phabricator.wikimedia.org/T430590#12105670 (10taavi) [14:49:08] 06tools-platform-team: [jobs-api] Check all the k8s objects related to a job when checking the up_to_date state - https://phabricator.wikimedia.org/T431724 (10dcaro) 03NEW [14:49:22] 10Toolforge, 06tools-platform-team: [jobs-api] Check all the k8s objects related to a job when checking the up_to_date state - https://phabricator.wikimedia.org/T431724#12105741 (10dcaro) [14:49:27] 10Tool-delintbot: Fix all sorts of bogus-image-options errors - https://phabricator.wikimedia.org/T421530#12105742 (10Redmin) a:05Redmin→03None [14:52:09] (03PS3) 10Jsn.sherman: docker compose: containerize XTools for local dev [labs/xtools] - 10https://gerrit.wikimedia.org/r/1308809 (https://phabricator.wikimedia.org/T280734) [14:52:29] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work, 13Patch-For-Review: Fix top-level issues in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428147#12105762 (10AGhirelli-WMF) 05Open→03In progress [14:57:59] (03update) 10dcaro: runtime: split create_job per-job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/330 [15:00:16] 10Toolforge, 06tools-platform-team: [jobs-api] Centralize test env config in pyproject.toml - https://phabricator.wikimedia.org/T431727 (10TLepage-WMF) 03NEW [15:00:31] 10Toolforge, 06tools-platform-team: [jobs-api] Centralize test env config in pyproject.toml - https://phabricator.wikimedia.org/T431727#12105837 (10TLepage-WMF) a:03TLepage-WMF [15:00:39] (03update) 10dcaro: k8s: use K8sKind instead of directly the k8s kinds [repos/cloud/toolforge/jobs-api] (split_create_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/348 [15:08:56] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:19:10] (03update) 10dcaro: k8s: use K8sKind instead of directly the k8s kinds [repos/cloud/toolforge/jobs-api] (split_create_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/348 [15:28:16] (03CR) 10Alien4444: "Er, I certainly didn't expect this patch to have performance issues." [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276371 (https://phabricator.wikimedia.org/T198872) (owner: 10Alien4444) [15:29:44] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [lima-kilo] check for k8s health (services running) / ldap service container running prior to executing account injection - https://phabricator.wikimedia.org/T431686#12106003 (10dcaro) Hmm... I agree that the check might be useful, but I note also th... [15:31:59] (03CR) 10Alien4444: "*Correction: the mysql optimizer uses `log_times` and not `log_actor_type_time` as I said, so it's rather that it doesn't index by actor." [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276371 (https://phabricator.wikimedia.org/T198872) (owner: 10Alien4444) [15:39:13] 10Tool-documentation, 10Adiutor, 06Wiki-Mentor-Africa, 07good first task: [Documentation] Update and improve docs for Adiutor tool - https://phabricator.wikimedia.org/T390392#12106031 (10Priyankar22) Hi @TBurmeister, thanks for the feedback! I've updated the draft: fixed the CSD acronym and removed the "Ho... [15:51:09] (03CR) 10Alien4444: "Ok, this is getting weirder and weirder. It appears to only use this wrong index for some rare users: I've only found you so far (actor_id" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276371 (https://phabricator.wikimedia.org/T198872) (owner: 10Alien4444) [15:54:29] 06cloud-services-team, 10PAWS, 06tools-platform-team: PAWS disk space is running out - https://phabricator.wikimedia.org/T429578#12106111 (10BLiviero-WMF) the amount of paws disk space available is low and getting worse as far as i can see. what actions are going to make a measurable difference? [15:57:20] 10Tool-documentation, 10Adiutor, 06Wiki-Mentor-Africa, 07good first task: [Documentation] Update and improve docs for Adiutor tool - https://phabricator.wikimedia.org/T390392#12106134 (10TBurmeister) Other than @ mentioning Dogu like we have in this thread, posting on Talk:Adiutor is the other best way I k... [16:14:33] 06cloud-services-team, 10PAWS, 06tools-platform-team: PAWS disk space is running out - https://phabricator.wikimedia.org/T429578#12106172 (10BLiviero-WMF) clarification: look at paws-nfs-2 space [16:22:44] 10PAWS, 06tools-platform-team: [paws] increase the spawner consecutive_failure_limit - https://phabricator.wikimedia.org/T431740 (10dcaro) 03NEW [16:59:20] 10PAWS, 06tools-platform-team: [bug] - https://phabricator.wikimedia.org/T431744 (10Soumendrapatra21) 03NEW [17:18:22] 10Cloud-Services, 06collaboration-services, 06Data-Persistence, 06Infrastructure-Foundations, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12106350 (10cmooney) [17:47:14] FIRING: ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [17:48:33] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 6 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:52:14] RESOLVED: [6x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [17:53:33] FIRING: [4x] ProbeDown: Service tools-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:57:44] FIRING: [22x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-8.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [17:58:33] RESOLVED: ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:59:28] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes [18:02:44] RESOLVED: [24x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [20:08:50] 10Wikibugs, 13Patch-For-Review: Wikibugs testing task - https://phabricator.wikimedia.org/T90594#12106874 (10bd808) test [20:10:33] 10Wikibugs: Wikibugs hasn't reported any new events since 17:47 UTC - https://phabricator.wikimedia.org/T431756#12106880 (10bd808) 05In progress→03Resolved `lang=irc [20:08] wikibugs (~wikibugs@wikimedia/bot/pywikibugs) joined the channel [20:08] ChanServ sets mode +v wikibugs [20:08] < wikibugs> Wikib... [20:13:14] 10Wikibugs: Wikibugs hasn't reported any new events since 17:47 UTC - https://phabricator.wikimedia.org/T431756#12106888 (10SomeRandomDeveloper) Thanks! [20:15:44] (03update) 10renovatebot: Update dependency typescript to v7 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/20 [20:24:41] 10PAWS, 06tools-platform-team: [bug] Notebook container not spawing for user Soumendrapatra21 - https://phabricator.wikimedia.org/T431744#12106931 (10bd808) [21:14:30] (03update) 10renovatebot: Update pnpm to v11.11.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/4 [21:14:34] (03open) 10renovatebot: Update pnpm to v11.11.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/4 [21:15:32] (03update) 10renovatebot: Update pnpm to v11.11.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/39 [21:15:36] (03open) 10renovatebot: Update pnpm to v11.11.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/39 [21:16:31] (03update) 10renovatebot: Update pnpm to v11.11.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/22 [21:16:36] (03open) 10renovatebot: Update pnpm to v11.11.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/22 [21:56:10] 10wikitech.wikimedia.org, 10Wikimedia-Site-requests, 07Community-consensus-needed: Disable or restrict upload in Wikitech - https://phabricator.wikimedia.org/T431648#12107115 (10bd808) Media management is something that has not really been discussed since the conversion of Wikitech to a SUL wiki which change... [22:01:43] 10wikitech.wikimedia.org, 10Wikimedia-Site-requests, 07Community-consensus-needed: Disable or restrict upload in Wikitech - https://phabricator.wikimedia.org/T431648#12107122 (10bd808) If group protection is added, I would suggest that wikitech follow the lead of mediawiki.org and add an `uploaders` group so... [23:05:48] (03PS4) 10Jsn.sherman: docker compose: containerize XTools for local dev [labs/xtools] - 10https://gerrit.wikimedia.org/r/1308809 (https://phabricator.wikimedia.org/T280734) [23:08:58] (03CR) 10Jsn.sherman: docker compose: containerize XTools for local dev (032 comments) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1308809 (https://phabricator.wikimedia.org/T280734) (owner: 10Jsn.sherman) [23:29:48] (03CR) 10MusikAnimal: [C:03+2] "Works fantastically!!! I even tested the "Usage Stats" and that parts works perfectly too." [labs/xtools] - 10https://gerrit.wikimedia.org/r/1308809 (https://phabricator.wikimedia.org/T280734) (owner: 10Jsn.sherman) [23:30:30] (03Merged) 10jenkins-bot: docker compose: containerize XTools for local dev [labs/xtools] - 10https://gerrit.wikimedia.org/r/1308809 (https://phabricator.wikimedia.org/T280734) (owner: 10Jsn.sherman)