[01:08:38] 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 07Essential-Work, 06Moderator-Tools-Team (Kanban): twl: Replace deprecated Bullseye VMs in Cloud VPS - https://phabricator.wikimedia.org/T402054#12097716 (10jsn.sherman) PR: https://github.com/WikipediaLibrary/TWLight/pull/1541 [01:08:44] 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 07Essential-Work, 06Moderator-Tools-Team (Kanban): twl: Replace deprecated Bullseye VMs in Cloud VPS - https://phabricator.wikimedia.org/T402054#12097717 (10jsn.sherman) 05Open→03In progress [01:15:26] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/34 [01:15:56] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [02:00:53] 10VPS-project-Wikistats: Rank page inaccessible (500 Error) - https://phabricator.wikimedia.org/T431462#12097769 (10Dzahn) a:03Dzahn [02:37:54] (03open) 10samwilson: Delete cached file on initial page load [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/15 (https://phabricator.wikimedia.org/T430577) [04:09:25] 10VPS-project-Wikistats: Add minwikiquote to wikistats - https://phabricator.wikimedia.org/T429948#12097865 (10Dzahn) 05Open→03Resolved a:03Dzahn ` MariaDB [wikistats]> insert into wikiquotes (prefix, lang, loclang, method) select prefix,lang,loclang,method from wikipedias where prefix="min"; Query O... [06:11:41] 10Tool-ghanasupremecases1, 03WMA-HackChallenge-2025: Update Color Scheme and Button Styles - https://phabricator.wikimedia.org/T412546#12097939 (10Pppery) p:05Low→03Triage a:03Fauwaz [06:49:33] (03PS1) 10Shashwatojha: Move image attribution below image to avoid obscuring content [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1308348 [06:55:54] (03update) 10samwilson: Improve CSS complexity test [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/14 (https://phabricator.wikimedia.org/T271595) [07:01:22] (03update) 10samwilson: Improve CSS complexity test [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/14 (https://phabricator.wikimedia.org/T271595) [07:36:26] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06tools-infrastructure-team: clean up nova IDs for newer cloudvirts - https://phabricator.wikimedia.org/T430933#12098106 (10fgiunchedi) a:05fgiunchedi→03None [07:59:45] 06cloud-services-team, 10Toolforge, 06tools-platform-team: Expose Toolforge service names via environment variables - https://phabricator.wikimedia.org/T151002#12098219 (10dcaro) [08:00:09] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [envvars-api,envvars-admission] Expose Toolforge service names via environment variables - https://phabricator.wikimedia.org/T151002#12098220 (10dcaro) [08:01:02] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [envvars-api,envvars-admission] Expose Toolforge service names via environment variables - https://phabricator.wikimedia.org/T151002#12098227 (10dcaro) Missing things: * toolsdb * replicas [08:01:46] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [envvars] show the 'global' envvars when running `toolforge envvars list` - https://phabricator.wikimedia.org/T394408#12098231 (10dcaro) [08:03:00] (03merge) 10dcaro: status: parse restarted_at with different format [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/344 [08:04:00] (03update) 10dcaro: bump_version: improve the message when bumping the version [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/142 [08:04:34] 06cloud-services-team, 10Striker, 06tools-platform-team: Cannot join Toolforge: Striker throws IntegrityError while saving user - https://phabricator.wikimedia.org/T429759#12098241 (10Kavaljeet_Singh) Thanks for investigating and identifying the issue. I do not intentionally need two Developer accounts. I o... [08:04:45] 10Toolforge, 06tools-platform-team: clis: only create tag on merge of the release patch - https://phabricator.wikimedia.org/T422452#12098254 (10dcaro) Quick doc-oriented fix until we get around automation: https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/142 [08:06:35] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.530-20260708080314-e696102e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1323 (https://phabricator.wikimedia.org/T431385) [08:06:43] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.530-20260708080314-e696102e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1323 (https://phabricator.wikimedia.org/T431385) [08:15:29] (03update) 10dcaro: k8s: use job_type for label generating functions [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/328 [08:23:55] 10Cloud-VPS (Project-requests): Request creation of wdqs-qlever-build-runner VPS project - https://phabricator.wikimedia.org/T431531 (10gmodena) 03NEW [08:24:10] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [08:24:33] (03Abandoned) 10Nikerabbit: Localisation updates from https://translatewiki.net. [labs/tools/wikiinfo] - 10https://gerrit.wikimedia.org/r/1306257 (owner: 10L10n-bot) [08:24:37] (03Abandoned) 10Nikerabbit: Localisation updates from https://translatewiki.net. [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1306233 (owner: 10L10n-bot) [08:24:45] (03Abandoned) 10Nikerabbit: Localisation updates from https://translatewiki.net. [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1307121 (owner: 10L10n-bot) [08:24:53] (03Abandoned) 10Nikerabbit: Localisation updates from https://translatewiki.net. [labs/tools/wikiinfo] - 10https://gerrit.wikimedia.org/r/1307135 (owner: 10L10n-bot) [08:25:15] (03Abandoned) 10Nikerabbit: Localisation updates from https://translatewiki.net. [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1307775 (owner: 10L10n-bot) [08:31:23] 10Cloud-VPS (Project-requests): Request creation of wdp-gitlab-runner VPS project - https://phabricator.wikimedia.org/T431531#12098362 (10gmodena) [08:31:55] 10Cloud-VPS (Project-requests), 06Wikidata Platform Team: Request creation of wdp-gitlab-runner VPS project - https://phabricator.wikimedia.org/T431531#12098372 (10dcaro) Hi! Given that it's one VM, and might be closely related, is this something that can be part of the query-service project? Or wikiqlever? [08:36:11] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [08:54:36] 10Cloud-VPS (Project-requests), 06Wikidata Platform Team: Request creation of wdp-gitlab-runner VPS project - https://phabricator.wikimedia.org/T431531#12098509 (10gmodena) Hi @dcaro ! AFAIK those are community driven hosting projects that run their own query service. Wikidata Platform we would need somethin... [08:55:38] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [08:59:38] (03update) 10dcaro: [values.yaml] add support for webservice-defaults [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/23 (https://phabricator.wikimedia.org/T431403) (owner: 10raymond-ndibe) [09:07:07] 10Tool-schedule-deployment: Allow patch authors to remove their patches from deployment windows - https://phabricator.wikimedia.org/T431521#12098575 (10Aklapper) [09:08:28] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [09:08:59] (03approved) 10dcaro: jobs-api: bump to 0.0.530-20260708080314-e696102e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1323 (https://phabricator.wikimedia.org/T431385) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [09:09:04] (03merge) 10dcaro: jobs-api: bump to 0.0.530-20260708080314-e696102e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1323 (https://phabricator.wikimedia.org/T431385) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [09:17:23] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12098654 (10TLepage-WMF) [09:18:48] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12098665 (10TLepage-WMF) [09:34:27] 10Toolforge, 06tools-platform-team: [components-api] Errors deploying sample-complex-app in toolsbeta - https://phabricator.wikimedia.org/T431385#12098748 (10dcaro) 05In progress→03Resolved [09:42:37] 10Cloud-VPS (Project-requests), 06Wikidata Platform Team: Request creation of wdp-gitlab-runner VPS project - https://phabricator.wikimedia.org/T431531#12098800 (10dcaro) >>! In T431531#12098509, @gmodena wrote: > Hi @dcaro ! > > AFAIK those are community driven hosting projects that run their own query servi... [09:48:49] 10Cloud-VPS (Project-requests), 06Wikidata Platform Team: Request creation of wdp-gitlab-runner VPS project - https://phabricator.wikimedia.org/T431531#12098838 (10gmodena) >>! In T431531#12098800, @dcaro wrote: > 👍 just checking :) That was a good point! Hopefully a broader audience can benefit from this pro... [09:52:49] 06cloud-services-team, 10Quarry, 06RoadToWiki, 13Patch-For-Review, and 2 others: "1 rows" should be "1 row" - https://phabricator.wikimedia.org/T419564#12098855 (10Nihar_Chakravarti) [10:01:41] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12098925 (10ops-monitoring-bot) Rebooting db-test1001.eqiad.wmnet [10:05:07] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12098934 (10ops-monitoring-bot) Rebooting db-test1001.eqiad.wmnet [10:05:12] 10Cloud-VPS (Project-requests), 06Wikidata Platform Team: Request creation of wdp-gitlab-runner VPS project - https://phabricator.wikimedia.org/T431531#12098939 (10fgiunchedi) +1 LGTM [10:08:34] 10Toolforge, 06tools-platform-team: Create a demo app for demonstration - https://phabricator.wikimedia.org/T431550 (10komla) 03NEW [10:08:47] 10Toolforge, 06tools-platform-team: Create a demo app for demonstration - https://phabricator.wikimedia.org/T431550#12098961 (10komla) 05Open→03Resolved [10:10:33] 10Cloud-VPS (Project-requests), 06Wikidata Platform Team: Request creation of wdp-gitlab-runner VPS project - https://phabricator.wikimedia.org/T431531#12098967 (10dcaro) a:03dcaro [10:11:22] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T431410) [10:17:00] 10Tool-wmf-openapi-linter, 06Tech-Docs-Team, 03[MWI] FY2025-26 Q4, 07OKR-Work: [Hypothesis] 5.2.5b: Productionalize API spec linting - https://phabricator.wikimedia.org/T422476#12099030 (10KBach) [10:20:49] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T431410) [10:36:59] (03open) 10countcount: Extract IP addresses glued to temporary-account usernames [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/37 [10:40:39] (03update) 10countcount: Extract IP addresses glued to temporary-account usernames [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/37 [10:40:55] (03update) 10countcount: Extract IP addresses glued to temporary-account usernames [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/37 [10:41:30] (03update) 10countcount: Extract IP addresses glued to temporary-account usernames [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/37 [10:41:46] (03merge) 10countcount: Extract IP addresses glued to temporary-account usernames [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/37 [10:41:59] (03update) 10countcount: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/34 (owner: 10renovatebot) [10:43:11] (03merge) 10countcount: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/34 (owner: 10renovatebot) [11:43:31] 10Tool-translatetagger, 06Indic-TechCom, 10LPL sprints, 10MediaWiki-extensions-Translate, 07Community-collaboration: Integrate TranslateTagger functions/workflow into Special:PagePreparation of Translate Extension - https://phabricator.wikimedia.org/T430257#12099445 (10Nikerabbit) [11:53:11] (03update) 10dcaro: cleanup by image name [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/165 [11:54:36] (03update) 10dcaro: cleanup by image name [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/165 [11:57:45] (03update) 10dcaro: cleanup by image name [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/165 [11:59:06] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Epic: Add pre-flight validity checks to the `webservice-runner` script in webservice images - https://phabricator.wikimedia.org/T431563 (10Raymond_Ndibe) 03NEW [12:03:13] (03update) 10tlepage: runtimes/k8s: fix flaky test_get_one_off_job_status [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/338 (https://phabricator.wikimedia.org/T430550) [12:13:16] !log dcaro@cloudcumin1001 wdp-gitlab-runner START - Cookbook wmcs.vps.create_project for project wdp-gitlab-runner in eqiad1 (T431531) [12:13:17] dcaro@cloudcumin1001: Unknown project "wdp-gitlab-runner" [12:13:18] T431531: Request creation of wdp-gitlab-runner VPS project - https://phabricator.wikimedia.org/T431531 [12:14:02] (03update) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project wdp-gitlab-runner [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/320 (https://phabricator.wikimedia.org/T431531) [12:14:05] (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project wdp-gitlab-runner [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/320 (https://phabricator.wikimedia.org/T431531) [12:17:14] dcaro@cloudcumin1001 create_project (PID 460679) is awaiting input [12:17:23] 10Toolforge, 06tools-platform-team: Regression: Images built with the build service images do no longer contain/configure locales specified in .locales - https://phabricator.wikimedia.org/T428230#12099650 (10dcaro) 05Open→03In progress [12:17:47] (03PS1) 10Raymond Ndibe: webservice-runner: Add pre-flight validity checks [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1308620 (https://phabricator.wikimedia.org/T431563) [12:18:07] 06cloud-services-team, 10wikitech.wikimedia.org, 10Observability-Alerting, 13Patch-For-Review: Move wikitech-static monitoring off Icinga - https://phabricator.wikimedia.org/T362397#12099654 (10fgiunchedi) And another one for your eyes @Andrew https://gitlab.wikimedia.org/repos/sre/wikitech-static-docker/-... [12:20:50] 10Toolforge, 06tools-platform-team: Regression: Images built with the build service images do no longer contain/configure locales specified in .locales - https://phabricator.wikimedia.org/T428230#12099670 (10dcaro) 05In progress→03Resolved [12:22:15] 10Cloud-VPS (Project-requests), 06Wikidata Platform Team, 13Patch-For-Review: Request creation of wdp-gitlab-runner VPS project - https://phabricator.wikimedia.org/T431531#12099678 (10dcaro) [12:22:16] 10Cloud-VPS (Project-requests), 06Wikidata Platform Team, 13Patch-For-Review: Request creation of wdp-gitlab-runner VPS project - https://phabricator.wikimedia.org/T431531#12099679 (10dcaro) 05Open→03In progress [12:23:00] (03approved) 10dcaro: projects: added project wdp-gitlab-runner [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/320 (https://phabricator.wikimedia.org/T431531) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [12:23:18] (03merge) 10dcaro: projects: added project wdp-gitlab-runner [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/320 (https://phabricator.wikimedia.org/T431531) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [12:24:56] !log dcaro@cloudcumin1001 wdp-gitlab-runner END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project wdp-gitlab-runner in eqiad1 (T431531) [12:24:58] dcaro@cloudcumin1001: Unknown project "wdp-gitlab-runner" [12:24:58] T431531: Request creation of wdp-gitlab-runner VPS project - https://phabricator.wikimedia.org/T431531 [12:28:14] 10Toolforge, 06tools-platform-team: Regression: Images built with the build service images do no longer contain/configure locales specified in .locales - https://phabricator.wikimedia.org/T428230#12099709 (10dcaro) 05Resolved→03Open [12:31:37] 10Cloud-VPS (Project-requests), 06Wikidata Platform Team: Request creation of wdp-gitlab-runner VPS project - https://phabricator.wikimedia.org/T431531#12099719 (10dcaro) Done, enjoy :) [12:31:43] 10Cloud-VPS (Project-requests), 06Wikidata Platform Team: Request creation of wdp-gitlab-runner VPS project - https://phabricator.wikimedia.org/T431531#12099721 (10dcaro) 05In progress→03Resolved [12:32:26] 06cloud-services-team, 10Striker, 06tools-platform-team: [striker] Unable to create phabricator project for tool odivoir – HTTP 500 Internal Server Error on toolsadmin - https://phabricator.wikimedia.org/T429857#12099727 (10dcaro) 05Duplicate→03Resolved [12:37:11] 10Cloud-VPS (Project-requests), 06Wikidata Platform Team: Request creation of wdp-gitlab-runner VPS project - https://phabricator.wikimedia.org/T431531#12099741 (10gmodena) Terrific! Thank you so much for picking this task up so quickly. Now let's build :) [12:39:15] (03CR) 10Jsn.sherman: "I did a quick look at the client subnet distribution in the logs, which should be a strong signal of what symphony is getting. Lots of emp" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T430888) (owner: 10MusikAnimal) [12:48:59] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: Add pre-flight validity checks to the `webservice-runner` script in webservice images - https://phabricator.wikimedia.org/T431563#12099846 (10Raymond_Ndibe) [12:56:16] (03update) 10raymond-ndibe: [values.yaml] add support for webservice-defaults [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/23 (https://phabricator.wikimedia.org/T431403) [12:56:20] (03update) 10raymond-ndibe: [values.yaml] add support for webservice-defaults [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/23 (https://phabricator.wikimedia.org/T431403) [13:08:38] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'cloudinfra-idp' [13:08:44] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Upgrade IDP server in cloudinfra project off of Bullseye - https://phabricator.wikimedia.org/T402006#12099969 (10taavi) a:03taavi [13:11:50] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Upgrade IDP server in cloudinfra project off of Bullseye - https://phabricator.wikimedia.org/T402006#12099990 (10taavi) [13:11:52] 06cloud-services-team, 10Cloud-VPS: [cloudinfra] Upgrade cloudinfra-idp-* to bookworm - https://phabricator.wikimedia.org/T373840#12099994 (10taavi) →14Duplicate dup:03T402006 [13:12:19] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'cloudinfra-idp' [13:13:59] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.refresh_puppet_certs on cloudinfra-idp-2.cloudinfra.eqiad1.wikimedia.cloud [13:16:27] !log taavi@cloudcumin1001 cloudinfra END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on cloudinfra-idp-2.cloudinfra.eqiad1.wikimedia.cloud [13:17:39] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.refresh_puppet_certs on cloudinfra-idp-2.cloudinfra.eqiad1.wikimedia.cloud [13:22:07] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on cloudinfra-idp-2.cloudinfra.eqiad1.wikimedia.cloud [13:24:28] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.remove_instance for instance cloudinfra-idp-2 [13:25:18] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance cloudinfra-idp-2 [13:25:47] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'cloudinfra-idp' [13:31:59] (03PS4) 10Jsn.sherman: RateLimitSubscriber: anon db traffic + rate limits [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T430888) (owner: 10MusikAnimal) [13:35:47] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'cloudinfra-idp' [13:43:53] 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 07Essential-Work, 06Moderator-Tools-Team (Kanban): twl: Replace deprecated Bullseye VMs in Cloud VPS - https://phabricator.wikimedia.org/T402054#12100129 (10Samwalton9-WMF) a:03jsn.sherman [13:49:36] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12100150 (10ops-monitoring-bot) Rebooting db-test1001.eqiad.wmnet [13:53:17] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team: Upgrade IDP server in cloudinfra project off of Bullseye - https://phabricator.wikimedia.org/T402006#12100175 (10taavi) Hi @MoritzMuehlenhoff @SLyngshede-WMF, I've installed CAS 7 (more specifically, `7.3.7.3+wmf13u1`) on a Trixie host (`c... [13:53:31] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team: Upgrade IDP server in cloudinfra project off of Bullseye - https://phabricator.wikimedia.org/T402006#12100179 (10taavi) [13:58:33] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12100245 (10ops-monitoring-bot) Rebooting db-test1001.eqiad.wmnet [14:02:34] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12100266 (10ops-monitoring-bot) Reboot of db-test1001.eqiad.wmnet completed [14:03:17] (03update) 10raymond-ndibe: [values.yaml] add support for webservice-defaults [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/23 (https://phabricator.wikimedia.org/T431403) [14:07:30] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12100299 (10ops-monitoring-bot) Rebooting db-test1002.eqiad.wmnet [14:08:49] (03update) 10raymond-ndibe: k8s/jobs.py: set PORT env whenever a job exposes a port [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/341 (https://phabricator.wikimedia.org/T431274) [14:10:01] (03CR) 10Jsn.sherman: "Claude noticed that the language selection helper was generating session db lookup traffic too. We were skipping those queries on the init" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T430888) (owner: 10MusikAnimal) [14:10:08] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12100327 (10ops-monitoring-bot) Reboot of db-test1002.eqiad.wmnet completed [14:10:14] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12100329 (10ops-monitoring-bot) Rebooting db-test1003.eqiad.wmnet [14:11:36] FIRING: NfsAlmostFull: The NFS drive is over 85% capacity (currently 92.36%) at host paws-nfs-2 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull [14:13:26] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12100365 (10ops-monitoring-bot) Reboot of db-test1003.eqiad.wmnet completed [14:13:33] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-cli, jobs-api] support --webservice option - https://phabricator.wikimedia.org/T428898#12100366 (10aputhin) p:05Triage→03High [14:16:14] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team: Upgrade IDP server in cloudinfra project off of Bullseye - https://phabricator.wikimedia.org/T402006#12100377 (10MoritzMuehlenhoff) >>! In T402006#12100175, @taavi wrote: > Hi @MoritzMuehlenhoff @SLyngshede-WMF, I've installed CAS 7 (more... [14:18:45] 06cloud-services-team, 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [jobs-api] Refactor before webservice support - https://phabricator.wikimedia.org/T359804#12100398 (10aputhin) a:05Raymond_Ndibe→03dcaro [14:19:18] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12100405 (10ops-monitoring-bot) Rebooting db-test1002.eqiad.wmnet [14:21:37] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-api] support exposing continuous jobs to the internet - https://phabricator.wikimedia.org/T423408#12100408 (10aputhin) 05In progress→03Stalled [14:22:07] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-cli] support publishing continuous job to the internet - https://phabricator.wikimedia.org/T423410#12100410 (10aputhin) 05In progress→03Stalled [14:22:22] 10Tool-inteGraality: Support reference columns in property dashboards - https://phabricator.wikimedia.org/T428637#12100414 (10JeanFred) thinking more, I’m leaning towards the syntax: * `P123/S*`: any reference * `P123/S+`: any good reference * `P123/S456`: specific reference property This is somewhat regex insp... [14:29:48] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: Add pre-flight validity checks to the `webservice-runner` script in webservice images - https://phabricator.wikimedia.org/T431563#12100448 (10aputhin) p:05Triage→03Medium [14:33:01] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12100484 (10ops-monitoring-bot) Rebooting db-test1002.eqiad.wmnet [14:33:29] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-cli, jobs-api] support --webservice option - https://phabricator.wikimedia.org/T428898#12100487 (10Raymond_Ndibe) [14:33:39] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Make webservice-cli use jobs-api - https://phabricator.wikimedia.org/T431421#12100495 (10aputhin) p:05Triage→03Medium [14:34:21] 10Toolforge (Push-to-Deploy), 06tools-platform-team: jobs-api, jobs-cli: support --webservice option in toolforge jobs run - https://phabricator.wikimedia.org/T431410#12100499 (10Raymond_Ndibe) →14Duplicate dup:03T428898 [14:34:22] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-cli, jobs-api] support --webservice option - https://phabricator.wikimedia.org/T428898#12100501 (10Raymond_Ndibe) [14:38:23] (03PS5) 10Jsn.sherman: RateLimitSubscriber: anon db traffic + rate limits [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T430888) (owner: 10MusikAnimal) [14:38:32] FIRING: InstanceDown: Project gitlab-runners instance runner-1035 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:38:41] 10Toolforge (Push-to-Deploy), 06tools-platform-team: webservice-cli: detect HTTPRoutes created by the jobs framework - https://phabricator.wikimedia.org/T431417#12100515 (10aputhin) [14:38:45] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Epic, 13Patch-For-Review: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#12100516 (10aputhin) [14:38:47] 10Tool-inteGraality: Support reference columns in property dashboards - https://phabricator.wikimedia.org/T428637#12100517 (10JeanFred) [14:39:03] 10Toolforge (Push-to-Deploy), 06tools-platform-team: webservice-cli: detect HTTPRoutes created by the jobs framework - https://phabricator.wikimedia.org/T431417#12100518 (10aputhin) p:05Triage→03High [14:39:33] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: nova: remove the network-ovs aggregate - https://phabricator.wikimedia.org/T431584 (10Andrew) 03NEW [14:39:50] 10Toolforge (Push-to-Deploy), 06tools-platform-team: image-config: add webservice-defaults to standard images - https://phabricator.wikimedia.org/T431403#12100537 (10aputhin) p:05Triage→03High [14:40:25] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: jobs-api: resolve some env variable differences between jobs-api and webservice-cli - https://phabricator.wikimedia.org/T431274#12100542 (10aputhin) p:05Triage→03High [14:41:07] (03CR) 10Andrew Bogott: [C:03+1] "this seems a lot better!" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1307689 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [14:41:10] 10Tool-inteGraality: Support reference columns in property dashboards - https://phabricator.wikimedia.org/T428637#12100553 (10JeanFred) [14:42:27] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-cli] Create aliases for the subcommands - https://phabricator.wikimedia.org/T430988#12100555 (10dcaro) [14:42:34] (03CR) 10Andrew Bogott: [C:04-1] "Not a strong opinion, but I'm not 100% sure we will never want this in the future. Aggregates are still useful for some things (e.g. the m" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1307690 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [14:46:02] 10Tool-inteGraality: Support reference columns in property dashboards - https://phabricator.wikimedia.org/T428637#12100600 (10JeanFred) [14:46:24] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-cli] Create aliases for the subcommands - https://phabricator.wikimedia.org/T430988#12100602 (10aputhin) How many tools are using the existing commands? How big is the impact of deprecating `toolforge components [...]`? [14:47:28] 10Tool-inteGraality: Support reference columns in property dashboards - https://phabricator.wikimedia.org/T428637#12100603 (10JeanFred) [14:48:19] 10Tool-inteGraality: Support reference columns in property dashboards - https://phabricator.wikimedia.org/T428637#12100616 (10JeanFred) [14:55:37] 10Tool-inteGraality: Support reference columns in property dashboards - https://phabricator.wikimedia.org/T428637#12100664 (10JeanFred) [15:02:42] 10Tool-inteGraality: Support reference columns in property dashboards - https://phabricator.wikimedia.org/T428637#12100720 (10JeanFred) [15:03:16] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12100723 (10ops-monitoring-bot) Reboot of db-test1002.eqiad.wmnet completed [15:06:11] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12100735 (10ops-monitoring-bot) Reboot of db-test1002.eqiad.wmnet completed [15:06:50] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12100737 (10ops-monitoring-bot) Reboot of db-test1003.eqiad.wmnet completed [15:08:09] 06cloud-services-team, 10Striker, 06tools-platform-team: [striker] Unable to create phabricator project for tool odivoir – HTTP 500 Internal Server Error on toolsadmin - https://phabricator.wikimedia.org/T429857#12100739 (10JJMC89) →14Duplicate dup:03T430120 [15:08:12] 06cloud-services-team, 10Striker, 06tools-platform-team, 06Release-Engineering-Team (Doing 😎): [striker] Failing to create new phabricator projects - https://phabricator.wikimedia.org/T430120#12100740 (10JJMC89) [15:08:51] 10Tool-inteGraality: Support reference columns in property dashboards - https://phabricator.wikimedia.org/T428637#12100743 (10JeanFred) [15:18:49] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-cli] Create aliases for the subcommands - https://phabricator.wikimedia.org/T430988#12100768 (10dcaro) Very rough lower limit estimation of tools that ever used components: ` root@tools-nfs-3:/srv/tools/project# find . -maxdepth 2 -iname .bash_hi... [15:24:30] 10Cloud-VPS (Quota-requests): Quota increase request for project kafka-infrastructure - https://phabricator.wikimedia.org/T431590 (10elukey) 03NEW [15:25:26] 10Tool-inteGraality: Support reference columns in property dashboards - https://phabricator.wikimedia.org/T428637#12100800 (10JeanFred) [15:30:13] (03merge) 10gabina: Add batch POST endpoint for counting references across many revisions [toolforge-repos/reference-counter] - 10https://gitlab.wikimedia.org/toolforge-repos/reference-counter/-/merge_requests/1 (owner: 10ragesoss) [15:35:21] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12100841 (10CWilliams-WMF) Apologies for the noise :) Some of it came from testing out a patch on t... [15:48:23] (03approved) 10dcaro: runtimes/k8s: fix flaky test_get_one_off_job_status [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/338 (https://phabricator.wikimedia.org/T430550) (owner: 10tlepage) [15:49:14] 10Toolforge (Push-to-Deploy), 06tools-platform-team: webservice-cli: detect conflict with HTTPRoutes created by the jobs framework - https://phabricator.wikimedia.org/T431417#12100929 (10Raymond_Ndibe) [15:50:00] (03PS6) 10Jsn.sherman: RateLimitSubscriber: anon db traffic + rate limits [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T430888) (owner: 10MusikAnimal) [15:51:48] (03open) 10raymond-ndibe: kubernetes.py: detect conflict with httproutes created by jobs-api [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/114 (https://phabricator.wikimedia.org/T431417) [15:59:30] (03update) 10raymond-ndibe: cleanup by image name [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/165 (owner: 10dcaro) [15:59:30] (03approved) 10raymond-ndibe: cleanup by image name [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/165 (owner: 10dcaro) [16:00:21] (03open) 10r4356th: Make conversion of font tags using relative sizes work even when non-numeric values are used [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/19 (https://phabricator.wikimedia.org/T426389) [16:00:51] 10Tool-delintbot, 13Patch-For-Review: Make conversion of font tags using relative sizes work even when non-numeric values are used - https://phabricator.wikimedia.org/T426389#12100994 (10Redmin) [16:02:08] 10Tool-delintbot, 13Patch-For-Review: Make conversion of font tags using relative sizes work even when non-numeric values are used - https://phabricator.wikimedia.org/T426389#12101008 (10Redmin) p:05Triage→03High [16:07:15] (03update) 10r4356th: Make conversion of font tags using relative sizes work even when non-numeric values are used [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/19 (https://phabricator.wikimedia.org/T426389) [16:07:54] (03merge) 10r4356th: Make conversion of font tags using relative sizes work even when non-numeric values are used [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/19 (https://phabricator.wikimedia.org/T426389) [16:10:54] (03PS1) 10Btullis: Fix the dummy secrets for the new an-test hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1308679 (https://phabricator.wikimedia.org/T406746) [16:11:05] (03CR) 10Btullis: [V:03+2 C:03+2] Fix the dummy secrets for the new an-test hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1308679 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [16:11:39] 10Tool-delintbot, 13Patch-For-Review: Make conversion of font tags using relative sizes work even when non-numeric values are used - https://phabricator.wikimedia.org/T426389#12101028 (10Redmin) 05Open→03Resolved a:03Redmin The command that had failed just completed successfully! [16:14:35] (03update) 10renovatebot: Update dependency typescript to v7 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/2 [16:14:38] (03open) 10renovatebot: Update dependency typescript to v7 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/2 [16:15:36] (03update) 10renovatebot: Update dependency typescript to v7 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/38 [16:15:39] (03open) 10renovatebot: Update dependency typescript to v7 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/38 [16:16:37] (03update) 10renovatebot: Update dependency typescript to v7 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/20 [16:16:40] (03open) 10renovatebot: Update dependency typescript to v7 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/20 [16:18:58] (03CR) 10MusikAnimal: [C:03+2] UserRights: fix removal of redundant auto-removals [labs/xtools] - 10https://gerrit.wikimedia.org/r/1308215 (owner: 10Alien4444) [16:19:41] (03Merged) 10jenkins-bot: UserRights: fix removal of redundant auto-removals [labs/xtools] - 10https://gerrit.wikimedia.org/r/1308215 (owner: 10Alien4444) [16:24:35] (03PS1) 10MusikAnimal: Authorship: add kowiki, mtwiki and zhwiki [labs/xtools] - 10https://gerrit.wikimedia.org/r/1308681 (https://phabricator.wikimedia.org/T372340) [16:28:52] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'syslog-server-audit' [16:29:00] !log taavi@cloudcumin1001 cloudinfra END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'syslog-server-audit' [16:31:11] (03CR) 10Jsn.sherman: "Okay, the switcher no longer has `uselang` in outbound links. The Apache mismatch check should be safe to turn back on. Legit users have `" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T430888) (owner: 10MusikAnimal) [16:31:31] (03CR) 10MusikAnimal: [C:03+2] Authorship: add kowiki, mtwiki and zhwiki [labs/xtools] - 10https://gerrit.wikimedia.org/r/1308681 (https://phabricator.wikimedia.org/T372340) (owner: 10MusikAnimal) [16:32:13] (03Merged) 10jenkins-bot: Authorship: add kowiki, mtwiki and zhwiki [labs/xtools] - 10https://gerrit.wikimedia.org/r/1308681 (https://phabricator.wikimedia.org/T372340) (owner: 10MusikAnimal) [16:33:48] (03PS1) 10Majavah: vps: create_instance: Add flag to specify number of first server [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1308683 [16:34:39] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'syslog-server-audit' [16:38:40] (03update) 10dcaro: k8s: use job_type for label generating functions [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/328 [16:40:37] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'syslog-server-audit' [16:45:11] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation), 07IPv6: Refresh cloudinfra central syslog audit servers to run on Trixie and enable IPv6 - https://phabricator.wikimedia.org/T402007#12101145 (10taavi) a:03taavi [16:46:18] (03update) 10taavi: eqiad1: cloudinfra: Manage syslog audit svc records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/321 [16:46:19] (03update) 10taavi: Remove import blocks [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/322 [16:46:20] (03update) 10taavi: eqiad1: cloudinfra: Move syslogaudit1.svc to new syslog server [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/323 (https://phabricator.wikimedia.org/T402007) [16:46:33] (03open) 10taavi: Remove import blocks [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/322 [16:46:34] (03open) 10taavi: eqiad1: cloudinfra: Manage syslog audit svc records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/321 [16:46:35] (03open) 10taavi: eqiad1: cloudinfra: Move syslogaudit1.svc to new syslog server [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/323 (https://phabricator.wikimedia.org/T402007) [16:49:18] (03update) 10taavi: eqiad1: cloudinfra: Move syslogaudit1.svc to new syslog server [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/323 (https://phabricator.wikimedia.org/T402007) [16:49:19] (03update) 10taavi: eqiad1: cloudinfra: Manage syslog audit svc records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/321 [16:49:26] (03update) 10taavi: Remove import blocks [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/322 [16:49:33] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/321 [16:49:52] !log taavi@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/321 [16:53:02] (03update) 10taavi: eqiad1: cloudinfra: Move syslogaudit1.svc to new syslog server [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/323 (https://phabricator.wikimedia.org/T402007) [16:53:03] (03update) 10taavi: eqiad1: cloudinfra: Manage syslog audit svc records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/321 [16:53:06] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/321 [16:53:06] (03update) 10taavi: Remove import blocks [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/322 [16:53:27] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/321 [16:54:04] (03open) 10andrew: Add 'magnum' service project and a couple default templates in codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/324 (https://phabricator.wikimedia.org/T393782 https://phabricator.wikimedia.org/T429164) [16:54:46] (03update) 10taavi: eqiad1: cloudinfra: Move syslogaudit1.svc to new syslog server [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/323 (https://phabricator.wikimedia.org/T402007) [16:54:46] (03update) 10taavi: eqiad1: cloudinfra: Manage syslog audit svc records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/321 [16:54:49] (03update) 10taavi: Remove import blocks [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/322 [16:55:24] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/321 [16:55:46] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/321 [16:56:07] (03merge) 10taavi: eqiad1: cloudinfra: Manage syslog audit svc records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/321 [16:56:16] (03update) 10taavi: Remove import blocks [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/322 [16:56:30] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [16:56:42] (03update) 10andrew: Add 'magnum' service project and a couple default templates in codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/324 (https://phabricator.wikimedia.org/T393782) [16:57:11] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [16:58:07] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/322 [16:58:25] (03CR) 10Jsn.sherman: "As it currently stands, this patch takes us from the language selector providing 128 language links that fire full page renders down to 0 " [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T430888) (owner: 10MusikAnimal) [16:58:28] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/322 [16:58:51] (03merge) 10taavi: Remove import blocks [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/322 [16:58:56] (03update) 10taavi: eqiad1: cloudinfra: Move syslogaudit1.svc to new syslog server [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/323 (https://phabricator.wikimedia.org/T402007) [16:59:27] (03close) 10andrew: Add 'magnum' service project and a couple default templates in codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/324 (https://phabricator.wikimedia.org/T393782) [17:00:21] (03open) 10andrew: Add 'magnum' service project and default templates in codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/325 (https://phabricator.wikimedia.org/T393782) [17:00:44] (03update) 10dcaro: runtime: remove diff_with_running_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/336 [17:00:49] (03update) 10dcaro: runtime: remove diff_with_running_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/336 [17:01:54] (03update) 10andrew: Add 'magnum' service project and default templates in codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/325 (https://phabricator.wikimedia.org/T393782) [17:03:11] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/323 [17:03:33] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/323 [17:04:36] (03merge) 10taavi: eqiad1: cloudinfra: Move syslogaudit1.svc to new syslog server [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/323 (https://phabricator.wikimedia.org/T402007) [17:04:44] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [17:05:18] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [17:06:32] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: Ensure cloudvirt capacity is more evenly spread out among racks - https://phabricator.wikimedia.org/T424658#12101239 (10VRiley-WMF) @fgiunchedi I was curious to know if there is an estimated date for these moves? Will w... [17:09:26] (03update) 10andrew: Add 'magnum' service project and default templates in codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/325 (https://phabricator.wikimedia.org/T393782) [17:10:21] (03update) 10dcaro: runtime: remove diff_with_running_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/336 [17:10:44] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'syslog-server-audit' [17:10:47] (03update) 10dcaro: runtime: remove diff_with_running_job [repos/cloud/toolforge/jobs-api] (use_job_kind_for_labels) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/336 [17:16:48] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'syslog-server-audit' [17:17:46] PROBLEM - SSH on cloudvirt1071 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [17:19:44] PROBLEM - Host cloudvirt1071 is DOWN: PING CRITICAL - Packet loss = 100% [17:20:15] (03update) 10taavi: eqiad1: cloudinfra: Move syslogaudit2.svc to new syslog server [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/326 (https://phabricator.wikimedia.org/T402007) [17:20:18] (03open) 10taavi: eqiad1: cloudinfra: Move syslogaudit2.svc to new syslog server [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/326 (https://phabricator.wikimedia.org/T402007) [17:20:21] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/326 [17:20:43] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/326 [17:20:50] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1071 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [17:21:00] (03merge) 10taavi: eqiad1: cloudinfra: Move syslogaudit2.svc to new syslog server [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/326 (https://phabricator.wikimedia.org/T402007) [17:21:02] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [17:21:28] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [17:21:36] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance syslog-server-audit-4 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [17:22:10] FIRING: CloudVirtDown: Cloudvirt node cloudvirt1071 is down. #page - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CloudVirtDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1071 - https://alerts.wikimedia.org/?q=alertname%3DCloudVirtDown [17:24:36] FIRING: InstanceDown: Project cloudinfra instance syslog-server-audit02 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:25:36] RECOVERY - SSH on cloudvirt1071 is OK: SSH OK - OpenSSH_10.0p2 Debian-7+deb13u4 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [17:25:38] RECOVERY - Host cloudvirt1071 is UP: PING OK - Packet loss = 0%, RTA = 0.33 ms [17:28:29] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#12101343 (10Ottomata) [17:29:36] RESOLVED: InstanceDown: Project cloudinfra instance syslog-server-audit02 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:30:50] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1071 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [17:36:16] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'cloudinfra-db' [17:36:36] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance syslog-server-audit-4 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [17:37:37] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#12101382 (10Ottomata) > webrequest.dumps vs dumps.webrequest: I prefer the latter. I could be convin... [17:39:40] RESOLVED: CloudVirtDown: Cloudvirt node cloudvirt1071 is down. #page - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CloudVirtDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1071 - https://alerts.wikimedia.org/?q=alertname%3DCloudVirtDown [17:41:18] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'cloudinfra-db' [17:42:18] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.remove_instance for instance cloudinfra-db-3 [17:43:08] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance cloudinfra-db-3 [17:43:12] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#12101415 (10Ottomata) > webrequest.dumps vs dumps.webrequest A related bikeshed for a 'webrequest' st... [17:43:47] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'cloudinfra-db' [17:48:36] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'cloudinfra-db' [17:49:19] 06cloud-services-team, 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: New thermal paste for cloudvirt1071.eqiad.wmnet - https://phabricator.wikimedia.org/T431429#12101448 (10VRiley-WMF) I am currently updating the firmware for bios and iDRAC. My next step will be to apply the thermal paste. [17:49:27] (03CR) 10MusikAnimal: "Yes! Go for it. You can freely use xtools-dev for testing." [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T430888) (owner: 10MusikAnimal) [17:51:50] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation), 07IPv6: Refresh cloudinfra central syslog audit servers to run on Trixie and enable IPv6 - https://phabricator.wikimedia.org/T402007#12101464 (10taavi) This is done except I need to remove the old VMs in a couple of days. [17:52:01] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS (Debian Bullseye Deprecation): Upgrade cloudinfra database hosts off of Bullseye - https://phabricator.wikimedia.org/T402005#12101465 (10taavi) a:03taavi [17:53:29] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS (Debian Bullseye Deprecation): Upgrade cloudinfra database hosts off of Bullseye - https://phabricator.wikimedia.org/T402005#12101466 (10taavi) On `cloudinfra-db04`: ` MariaDB [(none)]> stop slave; Query OK, 0 rows affected (0.029 sec) MariaDB [(none)]> s... [17:59:36] FIRING: InstanceDown: Project cloudinfra instance cloudinfra-db04 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:01:22] 10Tool-wiki-gender-stats: [VERSION] 0.3.0 - https://phabricator.wikimedia.org/T427979#12101482 (10Danya) [18:01:23] 10Tool-wiki-gender-stats: Header items overflow on mobile screens - https://phabricator.wikimedia.org/T422344#12101481 (10Danya) [18:04:36] RESOLVED: [2x] InstanceDown: Project cloudinfra instance cloudinfra-db-3 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:09:20] 06cloud-services-team, 10Striker: User should be told name of existing Developer account when SUL is already in use - https://phabricator.wikimedia.org/T294767#12101526 (10bd808) [18:09:23] 06cloud-services-team, 10Striker, 06tools-platform-team: Cannot join Toolforge: Striker throws IntegrityError while saving user - https://phabricator.wikimedia.org/T429759#12101522 (10bd808) 05Open→03Resolved a:03bd808 Per T429759#12098241 I took these actions to unblock things: * Blocked the unuse... [18:15:35] 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 07Essential-Work, 06Moderator-Tools-Team (Kanban): wikilink: Replace deprecated Bullseye VM in Cloud VPS - https://phabricator.wikimedia.org/T402055#12101563 (10jsn.sherman) 05Stalled→03In progress [18:32:29] (03open) 10andrew: Change testlabs fip quota in codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/327 [18:34:25] 06cloud-services-team, 10Toolforge: Upgrade Toolforge Redis cluster to Bookworm or newer - https://phabricator.wikimedia.org/T360704#12101644 (10taavi) a:03taavi [18:36:25] (03open) 10taavi: Add and import initial Redis service module [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/106 (https://phabricator.wikimedia.org/T360704) [18:37:31] (03update) 10taavi: Add and import initial Redis service module [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/106 (https://phabricator.wikimedia.org/T360704) [18:39:30] (03update) 10taavi: Add and import initial Redis service module [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/106 (https://phabricator.wikimedia.org/T360704) [18:40:48] 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 07Essential-Work, 06Moderator-Tools-Team (Kanban): wikilink: Replace deprecated Bullseye VM in Cloud VPS - https://phabricator.wikimedia.org/T402055#12101662 (10jsn.sherman) 05In progress→03Open [18:44:45] (03update) 10andrew: Change testlabs fip quota in codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/327 [18:52:59] PROBLEM - Host cloudvirt1071 is DOWN: PING CRITICAL - Packet loss = 100% [18:56:50] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1071 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [18:57:10] FIRING: CloudVirtDown: Cloudvirt node cloudvirt1071 is down. #page - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CloudVirtDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1071 - https://alerts.wikimedia.org/?q=alertname%3DCloudVirtDown [19:06:30] (03close) 10andrew: Change testlabs fip quota in codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/327 [19:06:47] RECOVERY - Host cloudvirt1071 is UP: PING OK - Packet loss = 0%, RTA = 0.30 ms [19:06:49] (03update) 10andrew: Add 'magnum' service project and default templates in codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/325 (https://phabricator.wikimedia.org/T393782) [19:11:50] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1071 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [19:12:10] RESOLVED: CloudVirtDown: Cloudvirt node cloudvirt1071 is down. #page - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CloudVirtDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1071 - https://alerts.wikimedia.org/?q=alertname%3DCloudVirtDown [19:12:58] (03close) 10andrew: Add 'magnum' service project and default templates in codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/325 (https://phabricator.wikimedia.org/T393782) [19:13:30] (03open) 10andrew: Add 'magnum' service project in and default templates in codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/328 (https://phabricator.wikimedia.org/T393782) [19:14:22] 06cloud-services-team, 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: New thermal paste for cloudvirt1071.eqiad.wmnet - https://phabricator.wikimedia.org/T431429#12101802 (10VRiley-WMF) I have applied the thermal paste and the BIOS and iDRAC are up to date with firmware. @Andrew would you be able to look thi... [19:17:53] !log andrew@cloudcumin1001 magnum START - Cookbook wmcs.vps.create_project for project magnum in codfw1dev [19:18:09] (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project magnum [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/329 [19:18:50] !log andrew@cloudcumin1001 magnum END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project magnum in codfw1dev [19:19:43] !log andrew@cloudcumin1001 magnum START - Cookbook wmcs.vps.create_project for project magnum in codfw1dev [19:19:50] !log andrew@cloudcumin1001 magnum END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project magnum in codfw1dev [19:20:17] (03close) 10andrew: projects: added project magnum [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/329 (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [19:20:35] (03close) 10andrew: Add 'magnum' service project in and default templates in codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/328 (https://phabricator.wikimedia.org/T393782) [19:20:36] !log andrew@cloudcumin1001 magnum START - Cookbook wmcs.vps.create_project for project magnum in codfw1dev [19:20:48] (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project magnum [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/330 [19:21:47] (03merge) 10andrew: projects: added project magnum [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/330 (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [19:22:59] !log andrew@cloudcumin1001 magnum END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project magnum in codfw1dev [19:23:19] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [19:23:52] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [19:25:33] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [19:25:51] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [19:38:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:58:50] 06cloud-services-team: Update all cloud-vps Bullseye NFS servers - https://phabricator.wikimedia.org/T429793#12101969 (10jsn.sherman) Today for twl, I: - removed the nfs heira config from client VMs - unmounted and commented fstab entry on same - shutdown nfs server - deleted server and volume - removed the nfs... [20:05:50] 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 07Essential-Work, 06Moderator-Tools-Team (Kanban): twl: Replace deprecated Bullseye VMs in Cloud VPS - https://phabricator.wikimedia.org/T402054#12101997 (10jsn.sherman) @taavi I did a dry run on replacement using our new rootless docker s... [20:10:32] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [20:15:03] (03open) 10andrew: Add a couple of default global magnum templates. [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/331 (https://phabricator.wikimedia.org/T429164) [20:25:32] RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [20:31:31] 06cloud-services-team, 10Openstack-Magnum, 13Patch-For-Review: Create two or three shared magnum templates - https://phabricator.wikimedia.org/T429164#12102077 (10Andrew) I have a start on this https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/331 But tofu-infra seems to not un... [20:40:03] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: Do something with cloudcontrol100[8-10]-dev - https://phabricator.wikimedia.org/T406630#12102104 (10Andrew) [20:40:07] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: Do something with cloudcontrol100[8-10]-dev - https://phabricator.wikimedia.org/T406630#12102106 (10Andrew) 05Open→03Resolved [20:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks