[00:11:48] <wikibugs>	 (03open) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505)
[00:14:08] <wikibugs>	 (03update) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505)
[00:14:17] <wikibugs>	 (03update) 10bd808: puppet_prefix: Generate YAML with `yamlencode` equivalent [repos/cloud/cloud-vps/terraform-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/terraform-cloudvps/-/merge_requests/10 (https://phabricator.wikimedia.org/T397994 https://phabricator.wikimedia.org/T398643)
[00:15:40] <wikibugs>	 (03update) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505)
[00:16:55] <wikibugs>	 (03update) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505)
[00:20:23] <wikibugs>	 (03update) 10bd808: puppet_prefix: Generate YAML with `yamlencode` equivalent [repos/cloud/cloud-vps/terraform-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/terraform-cloudvps/-/merge_requests/10 (https://phabricator.wikimedia.org/T397994 https://phabricator.wikimedia.org/T398643)
[00:23:57] <wikibugs>	 (03update) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505)
[00:28:00] <wikibugs>	 (03update) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505)
[00:28:23] <wikibugs>	 (03update) 10bd808: puppet_prefix: Generate YAML with `yamlencode` equivalent [repos/cloud/cloud-vps/terraform-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/terraform-cloudvps/-/merge_requests/10 (https://phabricator.wikimedia.org/T397994 https://phabricator.wikimedia.org/T398643)
[00:41:25] <wikibugs>	 (03update) 10bd808: puppet_prefix: Generate YAML with `yamlencode` equivalent [repos/cloud/cloud-vps/terraform-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/terraform-cloudvps/-/merge_requests/10 (https://phabricator.wikimedia.org/T397994 https://phabricator.wikimedia.org/T398643)
[00:43:55] <wikibugs>	 (03update) 10bd808: puppet_prefix: Generate YAML with `yamlencode` equivalent [repos/cloud/cloud-vps/terraform-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/terraform-cloudvps/-/merge_requests/10 (https://phabricator.wikimedia.org/T397994 https://phabricator.wikimedia.org/T398643)
[01:03:01] <wikibugs>	 (03update) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505)
[01:11:17] <wikibugs>	 (03update) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505)
[01:23:19] <jinxer-wm>	 FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[01:52:45] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
[01:52:47] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy
[02:03:19] <jinxer-wm>	 RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[02:35:19] <jinxer-wm>	 FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[02:40:19] <jinxer-wm>	 RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[02:51:19] <jinxer-wm>	 FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[02:57:56] <wikibugs>	 (03update) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505)
[03:31:19] <jinxer-wm>	 RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[03:31:43] <wikibugs>	 (03update) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505)
[04:31:10] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
[04:31:13] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy
[06:52:47] <wikibugs>	 (03PS1) 10Stevemunene: druid-public: Add dummy keytabs for new hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1182691 (https://phabricator.wikimedia.org/T397441)
[07:02:57] <wikibugs>	 (03PS1) 10Muehlenhoff: Remove obsolete stub secrets [labs/private] - 10https://gerrit.wikimedia.org/r/1182695 (https://phabricator.wikimedia.org/T360636)
[07:03:39] <wikibugs>	 (03update) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505)
[07:06:24] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
[07:06:27] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy
[07:17:22] <wikibugs>	 (03update) 10taavi: puppet_prefix: Generate YAML with `yamlencode` equivalent [repos/cloud/cloud-vps/terraform-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/terraform-cloudvps/-/merge_requests/10 (https://phabricator.wikimedia.org/T397994 https://phabricator.wikimedia.org/T398643) (owner: 10bd808)
[07:17:26] <wikibugs>	 (03update) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505)
[07:21:46] <wikibugs>	 (03update) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505)
[07:23:06] <wikibugs>	 (03approved) 10taavi: puppet_prefix: Generate YAML with `yamlencode` equivalent [repos/cloud/cloud-vps/terraform-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/terraform-cloudvps/-/merge_requests/10 (https://phabricator.wikimedia.org/T397994 https://phabricator.wikimedia.org/T398643) (owner: 10bd808)
[07:25:30] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Phase out DSA keys for SSH access (ssh-dss) - https://phabricator.wikimedia.org/T177371#11127487 (10MoritzMuehlenhoff) Support for DSA was removed in OpenSSH 10, which is the version in Debian Trixie: https://www...
[07:26:05] <wikibugs>	 (03update) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505)
[07:34:26] <wikibugs>	 (03update) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505)
[07:35:04] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Phase out DSA keys for SSH access (ssh-dss) - https://phabricator.wikimedia.org/T177371#11127495 (10taavi) Grepping the auth logs seems to think those are no longer in use: `lang=shell-session taavi@tools-bastion...
[07:38:15] <wikibugs>	 (03update) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505)
[07:41:33] <wikibugs>	 (03update) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505)
[07:51:08] <wikibugs>	 06cloud-services-team, 10Toolforge: Loki usage - https://phabricator.wikimedia.org/T401151#11127530 (10taavi) In general we have rate limiting in place so a single tool shouldn't be able to cause problems for other tools. That being said, while direct comparisons between Loki and NFS storage sizes are difficul...
[08:12:30] <wikibugs>	 06cloud-services-team, 10Toolforge: "toolforge-jobs list" error - "TjfCliError: Unable to find image in the supported list or harbor" - https://phabricator.wikimedia.org/T402724#11127592 (10dcaro) A quick check works for me using the same image: ` tools.sample-static-buildpack-app@tools-bastion-13:~$ cat jobs....
[08:13:56] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Catalyst: Quota increase request for catalyst-dev - https://phabricator.wikimedia.org/T402521#11127598 (10jnuche) 05Resolved→03Open Thank you for your support with this!  Unfortunately the quotas were not increased by 32 CPUs, 64GB mem and 670GB disk...
[08:16:01] <wikibugs>	 (03update) 10dcaro: build: allow re-using builds across components [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/118 (https://phabricator.wikimedia.org/T401893) (owner: 10damian)
[08:20:01] <wm-bot2>	 !log dcaro@acme catalyst-dev START - Cookbook wmcs.openstack.quota_increase (T402521)
[08:20:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Catalyst-dev/SAL
[08:20:06] <stashbot>	 T402521: Quota increase request for catalyst-dev - https://phabricator.wikimedia.org/T402521
[08:20:09] <wm-bot2>	 !log dcaro@acme catalyst-dev END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) (T402521)
[08:20:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Catalyst-dev/SAL
[08:20:13] <wm-bot2>	 !log dcaro@acme catalyst-dev START - Cookbook wmcs.openstack.quota_increase (T402521)
[08:20:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Catalyst-dev/SAL
[08:20:19] <wm-bot2>	 !log dcaro@acme catalyst-dev END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T402521)
[08:20:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Catalyst-dev/SAL
[08:22:49] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Catalyst: Quota increase request for catalyst-dev - https://phabricator.wikimedia.org/T402521#11127620 (10dcaro) 05Open→03Resolved Oops, xd, there you go: ` root@cloudcontrol1006:~# sudo wmcs-openstack quota show catalyst-dev | grep ' \(cores\|gi...
[08:23:23] <wikibugs>	 (03CR) 10Brouberol: [C:03+1] druid-public: Add dummy keytabs for new hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1182691 (https://phabricator.wikimedia.org/T397441) (owner: 10Stevemunene)
[08:25:24] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation), 10tofu.wmcloud.org: Replace tf-registry-2.terraform with new Trixie instance in tofu project - https://phabricator.wikimedia.org/T401814#11127631 (10taavi)
[08:26:50] <wikibugs>	 14VPS-project-icinga2: Add icinga2 support for puppetdb - https://phabricator.wikimedia.org/T183879#11127635 (10taavi) 05Open→03Invalid Marking #vps-project-icinga2 tasks as invalid and archiving the Phabricator project as the Cloud VPS project does not exist anymore.
[08:26:53] <wikibugs>	 14VPS-project-icinga2: Setup CI for Icinga2 Repo - https://phabricator.wikimedia.org/T180267#11127638 (10taavi) 05Open→03Invalid Marking #vps-project-icinga2 tasks as invalid and archiving the Phabricator project as the Cloud VPS project does not exist anymore.
[08:27:43] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Catalyst: Quota increase request for catalyst-dev - https://phabricator.wikimedia.org/T402521#11127641 (10jnuche) Thanks a mil :)
[08:29:58] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Phase out DSA keys for SSH access (ssh-dss) - https://phabricator.wikimedia.org/T177371#11127650 (10MoritzMuehlenhoff) >>! In T177371#11127495, @taavi wrote: > Grepping the auth logs seems to think those are no l...
[08:32:05] <wikibugs>	 (03approved) 10dcaro: build: allow re-using builds across components [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/118 (https://phabricator.wikimedia.org/T401893) (owner: 10damian)
[08:32:17] <wikibugs>	 (03merge) 10dcaro: build: allow re-using builds across components [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/118 (https://phabricator.wikimedia.org/T401893) (owner: 10damian)
[08:34:53] <wikibugs>	 06cloud-services-team, 10Toolforge: Establish an internal system or a recommended external system for monitoring user-created Toolforge web services - https://phabricator.wikimedia.org/T53434#11127662 (10taavi)
[08:35:15] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.153-20250828083225-788c69d6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/935 (https://phabricator.wikimedia.org/T401893)
[08:35:15] <wikibugs>	 06cloud-services-team, 10Striker: Preparation for api for community-labs-monitoring - https://phabricator.wikimedia.org/T157847#11127664 (10taavi) 05Open→03Invalid Marking #community-labs-monitoring tasks as invalid since the Cloud VPS project was deleted in https://wikitech.wikimedia.org/wiki/News/202...
[08:35:57] <wikibugs>	 06cloud-services-team, 10Striker: Preparation for api for community-labs-monitoring - https://phabricator.wikimedia.org/T157847#11127670 (10taavi)
[08:35:58] <wikibugs>	 06cloud-services-team, 10Toolforge: Establish an internal system or a recommended external system for monitoring user-created Toolforge web services - https://phabricator.wikimedia.org/T53434#11127671 (10taavi)
[08:37:21] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api
[08:42:08] <wikibugs>	 (03update) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505)
[08:42:20] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
[08:42:33] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api
[08:43:20] <wikibugs>	 (03open) 10taavi: tofu-provisioning: Cache plugin directory [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/69 (https://phabricator.wikimedia.org/T403028)
[08:43:24] <wikibugs>	 (03update) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505)
[08:43:25] <wikibugs>	 (03update) 10taavi: tofu-provisioning: Cache plugin directory [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/69 (https://phabricator.wikimedia.org/T403028)
[08:43:43] <wikibugs>	 (03update) 10taavi: tofu-provisioning: Cache plugin directory [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/69 (https://phabricator.wikimedia.org/T403028)
[08:43:48] <wikibugs>	 (03update) 10taavi: tofu-provisioning: Cache plugin directory [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/69 (https://phabricator.wikimedia.org/T403028)
[08:43:50] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: toolforge tofu-provisioning: Cache terraform-provider-openstack binary somewhere - https://phabricator.wikimedia.org/T403028#11127685 (10taavi) p:05Triage→03High a:03taavi
[08:48:04] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
[08:49:31] <wikibugs>	 (03update) 10taavi: tofu-provisioning: Cache plugin directory [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/69 (https://phabricator.wikimedia.org/T403028)
[08:52:20] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Upgrade cloudcumin hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403153 (10taavi) 03NEW
[08:52:50] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154 (10taavi) 03NEW
[08:55:11] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Phase out DSA keys for SSH access (ssh-dss) - https://phabricator.wikimedia.org/T177371#11127717 (10taavi) If we're just removing the host key, then I think it's fine to merge the patch at any time. The user keys...
[08:56:48] <wikibugs>	 06cloud-services-team, 10Toolforge, 07IPv6, 13Patch-For-Review: Upgrade Toolforge bastions to Trixie and enable IPv6 - https://phabricator.wikimedia.org/T392510#11127718 (10taavi)
[09:13:15] <wikibugs>	 (03approved) 10dcaro: components-api: bump to 0.0.153-20250828083225-788c69d6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/935 (https://phabricator.wikimedia.org/T401893) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[09:13:19] <wikibugs>	 (03merge) 10dcaro: components-api: bump to 0.0.153-20250828083225-788c69d6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/935 (https://phabricator.wikimedia.org/T401893) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[09:19:39] <wikibugs>	 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [components-api] Allow reusing another component build - https://phabricator.wikimedia.org/T401893#11127758 (10dcaro) Deployed and docs added https://wikitech.wikimedia.org/wiki/Help:Toolforge/Deploy_your_tool#Reusing_builds_between_components
[09:21:16] <wikibugs>	 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [components-api] Allow reusing another component build - https://phabricator.wikimedia.org/T401893#11127761 (10dcaro) a:03DamianZaremba
[09:21:19] <wikibugs>	 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [components-api] Allow reusing another component build - https://phabricator.wikimedia.org/T401893#11127763 (10dcaro) 05Open→03Resolved
[09:55:38] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
[09:55:40] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy
[10:00:12] <wikibugs>	 (03update) 10dcaro: [tool-config] handle unset and default arguments consistently [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/123 (https://phabricator.wikimedia.org/T402572) (owner: 10raymond-ndibe)
[10:03:11] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Phase out DSA keys for SSH access (ssh-dss) - https://phabricator.wikimedia.org/T177371#11127854 (10MoritzMuehlenhoff) Sounds good, I'll merge this later the day.
[10:05:28] <wikibugs>	 (03CR) 10David Caro: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1182632 (https://phabricator.wikimedia.org/T392510) (owner: 10Majavah)
[10:05:50] <wikibugs>	 (03CR) 10David Caro: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1182631 (owner: 10Majavah)
[10:06:11] <wikibugs>	 (03CR) 10Majavah: [C:03+2] build: Remove unsupported Python versions [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1182631 (owner: 10Majavah)
[10:06:14] <wikibugs>	 (03CR) 10Majavah: [C:03+2] inventory: Add new toolsbeta bastion [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1182632 (https://phabricator.wikimedia.org/T392510) (owner: 10Majavah)
[10:09:07] <wikibugs>	 06cloud-services-team, 10Toolforge: Update Toolforge Tcl image to a supported Debian release - https://phabricator.wikimedia.org/T400256#11127861 (10taavi) a:03taavi
[10:09:18] <wikibugs>	 06cloud-services-team, 10Toolforge, 07Documentation, 07Kubernetes: Figure out and document how to call the Kubernetes API as your tool user from inside a pod - https://phabricator.wikimedia.org/T321919#11127862 (10dcaro) >>! In T321919#11126898, @Anomie wrote: >>>! In T321919#10311951, @dcaro wrote: >> In...
[10:10:15] <wikibugs>	 (03Merged) 10jenkins-bot: build: Remove unsupported Python versions [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1182631 (owner: 10Majavah)
[10:10:15] <wikibugs>	 (03Merged) 10jenkins-bot: inventory: Add new toolsbeta bastion [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1182632 (https://phabricator.wikimedia.org/T392510) (owner: 10Majavah)
[10:11:01] <wikibugs>	 (03PS1) 10Majavah: mariadb-sssd: Build on Trixie [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1182805
[10:11:01] <wikibugs>	 (03PS1) 10Majavah: tcl86-sssd: Build on Trixie [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1182806 (https://phabricator.wikimedia.org/T400256)
[10:15:13] <wikibugs>	 (03open) 10taavi: toolsbeta: Remove floating IP for toolsbeta-bastion-6 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/70 (https://phabricator.wikimedia.org/T392510)
[10:15:15] <wikibugs>	 (03update) 10taavi: toolsbeta: Remove floating IP for toolsbeta-bastion-6 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/70 (https://phabricator.wikimedia.org/T392510)
[10:22:02] <wikibugs>	 (03update) 10taavi: toolsbeta: Remove floating IP for toolsbeta-bastion-6 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/70 (https://phabricator.wikimedia.org/T392510)
[10:23:07] <wikibugs>	 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [harbor,infra] gather stats about object storage qutoa usage and add an alert when tools is getting out of quota - https://phabricator.wikimedia.org/T402932#11127892 (10dcaro)
[11:10:35] <wikibugs>	 (03CR) 10Stevemunene: [V:03+2 C:03+2] druid-public: Add dummy keytabs for new hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1182691 (https://phabricator.wikimedia.org/T397441) (owner: 10Stevemunene)
[11:14:43] <wikibugs>	 10Toolforge (Toolforge iteration 24): [components-api] Allow reusing another component build - https://phabricator.wikimedia.org/T401893#11127988 (10DamianZaremba) Confirmed working in production with https://github.com/cluebotng/component-configs/blob/main/cluebotng-review.yaml ` tools.cluebotng-review@tool...
[11:45:18] <wikibugs>	 (03approved) 10dcaro: toolsbeta: Remove floating IP for toolsbeta-bastion-6 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/70 (https://phabricator.wikimedia.org/T392510) (owner: 10taavi)
[11:45:39] <wikibugs>	 (03update) 10taavi: toolsbeta: Remove floating IP for toolsbeta-bastion-6 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/70 (https://phabricator.wikimedia.org/T392510)
[11:45:44] <wikibugs>	 (03merge) 10taavi: toolsbeta: Remove floating IP for toolsbeta-bastion-6 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/70 (https://phabricator.wikimedia.org/T392510)
[11:55:11] <wikibugs>	 10cloud-services-team (FY2025/26-Q1): [ceph] 2025-08-27 ceph outage when bringing in a big osd host all at once (cloudcephosd1048) - https://phabricator.wikimedia.org/T403043#11128097 (10dcaro) This is interesting, though they did not find any clear solutions or ways to reproduce the issues https://indico.cern.c...
[12:09:11] <wikibugs>	 06cloud-services-team, 10Toolforge: "toolforge-jobs list" error - "TjfCliError: Unable to find image in the supported list or harbor" - https://phabricator.wikimedia.org/T402724#11128151 (10Bamyers99) "toolforge-jobs list" is working now.
[12:12:51] <wikibugs>	 10Cloud-VPS (Quota-requests), 10WLM-Italy (WLM-Italy-WebApp): Quota increase for Wiki Loves Monuments app - https://phabricator.wikimedia.org/T403165 (10Ferdi2005) 03NEW
[12:20:01] <wikibugs>	 (03merge) 10taavi: puppet_prefix: Generate YAML with `yamlencode` equivalent [repos/cloud/cloud-vps/tofu-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/-/merge_requests/10 (https://phabricator.wikimedia.org/T397994 https://phabricator.wikimedia.org/T398643) (owner: 10bd808)
[12:20:19] <wikibugs>	 06cloud-services-team, 10Toolforge: [components-api] rebuilds un-changed images - https://phabricator.wikimedia.org/T403167 (10DamianZaremba) 03NEW
[12:22:48] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Upgrade cloudcumin hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403153#11128212 (10MoritzMuehlenhoff) While cumin is available in Debian I think it makes sense if cloudcumin would also stick with Bookworm like the main nodes, otherwise Spicerack etc. need to...
[12:26:25] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Upgrade cloudcumin hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403153#11128229 (10Volans) +1 for me to upgrade them to bookworm for simplicity and to be in sync with the cumin hosts.
[12:28:18] <wikibugs>	 10cloud-services-team (FY2025/26-Q1): [ceph] 2025-08-27 ceph outage when bringing in a big osd host all at once (cloudcephosd1048) - https://phabricator.wikimedia.org/T403043#11128232 (10dcaro) During that time, I don't see any memory issues on any osds or the mons, the only one that shows a noticeable bump is 1...
[12:28:27] <wikibugs>	 (03open) 10taavi: Update repository URL [repos/cloud/cloud-vps/tofu-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/-/merge_requests/11
[12:28:30] <wikibugs>	 (03update) 10taavi: Update repository URL [repos/cloud/cloud-vps/tofu-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/-/merge_requests/11
[12:30:57] <wikibugs>	 06cloud-services-team, 10Toolforge: [webservice] returning default 404 even though webservice is healthy - https://phabricator.wikimedia.org/T403168 (10DamianZaremba) 03NEW
[12:33:30] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: KernelErrors Server cloudcephosd1052 logged kernel errors - https://phabricator.wikimedia.org/T402938#11128267 (10Jclark-ctr) Resolving this ticket since we will be replacing Nic with one that matches existing servers
[12:33:38] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: KernelErrors Server cloudcephosd1052 logged kernel errors - https://phabricator.wikimedia.org/T402938#11128268 (10Jclark-ctr) 05Open→03Resolved
[12:35:13] <wikibugs>	 (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/24
[12:46:48] <wikibugs>	 06cloud-services-team, 10Toolforge: [components-api] rebuilds un-changed images - https://phabricator.wikimedia.org/T403167#11128295 (10dcaro) This might be related to retention of builds and such, looking  Some notes, I can see that the latest build has the resolved ref correctly: ` tools.cluebotng-review@too...
[12:49:27] <wikibugs>	 10cloud-services-team (FY2025/26-Q1), 10Data-Services, 13Patch-For-Review: [wikireplicas] Refactor maintenance scripts to allow local testing - https://phabricator.wikimedia.org/T395266#11128300 (10fnegri) 05In progress→03Resolved The scripts were migrated to the new https://gitlab.wikimedia.org/repo...
[12:53:40] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
[12:53:42] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy
[12:54:44] <wikibugs>	 06cloud-services-team, 10Toolforge: [components-api] rebuilds un-changed images - https://phabricator.wikimedia.org/T403167#11128325 (10dcaro) Oh yep, I think I know what might be hapenning.  We have a retention for builds (different for successful and failed ones), and when that number is reached, it deletes...
[13:03:07] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 24): "toolforge-jobs list" error - "TjfCliError: Unable to find image in the supported list or harbor" - https://phabricator.wikimedia.org/T402724#11128367 (10dcaro) 05Open→03Resolved a:03dcaro awesome, if it happens again feel free to re-open
[13:10:18] <wikibugs>	 06cloud-services-team, 10Toolforge: [bulids-api] Figure out how to handle better build retention - https://phabricator.wikimedia.org/T403172 (10dcaro) 03NEW
[13:10:34] <wikibugs>	 06cloud-services-team, 10Toolforge: [components-api] rebuilds un-changed images - https://phabricator.wikimedia.org/T403167#11128414 (10DamianZaremba) That makes sense, I can see a limited number of builds: ` tools.cluebotng-review@tools-bastion-13:~$ toolforge build list build_id...
[13:14:29] <wikibugs>	 06cloud-services-team, 10Toolforge: [components-api] rebuilds un-changed images - https://phabricator.wikimedia.org/T403167#11128428 (10dcaro) > It would be quite nice if the build service reported back builds still present in harbour and/or a way to query harbor directly for components-api, which is the real...
[13:21:28] <wikibugs>	 06cloud-services-team, 10Toolforge: [components-api] rebuilds un-changed images - https://phabricator.wikimedia.org/T403167#11128459 (10DamianZaremba) >>! In T403167#11128428, @dcaro wrote: >> It would be quite nice if the build service reported back builds still present in harbour and/or a way to query harbor...
[13:28:13] <wikibugs>	 06cloud-services-team, 10Toolforge: [components-api] rebuilds un-changed images - https://phabricator.wikimedia.org/T403167#11128497 (10dcaro) > Aside from builds and internal tooling is anything else using harbor?  Not really, k8s uses it directly to pull the images, but it would not care if it's harbor or an...
[13:31:02] <wikibugs>	 (03CR) 10Andrew Bogott: "I haven't tested it in ages but it would still be nice to have." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/998492 (owner: 10Andrew Bogott)
[13:41:55] <wikibugs>	 06cloud-services-team, 10Toolforge: [components-api] retry internal api requests on failure - https://phabricator.wikimedia.org/T403175 (10DamianZaremba) 03NEW
[13:44:40] <wikibugs>	 06cloud-services-team, 10Toolforge: [components-api] Intermittent internal API failures / retry internal requests - https://phabricator.wikimedia.org/T403175#11128538 (10DamianZaremba)
[13:49:17] <jinxer-wm>	 FIRING: JobUnavailable: Reduced availability for job ebpf_exporter_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[13:49:29] <wikibugs>	 06cloud-services-team: JobUnavailable Reduced availability for job ebpf_exporter_eqiad in cloud@eqiad - https://phabricator.wikimedia.org/T403177 (10phaultfinder) 03NEW
[14:02:06] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Rename terraform-cloudvps repo to tofu-cloudvps - https://phabricator.wikimedia.org/T403178 (10taavi) 03NEW p:05Triage→03Low
[14:02:16] <wikibugs>	 (03update) 10taavi: Update repository URL [repos/cloud/cloud-vps/tofu-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/-/merge_requests/11 (https://phabricator.wikimedia.org/T403178)
[14:02:20] <wikibugs>	 (03update) 10taavi: Update repository URL [repos/cloud/cloud-vps/tofu-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/-/merge_requests/11 (https://phabricator.wikimedia.org/T403178)
[14:04:07] <wikibugs>	 (03update) 10taavi: Update repository URL [repos/cloud/cloud-vps/tofu-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/-/merge_requests/11 (https://phabricator.wikimedia.org/T403178)
[14:04:09] <wikibugs>	 (03update) 10taavi: Update repository URL [repos/cloud/cloud-vps/tofu-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/-/merge_requests/11 (https://phabricator.wikimedia.org/T403178)
[14:15:10] <wikibugs>	 (03approved) 10raymond-ndibe: dump: skip unset keys [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/124 (owner: 10dcaro)
[14:15:58] <wikibugs>	 (03update) 10raymond-ndibe: dump: skip unset keys [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/124 (owner: 10dcaro)
[14:16:04] <wikibugs>	 (03update) 10raymond-ndibe: dump: skip unset keys [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/124 (owner: 10dcaro)
[14:16:48] <wikibugs>	 (03update) 10raymond-ndibe: api: add `include_unset` parameter to get_job and get_jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/205 (https://phabricator.wikimedia.org/T402569) (owner: 10dcaro)
[14:16:49] <wikibugs>	 (03approved) 10raymond-ndibe: api: add `include_unset` parameter to get_job and get_jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/205 (https://phabricator.wikimedia.org/T402569) (owner: 10dcaro)
[14:19:31] <wikibugs>	 (03open) 10fnegri: WIP: add pre-commit [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/5
[14:20:31] <wikibugs>	 06cloud-services-team, 10decommission-hardware, 13Patch-For-Review: decommission cloudcephosd1004-10015 - https://phabricator.wikimedia.org/T402881#11128725 (10Andrew)
[14:20:42] <wikibugs>	 (03update) 10dcaro: api: add `include_unset` parameter to get_job and get_jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/205 (https://phabricator.wikimedia.org/T402569)
[14:22:22] <wikibugs>	 06cloud-services-team, 10decommission-hardware, 13Patch-For-Review: decommission cloudcephosd1004-10015 - https://phabricator.wikimedia.org/T402881#11128743 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by andrew@cumin2002 for hosts: `cloudcephosd1004.eqiad.wmnet` - cloudcephosd1004.eqiad....
[14:22:47] <wikibugs>	 06cloud-services-team, 10decommission-hardware, 13Patch-For-Review: decommission cloudcephosd1004-10015 - https://phabricator.wikimedia.org/T402881#11128747 (10Andrew)
[14:22:57] <wikibugs>	 (03update) 10fnegri: WIP: add pre-commit [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/5
[14:27:37] <wikibugs>	 (03update) 10dcaro: api: add `include_unset` parameter to get_job and get_jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/205 (https://phabricator.wikimedia.org/T402569)
[14:38:05] <wikibugs>	 10Cloud-VPS (Quota-requests), 10WLM-Italy (WLM-Italy-WebApp): Quota increase for Wiki Loves Monuments app - https://phabricator.wikimedia.org/T403165#11128819 (10dcaro) +1
[14:43:49] <wikibugs>	 10Cloud-VPS (Quota-requests), 10Content-Transform-Team (Work In Progress): Quote increase request for wikitextexp - https://phabricator.wikimedia.org/T403114#11128863 (10dcaro) +1  For the disk, it's not possible to shrink an existing volume, so the process here would be to create two smaller volumes and move...
[14:54:17] <jinxer-wm>	 RESOLVED: JobUnavailable: Reduced availability for job ebpf_exporter_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[14:54:50] <wikibugs>	 (03merge) 10dcaro: dump: skip unset keys [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/124
[14:59:30] <wikibugs>	 06cloud-services-team: JobUnavailable Reduced availability for job ebpf_exporter_eqiad in cloud@eqiad - https://phabricator.wikimedia.org/T403177#11129020 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Fixed by https://gerrit.wikimedia.org/r/1182508
[15:06:44] <wikibugs>	 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: DRAFT Decision request - Improving lima-kilo developer experience - https://phabricator.wikimedia.org/T403051#11129057 (10taavi)
[15:41:32] <wikibugs>	 06cloud-services-team, 10Toolforge: [webservice] returning default 404 even though webservice is healthy - https://phabricator.wikimedia.org/T403168#11129192 (10bd808) This is a mysterious and highly intermittent problem that has persisted for years. The 404 comes from the fourohfour tool which is configured a...
[15:46:53] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
[15:49:52] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.undrain_node
[16:00:13] <wikibugs>	 06cloud-services-team, 10Toolforge, 07Documentation, 07Kubernetes: Figure out and document how to call the Kubernetes API as your tool user from inside a pod - https://phabricator.wikimedia.org/T321919#11129288 (10Anomie) In one script I use the information to display a message "job <name> is already runni...
[16:02:37] <wikibugs>	 (03open) 10fnegri: Add pre-commit and ruff linter [repos/cloud/wikireplicas-utils] (pre-commit) - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/6
[16:02:38] <wikibugs>	 (03update) 10fnegri: Add pre-commit and ruff linter [repos/cloud/wikireplicas-utils] (pre-commit) - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/6
[16:02:38] <wikibugs>	 (03update) 10fnegri: Remove build_deb script, use gitlab-ci-local [repos/cloud/wikireplicas-utils] (pre-commit) - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/7
[16:02:38] <wikibugs>	 (03open) 10fnegri: Remove build_deb script, use gitlab-ci-local [repos/cloud/wikireplicas-utils] (pre-commit) - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/7
[16:02:39] <wikibugs>	 06cloud-services-team, 10decommission-hardware, 13Patch-For-Review: decommission cloudcephosd1004-10015 - https://phabricator.wikimedia.org/T402881#11129299 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by andrew@cumin2002 for hosts: `cloudcephosd[1005-1009].eqiad.wmnet` - cloudcephosd1005...
[16:02:45] <wikibugs>	 (03update) 10fnegri: Add pre-commit and ruff linter [repos/cloud/wikireplicas-utils] (pre-commit) - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/6
[16:02:49] <wikibugs>	 (03update) 10fnegri: Remove build_deb script, use gitlab-ci-local [repos/cloud/wikireplicas-utils] (pre-commit) - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/7
[16:06:38] <wikibugs>	 (03update) 10fnegri: Remove build_deb script, use gitlab-ci-local [repos/cloud/wikireplicas-utils] (pre-commit) - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/7
[16:07:56] <wikibugs>	 (03update) 10fnegri: Add pre-commit and ruff linter [repos/cloud/wikireplicas-utils] (pre-commit) - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/6
[16:07:57] <wikibugs>	 (03update) 10fnegri: Remove build_deb script, use gitlab-ci-local [repos/cloud/wikireplicas-utils] (pre-commit) - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/7
[16:09:09] <wikibugs>	 (03update) 10fnegri: Add pre-commit and ruff linter [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/6
[16:09:33] <wikibugs>	 (03update) 10fnegri: Remove build_deb script, use gitlab-ci-local [repos/cloud/wikireplicas-utils] (pre-commit) - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/7
[16:09:43] <wikibugs>	 (03update) 10fnegri: Add pre-commit and ruff linter [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/6
[16:09:59] <wikibugs>	 (03close) 10fnegri: WIP: add pre-commit [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/5
[16:12:51] <wikibugs>	 (03update) 10fnegri: Remove build_deb script, use gitlab-ci-local [repos/cloud/wikireplicas-utils] (pre-commit) - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/7
[16:12:57] <wikibugs>	 (03update) 10fnegri: Add pre-commit and ruff linter [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/6
[16:19:23] <wikibugs>	 06cloud-services-team, 10Toolforge, 07Documentation, 07Kubernetes: Figure out and document how to call the Kubernetes API as your tool user from inside a pod - https://phabricator.wikimedia.org/T321919#11129380 (10dcaro) Hmm, I'm thinking on adding something like `runtime_info` to the job, so we can put so...
[16:22:07] <wikibugs>	 (03approved) 10dcaro: Add pre-commit and ruff linter [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/6 (owner: 10fnegri)
[16:27:57] <wikibugs>	 (03unapproved) 10dcaro: Add pre-commit and ruff linter [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/6 (owner: 10fnegri)
[16:31:28] <wikibugs>	 (03approved) 10dcaro: Remove build_deb script, use gitlab-ci-local [repos/cloud/wikireplicas-utils] (pre-commit) - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/7 (owner: 10fnegri)
[16:32:53] <wikibugs>	 (03update) 10fnegri: Add pre-commit and ruff linter [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/6
[16:33:56] <wikibugs>	 (03update) 10fnegri: Remove build_deb script, use gitlab-ci-local [repos/cloud/wikireplicas-utils] (pre-commit) - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/7
[16:37:15] <wikibugs>	 (03update) 10fnegri: Add pre-commit and ruff linter [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/6
[16:37:18] <wikibugs>	 (03update) 10fnegri: Add pre-commit and ruff linter [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/6
[16:41:13] <wikibugs>	 (03update) 10fnegri: Remove build_deb script, use gitlab-ci-local [repos/cloud/wikireplicas-utils] (pre-commit) - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/7
[16:41:57] <wikibugs>	 (03update) 10fnegri: Remove build_deb script, use gitlab-ci-local [repos/cloud/wikireplicas-utils] (pre-commit) - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/7
[16:45:08] <wikibugs>	 (03approved) 10dcaro: Add pre-commit and ruff linter [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/6 (owner: 10fnegri)
[16:45:34] <wikibugs>	 (03update) 10fnegri: Add pre-commit and ruff linter [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/6
[16:45:56] <wikibugs>	 (03update) 10fnegri: Remove build_deb script, use gitlab-ci-local [repos/cloud/wikireplicas-utils] (pre-commit) - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/7
[16:49:24] <wikibugs>	 (03CR) 10BryanDavis: [C:03+1] mariadb-sssd: Build on Trixie [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1182805 (owner: 10Majavah)
[16:49:33] <wikibugs>	 (03CR) 10BryanDavis: [C:03+1] tcl86-sssd: Build on Trixie [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1182806 (https://phabricator.wikimedia.org/T400256) (owner: 10Majavah)
[16:49:54] <wikibugs>	 (03CR) 10Majavah: [C:03+2] mariadb-sssd: Build on Trixie [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1182805 (owner: 10Majavah)
[16:49:56] <wikibugs>	 (03CR) 10Majavah: [C:03+2] tcl86-sssd: Build on Trixie [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1182806 (https://phabricator.wikimedia.org/T400256) (owner: 10Majavah)
[16:50:29] <wikibugs>	 (03Merged) 10jenkins-bot: mariadb-sssd: Build on Trixie [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1182805 (owner: 10Majavah)
[16:50:32] <wikibugs>	 (03Merged) 10jenkins-bot: tcl86-sssd: Build on Trixie [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1182806 (https://phabricator.wikimedia.org/T400256) (owner: 10Majavah)
[16:54:24] <wikibugs>	 (03approved) 10bd808: tofu-provisioning: Cache plugin directory [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/69 (https://phabricator.wikimedia.org/T403028) (owner: 10taavi)
[16:55:16] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Update Toolforge Tcl image to a supported Debian release - https://phabricator.wikimedia.org/T400256#11129481 (10taavi) 05Open→03Resolved
[16:55:21] <wikibugs>	 (03update) 10bd808: puppet_prefix: Generate YAML with `yamlencode` equivalent [repos/cloud/cloud-vps/tofu-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/-/merge_requests/10 (https://phabricator.wikimedia.org/T397994 https://phabricator.wikimedia.org/T398643)
[17:07:21] <wikibugs>	 06cloud-services-team, 10decommission-hardware: decommission cloudcephosd1004-10015 - https://phabricator.wikimedia.org/T402881#11129546 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by andrew@cumin2002 for hosts: `cloudcephosd1010.eqiad.wmnet` - cloudcephosd1010.eqiad.wmnet (**PASS**)   - D...
[17:10:48] <jinxer-wm>	 FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudcephosd1014:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[17:20:48] <jinxer-wm>	 FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudcephosd1015:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[17:25:41] <wikibugs>	 06cloud-services-team, 10Toolforge, 07Documentation, 07Kubernetes: Figure out and document how to call the Kubernetes API as your tool user from inside a pod - https://phabricator.wikimedia.org/T321919#11129631 (10Anomie) > This just can skip saying which pod it runs on :)  Which then makes more work for m...
[18:05:19] <wikibugs>	 06cloud-services-team, 10decommission-hardware, 13Patch-For-Review: decommission cloudcephosd1004-10015 - https://phabricator.wikimedia.org/T402881#11129732 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by andrew@cumin2002 for hosts: `cloudcephosd[1011-1015].eqiad.wmnet` - cloudcephosd1011...
[18:05:54] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-eqiad: decommission cloudcephosd1004-10015 - https://phabricator.wikimedia.org/T402881#11129735 (10Andrew) a:05Andrew→03None
[18:13:41] <wikibugs>	 06cloud-services-team, 10Toolforge: [bulids-api] Figure out how to handle better build retention - https://phabricator.wikimedia.org/T403172#11129751 (10DamianZaremba) Another aspect to retention: ` [step-export] 2025-08-28T18:07:17.699718692Z ERROR: failed to export: failed to write image to the following tag...
[18:45:01] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10DNS: Move some of wikimediacloud.org 185.15.56.0/23 to Netbox - https://phabricator.wikimedia.org/T268621#11129842 (10ssingh) Can someone from WMCS comment on the status of this task? We are cleaning up the DNS/Traffic tickets and hence the question. More specifically, i...
[19:15:38] <wikibugs>	 (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/24 (owner: 10l10n-bot)
[19:15:41] <wikibugs>	 (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/24 (owner: 10l10n-bot)
[19:48:51] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Rabbitmq, neutron-openvswitch-agent, and network outages - https://phabricator.wikimedia.org/T397783#11129981 (10Andrew) 05Open→03Invalid I stopped tried stopping neutron-openvswitch-agent on all cloudvirts in codfw1dev and that did not interrupt my ssh connection to...
[20:46:30] <wikibugs>	 (03open) 10bd808: Makefile: Add targets for format, tidy, and test [repos/cloud/cloud-vps/tofu-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/-/merge_requests/12
[20:46:39] <wikibugs>	 (03update) 10bd808: Makefile: Add targets for format, tidy, and test [repos/cloud/cloud-vps/tofu-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/-/merge_requests/12
[20:46:44] <wikibugs>	 (03update) 10bd808: Makefile: Add targets for format, tidy, and test [repos/cloud/cloud-vps/tofu-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/-/merge_requests/12
[20:50:14] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
[20:52:07] <wikibugs>	 (03approved) 10bd808: Update repository URL [repos/cloud/cloud-vps/tofu-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/-/merge_requests/11 (https://phabricator.wikimedia.org/T403178) (owner: 10taavi)
[20:58:19] <wikibugs>	 10Cloud-VPS (Quota-requests), 10Content-Transform-Team (Work In Progress): Quote increase request for wikitextexp - https://phabricator.wikimedia.org/T403114#11130105 (10ssastry) Thanks! If you can bump up the volume quota with the other quotas, I can work on the rest and report back.
[21:25:00] <wikibugs>	 10VPS-Projects, 10Content-Transform-Team (Work In Progress), 07Essential-Work: Request new VPS for Content Transform Team Visual Diff testing - https://phabricator.wikimedia.org/T402836#11130165 (10ssastry) a:03ssastry
[23:53:30] <wikibugs>	 06cloud-services-team, 10Toolforge: [components-api] rebuilds un-changed images - https://phabricator.wikimedia.org/T403167#11130549 (10DamianZaremba) Here is an inverted example - builds-api returned the image, but the image no longer existed in harbour (`--force-build` required): ` tools.cluebotng-monitoring...