[00:11:06] (03approved) 10raymond-ndibe: [maintain-harbor] specify config through environment variables [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/36 (https://phabricator.wikimedia.org/T358225) [00:11:15] (03merge) 10raymond-ndibe: [maintain-harbor] specify config through environment variables [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/36 (https://phabricator.wikimedia.org/T358225) [00:11:16] (03update) 10raymond-ndibe: [maintain-harbor] Move to become a toolforge component [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/34 (https://phabricator.wikimedia.org/T358225) [00:14:44] 06cloud-services-team, 10Cloud-VPS: Extend Tofu provider to allow configuration via environment variables - https://phabricator.wikimedia.org/T321250#10366715 (10stwalkerster) a:03stwalkerster I've got an implementation of this working. Using the following OpenTofu code: `lang=hcl, name=main.tf terraform {... [00:15:24] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [00:15:43] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [00:22:09] (03open) 10stwalkerster: Allow configuring the provider by environment variables [repos/cloud/cloud-vps/terraform-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/terraform-cloudvps/-/merge_requests/2 [00:27:30] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [00:27:49] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [00:31:13] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [00:31:29] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [00:32:38] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [00:32:55] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [00:33:49] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [00:34:04] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [00:47:50] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [00:48:16] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [00:50:13] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [00:50:28] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [00:50:42] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [00:52:33] (03approved) 10raymond-ndibe: [toolforge-deploy] add --branch option [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/621 [00:52:36] (03merge) 10raymond-ndibe: [toolforge-deploy] add --branch option [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/621 [00:52:39] (03update) 10raymond-ndibe: [toolforge-deploy] deploy maintain-harbor [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/563 (https://phabricator.wikimedia.org/T358225) [00:56:12] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [00:56:29] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [01:22:50] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [01:28:29] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [02:32:03] 06Toolforge-standards-committee: Adoption request for ftools - https://phabricator.wikimedia.org/T381138 (10DatGuy) 03NEW [03:26:23] (03open) 10raymond-ndibe: [toolforge-deploy] bug fixes [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/622 [03:26:24] (03approved) 10raymond-ndibe: [toolforge-deploy] bug fixes [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/622 [03:26:43] (03merge) 10raymond-ndibe: [toolforge-deploy] bug fixes [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/622 [03:27:16] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [03:31:13] (03update) 10raymond-ndibe: [toolforge-deploy] deploy maintain-harbor [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/563 (https://phabricator.wikimedia.org/T358225) [03:31:19] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [03:32:51] (03update) 10raymond-ndibe: [toolforge-deploy] deploy maintain-harbor [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/563 (https://phabricator.wikimedia.org/T358225) [03:34:11] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [03:34:43] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [03:34:52] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [03:37:38] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [03:37:56] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [03:43:47] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [03:44:39] (03approved) 10raymond-ndibe: [toolforge-deploy] deploy maintain-harbor [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/563 (https://phabricator.wikimedia.org/T358225) [03:44:46] (03merge) 10raymond-ndibe: [toolforge-deploy] deploy maintain-harbor [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/563 (https://phabricator.wikimedia.org/T358225) [03:45:22] (03update) 10raymond-ndibe: [maintain-harbor] Move to become a toolforge component [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/34 (https://phabricator.wikimedia.org/T358225) [03:48:25] (03approved) 10raymond-ndibe: [maintain-harbor] Move to become a toolforge component [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/34 (https://phabricator.wikimedia.org/T358225) [03:48:31] (03merge) 10raymond-ndibe: [maintain-harbor] Move to become a toolforge component [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/34 (https://phabricator.wikimedia.org/T358225) [04:06:56] FIRING: SystemdUnitDown: The service unit purge_vm_rbd_images.service is in failed status on host cloudcontrol1005. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [04:08:20] (03open) 10raymond-ndibe: [toolforge-deploy] fix help conditional regex [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/623 (https://phabricator.wikimedia.org/T358225) [04:08:21] (03approved) 10raymond-ndibe: [toolforge-deploy] fix help conditional regex [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/623 (https://phabricator.wikimedia.org/T358225) [04:08:27] (03merge) 10raymond-ndibe: [toolforge-deploy] fix help conditional regex [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/623 (https://phabricator.wikimedia.org/T358225) [04:11:42] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: maintain-harbor: bump to 0.0.20-20241129041008-d8e4a15f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/624 [04:13:18] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [04:13:40] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor [04:14:09] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [04:14:27] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor [04:39:27] (03open) 10raymond-ndibe: [toolforg-deploy] append protocol to harbor url [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/625 (https://phabricator.wikimedia.org/T358225) [04:39:28] (03approved) 10raymond-ndibe: [toolforg-deploy] append protocol to harbor url [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/625 (https://phabricator.wikimedia.org/T358225) [04:39:33] (03merge) 10raymond-ndibe: [toolforg-deploy] append protocol to harbor url [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/625 (https://phabricator.wikimedia.org/T358225) [04:39:42] (03update) 10raymond-ndibe: maintain-harbor: bump to 0.0.20-20241129041008-d8e4a15f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/624 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [04:40:16] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [04:40:36] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor [04:44:07] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1098932 (owner: 10L10n-bot) [04:50:29] (03open) 10raymond-ndibe: [toolforge-deploy] fix failing maintain-harbor test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/626 (https://phabricator.wikimedia.org/T358225) [04:50:42] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:50:43] (03update) 10raymond-ndibe: maintain-harbor: bump to 0.0.20-20241129041008-d8e4a15f [repos/cloud/toolforge/toolforge-deploy] (fix_failing_maintain_harbor_test) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/624 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [04:50:55] (03update) 10raymond-ndibe: maintain-harbor: bump to 0.0.20-20241129041008-d8e4a15f [repos/cloud/toolforge/toolforge-deploy] (fix_failing_maintain_harbor_test) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/624 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [04:51:22] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [04:51:44] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor [04:52:39] (03approved) 10raymond-ndibe: [toolforge-deploy] fix failing maintain-harbor test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/626 (https://phabricator.wikimedia.org/T358225) [04:52:43] (03merge) 10raymond-ndibe: [toolforge-deploy] fix failing maintain-harbor test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/626 (https://phabricator.wikimedia.org/T358225) [04:52:44] (03update) 10raymond-ndibe: maintain-harbor: bump to 0.0.20-20241129041008-d8e4a15f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/624 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [04:54:18] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [04:54:36] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor [05:00:37] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [05:00:56] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor [06:01:56] FIRING: SystemdUnitDown: The systemd unit purge_vm_rbd_images.service on node cloudcontrol1005 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [06:02:03] 06cloud-services-team: SystemdUnitDown The systemd unit purge_vm_rbd_images.service on node cloudcontrol1005 has been failing for more than two hours. - https://phabricator.wikimedia.org/T381143 (10phaultfinder) 03NEW [06:07:16] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Prepare and check storage layer for arbcom_zhwiki - https://phabricator.wikimedia.org/T381086#10366983 (10Marostegui) @ABran-WMF I have merged the change, whenever you have a chance please restart all instances of all sanitarium hosts. Thanks. [06:30:28] (03open) 10raymond-ndibe: [toolforge-deploy] curl authentication with netrc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/627 (https://phabricator.wikimedia.org/T358225) [06:31:18] (03update) 10raymond-ndibe: [toolforge-deploy] curl authentication with netrc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/627 (https://phabricator.wikimedia.org/T358225) [06:31:35] (03approved) 10raymond-ndibe: [toolforge-deploy] curl authentication with netrc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/627 (https://phabricator.wikimedia.org/T358225) [06:31:40] (03merge) 10raymond-ndibe: [toolforge-deploy] curl authentication with netrc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/627 (https://phabricator.wikimedia.org/T358225) [06:32:27] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [06:32:50] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor [06:34:19] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [06:34:38] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor [06:39:02] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [06:39:24] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor [06:53:03] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [06:53:25] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor [07:01:45] 10Tools: contributionsurveyor tool not working - https://phabricator.wikimedia.org/T198443#10367011 (10JJMC89) 05Open→03Declined The tool is now completely offline. [07:04:04] (03open) 10raymond-ndibe: [toolforge-deploy] skip retention test for now [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/628 (https://phabricator.wikimedia.org/T358225) [07:04:08] (03merge) 10raymond-ndibe: [toolforge-deploy] skip retention test for now [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/628 (https://phabricator.wikimedia.org/T358225) [07:04:45] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [07:05:55] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor [07:07:11] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [07:08:10] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor [07:49:05] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Update cloudcephmon secrets [labs/private] - 10https://gerrit.wikimedia.org/r/1098988 (https://phabricator.wikimedia.org/T364870) (owner: 10Muehlenhoff) [07:52:43] 06cloud-services-team, 10Toolforge: rsyslogd tls issues on tools-nfs-2 - https://phabricator.wikimedia.org/T380829#10367040 (10taavi) [07:55:29] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): [jobs-api] crashing - https://phabricator.wikimedia.org/T380832#10367037 (10taavi) 05Open→03Resolved AFAICS this is resolved and fixing the alerts / docs are being tracked in separate tasks. [08:26:21] Change on 12wikitech.wikimedia.org a page News/Cloud VPS VXLAN IPv6 migration was modified, changed by Taavi link https://wikitech.wikimedia.org/w/index.php?diff=2249380 edit summary: [08:28:18] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [08:29:36] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor [08:31:16] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [08:32:30] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor [08:32:50] Change on 12wikitech.wikimedia.org a page Help:Toolforge/Building container images was modified, changed by Taavi link https://wikitech.wikimedia.org/w/index.php?diff=2249389 edit summary: fix sort [08:32:51] (03update) 10sstefanova: deploy-token: split create/refresh into POST/PUT endpoints [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/43 (https://phabricator.wikimedia.org/T380706) [08:32:55] Change on 12wikitech.wikimedia.org a page Help:Toolforge/Envvars was modified, changed by Taavi link https://wikitech.wikimedia.org/w/index.php?diff=2249390 edit summary: fix sort [08:33:09] (03approved) 10sstefanova: deploy-token: split create/refresh into POST/PUT endpoints [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/43 (https://phabricator.wikimedia.org/T380706) [08:33:13] (03merge) 10sstefanova: deploy-token: split create/refresh into POST/PUT endpoints [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/43 (https://phabricator.wikimedia.org/T380706) [08:33:23] Change on 12wikitech.wikimedia.org a page Help:Toolforge/Kubernetes/Reverse proxy was modified, changed by Taavi link https://wikitech.wikimedia.org/w/index.php?diff=2249391 edit summary: fix sort [08:33:35] Change on 12wikitech.wikimedia.org a page Help:Toolforge/Jobs framework was modified, changed by Taavi link https://wikitech.wikimedia.org/w/index.php?diff=2249392 edit summary: fix sort [08:33:40] Change on 12wikitech.wikimedia.org a page Help:Toolforge/Logs was modified, changed by Taavi link https://wikitech.wikimedia.org/w/index.php?diff=2249393 edit summary: fix sort [08:33:56] Change on 12wikitech.wikimedia.org a page Help:Toolforge/Rust was modified, changed by Taavi link https://wikitech.wikimedia.org/w/index.php?diff=2249394 edit summary: fix sort [08:35:02] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [08:35:48] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.71-20241129083321-0a425581 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/629 (https://phabricator.wikimedia.org/T380706) [08:36:16] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor [08:37:38] Change on 12wikitech.wikimedia.org a page Help:Toolforge/Building container images/My first Buildpack Django tool was modified, changed by Taavi link https://wikitech.wikimedia.org/w/index.php?diff=2249398 edit summary: fix category sorting [08:37:48] Change on 12wikitech.wikimedia.org a page Help:Toolforge/Building container images/My first Buildpack Python tool was modified, changed by Taavi link https://wikitech.wikimedia.org/w/index.php?diff=2249399 edit summary: fix sort [08:40:31] Change on 12wikitech.wikimedia.org a page Help:Toolforge/Web/Lighttpd was modified, changed by Taavi link https://wikitech.wikimedia.org/w/index.php?diff=2249403 edit summary: fix sort [08:42:24] Change on 12wikitech.wikimedia.org a page Help:Toolforge/Tool accounts was modified, changed by Taavi link https://wikitech.wikimedia.org/w/index.php?diff=2249409 edit summary: fix sort [08:49:09] (03update) 10sstefanova: cli: show deleted objects [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/5 (https://phabricator.wikimedia.org/T379091) [08:50:42] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:56:29] (03update) 10sstefanova: cli: show deleted objects [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/5 (https://phabricator.wikimedia.org/T379091) [08:57:21] (03update) 10sstefanova: cli: show deleted objects [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/5 (https://phabricator.wikimedia.org/T379091) [08:57:28] (03merge) 10sstefanova: cli: show deleted objects [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/5 (https://phabricator.wikimedia.org/T379091) [08:59:17] 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [components-api, components-cli] deploy-token: separate create from update - https://phabricator.wikimedia.org/T380706#10367088 (10Slst2020) [09:11:05] 06Toolforge-standards-committee: Adoption request for ftools - https://phabricator.wikimedia.org/T381138#10367101 (10SD0001) I'm not seeing a concrete reason to adopt the tool: * Source code is publicly available and can be used to deploy as a new tool * The tool doesn't use the database or other persistent stor... [10:01:56] FIRING: SystemdUnitDown: The systemd unit purge_vm_rbd_images.service on node cloudcontrol1005 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [10:16:57] Change on 12wikitech.wikimedia.org a page News was modified, changed by Taavi link https://wikitech.wikimedia.org/w/index.php?diff=2249444 edit summary: [+88] [10:46:29] (03open) 10sstefanova: cli: Improve deploy-token command UX and safety [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T380706) [11:25:26] (03update) 10sstefanova: cli: Improve deploy-token command UX and safety [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T380706) [11:48:48] (03update) 10sstefanova: cli: Improve deploy-token command UX and safety [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T380706) [12:01:50] (03update) 10sstefanova: cli: Improve deploy-token command UX and safety [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T380706) [12:16:55] (03update) 10sstefanova: cli: Improve deploy-token command UX and safety [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T380706) [12:50:01] Change on 12wikitech.wikimedia.org a page Help:Toolforge/My first static tool was modified, changed by Taavi link https://wikitech.wikimedia.org/w/index.php?diff=2249492 edit summary: removed [[Category:Static]] using [[Help:Gadget-HotCat|HotCat]] [12:50:42] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:59:52] Change on 12wikitech.wikimedia.org a page Help:Toolforge/Python was modified, changed by Taavi link https://wikitech.wikimedia.org/w/index.php?diff=2249498 edit summary: [14:01:57] FIRING: SystemdUnitDown: The systemd unit purge_vm_rbd_images.service on node cloudcontrol1005 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:25:16] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 05Goal, 13Patch-For-Review: [toolsdb] Upgrade to MariaDB 10.6 - https://phabricator.wikimedia.org/T352206#10368149 (10fnegri) Below is the summary of the actions performed during the upgrade on 2024-11-25 (based on the [proced... [14:28:54] 06Toolforge-standards-committee: Adoption request for ftools - https://phabricator.wikimedia.org/T381138#10368154 (10DatGuy) I would say the two threads less than a day apart point to its active use, as well as it being part of the [[ https://en.wikipedia.org/wiki/Wikipedia:Blocking_IP_addresses#Range_blocks | r... [16:50:42] FIRING: CloudVPSDesignateLeaks: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:10:22] 06cloud-services-team, 10Cloud-VPS: [tofu] [designate] [pdns] Swapping a CNAME and an A record can cause a loop - https://phabricator.wikimedia.org/T381180 (10fnegri) 03NEW [18:01:57] FIRING: SystemdUnitDown: The systemd unit purge_vm_rbd_images.service on node cloudcontrol1005 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [18:50:19] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 05Goal: [toolsdb] test failover procedure - https://phabricator.wikimedia.org/T344719#10368688 (10fnegri) 05Open→03Resolved This procedure was tested during {T352206} and several improvements were added to wikitech. [18:52:08] (03update) 10sstefanova: cli: add deployment subcommands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T379091) [20:50:42] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:01:07] (03update) 10bd808: Add #wikimedia-traffic [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/17 (https://phabricator.wikimedia.org/T381110) (owner: 10urbanecm) [21:07:10] (03merge) 10bd808: channels: Add wikimedia-perf and wikimedia-perf-bots [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/20 [21:08:45] (03update) 10bd808: Add #wikimedia-traffic [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/17 (https://phabricator.wikimedia.org/T381110) (owner: 10urbanecm) [21:14:53] 10VPS-Projects, 10fundraising-tech-ops, 10Puppet (Puppet 7.0): Update puppet civicrm-prototype puppetmaster - https://phabricator.wikimedia.org/T361595#10368769 (10Andrew) Hello @Dwisehaupt -- I don't know how things got into this state but I have tried to sort things out a bit. I believe the puppetserver is... [21:18:32] (03open) 10bd808: wikimedia-perf-bots: Autovoice icinga-wm and jinxer-wm [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/21 [21:21:58] (03merge) 10bd808: wikimedia-perf-bots: Autovoice icinga-wm and jinxer-wm [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/21 [21:26:15] (03update) 10bd808: Add #wikimedia-traffic [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/17 (https://phabricator.wikimedia.org/T381110) (owner: 10urbanecm) [21:41:46] Change on 12wikitech.wikimedia.org a page News/2020 Kubernetes cluster migration was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=2249552 edit summary: /* See also */ link update [21:42:12] Change on 12wikitech.wikimedia.org a page News/Toolforge Elasticsearch v7 migration was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=2249553 edit summary: /* See also */ link update [21:42:37] Change on 12wikitech.wikimedia.org a page News/Stretch deprecation was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=2249554 edit summary: /* See also */ link update [21:44:24] Change on 12wikitech.wikimedia.org a page News/2024 Buster deprecation was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=2249557 edit summary: /* See also */ link update [21:46:57] RESOLVED: SystemdUnitDown: The systemd unit purge_vm_rbd_images.service on node cloudcontrol1005 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [21:53:33] Change on 12wikitech.wikimedia.org a page Help:Toolforge/Web was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=2249570 edit summary: /* See also */ link update [21:55:01] Change on 12wikitech.wikimedia.org a page News/Toolforge Elasticsearch migration was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=2249573 edit summary: update link to avoid redirect [21:58:49] Change on 12wikitech.wikimedia.org a page News/2024 Toolforge Grid Engine deprecation was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=2249580 edit summary: update link to avoid redirect [22:01:19] Change on 12wikitech.wikimedia.org a page News/2021 Stretch deprecation was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=2249585 edit summary: /* See also */ link update [22:01:54] Change on 12wikitech.wikimedia.org a page Help:Toolforge/Jobs framework was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=2249586 edit summary: /* See also */ link update [22:04:02] Change on 12wikitech.wikimedia.org a page News/2021 Stretch deprecation was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=2249589 edit summary: update links to avoid redirect [22:06:05] (03update) 10jjmc89: Add #wikimedia-traffic [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/17 (https://phabricator.wikimedia.org/T381110) (owner: 10urbanecm) [22:06:08] (03approved) 10jjmc89: Add #wikimedia-traffic [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/17 (https://phabricator.wikimedia.org/T381110) (owner: 10urbanecm) [22:06:16] Change on 12wikitech.wikimedia.org a page News/Trusty deprecation was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=2249594 edit summary: update links to avoid redirect [22:06:42] (03unapproved) 10jjmc89: Add #wikimedia-traffic [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/17 (https://phabricator.wikimedia.org/T381110) (owner: 10urbanecm) [22:08:47] Change on 12wikitech.wikimedia.org a page Help:Toolforge/Grid/Stretch Migration was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=2249598 edit summary: update link to avoid redirect [22:14:01] Change on 12wikitech.wikimedia.org a page News/2019 Toolforge Trusty deprecation was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=2249607 edit summary: update link to avoid redirect [22:16:09] Change on 12wikitech.wikimedia.org a page News/2019 Toolforge Trusty deprecation was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=2249610 edit summary: update link to avoid redirect [22:16:46] Change on 12wikitech.wikimedia.org a page News/2019 Jessie deprecation was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=2249611 edit summary: /* See also */ link update [22:18:44] Change on 12wikitech.wikimedia.org a page News was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=2249613 edit summary: [+5] /* Recently completed */ update link to avoid redirect [22:21:45] Change on 12wikitech.wikimedia.org a page News/Wikireplicas 2020 Redesign was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=2249618 edit summary: update link to avoid redirect [22:24:37] Change on 12wikitech.wikimedia.org a page News/2020 Wiki Replicas Redesign was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=2249622 edit summary: update link to avoid redirect [22:26:13] Change on 12wikitech.wikimedia.org a page News was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=2249625 edit summary: [+5] update link to avoid redirect [22:29:28] Change on 12wikitech.wikimedia.org a page News was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=2249631 edit summary: [+5] update link to avoid redirect [22:30:42] RESOLVED: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:32:07] Change on 12wikitech.wikimedia.org a page News/Cloud VPS 2018 Purge was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=2249635 edit summary: /* See also */ link update [22:55:55] Change on 12wikitech.wikimedia.org a page News was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=2249654 edit summary: [-286] link updates; move some things from current to recently completed [23:05:04] (03update) 10bd808: Add #wikimedia-traffic [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/17 (https://phabricator.wikimedia.org/T381110) (owner: 10urbanecm) [23:05:14] (03update) 10bd808: Add #wikimedia-traffic [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/17 (https://phabricator.wikimedia.org/T381110) (owner: 10urbanecm) [23:09:07] (03approved) 10jjmc89: Add #wikimedia-traffic [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/17 (https://phabricator.wikimedia.org/T381110) (owner: 10urbanecm) [23:40:47] (03open) 10bd808: dev: Add pre-commit config and run under CI [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/22 [23:44:35] (03update) 10bd808: dev: Add pre-commit config and run under CI [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/22 [23:48:23] (03update) 10bd808: dev: Add pre-commit config and run under CI [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/22 [23:53:44] (03update) 10bd808: dev: Add pre-commit config and run under CI [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/22 [23:55:12] (03update) 10bd808: dev: Add pre-commit config and run under CI [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/22