[00:02:49] 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T381452#10378064 (10LibUp-bot) [00:02:55] 06cloud-services-team, 10Toolforge: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T381453#10378066 (10LibUp-bot) [00:53:50] (03open) 10raymond-ndibe: [toolforge-deploy] default to main if branch not in repo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/632 (https://phabricator.wikimedia.org/T358225) [00:55:57] (03update) 10raymond-ndibe: components-api: bump to 0.0.71-20241129083321-0a425581 [repos/cloud/toolforge/toolforge-deploy] (default_to_main_if_no_branch) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/629 (https://phabricator.wikimedia.org/T380706) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:56:04] (03update) 10raymond-ndibe: components-api: bump to 0.0.71-20241129083321-0a425581 [repos/cloud/toolforge/toolforge-deploy] (default_to_main_if_no_branch) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/629 (https://phabricator.wikimedia.org/T380706) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:56:30] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [00:57:48] (03update) 10raymond-ndibe: [toolforge-weld] refactor parse_quantity [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/64 (https://phabricator.wikimedia.org/T361120) [00:58:13] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [00:58:38] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [00:58:48] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [01:00:30] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [01:00:40] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [01:04:03] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [01:10:38] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [01:12:01] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [01:12:07] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [01:14:34] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [01:14:40] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [01:15:15] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [01:15:21] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [01:16:54] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [01:17:00] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [01:17:31] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [01:17:39] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [01:18:33] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [01:18:39] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [01:19:56] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [01:21:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:23:18] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [01:24:44] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [01:30:09] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [01:30:20] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [01:30:27] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [01:31:06] (03update) 10raymond-ndibe: components-api: bump to 0.0.71-20241129083321-0a425581 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/629 (https://phabricator.wikimedia.org/T380706) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [01:31:26] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [01:31:31] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [01:31:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:52:45] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10378154 [03:21:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:31:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:02:09] 06cloud-services-team, 10Toolforge: Unable to Connect to Database for New Toolforge Project "wiki-talents" - https://phabricator.wikimedia.org/T381457 (10UkrFace) 03NEW [07:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:47:42] 10Tool-translatetagger, 06Indic-TechCom, 10Technical-Tool-Request: Tool to convert Wikitext into translatable wiki-text - https://phabricator.wikimedia.org/T372243#10378441 (10Aafi) I have been looking around this since my conversation with Gopavasanth at Wikimedia Technology Summit in Hyderabad. It is a goo... [08:59:43] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10378577 [09:21:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:31:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:43:52] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 10PAWS: Restrict outbound connectivity from PAWS hosts - https://phabricator.wikimedia.org/T381373#10378670 (10cmooney) >>! In T381373#10375740, @aborrero wrote: > You may be aware of this, but let me note for the record: PAWS virtual machines are dynam... [11:32:39] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 10PAWS: Restrict outbound connectivity from PAWS hosts - https://phabricator.wikimedia.org/T381373#10379042 (10fnegri) > That said, how often is the system rebuilt? I would if possible like to keep the specific NAT rule in place for now, so that maybe i... [12:10:10] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 10PAWS: Restrict outbound connectivity from PAWS hosts - https://phabricator.wikimedia.org/T381373#10379264 (10cmooney) p:05High→03Medium >>! In T381373#10379042, @fnegri wrote: > Yes I think it's unlikely we'll have to rebuild the cluster before 1... [12:10:12] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 10PAWS: Restrict outbound connectivity from PAWS hosts - https://phabricator.wikimedia.org/T381373#10379266 (10rook) >>! In T381373#10379042, @fnegri wrote: >> That said, how often is the system rebuilt? I would if possible like to keep the specific NAT... [12:31:52] 10VPS-project-Codesearch: Please add patchdemo to codesearch index - https://phabricator.wikimedia.org/T333073#10379360 (10Ladsgroup) The devools is there, where is patchdemo now? https://github.com/MatmaRex/patchdemo is RO [12:45:12] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10379441 [12:51:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:01:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:19:02] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/55 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:21:09] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/44 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:22:30] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [13:22:49] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/44 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:22:56] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/44 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:25:04] (03update) 10sstefanova: components-api: bump to 0.0.71-20241129083321-0a425581 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/629 (https://phabricator.wikimedia.org/T380706) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:26:28] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.71-20241129083321-0a425581 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/629 (https://phabricator.wikimedia.org/T380706) [13:27:55] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/117 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:27:56] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/117 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:28:01] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/117 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:29:20] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [13:32:57] (03update) 10sstefanova: components-api: bump to 0.0.71-20241129083321-0a425581 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/629 (https://phabricator.wikimedia.org/T380706) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:32:59] (03approved) 10sstefanova: components-api: bump to 0.0.71-20241129083321-0a425581 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/629 (https://phabricator.wikimedia.org/T380706) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:33:06] (03merge) 10sstefanova: components-api: bump to 0.0.71-20241129083321-0a425581 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/629 (https://phabricator.wikimedia.org/T380706) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:35:19] (03update) 10sstefanova: [toolforge-deploy] more bug fixes [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/630 (https://phabricator.wikimedia.org/T358225) (owner: 10raymond-ndibe) [13:35:58] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: builds-api: bump to 0.0.177-20241204132811-dde183c5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/633 [13:36:19] (03update) 10sstefanova: builds-api: bump to 0.0.177-20241204132811-dde183c5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/633 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:37:05] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [13:38:19] !log sstefanova@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [13:39:38] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [13:39:49] !log sstefanova@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [13:43:22] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/49 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:43:22] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/49 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:43:29] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/49 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:48:33] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: envvars-api: bump to 0.0.63-20241204134340-d383083d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/634 [13:53:35] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [13:54:41] !log sstefanova@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api [13:59:18] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [13:59:28] !log sstefanova@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api [14:10:51] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [14:13:46] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [14:14:09] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [14:20:00] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [14:20:11] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api [14:20:35] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [14:20:40] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [14:21:01] 06cloud-services-team, 10Cloud-VPS: Upgrade cloud-vps openstack to version 'Dalmation' - https://phabricator.wikimedia.org/T381499 (10Andrew) 03NEW [14:26:09] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [14:28:04] (03CR) 10Andrew Bogott: Get openstack project list from keystone (031 comment) [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/1093997 (https://phabricator.wikimedia.org/T379030) (owner: 10Andrew Bogott) [14:41:31] vivian-rook opened https://github.com/toolforge/paws/pull/467 [14:42:39] (03update) 10raymond-ndibe: [toolforge-deploy] default to main if branch not in repo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/632 (https://phabricator.wikimedia.org/T358225) [14:45:44] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [14:45:50] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' (T380893) [14:45:55] T380893: decommission cloudcephmon100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T380893 [14:46:41] (03update) 10raymond-ndibe: [toolforge-deploy] default to main if branch not in repo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/632 (https://phabricator.wikimedia.org/T358225) [14:46:54] !log sstefanova@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [14:47:23] (03update) 10raymond-ndibe: [toolforge-deploy] default to main if branch not in repo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/632 (https://phabricator.wikimedia.org/T358225) [14:47:37] (03update) 10raymond-ndibe: [toolforge-deploy] default to main if branch not in repo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/632 (https://phabricator.wikimedia.org/T358225) [14:47:52] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [14:48:40] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' (T380893) [14:49:25] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1035.eqiad.wmnet}' (T380731) [14:49:30] T380731: Reboots of Bookworm systems which use 6.1.115 - https://phabricator.wikimedia.org/T380731 [14:52:23] PROBLEM - Host cloudvirt1035 is DOWN: PING CRITICAL - Packet loss = 100% [14:52:28] FIRING: InstanceDown: Project cloudinfra instance cloudinfra-cloudvps-puppetserver-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:54:40] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [14:54:59] RECOVERY - Host cloudvirt1035 is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms [14:55:05] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1035.eqiad.wmnet}' (T380731) [14:55:11] T380731: Reboots of Bookworm systems which use 6.1.115 - https://phabricator.wikimedia.org/T380731 [14:57:14] FIRING: Kernel error: Server cloudvirt1035 may have kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Kernel_panic - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-panic-detector?orgId=1&var-instance=cloudvirt1035 - https://alerts.wikimedia.org/?q=alertname%3DKernel+error [14:57:14] FIRING: Kernel warning: Server cloudvirt1035 may have kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Kernel_panic - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-panic-detector?orgId=1&var-instance=cloudvirt1035 - https://alerts.wikimedia.org/?q=alertname%3DKernel+warning [14:57:18] 06cloud-services-team: Kernel error Server cloudvirt1035 may have kernel errors - https://phabricator.wikimedia.org/T381500 (10phaultfinder) 03NEW [14:57:23] (03update) 10raymond-ndibe: [toolforge-deploy] fix bug in setup_toolforge_deploy [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/632 (https://phabricator.wikimedia.org/T358225) [14:58:02] (03update) 10raymond-ndibe: envvars-api: bump to 0.0.63-20241204134340-d383083d [repos/cloud/toolforge/toolforge-deploy] (default_to_main_if_no_branch) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/634 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [14:58:12] (03update) 10raymond-ndibe: envvars-api: bump to 0.0.63-20241204134340-d383083d [repos/cloud/toolforge/toolforge-deploy] (default_to_main_if_no_branch) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/634 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:00:52] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [15:06:06] 10PAWS: upgrade jupyterlab - https://phabricator.wikimedia.org/T381501 (10rook) 03NEW [15:07:28] RESOLVED: InstanceDown: Project cloudinfra instance cloudinfra-cloudvps-puppetserver-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:09:28] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [15:09:41] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [15:11:13] !log raymond-ndibe@cloudcumin1001 tools END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api [15:11:17] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [15:12:11] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10380054 (10KMontalva-WMF) |**Wikitech account/LDAP:**| kevmon| |**SUL account**| KMontalva-WMF| |**Account linked on [[ https://idm.wikimedia.org/ | IDM ]]** |Y| |**I have visited [[ https:/... [15:14:57] vivian-rook closed https://github.com/toolforge/paws/pull/467 [15:16:22] 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T381452#10380066 (10rook) https://github.com/toolforge/paws/pull/467 [15:16:23] vivian-rook opened https://github.com/toolforge/paws/pull/468 [15:16:27] 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T381452#10380067 (10rook) 05Open→03Resolved a:03rook [15:16:29] 10PAWS: upgrade jupyterlab - https://phabricator.wikimedia.org/T381501#10380071 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/468 [15:18:37] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [15:19:35] (03update) 10raymond-ndibe: envvars-api: bump to 0.0.63-20241204134340-d383083d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/634 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:19:43] 10PAWS: New upstream release for OpenRefine - https://phabricator.wikimedia.org/T380658#10380074 (10rook) →14Duplicate dup:03T380436 [15:19:44] 10PAWS: New upstream release for OpenRefine - https://phabricator.wikimedia.org/T380436#10380076 (10rook) [15:20:14] 10PAWS: Upgrade to k8s 1.28 - https://phabricator.wikimedia.org/T381503 (10rook) 03NEW [15:20:41] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10380098 (10Ladsgroup) https://wikitech.wikimedia.org/wiki/Special:Contributions/Kevmon This account is not registered in wikitech. Since you're new (welcome to the foundation!) Let me see wh... [15:22:17] (03approved) 10raymond-ndibe: [toolforge-deploy] fix bug in setup_toolforge_deploy [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/632 (https://phabricator.wikimedia.org/T358225) [15:22:35] (03merge) 10raymond-ndibe: [toolforge-deploy] fix bug in setup_toolforge_deploy [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/632 (https://phabricator.wikimedia.org/T358225) [15:23:43] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [15:24:02] (03approved) 10raymond-ndibe: envvars-api: bump to 0.0.63-20241204134340-d383083d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/634 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:24:06] (03merge) 10raymond-ndibe: envvars-api: bump to 0.0.63-20241204134340-d383083d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/634 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:24:31] (03update) 10raymond-ndibe: functional tests: add components-api tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/631 (https://phabricator.wikimedia.org/T379092) (owner: 10sstefanova) [15:25:48] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10380119 (10Ladsgroup) I force created the local account, can you try again? [15:26:00] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [15:26:14] !log sstefanova@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [15:26:22] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [15:26:47] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [15:26:51] (03update) 10raymond-ndibe: builds-api: bump to 0.0.177-20241204132811-dde183c5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/633 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:27:16] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [15:27:26] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [15:27:46] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [15:27:56] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [15:28:53] (03PS1) 10Ladsgroup: Add Patch demo [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1100472 (https://phabricator.wikimedia.org/T333073) [15:29:27] (03CR) 10Ladsgroup: [C:03+2] Add Patch demo [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1100472 (https://phabricator.wikimedia.org/T333073) (owner: 10Ladsgroup) [15:29:47] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [15:29:58] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [15:30:23] (03Merged) 10jenkins-bot: Add Patch demo [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1100472 (https://phabricator.wikimedia.org/T333073) (owner: 10Ladsgroup) [15:30:38] 06cloud-services-team, 10Cloud-VPS: openstack network problems (November 2024) - https://phabricator.wikimedia.org/T380882#10380135 (10fnegri) [15:30:40] 06cloud-services-team, 10decommission-hardware, 13Patch-For-Review: decommission cloudcephmon100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T380893#10380139 (10Andrew) [15:31:23] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [15:33:55] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [15:35:14] 10VPS-project-Codesearch: Please add patchdemo to codesearch index - https://phabricator.wikimedia.org/T333073#10380153 (10Ladsgroup) 05Open→03Resolved It'll be there by the next 24 hours. [15:37:10] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [15:39:43] (03update) 10raymond-ndibe: builds-api: bump to 0.0.177-20241204132811-dde183c5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/633 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:40:46] (03approved) 10raymond-ndibe: builds-api: bump to 0.0.177-20241204132811-dde183c5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/633 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:40:49] (03merge) 10raymond-ndibe: builds-api: bump to 0.0.177-20241204132811-dde183c5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/633 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:45:31] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [15:45:36] !log sstefanova@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [15:52:29] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/130 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:52:30] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/130 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:52:35] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/130 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:54:14] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/22 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:54:17] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/22 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:54:21] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/22 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:55:41] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: jobs-api: bump to 0.0.341-20241204155248-0a8cee40 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/635 [15:57:31] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: volume-admission: bump to 0.0.60-20241204155432-f73a7ddb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/636 [16:02:12] (03update) 10sstefanova: jobs-api: bump to 0.0.341-20241204155248-0a8cee40 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/635 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:03:04] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [16:10:18] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [16:13:33] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [16:14:41] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/15 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:14:42] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/15 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:14:52] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/15 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:17:08] vivian-rook closed https://github.com/toolforge/paws/pull/468 [16:17:36] 10PAWS: upgrade jupyterlab - https://phabricator.wikimedia.org/T381501#10380309 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/468 [16:17:54] 10PAWS: upgrade jupyterlab - https://phabricator.wikimedia.org/T381501#10380310 (10rook) 05Open→03Resolved a:03rook [16:18:10] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: ingress-admission: bump to 0.0.55-20241204161503-a4c0b7d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/637 [16:18:40] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10380323 (10KMontalva-WMF) > welcome to the foundation! Thank you! > I force created the local account, can you try again? That worked a treat, thank you! My understanding from the "migrat... [16:19:06] FIRING: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:21:41] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [16:24:06] FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:31:35] 10Tool-translatetagger: Improvise footer design for mobile devices - https://phabricator.wikimedia.org/T376169#10380365 (10Kuldeepburjbhalaike) a:03Kuldeepburjbhalaike [16:34:06] RESOLVED: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:34:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [16:34:16] 10cloud-services-team (FY2024/2025-Q1-Q2): Subdomain for catalyst-dev project - https://phabricator.wikimedia.org/T381508#10380371 (10fnegri) p:05Triage→03Medium a:03fnegri [16:34:36] (03approved) 10sstefanova: jobs-api: bump to 0.0.341-20241204155248-0a8cee40 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/635 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:34:38] (03merge) 10sstefanova: jobs-api: bump to 0.0.341-20241204155248-0a8cee40 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/635 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:35:25] (03update) 10sstefanova: volume-admission: bump to 0.0.60-20241204155432-f73a7ddb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/636 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:35:25] (03update) 10sstefanova: volume-admission: bump to 0.0.60-20241204155432-f73a7ddb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/636 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:35:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:35:59] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [16:37:57] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS (Quota-requests): Subdomain for catalyst-dev project - https://phabricator.wikimedia.org/T381508#10380384 (10JJMC89) [16:38:27] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/216 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:38:30] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/216 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:38:34] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/216 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:39:00] 06cloud-services-team, 10decommission-hardware, 13Patch-For-Review: decommission cloudcephmon100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T380893#10380386 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by andrew@cumin1002 for hosts: `cloudcephmon1001.eqiad.wmnet` - cloudcephmon1... [16:39:21] (03approved) 10sstefanova: bump_version: copy from jobs-api [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/67 (owner: 10dcaro) [16:39:23] (03update) 10sstefanova: bump_version: copy from jobs-api [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/67 (owner: 10dcaro) [16:39:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudcephmon1003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [16:43:22] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [16:45:34] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10380396 (10Ladsgroup) Nothing was wrong on your side. It's because we are at the middle of a migration and things are a bit whacky. It'll be fixed soon for future cases. Sorry for the inconv... [16:47:38] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [16:54:13] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [16:54:47] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission [16:55:20] (03approved) 10sstefanova: volume-admission: bump to 0.0.60-20241204155432-f73a7ddb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/636 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:55:23] (03merge) 10sstefanova: volume-admission: bump to 0.0.60-20241204155432-f73a7ddb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/636 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:55:57] (03update) 10sstefanova: ingress-admission: bump to 0.0.55-20241204161503-a4c0b7d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/637 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:55:58] (03update) 10sstefanova: ingress-admission: bump to 0.0.55-20241204161503-a4c0b7d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/637 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:56:42] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/16 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:57:00] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/16 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:57:06] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/16 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:57:34] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/18 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:57:40] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/18 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:57:43] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/18 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:58:39] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [17:00:02] 06cloud-services-team, 10decommission-hardware, 13Patch-For-Review: decommission cloudcephmon100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T380893#10380439 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by andrew@cumin1002 for hosts: `cloudcephmon1002.eqiad.wmnet` - cloudcephmon1... [17:00:50] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: registry-admission: bump to 0.0.55-20241204165756-6cff3221 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/638 [17:01:21] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: envvars-admission: bump to 0.0.23-20241204165719-1295007c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/639 [17:02:17] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission [17:03:56] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission [17:05:14] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/55 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:10:51] (03PS2) 10Bartosz Dziewoński: Update Patch demo repo [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1100495 (https://phabricator.wikimedia.org/T333073) [17:11:41] (03update) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/45 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:11:48] (03CR) 10Ladsgroup: [C:03+2] Update Patch demo repo [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1100495 (https://phabricator.wikimedia.org/T333073) (owner: 10Bartosz Dziewoński) [17:11:51] 06cloud-services-team, 10decommission-hardware, 13Patch-For-Review: decommission cloudcephmon100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T380893#10380471 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by andrew@cumin1002 for hosts: `cloudcephmon1003.eqiad.wmnet` - cloudcephmon1... [17:11:57] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission [17:12:00] 10VPS-project-Codesearch, 13Patch-For-Review: Please add patchdemo to codesearch index - https://phabricator.wikimedia.org/T333073#10380468 (10matmarex) 05Resolved→03Open https://gitlab.wikimedia.org/repos/ci-tools/patchdemo is also outdated. The current repo is https://gitlab.wikimedia.org/repos/qte/catal... [17:12:13] (03approved) 10sstefanova: ingress-admission: bump to 0.0.55-20241204161503-a4c0b7d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/637 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:12:17] (03merge) 10sstefanova: ingress-admission: bump to 0.0.55-20241204161503-a4c0b7d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/637 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:12:42] (03Merged) 10jenkins-bot: Update Patch demo repo [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1100495 (https://phabricator.wikimedia.org/T333073) (owner: 10Bartosz Dziewoński) [17:13:21] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 13Patch-For-Review: decommission cloudcephmon100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T380893#10380484 (10Andrew) [17:16:39] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' (T380893) [17:16:39] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) on host 'cloudvirt1035.eqiad.wmnet' (T380893) [17:16:45] T380893: decommission cloudcephmon100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T380893 [17:16:45] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' [17:18:04] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' [17:20:35] (03update) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/45 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:21:56] (03update) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/45 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:22:36] (03update) 10sstefanova: envvars-admission: bump to 0.0.23-20241204165719-1295007c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/639 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:22:43] (03update) 10sstefanova: envvars-admission: bump to 0.0.23-20241204165719-1295007c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/639 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:23:18] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [17:30:06] FIRING: ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_toolserver_org_redirects_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:30:22] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [17:30:53] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [17:35:06] RESOLVED: ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_toolserver_org_redirects_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:36:06] FIRING: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:38:39] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [17:38:40] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/7 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:38:42] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/7 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:38:45] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/7 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:41:06] FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:51:06] RESOLVED: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:53:36] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS (Quota-requests): Subdomain for catalyst-dev project - https://phabricator.wikimedia.org/T381508#10380751 (10fnegri) Docs are here: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Web_proxy#Enable_per-project_subdomain_delegation [18:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:17:30] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10380834 [18:21:06] FIRING: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:26:06] FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:29:46] 10VPS-project-Codesearch: Please add patchdemo to codesearch index - https://phabricator.wikimedia.org/T333073#10380886 (10matmarex) 05Open→03Resolved [18:32:23] 06cloud-services-team: Kernel error Server cloudvirt1035 may have kernel errors - https://phabricator.wikimedia.org/T381500#10380910 (10fnegri) 05Open→03Resolved a:03fnegri This was rebooted today by @Andrew (T380893) and a kernel error was logged during shutdown: ` root@cloudvirt1035:~# journalctl --... [18:51:06] RESOLVED: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:07:34] FIRING: DiskSpace: Disk space cloudbackup1002-dev:9100:/srv/cinder-backups 0% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1002-dev - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [19:08:56] FIRING: SystemdUnitDown: The service unit backup_cinder_volumes.service is in failed status on host cloudbackup1002-dev. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1002-dev - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [19:11:16] FIRING: ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:13:12] 10Cloud-Services, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54 - https://phabricator.wikimedia.org/T380503#10381052 (10VRiley-WMF) Understood, I will close this this and ask for a replacement! [19:13:24] 10Cloud-Services, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54 - https://phabricator.wikimedia.org/T380503#10381053 (10VRiley-WMF) 05Open→03Resolved [19:13:31] (03approved) 10sstefanova: envvars-admission: bump to 0.0.23-20241204165719-1295007c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/639 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [19:13:34] (03merge) 10sstefanova: envvars-admission: bump to 0.0.23-20241204165719-1295007c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/639 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [19:13:56] FIRING: [2x] SystemdUnitDown: The service unit backup_cinder_volumes.service is in failed status on host cloudbackup1002-dev. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1002-dev - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [19:14:05] (03update) 10sstefanova: registry-admission: bump to 0.0.55-20241204165756-6cff3221 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/638 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [19:14:50] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [19:14:56] (03update) 10sstefanova: registry-admission: bump to 0.0.55-20241204165756-6cff3221 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/638 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [19:15:57] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/78 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [19:16:16] RESOLVED: ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:18:24] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/78 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [19:18:28] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/78 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [19:19:39] FIRING: [3x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:21:43] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [19:22:35] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/11 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [19:22:37] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/11 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [19:22:40] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/11 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [19:23:33] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [19:24:38] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: jobs-emailer: bump to 0.0.45-20241204192251-ef3470d9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/640 [19:24:39] FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:24:40] PROBLEM - Disk space on cloudbackup1002-dev is CRITICAL: DISK CRITICAL - free space: /srv/cinder-backups 0MiB (0% inode=96%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=cloudbackup1002-dev&var-datasource=eqiad+prometheus/ops [19:26:11] !log sstefanova@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission [19:26:41] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [19:33:55] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [19:35:24] (03approved) 10sstefanova: registry-admission: bump to 0.0.55-20241204165756-6cff3221 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/638 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [19:35:26] (03merge) 10sstefanova: registry-admission: bump to 0.0.55-20241204165756-6cff3221 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/638 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [19:35:52] (03update) 10sstefanova: jobs-emailer: bump to 0.0.45-20241204192251-ef3470d9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/640 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [19:36:12] (03update) 10sstefanova: jobs-emailer: bump to 0.0.45-20241204192251-ef3470d9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/640 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [19:37:07] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [19:37:14] !log sstefanova@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer [19:38:22] (03update) 10sstefanova: jobs-emailer: bump to 0.0.45-20241204192251-ef3470d9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/640 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [19:45:33] (03update) 10sstefanova: add prometheus stats [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/10 (https://phabricator.wikimedia.org/T320284 https://phabricator.wikimedia.org/T379924) (owner: 10dcaro) [19:45:40] (03update) 10sstefanova: add prometheus stats [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/10 (https://phabricator.wikimedia.org/T320284 https://phabricator.wikimedia.org/T379924) (owner: 10dcaro) [19:59:39] FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:04:39] FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:19:39] FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:22:34] RESOLVED: DiskSpace: Disk space cloudbackup1002-dev:9100:/srv/cinder-backups 0% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1002-dev - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [20:24:39] RESOLVED: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:24:41] RECOVERY - Disk space on cloudbackup1002-dev is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=cloudbackup1002-dev&var-datasource=eqiad+prometheus/ops [20:50:16] FIRING: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:55:16] RESOLVED: [3x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:20:06] FIRING: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:25:06] FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:28:17] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10381549 [21:35:06] RESOLVED: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:32:06] FIRING: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:37:06] FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:42:00] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS (Quota-requests): Subdomain for catalyst-dev project - https://phabricator.wikimedia.org/T381508#10381710 (10bd808) [x] Delegate the specific subdomain in Designate to the project using wmcs-makedomain `lang=shell-session root@cloudcontrol1005:~# wmcs-make... [22:47:06] RESOLVED: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:53:28] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS (Quota-requests): Subdomain for catalyst-dev project - https://phabricator.wikimedia.org/T381508#10381726 (10bd808) I figured out that the `id` refers to the Designate zone record. For catalyst-dev.wmcloud.org that is `35699886-add9-4ad3-88d7-5b2829f8c72c`... [23:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:07:57] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS (Quota-requests): Subdomain for catalyst-dev project - https://phabricator.wikimedia.org/T381508#10381766 (10bd808) 05Open→03Resolved a:05fnegri→03bd808 The `profile::wmcs::novaproxy::supported_zones` data from T381508#10381726 was not quite ri... [23:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks