[00:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:48:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudidm2001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [04:25:39] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 05Goal: [toolsdb] Upgrade to MariaDB 10.6 - https://phabricator.wikimedia.org/T352206#10178220 (10Pintoch) The latest version of Django (5.1.1) only supports MariaDB 10.5 or higher, and ToolsDB is currently running 10.4.29, so that's a hurdle I enco... [05:17:09] (03PS5) 10Raymond Ndibe: [wmcs-cookbooks.depool_and_remove_node] force node delete with --force [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075348 (https://phabricator.wikimedia.org/T375158) [05:18:07] 10Toolforge (Toolforge iteration 15), 13Patch-For-Review: add --force to wmcs.toolforge.remove_k8s_node cookbook - https://phabricator.wikimedia.org/T375158#10178229 (10Raymond_Ndibe) 05Open→03In progress [05:34:49] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request: To strictly enforce semantic versioning rules for toolforge services' APIs or not - https://phabricator.wikimedia.org/T373072#10178232 (10Raymond_Ndibe) Initially I was on the side of **Option 4**. But maybe that's too extreme... [05:36:33] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge (Toolforge iteration 15): Decision Request: To strictly enforce semantic versioning rules for toolforge services' APIs or not - https://phabricator.wikimedia.org/T373072#10178233 (10Raymond_Ndibe) [06:20:12] (03open) 10raymond-ndibe: [volume-admission] update go.mod packages [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/18 (https://phabricator.wikimedia.org/T359641) [06:20:18] (03update) 10raymond-ndibe: [volume-admission] update go.mod packages [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/18 (https://phabricator.wikimedia.org/T359641) [06:20:57] (03update) 10raymond-ndibe: [volume-admission] update go.mod packages [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/18 (https://phabricator.wikimedia.org/T359641) [06:21:25] (03update) 10raymond-ndibe: [volume-admission] update go.mod packages [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/18 (https://phabricator.wikimedia.org/T359641) [06:23:46] (03open) 10raymond-ndibe: [registry-admission] update go.mod packages [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/14 (https://phabricator.wikimedia.org/T359641) [06:30:14] (03open) 10raymond-ndibe: [ingress-admission] update go.mod packages [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/11 (https://phabricator.wikimedia.org/T359641) [06:30:22] (03update) 10raymond-ndibe: [ingress-admission] update go.mod packages [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/11 (https://phabricator.wikimedia.org/T359641) [06:40:05] (03update) 10raymond-ndibe: [toolforge-weld] move _display_message into toolforge weld [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/46 [06:43:55] !log raymondndibe@wmf3402 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [06:43:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [06:44:22] !log raymondndibe@wmf3402 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld [06:44:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [06:48:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudidm2001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [06:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:52:04] !log raymondndibe@wmf3402 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [06:52:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [06:52:34] !log raymondndibe@wmf3402 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld [06:52:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [06:53:47] (03approved) 10raymond-ndibe: [toolforge-weld] move _display_message into toolforge weld [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/46 [06:54:04] (03merge) 10raymond-ndibe: [toolforge-weld] move _display_message into toolforge weld [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/46 [06:54:28] (03approved) 10raymond-ndibe: [envvars-cli] remove display_messages [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/57 [06:54:35] (03merge) 10raymond-ndibe: [envvars-cli] remove display_messages [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/57 [06:54:40] (03approved) 10raymond-ndibe: [jobs-cli] remove _display_messages [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/62 [06:54:43] (03merge) 10raymond-ndibe: [jobs-cli] remove _display_messages [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/62 [06:54:54] (03approved) 10raymond-ndibe: [builds-cli] remove _display_messages [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/69 [06:54:57] (03merge) 10raymond-ndibe: [builds-cli] remove _display_messages [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/69 [06:58:48] (03PS1) 10Raymond Ndibe: [wmcs-cookbooks.toolforge_deploy] allow for deploying toolforge-weld [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075777 [07:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:17:53] (03open) 10raymond-ndibe: d/changelog: bump to 1.6.2 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/55 [07:27:30] (03open) 10raymond-ndibe: d/changelog: bump to 0.0.19 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/89 [07:37:08] (03open) 10raymond-ndibe: d/changelog: bump to 16.1.4 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/70 [07:37:34] (03open) 10raymond-ndibe: d/changelog: bump to 0.0.11 [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/62 [07:40:03] !log raymondndibe@wmf3402 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [07:40:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [07:42:33] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814) [07:42:34] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814) [07:42:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [07:42:41] T372814: Put cloudcephosd10[39-41] into service - https://phabricator.wikimedia.org/T372814 [07:42:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [07:45:22] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814) [07:45:23] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814) [07:45:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [07:45:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [07:45:45] !log raymondndibe@wmf3402 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld [07:45:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [07:46:07] !log raymondndibe@wmf3402 tools START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [07:46:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [07:46:19] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814) [07:46:24] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814) [07:46:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [07:46:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [07:47:03] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814) [07:47:06] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814) [07:47:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [07:47:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [07:48:03] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814) [07:48:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [07:48:10] T372814: Put cloudcephosd10[39-41] into service - https://phabricator.wikimedia.org/T372814 [07:53:32] !log raymondndibe@wmf3402 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld [07:53:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [07:53:42] (03PS1) 10David Caro: proxy: add socks_proxy_enable option to toggle on the proxy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075843 [07:54:44] (03approved) 10raymond-ndibe: d/changelog: bump to 1.6.2 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/55 [07:54:51] (03merge) 10raymond-ndibe: d/changelog: bump to 1.6.2 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/55 [07:55:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [07:56:08] !log raymondndibe@wmf3402 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-cli [07:56:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [07:58:42] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Put cloudcephosd10[39-41] into service - https://phabricator.wikimedia.org/T372814#10178399 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dcaro@cumin1002 for host cloudcephosd1040.eqiad.wmnet with OS bul... [07:59:02] !log raymondndibe@wmf3402 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli [07:59:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [07:59:31] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814) [07:59:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [07:59:38] T372814: Put cloudcephosd10[39-41] into service - https://phabricator.wikimedia.org/T372814 [08:04:33] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814) [08:04:34] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814) [08:04:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:04:42] T372814: Put cloudcephosd10[39-41] into service - https://phabricator.wikimedia.org/T372814 [08:04:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:05:57] (03CR) 10Arturo Borrero Gonzalez: [C:03+1] "LGTM." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075843 (owner: 10David Caro) [08:07:13] (03update) 10raymond-ndibe: toolforge-weld: add build_deb.sh [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/23 [08:14:08] (03PS2) 10David Caro: proxy: add socks_proxy_enable option to toggle on the proxy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075843 [08:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:34:17] (03open) 10aborrero: secgroups: manage_default_secgroups yes by default [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/61 (https://phabricator.wikimedia.org/T357111) [08:39:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-idp-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [08:41:38] (03open) 10raymond-ndibe: [toolforge-deploy] update builds-api test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/532 [08:41:49] (03update) 10raymond-ndibe: [toolforge-deploy] update builds-api test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/532 [08:45:56] 06cloud-services-team, 10Cloud-VPS: tofu-infra: migrate default zone creation from the keystone hook - https://phabricator.wikimedia.org/T375720 (10aborrero) 03NEW [08:48:52] 06cloud-services-team, 10Cloud-VPS: tofu-infra: migrate default zone creation from the keystone hook - https://phabricator.wikimedia.org/T375720#10178539 (10aborrero) p:05Triage→03Medium [08:51:34] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge: [toolforge-prometheus] upgrade to bookworm - https://phabricator.wikimedia.org/T375523#10178545 (10taavi) [08:52:51] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814) [08:52:52] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814) [08:53:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:53:02] T372814: Put cloudcephosd10[39-41] into service - https://phabricator.wikimedia.org/T372814 [08:53:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:53:30] (03CR) 10David Caro: [C:03+2] proxy: add socks_proxy_enable option to toggle on the proxy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075843 (owner: 10David Caro) [08:54:32] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814) [08:54:33] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814) [08:54:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:54:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:54:56] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814) [08:54:56] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814) [08:55:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:55:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:55:26] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814) [08:55:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:55:34] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814) [08:55:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:55:58] (03PS3) 10David Caro: proxy: add socks_proxy_enable option to toggle on the proxy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075843 [08:55:59] (03PS1) 10David Caro: proxy: fix negative condition [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075856 [08:56:20] (03CR) 10David Caro: [C:03+2] proxy: fix negative condition [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075856 (owner: 10David Caro) [08:56:38] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814) [08:56:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:56:45] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814) [08:56:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:56:53] (03PS2) 10David Caro: proxy: fix negative condition [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075856 [08:57:35] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814) [08:57:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:59:01] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: Put cloudcephosd10[39-41] into service - https://phabricator.wikimedia.org/T372814#10178564 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dcaro@cumin1002 for host cloudcephosd1040.eqiad.wmnet with OS bullseye executed with errors... [08:59:18] (03CR) 10CI reject: [V:04-1] proxy: fix negative condition [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075856 (owner: 10David Caro) [08:59:30] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: Put cloudcephosd10[39-41] into service - https://phabricator.wikimedia.org/T372814#10178568 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dcaro@cumin1002 for host cloudcephosd1040.eqiad.wmnet with OS bullseye [09:00:18] (03Merged) 10jenkins-bot: proxy: add socks_proxy_enable option to toggle on the proxy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075843 (owner: 10David Caro) [09:00:18] (03CR) 10jenkins-bot: proxy: fix negative condition [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075856 (owner: 10David Caro) [09:20:00] (03approved) 10raymond-ndibe: [toolforge-deploy] update builds-api test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/532 [09:20:39] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [09:23:33] (03approved) 10dcaro: [toolforge-deploy] update builds-api test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/532 (owner: 10raymond-ndibe) [09:24:02] (03merge) 10raymond-ndibe: [toolforge-deploy] update builds-api test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/532 [09:31:55] 06cloud-services-team, 06Infrastructure-Foundations: Which team should receive alerts for cloudidm2001-dev? - https://phabricator.wikimedia.org/T375723 (10fnegri) 03NEW [09:40:24] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: Put cloudcephosd10[39-41] into service - https://phabricator.wikimedia.org/T372814#10178711 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dcaro@cumin1002 for host cloudcephosd1040.eqiad.wmnet with OS bullseye completed: - cloudce... [09:43:32] 06cloud-services-team, 06Infrastructure-Foundations, 13Patch-For-Review: Which team should receive alerts for cloudidm2001-dev? - https://phabricator.wikimedia.org/T375723#10178721 (10SLyngshede-WMF) Let's us just assign this host to IF, we'll be getting the questions in any case, so there's no point in goin... [09:44:14] 06cloud-services-team, 06Infrastructure-Foundations, 13Patch-For-Review: Which team should receive alerts for cloudidm2001-dev? - https://phabricator.wikimedia.org/T375723#10178726 (10SLyngshede-WMF) 05Open→03Resolved [09:46:09] 10cloud-services-team (FY2024/2025-Q1-Q2): [cloudinfra] Upgrade cloudinfra-idp-* to bookworm - https://phabricator.wikimedia.org/T373840#10178733 (10fnegri) Puppet is currently failing in the old bullseye instance `cloudinfra-idp-1`: ` Sep 26 09:23:03 cloudinfra-idp-1 puppet-agent[1050485]: Could not retrieve c... [09:52:12] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: Put cloudcephosd10[39-41] into service - https://phabricator.wikimedia.org/T372814#10178743 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dcaro@cumin1002 for host cloudcephosd1041.eqiad.wmnet with OS bullseye [09:59:08] !log raymondndibe@wmf3402 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli [09:59:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:00:06] (03approved) 10raymond-ndibe: d/changelog: bump to 0.0.19 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/89 [10:00:11] (03merge) 10raymond-ndibe: d/changelog: bump to 0.0.19 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/89 [10:04:40] !log raymondndibe@wmf3402 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli [10:04:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:05:00] !log raymondndibe@wmf3402 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli [10:05:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:11:41] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814) [10:11:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:11:48] T372814: Put cloudcephosd10[39-41] into service - https://phabricator.wikimedia.org/T372814 [10:12:04] !log raymondndibe@wmf3402 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli [10:12:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:12:36] !log raymondndibe@wmf3402 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [10:12:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:15:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [10:20:07] !log raymondndibe@wmf3402 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [10:20:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [10:20:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:20:36] !log raymondndibe@wmf3402 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [10:20:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:23:13] (03approved) 10raymond-ndibe: d/changelog: bump to 0.0.11 [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/62 [10:23:18] (03merge) 10raymond-ndibe: d/changelog: bump to 0.0.11 [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/62 [10:23:53] (03CR) 10David Caro: [C:03+2] proxy: fix negative condition [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075856 (owner: 10David Caro) [10:24:11] (03CR) 10Raymond Ndibe: [C:03+2] [wmcs-cookbooks.depool_and_remove_node] force node delete with --force [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075348 (https://phabricator.wikimedia.org/T375158) (owner: 10Raymond Ndibe) [10:25:09] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814) [10:25:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:25:16] T372814: Put cloudcephosd10[39-41] into service - https://phabricator.wikimedia.org/T372814 [10:25:21] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) (T372814) [10:25:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:26:13] (03CR) 10Raymond Ndibe: [C:03+2] toolforge.component.deploy: run tests by default [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1072162 (owner: 10David Caro) [10:26:35] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.undrain_node (T372814) [10:26:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:26:53] !log raymondndibe@wmf3402 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [10:26:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:27:54] (03approved) 10raymond-ndibe: d/changelog: bump to 16.1.4 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/70 [10:27:57] (03merge) 10raymond-ndibe: d/changelog: bump to 16.1.4 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/70 [10:28:33] (03CR) 10David Caro: [C:04-1] wmcs.vps.create_project: replace logic with message about deprecation (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1069994 (https://phabricator.wikimedia.org/T371393) (owner: 10Arturo Borrero Gonzalez) [10:33:36] (03Merged) 10jenkins-bot: proxy: fix negative condition [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075856 (owner: 10David Caro) [10:33:36] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: Put cloudcephosd10[39-41] into service - https://phabricator.wikimedia.org/T372814#10178851 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dcaro@cumin1002 for host cloudcephosd1041.eqiad.wmnet with OS bullseye completed: - cloudce... [10:33:37] (03Merged) 10jenkins-bot: [wmcs-cookbooks.depool_and_remove_node] force node delete with --force [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075348 (https://phabricator.wikimedia.org/T375158) (owner: 10Raymond Ndibe) [10:33:49] (03Merged) 10jenkins-bot: toolforge.component.deploy: run tests by default [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1072162 (owner: 10David Caro) [10:48:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudidm2001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [10:51:24] (03open) 10raymond-ndibe: [toolforge-weld] display_messages default True [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/56 [11:15:54] (03open) 10aborrero: eqiad1: default secgroup: allow prometheus from metricsinfra [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/62 (https://phabricator.wikimedia.org/T375111) [11:46:56] 10Tool-video-answer-tool, 06Future-Audiences: Update DYK dataset to include info on which also reached GA or FA - https://phabricator.wikimedia.org/T375731 (10Maryana) 03NEW [11:53:10] 10Tool-video-answer-tool, 06Future-Audiences, 07Spike: Investigate On This Day datasets available - https://phabricator.wikimedia.org/T375733 (10Maryana) 03NEW [12:00:25] (03PS1) 10David Caro: wmcs.vps.create_project: replace logic with message about deprecation [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075892 (https://phabricator.wikimedia.org/T371393) [12:00:27] (03PS1) 10David Caro: ceph.osd.bootstrap_and_add: make sure batch-size is an int [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075893 [12:04:08] (03CR) 10CI reject: [V:04-1] ceph.osd.bootstrap_and_add: make sure batch-size is an int [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075893 (owner: 10David Caro) [12:04:24] (03CR) 10CI reject: [V:04-1] wmcs.vps.create_project: replace logic with message about deprecation [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075892 (https://phabricator.wikimedia.org/T371393) (owner: 10David Caro) [12:08:24] (03PS2) 10David Caro: wmcs.vps.create_project: replace logic with message about deprecation [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075892 (https://phabricator.wikimedia.org/T371393) [12:11:07] 10cloud-services-team (FY2024/2025-Q1-Q2), 06Infrastructure-Foundations, 10netops: cloud: edge network suffers downtime if one cloudsw is down - https://phabricator.wikimedia.org/T375259#10179065 (10ayounsi) It would be useful to capture more data (eg. packet capture) next time this happens. The ICMP no rout... [12:13:00] (03CR) 10CI reject: [V:04-1] wmcs.vps.create_project: replace logic with message about deprecation [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075892 (https://phabricator.wikimedia.org/T371393) (owner: 10David Caro) [12:13:04] (03CR) 10CI reject: [V:04-1] ceph.osd.bootstrap_and_add: make sure batch-size is an int [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075893 (owner: 10David Caro) [12:14:16] (03PS3) 10David Caro: wmcs.vps.create_project: replace logic with message about deprecation [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075892 (https://phabricator.wikimedia.org/T371393) [12:14:16] (03PS3) 10David Caro: ceph.osd.bootstrap_and_add: make sure batch-size is an int [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075893 [12:28:10] 10cloud-services-team (FY2024/2025-Q1-Q2), 06Infrastructure-Foundations, 10netops: cloud: edge network suffers downtime if one cloudsw is down - https://phabricator.wikimedia.org/T375259#10179194 (10ayounsi) A few more info thanks to @aborrero on IRC. After 185.15.56.244, the packets towards 185.15.56.57 ar... [12:33:57] FIRING: SystemdUnitDown: The service unit wikitech_run_jobs.service is in failed status on host cloudweb1004. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudweb1004 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [12:38:57] RESOLVED: SystemdUnitDown: The service unit wikitech_run_jobs.service is in failed status on host cloudweb1004. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudweb1004 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [12:47:06] 10cloud-services-team (FY2024/2025-Q1-Q2), 06Infrastructure-Foundations, 10netops: cloud: edge network suffers downtime if one cloudsw is down - https://phabricator.wikimedia.org/T375259#10179269 (10ayounsi) Actually... `ssh: connect to host login.toolforge.org port 22: No route to host` is a red hearing, S... [12:49:46] (03CR) 10David Caro: [C:04-1] wmcs.vps.create_project: replace logic with message about deprecation (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1069994 (https://phabricator.wikimedia.org/T371393) (owner: 10Arturo Borrero Gonzalez) [12:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:03:55] 10cloud-services-team (FY2024/2025-Q1-Q2), 06Infrastructure-Foundations, 10netops: cloud: edge network suffers downtime if one cloudsw is down - https://phabricator.wikimedia.org/T375259#10179344 (10aborrero) In case they are useful, keepalived VRRP logs can be seen here: {P69421} [13:06:37] (03CR) 10David Caro: [C:03+2] "Tested locally:" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075893 (owner: 10David Caro) [13:06:47] (03PS4) 10David Caro: ceph.osd.bootstrap_and_add: make sure batch-size is an int [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075893 [13:06:55] (03CR) 10David Caro: ceph.osd.bootstrap_and_add: make sure batch-size is an int [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075893 (owner: 10David Caro) [13:11:29] (03Merged) 10jenkins-bot: ceph.osd.bootstrap_and_add: make sure batch-size is an int [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075893 (owner: 10David Caro) [13:14:08] 06cloud-services-team, 06Infrastructure-Foundations: Which team should receive alerts for cloudidm2001-dev? - https://phabricator.wikimedia.org/T375723#10179405 (10fnegri) Thanks @SLyngshede-WMF for the patch! If you haven't seen it already, there is currently an alert firing for this host: > Puppet has f... [13:20:04] (03CR) 10David Caro: [C:03+1] "LGTM, have you tested it yet?" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075777 (owner: 10Raymond Ndibe) [13:35:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:40:15] 10Data-Services, 06Data-Engineering, 06SRE, 06Trust and Safety Product Team, and 3 others: Hide the value of gb_address column in public replicas if gb_autoblock_parent_id is not null - https://phabricator.wikimedia.org/T371486#10179581 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-view... [13:45:22] 10Data-Services, 06Data-Engineering, 06SRE, 06Trust and Safety Product Team, and 3 others: Hide the value of gb_address column in public replicas if gb_autoblock_parent_id is not null - https://phabricator.wikimedia.org/T371486#10179597 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-view... [13:50:14] 10Data-Services, 06Data-Engineering, 06SRE, 06Trust and Safety Product Team, and 3 others: Hide the value of gb_address column in public replicas if gb_autoblock_parent_id is not null - https://phabricator.wikimedia.org/T371486#10179621 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-view... [14:02:21] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: Communication for Wikitech/Wikimedia Developer Account migration - https://phabricator.wikimedia.org/T373615#10179685 (10joanna_borun) [14:04:12] 10Data-Services, 06Data-Engineering, 06SRE, 06Trust and Safety Product Team, and 3 others: Hide the value of gb_address column in public replicas if gb_autoblock_parent_id is not null - https://phabricator.wikimedia.org/T371486#10179700 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-view... [14:12:02] 10Data-Services, 06Data-Engineering, 06SRE, 06Trust and Safety Product Team, and 3 others: Hide the value of gb_address column in public replicas if gb_autoblock_parent_id is not null - https://phabricator.wikimedia.org/T371486#10179747 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-view... [14:14:19] 10Data-Services, 06Data-Engineering, 06SRE, 06Trust and Safety Product Team, and 3 others: Hide the value of gb_address column in public replicas if gb_autoblock_parent_id is not null - https://phabricator.wikimedia.org/T371486#10179759 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-view... [14:15:46] 10Data-Services, 06Data-Engineering, 06SRE, 06Trust and Safety Product Team, and 3 others: Hide the value of gb_address column in public replicas if gb_autoblock_parent_id is not null - https://phabricator.wikimedia.org/T371486#10179767 (10fnegri) The cookbook is failing for a bunch of different reasons th... [14:17:02] 10Data-Services, 06Data-Engineering, 06SRE, 06Trust and Safety Product Team, and 3 others: Hide the value of gb_address column in public replicas if gb_autoblock_parent_id is not null - https://phabricator.wikimedia.org/T371486#10179781 (10Dreamy_Jazz) >>! In T371486#10179767, @fnegri wrote: > The cookbook... [14:28:10] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) (T372814) [14:28:12] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.undrain_node (T372814) [14:28:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:28:18] T372814: Put cloudcephosd10[39-41] into service - https://phabricator.wikimedia.org/T372814 [14:28:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:29:47] (03open) 10raymond-ndibe: [jobs-cli] remove display_messages default from cli [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/71 [14:31:36] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services: update-views cookbook doesn't handle filters correctly - https://phabricator.wikimedia.org/T375760 (10fnegri) 03NEW [14:33:02] 10Data-Services, 06Data-Engineering, 06SRE, 06Trust and Safety Product Team, and 2 others: Hide the value of gb_address column in public replicas if gb_autoblock_parent_id is not null - https://phabricator.wikimedia.org/T371486#10179872 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-view... [14:41:54] 10Data-Services, 06Data-Engineering, 06SRE, 06Trust and Safety Product Team, and 2 others: Hide the value of gb_address column in public replicas if gb_autoblock_parent_id is not null - https://phabricator.wikimedia.org/T371486#10179905 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-view... [14:48:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudidm2001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [14:52:17] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [14:52:58] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [14:53:16] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 13Patch-For-Review: update-views cookbook doesn't handle filters correctly - https://phabricator.wikimedia.org/T375760#10179952 (10fnegri) p:05Triage→03Medium [15:01:43] FIRING: Test-Rule: Testing transport from LibreNMS - https://alerts.wikimedia.org/?q=alertname%3DTest-Rule [15:01:49] 06cloud-services-team: Test-Rule This is a test alert - https://phabricator.wikimedia.org/T375767 (10phaultfinder) 03NEW [15:02:53] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 13Patch-For-Review: openstack: vxlan: verify nova proxy and floating IPs work with new VXLAN-based network - https://phabricator.wikimedia.org/T374828#10179995 (10aborrero) 05In progress→03Resolved nova proxy also works! `lang=shell-session $ c... [15:04:39] RECOVERY - Host cloudcephosd1025 is UP: PING WARNING - Packet loss = 66%, RTA = 35.95 ms [15:04:47] (03PS1) 10Vgutierrez: secrets: Add digicert-2024 dummy files [labs/private] - 10https://gerrit.wikimedia.org/r/1075938 [15:05:18] PROBLEM - SSH on cloudcephosd1025 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [15:05:56] (03CR) 10Ssingh: [C:03+1] "[-1, no cliche snake oil string]" [labs/private] - 10https://gerrit.wikimedia.org/r/1075938 (owner: 10Vgutierrez) [15:06:58] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-idp-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [15:07:57] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10180059 (10jcrespo) Hey, I don't know how the process is ongoing but @fnegri found a few centralauth tables on labswiki while sanitizing wikireplicas... [15:11:03] PROBLEM - Host cloudcephosd1025 is DOWN: PING CRITICAL - Packet loss = 100% [15:15:44] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10180118 (10fnegri) I only found the table `globalblocks` so far in the `labswiki` db, and it's empty, but there may be others. [15:18:09] (03approved) 10dcaro: [volume-admission] update go.mod packages [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/18 (https://phabricator.wikimedia.org/T359641) (owner: 10raymond-ndibe) [15:18:13] (03merge) 10dcaro: [volume-admission] update go.mod packages [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/18 (https://phabricator.wikimedia.org/T359641) (owner: 10raymond-ndibe) [15:21:12] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: volume-admission: bump to 0.0.56-20240926151825-d311e795 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/533 (https://phabricator.wikimedia.org/T359641) [15:23:02] 10Data-Services, 06Data-Engineering, 06SRE, 06Trust and Safety Product Team, and 2 others: Hide the value of gb_address column in public replicas if gb_autoblock_parent_id is not null - https://phabricator.wikimedia.org/T371486#10180139 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-view... [15:24:56] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission (T359641) [15:25:01] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [15:26:28] (03CR) 10Vgutierrez: [V:03+2 C:03+2] secrets: Add digicert-2024 dummy files [labs/private] - 10https://gerrit.wikimedia.org/r/1075938 (owner: 10Vgutierrez) [15:29:56] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission (T359641) [15:30:01] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [15:32:14] 10Data-Services, 06Data-Engineering, 06SRE, 06Trust and Safety Product Team, and 2 others: Hide the value of gb_address column in public replicas if gb_autoblock_parent_id is not null - https://phabricator.wikimedia.org/T371486#10180195 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-view... [15:38:35] 10Data-Services, 06Data-Engineering, 06SRE, 06Trust and Safety Product Team, and 2 others: Hide the value of gb_address column in public replicas if gb_autoblock_parent_id is not null - https://phabricator.wikimedia.org/T371486#10180226 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-view... [15:40:05] 10Data-Services, 06Data-Engineering, 06SRE, 06Trust and Safety Product Team, and 2 others: Hide the value of gb_address column in public replicas if gb_autoblock_parent_id is not null - https://phabricator.wikimedia.org/T371486#10180243 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-view... [15:43:16] (03approved) 10raymond-ndibe: [ingress-admission] update go.mod packages [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/11 (https://phabricator.wikimedia.org/T359641) [15:43:21] (03merge) 10raymond-ndibe: [ingress-admission] update go.mod packages [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/11 (https://phabricator.wikimedia.org/T359641) [15:43:26] (03approved) 10raymond-ndibe: [registry-admission] update go.mod packages [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/14 (https://phabricator.wikimedia.org/T359641) [15:43:30] (03merge) 10raymond-ndibe: [registry-admission] update go.mod packages [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/14 (https://phabricator.wikimedia.org/T359641) [15:46:59] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: ingress-admission: bump to 0.0.51-20240926154329-19cfe59e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/535 (https://phabricator.wikimedia.org/T359641) [15:46:59] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: registry-admission: bump to 0.0.51-20240926154338-be8dc0fd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/534 (https://phabricator.wikimedia.org/T359641) [15:47:03] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: ingress-admission: bump to 0.0.51-20240926154329-19cfe59e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/535 (https://phabricator.wikimedia.org/T359641) [15:51:07] !log raymondndibe@wmf3402 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [15:51:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:55:08] (03open) 10dcaro: Update file README.md [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/536 [15:57:30] !log raymondndibe@wmf3402 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [15:57:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:58:12] !log raymondndibe@wmf3402 tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [15:58:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:59:48] (03update) 10dcaro: builds-buidler: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 [16:02:20] 10Data-Services: maintain-views: skip new databases that have not been sanitized yet - https://phabricator.wikimedia.org/T375779 (10fnegri) 03NEW [16:02:25] (03update) 10dcaro: tekton: upgrade to v0.59.3 [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/61 [16:02:49] 10Data-Services: maintain-views: skip new databases that have not been sanitized yet - https://phabricator.wikimedia.org/T375779#10180394 (10fnegri) p:05Triage→03Medium [16:05:30] !log raymondndibe@wmf3402 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [16:06:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:06:14] (03approved) 10raymond-ndibe: volume-admission: bump to 0.0.56-20240926151825-d311e795 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/533 (https://phabricator.wikimedia.org/T359641) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:07:06] (03merge) 10raymond-ndibe: volume-admission: bump to 0.0.56-20240926151825-d311e795 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/533 (https://phabricator.wikimedia.org/T359641) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:07:35] (03update) 10raymond-ndibe: registry-admission: bump to 0.0.51-20240926154338-be8dc0fd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/534 (https://phabricator.wikimedia.org/T359641) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:08:32] !log raymondndibe@wmf3402 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [16:08:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:08:38] 10Data-Services: maintain-views fails on labswiki - https://phabricator.wikimedia.org/T375780 (10fnegri) 03NEW [16:08:48] 10Data-Services: maintain-views fails on labswiki - https://phabricator.wikimedia.org/T375780#10180433 (10fnegri) p:05Triage→03Medium [16:16:52] 06cloud-services-team: Test-Rule This is a test alert - https://phabricator.wikimedia.org/T375767#10180454 (10dcaro) 05Open→03Resolved a:03dcaro [16:18:31] !log raymondndibe@wmf3402 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission [16:18:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:18:59] !log raymondndibe@wmf3402 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [16:19:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:20:32] 10Data-Services: maintain-views fails on labswiki - https://phabricator.wikimedia.org/T375780#10180469 (10fnegri) @bd808 perhaps you know what's the problem here? Did maintain-views ever work on labswiki? I see there are some views in labswiki_p but maybe they were not created with maintain-views? [16:24:40] !log raymondndibe@wmf3402 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [16:24:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:25:18] (03approved) 10raymond-ndibe: registry-admission: bump to 0.0.51-20240926154338-be8dc0fd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/534 (https://phabricator.wikimedia.org/T359641) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:25:23] (03merge) 10raymond-ndibe: registry-admission: bump to 0.0.51-20240926154338-be8dc0fd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/534 (https://phabricator.wikimedia.org/T359641) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:25:45] (03approved) 10fnegri: Update file README.md [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/536 (owner: 10dcaro) [16:26:51] (03update) 10raymond-ndibe: ingress-admission: bump to 0.0.51-20240926154329-19cfe59e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/535 (https://phabricator.wikimedia.org/T359641) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:27:04] (03update) 10dcaro: Update file README.md [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/536 [16:27:29] (03merge) 10dcaro: Update file README.md [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/536 [16:31:34] (03update) 10raymond-ndibe: ingress-admission: bump to 0.0.51-20240926154329-19cfe59e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/535 (https://phabricator.wikimedia.org/T359641) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:32:56] 10Data-Services, 06Data-Engineering, 06SRE, 06Trust and Safety Product Team, and 2 others: Hide the value of gb_address column in public replicas if gb_autoblock_parent_id is not null - https://phabricator.wikimedia.org/T371486#10180512 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-view... [16:33:09] !log raymondndibe@wmf3402 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission [16:33:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:34:17] 10Data-Services, 06Data-Engineering, 06SRE, 06Trust and Safety Product Team, and 2 others: Hide the value of gb_address column in public replicas if gb_autoblock_parent_id is not null - https://phabricator.wikimedia.org/T371486#10180514 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-view... [16:37:07] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services: update-views cookbook doesn't handle filters correctly - https://phabricator.wikimedia.org/T375760#10180515 (10fnegri) 05Open→03Resolved [16:39:26] !log raymondndibe@wmf3402 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission [16:39:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:40:32] !log raymondndibe@wmf3402 tools START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission [16:40:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:45:51] !log raymondndibe@wmf3402 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission [16:45:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:46:25] (03approved) 10raymond-ndibe: ingress-admission: bump to 0.0.51-20240926154329-19cfe59e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/535 (https://phabricator.wikimedia.org/T359641) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:46:31] (03merge) 10raymond-ndibe: ingress-admission: bump to 0.0.51-20240926154329-19cfe59e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/535 (https://phabricator.wikimedia.org/T359641) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:47:21] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 15), 13Patch-For-Review: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641#10180565 (10Raymond_Ndibe) [16:50:55] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 06Data-Engineering, 06SRE, and 3 others: Hide the value of gb_address column in public replicas if gb_autoblock_parent_id is not null - https://phabricator.wikimedia.org/T371486#10180544 (10fnegri) 05Open→03Resolved p:05Triage→03High a... [16:53:02] 10Data-Services: maintain-views fails on labswiki - https://phabricator.wikimedia.org/T375780#10180581 (10fnegri) [16:53:05] 10Data-Services: maintain-views fails on labswiki - https://phabricator.wikimedia.org/T375780#10180576 (10taavi) I think maintain-views should work fine on labswiki. The real question to me is why is it trying to run on a `globaluser` table on that wiki since that table only exists on centralauth. Is there more... [16:53:52] 10Data-Services: maintain-views fails on labswiki - https://phabricator.wikimedia.org/T375780#10180585 (10fnegri) Stack trace: ` fnegri@an-redacteddb1001:~$ sudo maintain-views --databases labswiki --table globalblocks 2024-09-26 16:48:37,690 INFO Full views for labswiki: 2024-09-26 16:48:37,691 INFO Custom vie... [16:59:34] 10wikitech.wikimedia.org, 06Data Products, 06Data-Engineering, 06DBA, 07Schema-change: Please drop globalblocks table from labswiki - https://phabricator.wikimedia.org/T375783#10180620 (10taavi) [17:00:40] 10Data-Services: maintain-views fails on labswiki - https://phabricator.wikimedia.org/T375780#10180617 (10taavi) Ah. For some reason `labswiki` has a `globalblocks` table (which should only exist on `centralauth`), and now that the `globalblocks` view definition depends on `globaluser`, the view creation fails t... [17:07:00] 10Data-Services: maintain-views fails on labswiki - https://phabricator.wikimedia.org/T375780#10180661 (10fnegri) Right thanks! I should have spotted that, given that I've looked at the new view definition a hundred times since last week :) [17:08:40] 10Data-Services: maintain-views fails on labswiki - https://phabricator.wikimedia.org/T375780#10180676 (10fnegri) [17:08:43] 10wikitech.wikimedia.org, 06Data Products, 06Data-Engineering, 06DBA, 07Schema-change: Please drop globalblocks table from labswiki - https://phabricator.wikimedia.org/T375783#10180677 (10fnegri) [17:12:13] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 06Data-Engineering, 06SRE, and 3 others: Hide the value of gb_address column in public replicas if gb_autoblock_parent_id is not null - https://phabricator.wikimedia.org/T371486#10180670 (10fnegri) > However, I checked and the globalblocks tab... [17:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:21:03] (03PS1) 10JHathaway: Move wikimediafoundation.org out of secret puppet [labs/private] - 10https://gerrit.wikimedia.org/r/1075984 [18:24:23] (03CR) 10JHathaway: [C:03+2] Move wikimediafoundation.org out of secret puppet [labs/private] - 10https://gerrit.wikimedia.org/r/1075984 (owner: 10JHathaway) [18:24:24] (03CR) 10JHathaway: [V:03+2 C:03+2] Move wikimediafoundation.org out of secret puppet [labs/private] - 10https://gerrit.wikimedia.org/r/1075984 (owner: 10JHathaway) [18:48:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudidm2001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:33:12] 10wikitech.wikimedia.org, 06SRE, 10WikimediaDebug: With XWikimediaDebug enabled, wikitech.wikimedia.org gets redirected to foundation.wikimedia.org until Wikitech is on k8s - https://phabricator.wikimedia.org/T375795#10181190 (10bd808) [21:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:36:13] 14Toolforge Build Service: [tbs][builder] Explore adding support for third-party buildpacks - https://phabricator.wikimedia.org/T352389#10181685 (10bd808) 05Resolved→03Declined a:05dcaro→03None This task is linked from https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Roadmap as a plan... [22:48:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudidm2001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [22:54:05] (03open) 10raymond-ndibe: [builds-cli] update README.md [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/90 [23:01:25] (03update) 10raymond-ndibe: [builds-cli] update README.md [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/90 [23:14:47] (03open) 10raymond-ndibe: [jobs-cli] update README.md [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/72 [23:34:53] 10Tool-video-answer-tool, 06Future-Audiences: Ensure Image Attribution Is Readable - https://phabricator.wikimedia.org/T375830 (10derenrich) 03NEW [23:41:34] 10PAWS: Unable to start - https://phabricator.wikimedia.org/T375831 (10Kizule) 03NEW [23:43:26] 10PAWS: Unable to start - https://phabricator.wikimedia.org/T375831#10181780 (10Kizule) Also, there isn't a logo, and it looks broken. {F57560548} [23:43:29] 10PAWS: Unable to start - https://phabricator.wikimedia.org/T375831#10181781 (10Kizule) p:05Triage→03High [23:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks