[00:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [00:15:38] (03open) 10raymond-ndibe: [envvars-cli] update README.md [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/63 [00:48:13] (03open) 10raymond-ndibe: [toolforge-weld] bump_version.sh auto create commit and tag [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/57 [00:51:55] (03CR) 10Raymond Ndibe: "Yes I have. It works. merging now" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075777 (owner: 10Raymond Ndibe) [00:52:02] (03CR) 10Raymond Ndibe: [C:03+2] [wmcs-cookbooks.toolforge_deploy] allow for deploying toolforge-weld [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075777 (owner: 10Raymond Ndibe) [00:55:54] (03Merged) 10jenkins-bot: [wmcs-cookbooks.toolforge_deploy] allow for deploying toolforge-weld [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075777 (owner: 10Raymond Ndibe) [01:25:49] FIRING: PuppetFailure: Puppet has failed on cloudvirt1067:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [01:25:57] 06cloud-services-team: PuppetFailure Puppet failure on cloudvirt1067:9100 - https://phabricator.wikimedia.org/T375836 (10phaultfinder) 03NEW [01:35:49] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudcephosd1032:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [01:40:49] FIRING: [2x] PuppetFailure: Puppet has failed on cloudvirt1031:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [01:40:49] FIRING: [3x] PuppetZeroResources: Puppet has failed generate resources on cloudcephosd1032:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [01:40:56] 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T375485#10181893 (10phaultfinder) [01:45:49] FIRING: [5x] PuppetFailure: Puppet has failed on cloudnet1006:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [01:46:01] 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T375485#10181895 (10phaultfinder) [01:50:49] FIRING: [7x] PuppetFailure: Puppet has failed on cloudbackup1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [01:50:57] FIRING: [5x] PuppetZeroResources: Puppet has failed generate resources on cloudcephosd1032:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [01:51:00] 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T375485#10181903 (10phaultfinder) [01:55:48] FIRING: [10x] PuppetFailure: Puppet has failed on cloudbackup1002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [01:55:53] 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T375485#10181906 (10phaultfinder) [02:00:53] 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T375485#10181907 (10phaultfinder) [02:00:58] FIRING: [13x] PuppetFailure: Puppet has failed on cloudbackup1002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [02:01:24] FIRING: [5x] PuppetZeroResources: Puppet has failed generate resources on cloudcephosd1032:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [02:05:48] FIRING: [14x] PuppetFailure: Puppet has failed on cloudbackup1002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [02:05:59] 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T375485#10181913 (10phaultfinder) [02:10:48] FIRING: [14x] PuppetFailure: Puppet has failed on cloudbackup1002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [02:11:02] FIRING: [5x] PuppetZeroResources: Puppet has failed generate resources on cloudcephosd1032:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [02:15:49] FIRING: [13x] PuppetFailure: Puppet has failed on cloudbackup1002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [02:15:57] FIRING: [5x] PuppetZeroResources: Puppet has failed generate resources on cloudcephosd1032:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [02:25:49] RESOLVED: [8x] PuppetFailure: Puppet has failed on cloudbackup1002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [02:25:49] RESOLVED: [3x] PuppetZeroResources: Puppet has failed generate resources on cloudvirt1033:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [02:48:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudidm2001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [03:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:56:41] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Cloud VPS: extend tofu-infra to cover projects, users and roles - https://phabricator.wikimedia.org/T371393#10181986 (10aborrero) 05In progress→03Open [05:56:42] 06cloud-services-team, 10Cloud-VPS, 07Epic: Cloud VPS: extend tofu-infra coverage - https://phabricator.wikimedia.org/T370037#10181988 (10aborrero) 05In progress→03Open [06:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:43:57] FIRING: SystemdUnitDown: The service unit wikitech_run_jobs.service is in failed status on host cloudweb1004. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudweb1004 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [07:49:02] RESOLVED: SystemdUnitDown: The service unit wikitech_run_jobs.service is in failed status on host cloudweb1004. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudweb1004 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [08:13:33] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: openstack: cleanup and prepare security groups - https://phabricator.wikimedia.org/T375604#10182135 (10aborrero) 05In progress→03Resolved [08:21:07] 06cloud-services-team, 06Infrastructure-Foundations, 10netops, 06SRE: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847 (10aborrero) 03NEW [08:22:38] 06cloud-services-team, 06Infrastructure-Foundations, 10netops, 06SRE: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847#10182170 (10aborrero) 05Open→03In progress p:05Triage→03Medium [08:23:40] 06cloud-services-team, 06Infrastructure-Foundations, 10netops, 06SRE: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847#10182174 (10aborrero) [08:28:55] 06cloud-services-team, 06Infrastructure-Foundations, 10netops, 06SRE: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847#10182196 (10aborrero) [08:51:30] (03open) 10aborrero: codfw1dev: enable IPv6 in the vxlan flat network [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/63 (https://phabricator.wikimedia.org/T375847) [08:54:12] (03update) 10aborrero: codfw1dev: enable IPv6 in the vxlan flat network [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/63 (https://phabricator.wikimedia.org/T375847) [08:56:33] (03update) 10aborrero: codfw1dev: enable IPv6 in the vxlan flat network [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/63 (https://phabricator.wikimedia.org/T375847) [09:03:01] 10wikitech.wikimedia.org, 06Data Products, 06Data-Engineering, 06DBA, 07Schema-change: Please drop globalblocks table from labswiki - https://phabricator.wikimedia.org/T375783#10182256 (10Dreamy_Jazz) Once `labswiki` is merged into the production cluster, I presume we should be able to use the `centralau... [09:10:00] (03approved) 10fnegri: codfw1dev: enable IPv6 in the vxlan flat network [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/63 (https://phabricator.wikimedia.org/T375847) (owner: 10aborrero) [09:14:50] (03CR) 10FNegri: wmcs.vps.create_project: replace logic with message about deprecation (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075892 (https://phabricator.wikimedia.org/T371393) (owner: 10David Caro) [09:19:20] 10PAWS: Unable to start - https://phabricator.wikimedia.org/T375831#10182291 (10rook) Could you try this again? I seem to be able to log in on my account. It may be the case that quay.io was offline when you tried to connect to paws. [09:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:28:45] 10Data-Services: maintain-views fails on labswiki - https://phabricator.wikimedia.org/T375780#10182314 (10fnegri) [09:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:45:38] 10Data-Services: maintain-views fails on labswiki - https://phabricator.wikimedia.org/T375780#10182349 (10fnegri) [09:52:34] (03merge) 10aborrero: codfw1dev: enable IPv6 in the vxlan flat network [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/63 (https://phabricator.wikimedia.org/T375847) [09:55:41] (03open) 10aborrero: codfw1dev: subnet: specify ip_version as 6 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/64 (https://phabricator.wikimedia.org/T375847) [09:56:03] (03update) 10aborrero: codfw1dev: subnet: specify ip_version as 6 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/64 (https://phabricator.wikimedia.org/T375847) [09:57:33] 10Cloud-VPS: Frequent radosgw 500 errors with OpenTofu - https://phabricator.wikimedia.org/T360626#10182376 (10aborrero) this happened today in eqiad1. My previous theory that this only happened for codfw1dev is invalidated. [09:57:56] (03merge) 10aborrero: codfw1dev: subnet: specify ip_version as 6 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/64 (https://phabricator.wikimedia.org/T375847) [10:04:53] 06cloud-services-team, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847#10182409 (10aborrero) new instance creation will allocate an IPv6 by default for a VM: {F57561880} [10:14:51] 06cloud-services-team, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847#10182457 (10aborrero) however, instance creation itself failed: ` 2024-09-27 10:07:20.088 451966 ERROR nova.compute.manager [N... [10:14:52] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [10:16:04] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [10:37:34] 10PAWS: Unable to start - https://phabricator.wikimedia.org/T375831#10182499 (10Kizule) 05Open→03Invalid Yup, it's working now. Thanks! [10:54:24] 06cloud-services-team, 06Infrastructure-Foundations, 10netops, 06SRE: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847#10182562 (10aborrero) neutron virtual router has the right IPv6 address: {F57561976} [10:55:07] 10wikitech.wikimedia.org, 06Data Products, 06Data-Engineering, 06DBA, 07Schema-change: Please drop globalblocks table from labswiki - https://phabricator.wikimedia.org/T375783#10182553 (10Ladsgroup) 05Open→03Stalled We will do that in October 1. Let's mark this stalled until then. [10:56:50] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [11:00:47] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [11:06:27] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops, 07User-notice: Communication for Wikitech/Wikimedia Developer Account migration - https://phabricator.wikimedia.org/T373615#10182598 (10Ladsgroup) If we want this to be in the next week's tech news, this should be added to it now. [11:11:11] 06cloud-services-team, 06Infrastructure-Foundations, 10netops, 06SRE: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847#10182616 (10aborrero) The VM did not get the IPv6 assigned in the interface via dhcpv6 :-( `lang=shell-session aborrero@ipv6:~$ ip -br a lo... [11:12:02] 06cloud-services-team, 06Infrastructure-Foundations, 10netops, 06SRE: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847#10182629 (10aborrero) We got DNS integration half working: `lang=shell-session aborrero@ipv6:~$ host ipv6.cloudinfra-codfw1dev.codfw1dev.wikimedia.cl... [11:22:01] 06cloud-services-team, 06Infrastructure-Foundations, 10netops, 06SRE: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847#10182667 (10aborrero) I see the dhcp6 packets from my test VM arriving into neutron: ` 11:20:29.156995 IP6 fe80::f816:3eff:fe3e:4b38.546 > ff02::1:2.... [11:24:44] 06cloud-services-team, 10Cloud-VPS: [wmcs-cookbooks] Improve cloudvirt.drain and cloudvirt.set_maintenance - https://phabricator.wikimedia.org/T375867 (10fnegri) 03NEW [11:29:30] 06cloud-services-team, 10Cloud-VPS: [wmcs-cookbooks] Improve cloudvirt.drain and cloudvirt.set_maintenance - https://phabricator.wikimedia.org/T375867#10182684 (10fnegri) p:05Triage→03High Setting priority to High as this would be very useful when there's a node failure, like we had this month in {T375223}. [12:02:19] (03open) 10aborrero: codfw1dev: default secgroup: allow egress IPv6 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/65 (https://phabricator.wikimedia.org/T375847) [12:05:10] (03update) 10aborrero: codfw1dev: default secgroup: allow egress IPv6 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/65 (https://phabricator.wikimedia.org/T375847) [12:07:26] (03open) 10aborrero: subnets: cloud-flat-codfw1dev-v6: summarize addresses [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/66 (https://phabricator.wikimedia.org/T375847) [12:10:06] 06cloud-services-team, 10Cloud-VPS: tofu-infra: refactor repo structure - https://phabricator.wikimedia.org/T375283#10182798 (10fnegri) > What do you think about each project dir only having YAML files, and the tofu logic being in a generic directory? something like this I think this would be an improvement c... [12:12:20] (03approved) 10fnegri: codfw1dev: track novaproxy.codfw1dev.wmcloud.org record [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/57 (https://phabricator.wikimedia.org/T374828) (owner: 10aborrero) [12:18:35] (03approved) 10fnegri: zones, records: add common description to resources [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/58 (owner: 10aborrero) [12:19:00] (03approved) 10fnegri: codfw1dev: proxy-codfw1dev: track security groups [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/59 (owner: 10aborrero) [12:19:32] (03approved) 10fnegri: secgroups: for delete_default_rules, use null to avoid changes [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/60 (owner: 10aborrero) [12:19:53] (03approved) 10fnegri: secgroups: manage_default_secgroups yes by default [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/61 (https://phabricator.wikimedia.org/T357111) (owner: 10aborrero) [12:20:54] (03approved) 10fnegri: eqiad1: default secgroup: allow prometheus from metricsinfra [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/62 (https://phabricator.wikimedia.org/T375111) (owner: 10aborrero) [12:23:01] (03approved) 10fnegri: subnets: cloud-flat-codfw1dev-v6: summarize addresses [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/66 (https://phabricator.wikimedia.org/T375847) (owner: 10aborrero) [12:30:36] (03update) 10aborrero: codfw1dev: default secgroup: allow egress IPv6 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/65 (https://phabricator.wikimedia.org/T375847) [12:31:54] (03update) 10aborrero: zones, records: add common description to resources [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/58 [12:33:26] (03merge) 10aborrero: subnets: cloud-flat-codfw1dev-v6: summarize addresses [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/66 (https://phabricator.wikimedia.org/T375847) [12:35:20] (03update) 10aborrero: secgroups: for delete_default_rules, use null to avoid changes [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/60 [12:36:13] (03merge) 10aborrero: secgroups: for delete_default_rules, use null to avoid changes [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/60 [12:37:21] (03update) 10aborrero: codfw1dev: default secgroup: allow egress IPv6 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/65 (https://phabricator.wikimedia.org/T375847) [12:38:24] (03update) 10aborrero: codfw1dev: default secgroup: allow egress IPv6 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/65 (https://phabricator.wikimedia.org/T375847) [12:38:50] (03merge) 10aborrero: codfw1dev: default secgroup: allow egress IPv6 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/65 (https://phabricator.wikimedia.org/T375847) [12:40:31] (03update) 10aborrero: secgroups: manage_default_secgroups yes by default [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/61 (https://phabricator.wikimedia.org/T357111) [12:41:31] (03merge) 10aborrero: secgroups: manage_default_secgroups yes by default [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/61 (https://phabricator.wikimedia.org/T357111) [12:41:37] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:42:14] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [12:45:33] 06cloud-services-team, 10Cloud-VPS: tofu-infra: refactor repo structure - https://phabricator.wikimedia.org/T375283#10183017 (10aborrero) OK. Just to clarify, how would you handle the logic in the cookook then? and a _future_ CI integration? [12:47:15] (03update) 10aborrero: eqiad1: default secgroup: allow prometheus from metricsinfra [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/62 (https://phabricator.wikimedia.org/T375111) [12:47:27] (03PS1) 10D3r1ck01: README.md: Add pointer to frontend local setup documentation [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1076198 [12:48:11] (03merge) 10aborrero: eqiad1: default secgroup: allow prometheus from metricsinfra [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/62 (https://phabricator.wikimedia.org/T375111) [12:48:15] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:48:22] (03PS2) 10D3r1ck01: README.md: Add pointer to frontend local setup documentation [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1076198 [12:49:25] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [12:50:20] (03update) 10aborrero: codfw1dev: proxy-codfw1dev: track security groups [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/59 [12:51:35] (03merge) 10aborrero: codfw1dev: proxy-codfw1dev: track security groups [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/59 [12:51:43] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:52:35] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [12:56:10] (03CR) 10D3r1ck01: [C:03+2] wmcs: add the new gitlab repos [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1053538 (owner: 10David Caro) [12:57:03] (03Merged) 10jenkins-bot: wmcs: add the new gitlab repos [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1053538 (owner: 10David Caro) [12:59:08] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:59:27] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [12:59:31] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [13:00:13] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [13:00:52] (03update) 10aborrero: zones, records: add common description to resources [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/58 [13:02:13] (03merge) 10aborrero: zones, records: add common description to resources [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/58 [13:02:17] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [13:02:48] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [13:03:24] (03update) 10aborrero: codfw1dev: track novaproxy.codfw1dev.wmcloud.org record [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/57 (https://phabricator.wikimedia.org/T374828) [13:04:08] (03merge) 10aborrero: codfw1dev: track novaproxy.codfw1dev.wmcloud.org record [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/57 (https://phabricator.wikimedia.org/T374828) [13:04:30] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [13:05:46] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [13:07:05] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [13:07:15] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [13:07:17] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [13:08:03] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [13:20:18] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [13:20:47] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [13:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:14:48] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [14:14:58] 06cloud-services-team: PuppetFailure Puppet failure on cloudcontrol2006-dev:9100 - https://phabricator.wikimedia.org/T375884 (10phaultfinder) 03NEW [14:47:30] 10Tool-video-answer-tool, 06Future-Audiences: FA community call video demo - https://phabricator.wikimedia.org/T374878#10183432 (10Maryana) 05Open→03Resolved a:03Maryana [14:52:09] 10Tool-video-answer-tool, 06Future-Audiences: Generate first batch of videos with latest styling - https://phabricator.wikimedia.org/T375888 (10Maryana) 03NEW [14:56:14] 06cloud-services-team: PuppetFailure Puppet failure on cloudcontrol2006-dev:9100 - https://phabricator.wikimedia.org/T375884#10183494 (10aborrero) 05Open→03Resolved a:03aborrero this was a flap. Creating a ticket for every potential puppet agent error is going to be too overkill. [14:59:36] 10Tool-video-answer-tool, 06Future-Audiences, 07Spike: Research getting image attribution via API - https://phabricator.wikimedia.org/T374375#10183506 (10Maryana) 05Open→03Resolved [15:00:02] 10Tool-video-answer-tool, 06Future-Audiences, 07Spike: Research TTS options - https://phabricator.wikimedia.org/T374368#10183511 (10Maryana) 05Open→03Resolved [15:14:13] (03open) 10aborrero: codfw1dev: enable slaac mode for IPv6 network [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/67 (https://phabricator.wikimedia.org/T375847) [15:16:12] (03merge) 10aborrero: codfw1dev: enable slaac mode for IPv6 network [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/67 (https://phabricator.wikimedia.org/T375847) [15:16:14] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:18:22] 06cloud-services-team, 10Cloud-VPS: tofu-infra: refactor repo structure - https://phabricator.wikimedia.org/T375283#10183643 (10fnegri) //Caveat: this is all at "brainstorming" stage and we will need to experiment and find what works best.// Ideally, both the cookbook and the CI would detect which files were... [15:27:10] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [15:30:07] 06cloud-services-team, 10Toolforge, 10Pywikibot: unexpected root-owned files in /data/project/pywikibot/public_html - https://phabricator.wikimedia.org/T375279#10183701 (10valhallasw) Moved `public_html.old` to `public_html.old.20240927` to unblock this for now, however manually moving the directory is not a... [15:33:33] RESOLVED: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [15:35:16] (03open) 10aborrero: Revert "codfw1dev: enable slaac mode for IPv6 network" [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/68 (https://phabricator.wikimedia.org/T375847) [15:36:33] (03merge) 10aborrero: Revert "codfw1dev: enable slaac mode for IPv6 network" [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/68 (https://phabricator.wikimedia.org/T375847) [17:03:48] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2004-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [17:28:48] FIRING: [2x] PuppetFailure: Puppet has failed on cloudcontrol1005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [17:28:56] 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T375485#10184121 (10phaultfinder) [17:46:16] 06cloud-services-team: PuppetFailure Puppet failure on cloudcontrol2006-dev:9100 - https://phabricator.wikimedia.org/T373815#10184167 (10fnegri) 05Resolved→03Open This happened again, Puppet is failing on both cloudcontrol1005 and cloudcontrol2004-dev: ` root@cloudcontrol2004-dev:/srv/tofu-infra# run-puppe... [17:46:56] 06cloud-services-team: Puppet fails on cloudcontrol when updating /srv/tofu-infra - https://phabricator.wikimedia.org/T373815#10184170 (10fnegri) p:05Triage→03High [17:47:52] 06cloud-services-team: Puppet fails on cloudcontrol when updating /srv/tofu-infra - https://phabricator.wikimedia.org/T373815#10184178 (10fnegri) [17:48:00] 06cloud-services-team: Puppet fails on cloudcontrol when updating /srv/tofu-infra - https://phabricator.wikimedia.org/T373815#10184184 (10fnegri) [17:48:02] 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T375485#10184182 (10fnegri) →14Duplicate dup:03T373815 [17:48:16] 06cloud-services-team: PuppetFailure Puppet failure on cloudcontrol2004-dev:9100 - https://phabricator.wikimedia.org/T375534#10184176 (10fnegri) →14Duplicate dup:03T373815 [17:49:10] 06cloud-services-team: Puppet fails on cloudcontrol when updating /srv/tofu-infra - https://phabricator.wikimedia.org/T373815#10184203 (10fnegri) I fixed this manually with the same method: ` /srv/tofu-infra# sudo git branch -d remotes/origin/HEAD ` [17:52:20] 06cloud-services-team: PuppetFailure Puppet failure on cloudvirt1067:9100 - https://phabricator.wikimedia.org/T375836#10184209 (10fnegri) 05Open→03Resolved a:03fnegri Not failing anymore. From the logs, it looks like a one-off network issue: ` 2024-09-27T01:01:16.999596+00:00 cloudvirt1067 puppet-ag... [17:53:52] 06cloud-services-team: PuppetFailure Puppet failure on cloudcephosd1040:9100 - https://phabricator.wikimedia.org/T375484#10184215 (10fnegri) 05Open→03Resolved a:03fnegri This was related to {T372814} and is not failing anymore. [18:06:33] 10Toolforge, 07Epic: WE6.3.3: Toolforge Modernization Discovery - https://phabricator.wikimedia.org/T375914 (10Sarai-WMF) 03NEW [18:07:02] 10Toolforge, 07Epic: [Hypothesis] WE6.3.3: Toolforge Modernization Discovery - https://phabricator.wikimedia.org/T375914#10184257 (10Sarai-WMF) [18:07:24] 10Toolforge, 07Epic: [Hypothesis] WE6.3.3: Toolforge Modernization Discovery - https://phabricator.wikimedia.org/T375914#10184258 (10Sarai-WMF) a:03Sarai-WMF [18:08:48] FIRING: [2x] PuppetFailure: Puppet has failed on cloudcontrol1005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [18:13:48] RESOLVED: [2x] PuppetFailure: Puppet has failed on cloudcontrol1005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [18:55:06] 10VPS-project-Codesearch: Repositories selection is discarded when starting from a specific URL - https://phabricator.wikimedia.org/T375920 (10Od1n) 03NEW [19:17:48] 10Tool-video-answer-tool, 06Future-Audiences: Exclude Fair Use images from Wikipedia from appearing in videos - https://phabricator.wikimedia.org/T375925 (10Maryana) 03NEW [20:17:27] 10Tool-video-answer-tool, 06Future-Audiences: Add element to video preview to allow for text editing - https://phabricator.wikimedia.org/T375933 (10Maryana) 03NEW [20:21:19] 10Tool-video-answer-tool, 06Future-Audiences: Exclude Fair Use images from Wikipedia from appearing in videos - https://phabricator.wikimedia.org/T375925#10184709 (10Maryana) 05Open→03Invalid [20:22:20] 10Tool-video-answer-tool, 06Future-Audiences: Exclude Fair Use images from Wikipedia from appearing in videos - https://phabricator.wikimedia.org/T375925#10184711 (10Maryana) 05Invalid→03Resolved [20:32:22] (03approved) 10raymond-ndibe: [envvars-cli] update README.md [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/63 [20:32:26] (03merge) 10raymond-ndibe: [envvars-cli] update README.md [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/63 [20:33:00] (03approved) 10raymond-ndibe: [jobs-cli] update README.md [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/72 [20:33:03] (03merge) 10raymond-ndibe: [jobs-cli] update README.md [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/72 [20:33:10] (03approved) 10raymond-ndibe: [builds-cli] update README.md [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/90 [20:33:13] (03merge) 10raymond-ndibe: [builds-cli] update README.md [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/90 [20:33:20] (03approved) 10raymond-ndibe: [toolforge-weld] display_messages default True [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/56 [20:33:24] (03merge) 10raymond-ndibe: [toolforge-weld] display_messages default True [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/56 [20:33:39] (03update) 10raymond-ndibe: [jobs-cli] remove display_messages default from cli [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/71 [20:33:45] (03approved) 10raymond-ndibe: [jobs-cli] remove display_messages default from cli [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/71 [20:35:00] (03merge) 10raymond-ndibe: [jobs-cli] remove display_messages default from cli [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/71 [20:47:27] 10Tool-itwiki: Understand crash in BotCancellazioni - Error reported from cURL [error n. 16] - https://phabricator.wikimedia.org/T375936 (10valerio.bozzolan) 03NEW p:05Triage→03High [20:52:13] 10Tool-itwiki: Understand why BotCancellazioni does not start automagically (apparently after every forced wiki read-only mode) - https://phabricator.wikimedia.org/T375937 (10valerio.bozzolan) 03NEW p:05Triage→03High [20:52:42] 10Tool-itwiki: Understand why BotCancellazioni does not start automagically (apparently after every forced wiki read-only mode) - https://phabricator.wikimedia.org/T375937#10184791 (10valerio.bozzolan) [20:52:43] 10Tool-itwiki: Understand crash in BotCancellazioni - Error reported from cURL [error n. 16] - https://phabricator.wikimedia.org/T375936#10184792 (10valerio.bozzolan) [21:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:47:28] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate mwv-builder-03.mediawiki-vagrant.eqiad.wmflabs is about to expire in 27d 23h 58m 34s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [22:50:17] (03update) 10raymond-ndibe: [toolforge-weld] bump_version.sh auto create commit and tag [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/57 [23:18:06] (03approved) 10raymond-ndibe: [toolforge-weld] bump_version.sh auto create commit and tag [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/57 [23:18:10] (03merge) 10raymond-ndibe: [toolforge-weld] bump_version.sh auto create commit and tag [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/57 [23:22:15] (03open) 10raymond-ndibe: d/changelog: bump to 1.6.3 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/58 [23:44:16] !log raymondndibe@wmf3402 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [23:44:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [23:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:51:00] !log raymondndibe@wmf3402 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld [23:51:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [23:52:29] !log raymondndibe@wmf3402 tools START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [23:52:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:56:52] (03open) 10raymond-ndibe: d/changelog: bump to 16.1.5 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/73 [23:58:54] !log raymondndibe@wmf3402 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld [23:58:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL