[00:16:28] FIRING: InstanceDown: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:20:45] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [00:20:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [00:20:56] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [00:21:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [00:21:28] RESOLVED: InstanceDown: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:50:50] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [00:50:52] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [00:50:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [00:50:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [00:57:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [01:02:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [02:19:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:14:54] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [03:14:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [03:15:06] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [03:15:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [03:45:51] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [03:45:53] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [03:45:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [03:45:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [05:35:00] (03PS1) 10Majavah: archiver: Update for templatelinks normalization [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055557 [05:35:01] (03PS1) 10Majavah: cli: Configure a sensible logging format [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055558 [05:35:01] (03PS1) 10Majavah: Ignore another apicache folder [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055559 [05:36:52] (03CR) 10Majavah: [C:03+2] archiver: Update for templatelinks normalization [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055557 (owner: 10Majavah) [05:36:55] (03CR) 10Majavah: [C:03+2] cli: Configure a sensible logging format [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055558 (owner: 10Majavah) [05:36:59] (03CR) 10Majavah: [C:03+2] Ignore another apicache folder [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055559 (owner: 10Majavah) [05:38:24] (03Merged) 10jenkins-bot: archiver: Update for templatelinks normalization [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055557 (owner: 10Majavah) [05:38:33] (03Merged) 10jenkins-bot: cli: Configure a sensible logging format [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055558 (owner: 10Majavah) [05:38:33] (03Merged) 10jenkins-bot: Ignore another apicache folder [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055559 (owner: 10Majavah) [05:47:10] (03PS1) 10Majavah: cli: Fix level length [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055560 [05:47:10] (03PS1) 10Majavah: archiver: Fix page_title query escaping [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055561 [05:48:47] (03CR) 10Majavah: [C:03+2] archiver: Fix page_title query escaping [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055561 (owner: 10Majavah) [05:48:51] (03CR) 10Majavah: [C:03+2] cli: Fix level length [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055560 (owner: 10Majavah) [05:50:34] (03Merged) 10jenkins-bot: cli: Fix level length [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055560 (owner: 10Majavah) [05:50:34] (03Merged) 10jenkins-bot: archiver: Fix page_title query escaping [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055561 (owner: 10Majavah) [05:52:55] (03PS1) 10Majavah: Migrate to Python 3.11 [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055562 [05:53:24] (03CR) 10Majavah: [C:03+2] Migrate to Python 3.11 [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055562 (owner: 10Majavah) [05:54:21] (03Merged) 10jenkins-bot: Migrate to Python 3.11 [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055562 (owner: 10Majavah) [05:54:53] (03PS1) 10Majavah: Bump mypy compat to 3.11 [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055563 [06:10:32] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [06:10:44] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [06:19:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:40:49] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [06:40:51] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [06:40:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [06:40:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:07:06] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [09:07:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:07:18] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [09:07:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:37:49] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [09:37:51] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [09:37:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:37:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:44:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [09:49:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [10:02:43] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) [10:02:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:19:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:33:14] (03PS1) 10Majavah: Fix a bunch of strict mypy errors [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055580 [13:34:09] (03CR) 10CI reject: [V:04-1] Fix a bunch of strict mypy errors [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055580 (owner: 10Majavah) [13:34:40] (03CR) 10Majavah: [C:03+2] Bump mypy compat to 3.11 [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055563 (owner: 10Majavah) [13:35:40] (03Merged) 10jenkins-bot: Bump mypy compat to 3.11 [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055563 (owner: 10Majavah) [14:19:57] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:25:57] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace deployment-sessionstore04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370461#10000860 (10Andrew) >>! In T370461#10000308, @Southparkfan wrote:... [16:34:42] RESOLVED: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:18:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:33:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:46:43] (03merge) 10legoktm: Make things work on gitlab [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/1 (https://phabricator.wikimedia.org/T344744) (owner: 10bd808) [17:49:12] (03merge) 10legoktm: Make things work on gitlab [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/1 (https://phabricator.wikimedia.org/T344744) (owner: 10bd808) [17:54:20] (03open) 10legoktm: Pull ircservserv from GitLab, don't pin commit [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/2 [17:56:55] (03merge) 10legoktm: Pull ircservserv from GitLab, don't pin commit [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/2 [18:13:59] (03open) 10legoktm: Upgrade toml to 0.8 [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/2 [18:19:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:21:18] (03update) 10legoktm: Upgrade toml to 0.8 [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/2 [18:21:28] (03update) 10legoktm: Upgrade dependencies [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/2 [18:23:12] (03merge) 10legoktm: Upgrade dependencies [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/2 [18:29:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:20:53] (03open) 10legoktm: Switch to Toolforge jobs [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/3 [19:23:59] (03merge) 10legoktm: Switch to Toolforge jobs [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/3 [19:49:29] (03open) 10jjmc89: add new maintainers - bd808 and JJMC89 [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/4 [19:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:59:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:21:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-47 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [22:06:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-47 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [22:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:29:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks