[00:09:28] FIRING: NodeTextfileStale: Stale textfile for cloudvirt2004-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [00:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [00:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:09:28] FIRING: NodeTextfileStale: Stale textfile for cloudvirt2004-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [05:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:15:58] 10Toolforge, 07Documentation, 07good first task: Find and fix inaccuracies in Toolforge Django tutorial - https://phabricator.wikimedia.org/T245683#10147319 (10Aklapper) @Chickenleaf: Hi! This task has been assigned to you a while ago. Could you maybe share an update? Do you still plan to work on this task,... [07:19:28] RESOLVED: NodeTextfileStale: Stale textfile for cloudvirt2004-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [08:18:54] 10Toolforge: [builds-cli] No obvious way to delete individual `toolforge build` generated artifacts other than `toolforge clean` - https://phabricator.wikimedia.org/T368317#10147622 (10dcaro) >>! In T368317#10144873, @bd808 wrote: >>>! In T368317#10136747, @dcaro wrote: >> This is half-intentional, in the sense... [08:20:28] FIRING: NodeTextfileStale: Stale textfile for cloudvirt2004-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [08:45:50] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request: To strictly enforce semantic versioning rules for toolforge services' APIs or not - https://phabricator.wikimedia.org/T373072#10147717 (10dcaro) I would go with Option 2, as we already have some checks to make sure we bump the... [08:48:18] 10Tool-translatetagger: Add support for Wikitext Tables - https://phabricator.wikimedia.org/T374784#10147727 (10Gopavasanth) I just made some fixes to support a certain extent for the tables, more testing and tuning is required. https://github.com/indictechcom/translatable-wikitext-converter/pull/1 [09:12:30] PROBLEM - Host clouddb1013 is DOWN: PING CRITICAL - Packet loss = 100% [09:12:36] RECOVERY - Host clouddb1013 is UP: PING OK - Packet loss = 0%, RTA = 30.35 ms [09:12:42] PROBLEM - mysqld processes on clouddb1013 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [09:16:48] RECOVERY - mysqld processes on clouddb1013 is OK: PROCS OK: 2 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [09:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:22:17] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: openstack: instrument VXLAN-based flat network - https://phabricator.wikimedia.org/T374020#10147890 (10aborrero) 05In progress→03Resolved [09:22:56] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Migrate Cloud VPS instances to VXLAN based networks - https://phabricator.wikimedia.org/T364725#10147897 (10aborrero) [09:22:57] 06cloud-services-team, 10Cloud-VPS, 07Epic: Cloud VPS: extend tofu-infra coverage - https://phabricator.wikimedia.org/T370037#10147896 (10aborrero) [09:29:34] PROBLEM - Host clouddb1014 is DOWN: PING CRITICAL - Packet loss = 100% [09:29:35] RECOVERY - Host clouddb1014 is UP: PING OK - Packet loss = 0%, RTA = 30.31 ms [09:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:34:24] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: openstack: create some automation to migrate VMs from VLAN to VXLAN networks - https://phabricator.wikimedia.org/T374822 (10aborrero) 03NEW [09:39:07] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: openstack: create some automation to migrate VMs from VLAN to VXLAN networks - https://phabricator.wikimedia.org/T374822#10147982 (10aborrero) p:05Triage→03Low [09:41:32] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: openstack: updates to horizon for vxlan migration - https://phabricator.wikimedia.org/T374824 (10aborrero) 03NEW [09:47:31] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: openstack: updates to horizon for vxlan migration - https://phabricator.wikimedia.org/T374824#10148030 (10aborrero) [09:50:44] PROBLEM - mysqld processes on clouddb1015 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [09:56:44] RECOVERY - mysqld processes on clouddb1015 is OK: PROCS OK: 2 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [10:10:20] (03update) 10aborrero: tofu-infra: update openstack provider from 2.0.0 to 2.1.0 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45 [10:20:20] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 13Patch-For-Review: openstack: updates to horizon for vxlan migration - https://phabricator.wikimedia.org/T374824#10148258 (10aborrero) [10:20:44] 06cloud-services-team, 10Cloud-VPS, 07Epic, 07IPv6: Enable IPv6 on CloudVPS - https://phabricator.wikimedia.org/T37947#10148264 (10aborrero) additional tracking of the project happening in https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/IPv6/initial_deploy [10:21:48] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: openstack: vxlan: verify nova proxy and floating IPs work with new VXLAN-based network - https://phabricator.wikimedia.org/T374828 (10aborrero) 03NEW [10:24:47] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: openstack: vxlan: verify nova proxy and floating IPs work with new VXLAN-based network - https://phabricator.wikimedia.org/T374828#10148312 (10aborrero) p:05Triage→03Medium [10:29:20] PROBLEM - Host clouddb1016 is DOWN: PING CRITICAL - Packet loss = 100% [10:29:30] RECOVERY - Host clouddb1016 is UP: PING OK - Packet loss = 0%, RTA = 30.41 ms [10:29:31] PROBLEM - mysqld processes on clouddb1016 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [10:30:31] RECOVERY - mysqld processes on clouddb1016 is OK: PROCS OK: 2 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [10:34:46] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 05Goal: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#10148359 (10fnegri) [10:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:56:26] 10Tool-Global-user-contributions, 10Special:GlobalContributions, 06Stewards-and-global-tools, 07Epic, and 2 others: [Epic] Implement global user contributions feature - https://phabricator.wikimedia.org/T337089#10148439 (10kostajh) [11:00:42] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:16:31] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 13Patch-For-Review: openstack: updates to horizon for vxlan migration - https://phabricator.wikimedia.org/T374824#10148485 (10aborrero) 05Open→03In progress p:05Triage→03Low [11:19:05] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 13Patch-For-Review: openstack: updates to horizon for vxlan migration - https://phabricator.wikimedia.org/T374824#10148489 (10aborrero) the above patch seems to be enough for horizon to start creating VMs in the new subnet. [11:24:29] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: openstack: create some automation to migrate VMs from VLAN to VXLAN networks - https://phabricator.wikimedia.org/T374822#10148510 (10aborrero) [12:01:02] (03update) 10dcaro: [toolforge-deploy] test multi-replica support for continuous jobs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/521 (https://phabricator.wikimedia.org/T341066) (owner: 10raymond-ndibe) [12:20:29] FIRING: NodeTextfileStale: Stale textfile for cloudvirt2004-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [12:38:12] 06cloud-services-team, 10Cloud-VPS, 07Epic: cloud: tofu-infra: support neutron security groups - https://phabricator.wikimedia.org/T374835 (10aborrero) 03NEW [12:39:35] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641#10148693 (10dcaro) [12:40:31] 10Tool-translatetagger, 06Indic-TechCom, 10Technical-Tool-Request: Tool to convert Wikitext into translatable wiki-text - https://phabricator.wikimedia.org/T372243#10148695 (10Kuldeepburjbhalaike) @KCVelaga, i guess //translatetagger// is better name than //translatable-wikitext-converter// or suggesting: /... [12:50:28] 06cloud-services-team, 10Cloud-VPS, 07Epic: cloud: tofu-infra: support neutron security groups - https://phabricator.wikimedia.org/T374835#10148708 (10aborrero) 05Open→03In progress p:05Triage→03Medium [13:02:58] (03open) 10dcaro: tekton: upgrade to v0.60.2 [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/61 [13:03:25] (03open) 10aborrero: tofu-infra: add support for neutron security groups [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46 (https://phabricator.wikimedia.org/T374835) [13:04:01] (03approved) 10fnegri: tofu-infra: update openstack provider from 2.0.0 to 2.1.0 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45 (owner: 10aborrero) [13:27:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-57 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [13:32:20] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.26.15 to 1.27.16 (T359641) [13:32:21] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [13:32:22] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [13:34:54] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.26.15 to 1.27.16 (T359641) [13:34:55] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [13:41:04] PROBLEM - mysqld processes on clouddb1017 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [13:42:04] RECOVERY - mysqld processes on clouddb1017 is OK: PROCS OK: 2 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [13:48:37] (03update) 10dcaro: tekton: upgrade to v0.60.2 [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/61 [13:53:52] (03update) 10dcaro: tekton: upgrade to v0.60.2 [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/61 [13:55:07] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.26.15 to 1.27.16 (T359641) [13:55:09] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [13:55:09] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [14:00:46] PROBLEM - mysqld processes on clouddb1018 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [14:01:44] RECOVERY - mysqld processes on clouddb1018 is OK: PROCS OK: 2 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [14:03:12] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.26.15 to 1.27.16 (T359641) [14:03:13] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:03:14] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [14:08:20] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.26.15 to 1.27.16 (T359641) [14:08:21] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:08:21] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [14:19:10] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.26.15 to 1.27.16 (T359641) [14:19:11] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:19:12] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [14:19:55] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.26.15 to 1.27.16 (T359641) [14:19:55] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:25:17] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.26.15 to 1.27.16 (T359641) [14:25:19] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:25:19] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [14:26:37] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 10Infrastructure Security, 13Patch-For-Review: wikireplicas root access - https://phabricator.wikimedia.org/T344599#10149100 (10fnegri) 05Open→03Resolved [14:26:39] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.26.15 to 1.27.16 (T359641) [14:26:39] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:27:43] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.26.15 to 1.27.16 (T359641) [14:27:43] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:27:52] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45 [14:28:08] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45 [14:28:28] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45 [14:28:59] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45 [14:29:11] (03merge) 10aborrero: tofu-infra: update openstack provider from 2.0.0 to 2.1.0 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45 [14:29:18] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [14:30:04] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [14:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:37:55] 10Data-Services, 06Data-Engineering, 06Trust and Safety Product Team: Hide the value of gb_address column in public replicas if gb_autoblock_parent_id is not null - https://phabricator.wikimedia.org/T371486#10149274 (10Dreamy_Jazz) [14:40:48] 10Data-Services, 06Data-Engineering, 06Trust and Safety Product Team, 10Temporary accounts (Blockers to minor pilot wiki deployment): Hide the value of gb_address column in public replicas if gb_autoblock_parent_id is not null - https://phabricator.wikimedia.org/T371486#10149292 (10Dreamy_Jazz) [14:42:11] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.26.15 to 1.27.16 (T359641) [14:42:13] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:42:13] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [14:42:16] 10Data-Services, 06Data-Engineering, 10GlobalBlocking, 06Stewards-and-global-tools, and 2 others: Hide the value of gbw_address in the global_block_whitelist table if the associated gb_id has gb_autoblock_parent_id as not null - https://phabricator.wikimedia.org/T374855#10149306 (10Dreamy_Jazz) [14:42:21] !log raymond-ndibe@cloudcumin1001 tools END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-103 from 1.26.15 to 1.27.16 (T359641) [14:42:21] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:42:51] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.26.15 to 1.27.16 (T359641) [14:42:52] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:43:49] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.26.15 to 1.27.16 (T359641) [14:43:49] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:43:50] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.26.15 to 1.27.16 (T359641) [14:43:50] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:44:55] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.26.15 to 1.27.16 (T359641) [14:44:55] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:44:55] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.26.15 to 1.27.16 (T359641) [14:44:56] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:45:44] 10Data-Services, 06Data-Engineering, 06Trust and Safety Product Team, 10Temporary accounts (Blockers to minor pilot wiki deployment), 10Trust and Safety Product Sprint (Sprint Beatboxing (Sept 16-27)): Hide the value of gb_address column in public replic... - https://phabricator.wikimedia.org/T371486#10149316 [14:46:04] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.26.15 to 1.27.16 (T359641) [14:46:04] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:46:05] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.26.15 to 1.27.16 (T359641) [14:46:05] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:47:18] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.26.15 to 1.27.16 (T359641) [14:47:19] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.26.15 to 1.27.16 (T359641) [14:47:19] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:47:19] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [14:47:19] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:48:21] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.26.15 to 1.27.16 (T359641) [14:48:21] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:48:22] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.26.15 to 1.27.16 (T359641) [14:48:22] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:54:25] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-1 from 1.26.15 to 1.27.16 (T359641) [14:54:27] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:54:28] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [14:58:20] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.26.15 to 1.27.16 (T359641) [14:58:21] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [15:04:04] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-1 from 1.26.15 to 1.27.16 (T359641) [15:04:05] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [15:04:06] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [15:22:56] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.26.15 to 1.27.16 (T359641) [15:22:57] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [15:22:57] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [15:28:51] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-1 from 1.26.15 to 1.27.16 (T359641) [15:28:52] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [15:28:53] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [15:42:04] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.26.15 to 1.27.16 (T359641) [15:42:05] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [15:42:06] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [15:47:47] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-1 from 1.26.15 to 1.27.16 (T359641) [15:47:48] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [15:47:48] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [15:56:03] PROBLEM - mysqld processes on clouddb1020 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [15:57:03] RECOVERY - mysqld processes on clouddb1020 is OK: PROCS OK: 2 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [15:57:35] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 (T359641) [15:57:37] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [15:57:38] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [15:57:48] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [16:02:54] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 (T359641) [16:02:55] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:02:56] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [16:04:45] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.26.15 to 1.27.16 (T359641) [16:04:45] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:06:01] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.26.15 to 1.27.16 (T359641) [16:06:01] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:06:56] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.26.15 to 1.27.16 (T359641) [16:06:56] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:08:04] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.26.15 to 1.27.16 (T359641) [16:08:05] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.26.15 to 1.27.16 (T359641) [16:08:05] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:08:06] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [16:08:06] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:09:14] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.26.15 to 1.27.16 (T359641) [16:09:15] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:09:15] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.26.15 to 1.27.16 (T359641) [16:09:16] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:10:21] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.26.15 to 1.27.16 (T359641) [16:10:22] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:10:22] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.26.15 to 1.27.16 (T359641) [16:10:23] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:11:30] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.26.15 to 1.27.16 (T359641) [16:11:31] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.26.15 to 1.27.16 (T359641) [16:11:32] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:11:32] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:12:42] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.26.15 to 1.27.16 (T359641) [16:12:42] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:12:43] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.26.15 to 1.27.16 (T359641) [16:12:43] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:13:52] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.26.15 to 1.27.16 (T359641) [16:13:53] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.26.15 to 1.27.16 (T359641) [16:13:53] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:13:54] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [16:13:54] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:20:00] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-17 from 1.26.15 to 1.27.16 (T359641) [16:20:02] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:20:02] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [16:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:21:23] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 (T359641) [16:21:23] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:26:41] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 (T359641) [16:26:43] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:26:44] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [16:27:38] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.26.15 to 1.27.16 (T359641) [16:27:39] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:28:37] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.26.15 to 1.27.16 (T359641) [16:28:37] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:28:38] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.26.15 to 1.27.16 (T359641) [16:28:38] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:29:45] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.26.15 to 1.27.16 (T359641) [16:29:45] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:29:46] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.26.15 to 1.27.16 (T359641) [16:29:46] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:35:49] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-2 from 1.26.15 to 1.27.16 (T359641) [16:35:50] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:35:50] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [16:36:53] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2 (T359641) [16:36:53] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:42:10] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2 (T359641) [16:42:11] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:42:11] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [16:45:18] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.26.15 to 1.27.16 (T359641) [16:45:18] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:46:14] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.26.15 to 1.27.16 (T359641) [16:46:14] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:46:15] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.26.15 to 1.27.16 (T359641) [16:46:15] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:47:24] Raymond_Ndibe: I have seen these for a while now, just be aware none of these actually get logged, if you mentioned the project name first they would though [16:47:48] RESOLVED: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [16:48:00] maybe it's a bug report against the cookbook, not sure [16:49:44] mutante: can you explain a bit what you meant? I am currently upgrading toolforge tools k8s nodes to 1.27 [16:51:03] Raymond_Ndibe: if you look at the log lines above, the logging bot always says "unknown project" and doesn't actually log [16:51:17] that is because the format it expects is !log [16:51:32] but somehow your log lines start with user@host [16:51:49] not sure if that is the cookbook inserting it or manual [16:51:58] but it means the logs dont end up being logged [16:52:15] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-20 from 1.26.15 to 1.27.16 (T359641) [16:52:16] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:52:16] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [16:52:52] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 (T359641) [16:52:53] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:54:50] hmmmm. this is interesting. I am running the cookbooks as root@cloudcumin1001 [16:56:28] the only thing that might point out my username is the path that I am running the cookbook from which is `/home/raymond-ndibe` [16:58:10] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-20 (T359641) [16:58:12] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [16:58:12] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [17:02:16] mutante: yep, that's a cookbook issue I think (or at least, the bot), iirc there's some parsing going on in the bot to manage the user@host bit, but it seems it's not doing what's expected [17:02:43] maybe it does not like the '-' in the name :/ [17:02:49] will have to look a bit more into it [17:03:02] the interesting thing is that I am not running as `raymond-ndibe` [17:03:37] it might be interesting to find out where it's getting the username from. maybe path? [17:04:25] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.26.15 to 1.27.16 (T359641) [17:04:26] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [17:04:26] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [17:04:51] I created T374875 to follow up [17:04:51] T374875: [cookbook,sal] it does not seem to parse correctly the user@host header anymore - https://phabricator.wikimedia.org/T374875 [17:04:57] Raymond_Ndibe: it know even if you use sudo [17:06:06] dcaro: ack, thank you. So it's just a bug report then [17:06:07] yeaaa. but I've already done `sudo su`. Well we should look into it a bit more [17:10:11] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-20 from 1.26.15 to 1.27.16 (T359641) [17:10:12] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [17:10:12] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [17:26:32] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 (T359641) [17:26:33] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [17:26:33] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [17:31:49] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-20 (T359641) [17:31:50] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [17:31:51] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [17:34:18] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.26.15 to 1.27.16 (T359641) [17:34:18] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [17:40:02] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-20 from 1.26.15 to 1.27.16 (T359641) [17:40:04] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [17:40:04] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [17:40:39] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6 [17:40:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:40:59] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.26.15 to 1.27.16 (T359641) [17:41:00] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [17:46:24] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6 [17:46:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:46:38] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-20 from 1.26.15 to 1.27.16 (T359641) [17:46:38] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [17:46:38] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [17:47:10] (03update) 10dcaro: tekton: upgrade to v0.60.2 [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/61 [17:51:00] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 [17:51:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:56:53] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 [17:56:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:20:22] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.26.15 to 1.27.16 (T359641) [19:20:22] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [19:20:22] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [19:26:12] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-20 from 1.26.15 to 1.27.16 (T359641) [19:26:14] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [19:26:14] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [19:52:52] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 (T359641) [19:52:54] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [19:52:54] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [19:58:10] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-20 (T359641) [19:58:11] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [19:58:12] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [20:14:45] (03open) 10pwangai: Gerrit message improvements [toolforge-repos/sonarqubebot-experimental] - 10https://gitlab.wikimedia.org/toolforge-repos/sonarqubebot-experimental/-/merge_requests/1 (https://phabricator.wikimedia.org/T373109) [20:15:05] (03merge) 10pwangai: Gerrit message improvements [toolforge-repos/sonarqubebot-experimental] - 10https://gitlab.wikimedia.org/toolforge-repos/sonarqubebot-experimental/-/merge_requests/1 (https://phabricator.wikimedia.org/T373109) [20:20:29] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 (T359641) [20:20:30] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [20:20:30] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [20:25:47] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-20 (T359641) [20:25:48] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [20:25:48] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [20:28:49] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T359641) [20:28:49] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [20:38:37] !log raymond-ndibe@cloudcumin1001 tools Added a new k8s worker-nfs tools-k8s-worker-nfs-65.tools.eqiad1.wikimedia.cloud to the cluster [20:38:37] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster [20:38:38] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [20:38:38] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [20:40:33] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node for host tools-k8s-worker-nfs-20 (T359641) [20:40:33] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [20:40:34] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [20:45:49] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node (exit_code=99) for host tools-k8s-worker-nfs-20 [20:45:50] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [20:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:56:12] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 (T359641) [20:56:13] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [20:56:13] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [20:57:28] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on tools-puppetserver-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [21:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:02:17] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 (T359641) [21:02:18] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 (T359641) [21:02:19] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:02:19] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [21:02:19] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:08:23] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 (T359641) [21:08:24] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:08:24] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.26.15 to 1.27.16 (T359641) [21:08:25] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [21:08:25] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:09:31] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.26.15 to 1.27.16 (T359641) [21:09:31] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:09:32] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.26.15 to 1.27.16 (T359641) [21:09:32] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:10:40] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.26.15 to 1.27.16 (T359641) [21:10:40] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:10:41] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 (T359641) [21:10:41] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:16:45] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 (T359641) [21:16:46] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 (T359641) [21:16:46] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:16:47] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [21:16:47] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:17:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-27 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [21:21:28] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance tools-k8s-worker-nfs-65 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [21:22:49] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 (T359641) [21:22:50] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 (T359641) [21:22:50] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:22:50] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [21:22:50] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:28:54] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 (T359641) [21:28:55] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:28:55] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 (T359641) [21:28:56] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [21:28:56] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:35:00] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 (T359641) [21:35:01] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.26.15 to 1.27.16 (T359641) [21:35:02] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:35:02] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [21:35:02] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:36:09] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.26.15 to 1.27.16 (T359641) [21:36:09] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:36:10] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 (T359641) [21:36:10] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:42:13] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 (T359641) [21:42:14] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 (T359641) [21:42:15] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:42:15] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [21:42:15] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:48:18] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 (T359641) [21:48:19] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 (T359641) [21:48:20] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:48:20] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [21:48:20] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:54:22] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 (T359641) [21:54:23] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:54:23] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [21:54:24] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.26.15 to 1.27.16 (T359641) [21:54:24] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:55:34] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.26.15 to 1.27.16 (T359641) [21:55:34] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:55:35] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.26.15 to 1.27.16 (T359641) [21:55:35] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:56:44] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.26.15 to 1.27.16 (T359641) [21:56:44] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:56:45] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.26.15 to 1.27.16 (T359641) [21:56:45] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:57:54] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.26.15 to 1.27.16 (T359641) [21:57:54] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [21:57:55] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 (T359641) [21:57:55] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:01:34] 10cloud-services-team (FY2024/2025-Q1-Q2), 05Cloud-Services-Origin-User, 07Cloud-Services-Worktype-Unplanned: [cookbook,sal] it does not seem to parse correctly the user@host header anymore - https://phabricator.wikimedia.org/T374875#10151020 (10bd808) >>! In T374875#10150374, @dcaro wrote: > No, it's not yo... [22:03:56] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 (T359641) [22:03:57] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.26.15 to 1.27.16 (T359641) [22:03:57] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:03:57] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [22:03:58] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:05:04] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.26.15 to 1.27.16 (T359641) [22:05:04] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:05:05] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 (T359641) [22:05:05] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:11:09] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 (T359641) [22:11:10] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.26.15 to 1.27.16 (T359641) [22:11:11] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:11:11] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [22:11:11] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:12:16] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.26.15 to 1.27.16 (T359641) [22:12:16] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:12:17] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.26.15 to 1.27.16 (T359641) [22:12:17] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:13:25] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.26.15 to 1.27.16 (T359641) [22:13:25] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:13:26] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.26.15 to 1.27.16 (T359641) [22:13:26] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:14:32] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.26.15 to 1.27.16 (T359641) [22:14:32] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:14:33] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.26.15 to 1.27.16 (T359641) [22:14:33] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:15:39] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.26.15 to 1.27.16 (T359641) [22:15:39] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:15:40] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.26.15 to 1.27.16 (T359641) [22:15:40] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:16:46] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.26.15 to 1.27.16 (T359641) [22:16:47] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.26.15 to 1.27.16 (T359641) [22:16:48] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:16:48] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [22:16:48] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:17:51] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.26.15 to 1.27.16 (T359641) [22:17:51] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:17:52] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.26.15 to 1.27.16 (T359641) [22:17:52] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:19:03] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.26.15 to 1.27.16 (T359641) [22:19:03] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:19:04] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 (T359641) [22:19:04] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:25:05] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 (T359641) [22:25:07] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.26.15 to 1.27.16 (T359641) [22:26:16] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.26.15 to 1.27.16 (T359641) [22:26:17] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.26.15 to 1.27.16 (T359641) [22:27:25] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.26.15 to 1.27.16 (T359641) [22:27:26] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 (T359641) [22:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:33:30] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 (T359641) [22:33:32] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.26.15 to 1.27.16 (T359641) [22:33:32] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:33:32] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [22:33:32] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:34:43] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.26.15 to 1.27.16 (T359641) [22:34:43] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:34:44] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 (T359641) [22:34:44] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:40:46] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 (T359641) [22:40:47] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.26.15 to 1.27.16 (T359641) [22:40:47] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:40:48] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [22:40:48] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:41:55] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.26.15 to 1.27.16 (T359641) [22:41:56] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:41:56] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.26.15 to 1.27.16 (T359641) [22:41:56] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:43:10] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.26.15 to 1.27.16 (T359641) [22:43:10] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:43:10] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.26.15 to 1.27.16 (T359641) [22:43:11] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:44:16] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.26.15 to 1.27.16 (T359641) [22:44:16] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:44:17] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 (T359641) [22:44:17] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:50:21] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 (T359641) [22:50:22] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 (T359641) [22:50:22] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:50:22] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [22:50:22] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:56:24] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 (T359641) [22:56:26] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:56:26] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.26.15 to 1.27.16 (T359641) [22:56:26] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [22:56:26] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:57:33] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.26.15 to 1.27.16 (T359641) [22:57:33] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [22:57:34] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 (T359641) [22:57:34] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [23:03:31] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-28, tools-k8s-worker-nfs-29, tools-k8s-worker-nfs-30, tools-k8s-worker-nfs-31, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-46, tools-k8s-wor [23:03:31] ker-nfs-49, tools-k8s-worker-nfs-50 (T359641) [23:03:32] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [23:03:33] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [23:03:37] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 (T359641) [23:03:37] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [23:03:38] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 (T359641) [23:03:38] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [23:08:12] (03PS1) 10BryanDavis: sal: Expand regex used to check for "user@host" clause [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/1073296 (https://phabricator.wikimedia.org/T374875) [23:09:40] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 (T359641) [23:09:41] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 (T359641) [23:09:42] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [23:09:42] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [23:09:42] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [23:13:55] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of cpu - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [23:15:41] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 (T359641) [23:15:42] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 (T359641) [23:15:42] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [23:15:43] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [23:15:43] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [23:21:47] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 (T359641) [23:21:48] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 (T359641) [23:21:48] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [23:21:48] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [23:21:48] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [23:27:54] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 (T359641) [23:27:56] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [23:27:56] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [23:27:56] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 (T359641) [23:27:56] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [23:30:53] 10Tool-video-answer-tool, 06Future-Audiences, 07Spike: Investigate different options for animation of images - https://phabricator.wikimedia.org/T374367#10151190 (10Maryana) a:05derenrich→03None [23:32:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-27 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [23:33:40] 10Tool-video-answer-tool, 06Future-Audiences, 07Spike: Image animation exploration - https://phabricator.wikimedia.org/T374877#10151196 (10Maryana) 05Open→03Resolved [23:34:00] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 (T359641) [23:34:01] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.26.15 to 1.27.16 (T359641) [23:34:01] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [23:34:02] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [23:34:02] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [23:34:04] 10Tool-video-answer-tool, 06Future-Audiences, 07Spike: Image animation exploration - https://phabricator.wikimedia.org/T374877#10151197 (10Maryana) 05Resolved→03Open [23:34:19] 10Tool-video-answer-tool, 06Future-Audiences, 07Spike: Image animation exploration - https://phabricator.wikimedia.org/T374877#10151198 (10Maryana) 05Open→03Resolved [23:35:10] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.26.15 to 1.27.16 (T359641) [23:35:10] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [23:35:11] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.26.15 to 1.27.16 (T359641) [23:35:11] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [23:36:20] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.26.15 to 1.27.16 (T359641) [23:36:20] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [23:36:21] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.26.15 to 1.27.16 (T359641) [23:36:21] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [23:37:29] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.26.15 to 1.27.16 (T359641) [23:37:30] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [23:44:28] (03CR) 10BryanDavis: [C:03+2] sal: Expand regex used to check for "user@host" clause [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/1073296 (https://phabricator.wikimedia.org/T374875) (owner: 10BryanDavis) [23:45:28] (03Merged) 10jenkins-bot: sal: Expand regex used to check for "user@host" clause [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/1073296 (https://phabricator.wikimedia.org/T374875) (owner: 10BryanDavis) [23:45:37] !log raymond-ndibe@cloudcumin1001 tools END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-28, tools-k8s-worker-nfs-29, tools-k8s-worker-nfs-30, tools-k8s-worker-nfs-31, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-38, tools-k8s-worker- [23:45:38] nfs-46, tools-k8s-worker-nfs-49, tools-k8s-worker-nfs-50 (T359641) [23:45:39] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [23:45:39] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [23:47:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-27 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [23:48:06] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-31, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-36 (T359641) [23:48:09] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-49, tools-k8s-worker-nfs-50 (T359641)