[00:05:55] FIRING: MaxConntrack: Max conntrack at 84.99% on cloudvirt1039:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:55:55] RESOLVED: MaxConntrack: Max conntrack at 82.27% on cloudvirt1039:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [01:09:00] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [01:09:09] 06cloud-services-team: NovafullstackSustainedFailures Novafullstack tests have been failing for more than 5hours in eqiad - https://phabricator.wikimedia.org/T386988 (10phaultfinder) 03NEW [01:44:00] RESOLVED: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [02:29:13] (03update) 10raymond-ndibe: jobs: add job for managing harbor quotas [repos/cloud/toolforge/maintain-harbor] (refactor_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/22 (https://phabricator.wikimedia.org/T352417) (owner: 10sstefanova) [02:30:07] (03update) 10raymond-ndibe: jobs: add job for managing harbor quotas [repos/cloud/toolforge/maintain-harbor] (refactor_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/22 (https://phabricator.wikimedia.org/T352417) (owner: 10sstefanova) [02:31:04] (03update) 10raymond-ndibe: jobs: add job for managing harbor quotas [repos/cloud/toolforge/maintain-harbor] (refactor_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/22 (https://phabricator.wikimedia.org/T352417) (owner: 10sstefanova) [02:37:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [02:42:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [07:19:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-75 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [08:31:18] !log dcaro@urcuchillay globaleducation START - Cookbook wmcs.openstack.cloudvirt.vm_console [08:31:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Globaleducation/SAL [08:35:23] !log dcaro@urcuchillay globaleducation END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [08:35:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Globaleducation/SAL [08:48:37] 10Tool-erinnermich: [ErinnerMichBot] Query current page title before posting reminder - https://phabricator.wikimedia.org/T381563#10570240 (10Tkarcher) 05Open→03In progress Update available now in the development branch: https://gitlab.wikimedia.org/toolforge-repos/erinnermich/-/commit/10879f0216c0ed21c2614c... [08:55:22] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10570245 (10Boghog) [10:23:17] 06cloud-services-team, 10Toolforge: Toolforge: migrate to Debian Bookworm or later - https://phabricator.wikimedia.org/T387005 (10taavi) 03NEW [10:23:33] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS (Debian Buster Deprecation), 10Toolforge (Toolforge iteration 17), 07Epic, 05Goal: Toolforge: migrate to Debian Bullseye or later - https://phabricator.wikimedia.org/T311897#10570511 (10taavi) [10:24:19] 06cloud-services-team, 10Toolforge: Toolforge: migrate to Debian Bookworm or later - https://phabricator.wikimedia.org/T387005#10570514 (10taavi) [10:24:24] 06cloud-services-team, 10Toolforge, 07Kubernetes: [infra] Upgrade Toolforge K8s etcd nodes to Bookworm - https://phabricator.wikimedia.org/T361237#10570515 (10taavi) [10:25:18] 06cloud-services-team, 10Toolforge: Upgrade Toolsbeta Redis cluster to Bookworm - https://phabricator.wikimedia.org/T360704#10570516 (10taavi) [10:25:33] 06cloud-services-team, 10Toolforge: Toolforge: migrate to Debian Bookworm or later - https://phabricator.wikimedia.org/T387005#10570519 (10taavi) [10:25:34] 06cloud-services-team, 10Toolforge: Upgrade Toolsbeta Redis cluster to Bookworm - https://phabricator.wikimedia.org/T360704#10570518 (10taavi) [10:25:36] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Toolsbeta: migrate to Debian Bullseye or later - https://phabricator.wikimedia.org/T360699#10570520 (10taavi) [10:25:43] 06cloud-services-team, 10Toolforge: Upgrade Toolforge Redis cluster to Bookworm or newer - https://phabricator.wikimedia.org/T360704#10570521 (10taavi) [10:26:38] 06cloud-services-team, 10Toolforge: [toolforge-prometheus] upgrade to bookworm - https://phabricator.wikimedia.org/T375523#10570522 (10taavi) [10:26:39] 06cloud-services-team, 10Toolforge: Toolforge: migrate to Debian Bookworm or later - https://phabricator.wikimedia.org/T387005#10570523 (10taavi) [10:26:41] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS (Debian Buster Deprecation), 10Toolforge (Toolforge iteration 17), 07Epic, 05Goal: Toolforge: migrate to Debian Bullseye or later - https://phabricator.wikimedia.org/T311897#10570524 (10taavi) [11:24:14] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string Wikitech not found on https://wikitech-static.wikimedia.org:443/wiki/Main_Page?debug=true - 2517 bytes in 0.118 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [12:03:14] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29348 bytes in 0.204 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [12:10:37] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10570748 (10Ladsgroup) [12:10:41] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10570750 (10Ladsgroup) @Boghog please make a comment. [12:18:19] !log andrew@cloudcumin1001 uuidtest START - Cookbook wmcs.vps.create_project for project uuidtest in codfw1dev [12:18:20] andrew@cloudcumin1001: Unknown project "uuidtest" [12:18:29] (03open) 10group_199_bot_333a6c67971a471aeb1cf0b14ccf9f49: projects: added project uuidtest [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/153 [12:19:33] (03approved) 10andrew: projects: added project uuidtest [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/153 (owner: 10group_199_bot_333a6c67971a471aeb1cf0b14ccf9f49) [12:20:34] (03merge) 10andrew: projects: added project uuidtest [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/153 (owner: 10group_199_bot_333a6c67971a471aeb1cf0b14ccf9f49) [12:23:56] !log andrew@cloudcumin1001 uuidtest END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project uuidtest in codfw1dev [12:23:57] andrew@cloudcumin1001: Unknown project "uuidtest" [12:29:26] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [12:32:05] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [12:52:02] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 [12:57:33] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 [13:18:05] (03update) 10raymond-ndibe: jobs: add job for managing harbor quotas [repos/cloud/toolforge/maintain-harbor] (refactor_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/22 (https://phabricator.wikimedia.org/T352417) (owner: 10sstefanova) [13:24:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-75 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [13:24:33] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-75 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [13:29:33] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-75 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [13:30:33] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-75 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [13:49:18] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-75 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [14:47:41] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: [monitoring] KernelErrors alerts trigger incorrectly when a host is reimaged - https://phabricator.wikimedia.org/T386850#10571240 (10fnegri) 05In progress→03Resolved [15:01:39] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10571305 (10Boghog) |**Wikitech account/LDAP:**| Boghog|myaccount| |**SUL account**| mySULaccount| |**Account linked on [[ https://idm.wikimedia.org/ | IDM ]]IDM** |Y/N| |**I have visited [[... [15:51:53] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10571433 (10bd808) >>! In T376267#10571305, @Boghog wrote: > |**Wikitech account/LDAP:**| Boghog| > |**SUL account**| mySULaccount| > |**Account linked on [[ https://idm.wikimedia.org/ | IDM... [16:14:17] (03open) 10hridyeshgupta: Update display error message for Luthor when Wikisource is unavailable [toolforge-repos/luthor] - 10https://gitlab.wikimedia.org/toolforge-repos/luthor/-/merge_requests/3 [16:25:41] (03update) 10hridyeshgupta: Update display error message for Luthor when Wikisource is unavailable [toolforge-repos/luthor] - 10https://gitlab.wikimedia.org/toolforge-repos/luthor/-/merge_requests/3 [16:33:44] (03update) 10raymond-ndibe: jobs: add job for managing harbor quotas [repos/cloud/toolforge/maintain-harbor] (refactor_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/22 (https://phabricator.wikimedia.org/T352417) (owner: 10sstefanova) [16:38:47] (03update) 10raymond-ndibe: jobs: add job for managing harbor quotas [repos/cloud/toolforge/maintain-harbor] (refactor_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/22 (https://phabricator.wikimedia.org/T352417) (owner: 10sstefanova) [17:20:16] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10571712 (10Boghog) Thanks! I was able to login to my account and restart the citation-template-filling tool. The tool is now running again. [17:35:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-37 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [17:40:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-37 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [17:42:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [17:47:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [17:52:26] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [17:57:26] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [17:59:56] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10571813 (10Pkropf) |**Wikitech account/LDAP:**| pkropf| |**SUL account**|pkropf| |**Account linked on IDM** |N| |**I have visited [[ https://wikitech.wikimedia.org/wiki/MediaWiki:Loginprompt... [18:09:12] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10571843 (10Ladsgroup) You're SUL account is `Pkropf` (https://www.mediawiki.org/w/index.php?title=User:Pkropf) but no such user exists in wikitech: https://wikitech.wikimedia.org/wiki/User:P... [19:40:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-puppetserver-1 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:44:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance project-proxy-puppetserver-1 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:47:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance paws-puppetserver-1 on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:51:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-internal-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:55:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tools-puppetserver-01 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:02:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-puppetserver-1 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:05:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:06:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-cloudvps-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:17:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [21:32:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [22:24:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [22:29:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [22:40:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance tools-puppetserver-01 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:47:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-puppetserver-1 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:50:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:51:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-cloudvps-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:00:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-puppetserver-1 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:02:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance paws-puppetserver-1 on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:04:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance project-proxy-puppetserver-1 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:06:28] RESOLVED: [2x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-cloudvps-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:11:18] (03update) 10raymond-ndibe: jobs: add job for managing harbor quotas [repos/cloud/toolforge/maintain-harbor] (refactor_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/22 (https://phabricator.wikimedia.org/T352417) (owner: 10sstefanova) [23:32:35] (03open) 10raymond-ndibe: [toolforge-deploy] maintain-harbor use robot account [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417 https://phabricator.wikimedia.org/T361698) [23:32:42] (03update) 10raymond-ndibe: [toolforge-deploy] maintain-harbor use robot account [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417 https://phabricator.wikimedia.org/T361698) [23:32:52] (03update) 10raymond-ndibe: [toolforge-deploy] maintain-harbor use robot account [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417 https://phabricator.wikimedia.org/T361698) [23:44:20] (03update) 10raymond-ndibe: jobs: add job for managing harbor quotas [repos/cloud/toolforge/maintain-harbor] (refactor_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/22 (https://phabricator.wikimedia.org/T352417) (owner: 10sstefanova) [23:46:38] (03update) 10raymond-ndibe: jobs: add job for managing harbor quotas [repos/cloud/toolforge/maintain-harbor] (refactor_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/22 (https://phabricator.wikimedia.org/T352417) (owner: 10sstefanova) [23:52:08] (03update) 10raymond-ndibe: jobs: add job for managing harbor quotas [repos/cloud/toolforge/maintain-harbor] (refactor_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/22 (https://phabricator.wikimedia.org/T352417) (owner: 10sstefanova)