[00:02:05] 06cloud-services-team, 10Cloud-VPS: Unable to persistently set fs.inotify.max_user_instances and fs.inotify.max_user_watches - https://phabricator.wikimedia.org/T385530#10519459 (10bd808) There are 3 manifests in puppet that configure `fs.inotify.max_user_instances` and 2 of them also configure `fs.inotify.max... [00:06:26] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [00:10:54] 06cloud-services-team, 10Cloud-VPS: Unable to persistently set fs.inotify.max_user_instances and fs.inotify.max_user_watches - https://phabricator.wikimedia.org/T385530#10519498 (10bd808) Another reasonable place for a shared inotify sysctl config in Puppet would be `::base::sysctl::inotify`. A class there cou... [00:13:41] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [00:30:44] 06cloud-services-team, 10Cloud-VPS: Unable to persistently set fs.inotify.max_user_instances and fs.inotify.max_user_watches - https://phabricator.wikimedia.org/T385530#10519551 (10Andrew) > Oddly, the config changes remained in /etc/sysctl.conf, depsite seeming to have either not been applied or to have been... [00:32:56] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [00:44:56] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [00:46:41] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [00:51:41] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [01:11:41] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [01:27:26] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [01:44:41] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [02:04:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [02:15:56] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [02:28:56] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [03:18:56] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [03:19:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:20:41] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [03:22:35] FIRING: NetworkOutSaturated: Outgoing network saturation detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DNetworkOutSaturated [03:27:35] RESOLVED: NetworkOutSaturated: Outgoing network saturation detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DNetworkOutSaturated [03:36:35] FIRING: NetworkOutSaturated: Outgoing network saturation detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DNetworkOutSaturated [03:41:35] RESOLVED: NetworkOutSaturated: Outgoing network saturation detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DNetworkOutSaturated [03:55:41] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [04:00:41] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [04:14:56] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [04:27:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [04:52:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [05:13:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [05:23:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [05:34:41] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [05:59:41] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [06:24:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [06:39:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [06:54:28] FIRING: [5x] PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-etcd-23 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [06:59:28] FIRING: [23x] PuppetAgentNoResources: No Puppet resources found on instance tools-db-5 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:04:28] FIRING: [51x] PuppetAgentNoResources: No Puppet resources found on instance tools-acme-chief-4 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:06:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance extdist-06 on project extdist - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:06:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance cvn-app10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:08:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance etcd-discovery-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:09:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-puppetserver-1 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:09:28] FIRING: [66x] PuppetAgentNoResources: No Puppet resources found on instance tools-acme-chief-4 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:09:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:10:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [07:11:28] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:12:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-legacy-redirector-2 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:14:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance project-proxy-puppetserver-1 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:14:28] FIRING: [88x] PuppetAgentNoResources: No Puppet resources found on instance tools-acme-chief-4 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:17:28] FIRING: [8x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-docker-imagebuilder-2 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:19:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:19:28] FIRING: [104x] PuppetAgentNoResources: No Puppet resources found on instance tools-acme-chief-3 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:19:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:22:28] FIRING: [12x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-cumin-1 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:23:28] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-internal-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:24:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance project-proxy-puppetserver-1 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:24:28] FIRING: [110x] PuppetAgentNoResources: No Puppet resources found on instance tools-acme-chief-3 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:26:28] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:27:28] FIRING: [16x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-cumin-1 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:28:28] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-internal-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:29:28] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance maps-proxy-03 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:30:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [07:32:28] FIRING: [27x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-acme-chief-2 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:34:28] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance maps-proxy-03 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:35:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:37:28] FIRING: [32x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-acme-chief-2 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:39:28] FIRING: [6x] PuppetAgentNoResources: No Puppet resources found on instance maps-proxy-03 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:42:28] FIRING: [36x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-acme-chief-2 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:44:28] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:46:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [07:49:28] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-3 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:54:28] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-3 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [08:00:26] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [08:04:28] FIRING: [6x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-3 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [08:09:28] FIRING: [8x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-3 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [08:14:28] FIRING: [10x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [08:17:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [08:22:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [08:24:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [08:27:09] (03open) 10arinaigum: Added a local dev option in config. Updated config for 2025-01 processing... [toolforge-repos/events-impact-report] - 10https://gitlab.wikimedia.org/toolforge-repos/events-impact-report/-/merge_requests/3 [08:27:21] (03merge) 10arinaigum: Added a local dev option in config. Updated config for 2025-01 processing... [toolforge-repos/events-impact-report] - 10https://gitlab.wikimedia.org/toolforge-repos/events-impact-report/-/merge_requests/3 [09:09:00] 06cloud-services-team: Cloud VPS puppet breakage on 2025-02-04 related to puppet-enc - https://phabricator.wikimedia.org/T385553 (10aborrero) 03NEW [09:13:26] 06cloud-services-team, 10Cloud-VPS: Cloud VPS puppet breakage on 2025-02-04 related to puppet-enc - https://phabricator.wikimedia.org/T385553#10520039 (10taavi) [09:17:19] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Unable to persistently set fs.inotify.max_user_instances and fs.inotify.max_user_watches - https://phabricator.wikimedia.org/T385530#10520042 (10taavi) Instead of making a inotify-specific thing, any reason not to provide a profile to set arbitrary sysc... [09:17:50] 06cloud-services-team, 10Cloud-VPS: Cloud VPS puppet breakage on 2025-02-04 related to puppet-enc - https://phabricator.wikimedia.org/T385553#10520043 (10aborrero) p:05Triage→03Medium @taavi pointed out that this may be an unnattended upgrade of java: ` Start-Date: 2025-02-04 06:26:11 Commandline: /usr/b... [09:20:10] 06cloud-services-team, 10Cloud-VPS: Cloud VPS puppet breakage on 2025-02-04 related to puppet-enc - https://phabricator.wikimedia.org/T385553#10520049 (10aborrero) [09:20:14] 06cloud-services-team, 10Cloud-VPS: Cloud VPS: 2024-10-22 cloud-wide puppet problem related to java update - https://phabricator.wikimedia.org/T377803#10520050 (10aborrero) [09:22:12] 06cloud-services-team, 10Cloud-VPS: Cloud VPS puppet breakage on 2025-02-04 related to puppet-enc - https://phabricator.wikimedia.org/T385553#10520069 (10Urbanecm) Started to receive emails for `wmcz-stats` about this: ` urbanecm@wmcz-stats-wikinside01:~$ sudo run-puppet-agent Info: Using environment 'product... [09:23:04] 06cloud-services-team, 10Cloud-VPS: Cloud VPS puppet breakage on 2025-02-04 related to puppet-enc - https://phabricator.wikimedia.org/T385553#10520072 (10aborrero) restarting puppetservers fleetwide with: ` aborrero@cloudcumin1001:~ $ sudo cumin -x --force O{*} 'systemctl list-units --all | grep -q puppetserv... [09:41:28] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [09:51:28] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [09:55:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [09:56:28] RESOLVED: [4x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [09:56:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance extdist-06 on project extdist - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [09:59:58] RESOLVED: [4x] PuppetAgentNoResources: No Puppet resources found on instance maps-proxy-03 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:07:58] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:07:58] RESOLVED: [2x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-harbor-1 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:10:58] RESOLVED: [5x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:10:58] RESOLVED: [16x] PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-ingress-8 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:11:58] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance etcd-discovery-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:24:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [10:29:26] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [11:19:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:37:54] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Cloud VPS puppet breakage on 2025-02-04 related to puppet-enc - https://phabricator.wikimedia.org/T385553#10520508 (10aborrero) 05Open→03Resolved a:03aborrero [11:41:01] 06cloud-services-team, 10Horizon: Horizon: requires access to openstack.eqiad1.wikimediacloud.org port 25000 - https://phabricator.wikimedia.org/T385527#10520516 (10taavi) →14Duplicate dup:03T377055 [11:41:02] 06cloud-services-team, 10Horizon: Keystone auth endpoint should use a standard HTTPS port - https://phabricator.wikimedia.org/T377055#10520518 (10taavi) [11:52:10] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: replace cloudgw100[12] with spare 'second region' dev servers cloudnet100[78]-dev - https://phabricator.wikimedia.org/T382356#10520556 (10aborrero) some interesting grafana dashboards to watch: * https://grafana.wikimedia.org/d/00000... [12:23:24] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: replace cloudgw100[12] with spare 'second region' dev servers cloudnet100[78]-dev - https://phabricator.wikimedia.org/T382356#10520717 (10aborrero) DNS changes: * in https://netbox.wikimedia.org/ipam/ip-addresses/14994/ from `vlan110... [12:25:44] 06cloud-services-team, 10Horizon: Horizon: obsessive redirects during logins - https://phabricator.wikimedia.org/T383370#10520737 (10Andrew) 05Open→03Resolved a:03Andrew [12:33:47] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Unable to persistently set fs.inotify.max_user_instances and fs.inotify.max_user_watches - https://phabricator.wikimedia.org/T385530#10520755 (10Andrew) >>! In T385530#10520042, @taavi wrote: > Instead of making a inotify-specific thing, any reason not... [12:57:24] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: replace cloudgw100[12] with spare 'second region' dev servers cloudnet100[78]-dev - https://phabricator.wikimedia.org/T382356#10520852 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for... [12:57:53] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: replace cloudgw100[12] with spare 'second region' dev servers cloudnet100[78]-dev - https://phabricator.wikimedia.org/T382356#10520853 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for ho... [13:04:31] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: replace cloudgw100[12] with spare 'second region' dev servers cloudnet100[78]-dev - https://phabricator.wikimedia.org/T382356#10520889 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host... [13:04:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:12:09] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: replace cloudgw100[12] with spare 'second region' dev servers cloudnet100[78]-dev - https://phabricator.wikimedia.org/T382356#10520895 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for... [13:12:09] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: replace cloudgw100[12] with spare 'second region' dev servers cloudnet100[78]-dev - https://phabricator.wikimedia.org/T382356#10520912 (10aborrero) we decided to reimage the new hosts clodugw1004 and cloudgw1003 on bullseye rather th... [13:34:25] (03CR) 10Nikerabbit: [C:04-1] WIP Merge lego footer messages into one [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1003574 (https://phabricator.wikimedia.org/T355011) (owner: 10Amire80) [13:36:01] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: replace cloudgw100[12] with spare 'second region' dev servers cloudnet100[78]-dev - https://phabricator.wikimedia.org/T382356#10520987 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host c... [13:45:04] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: replace cloudgw100[12] with spare 'second region' dev servers cloudnet100[78]-dev - https://phabricator.wikimedia.org/T382356#10521011 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host... [14:03:01] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: replace cloudgw100[12] with spare 'second region' dev servers cloudnet100[78]-dev - https://phabricator.wikimedia.org/T382356#10521048 (10Andrew) Now the two hosts are: cloudgw1004 (active), cloudgw1001 (standby). 1002 is ready for... [14:03:09] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: replace cloudgw100[12] with spare 'second region' dev servers cloudnet100[78]-dev - https://phabricator.wikimedia.org/T382356#10521050 (10aborrero) cloudgw1004 took over cloudgw1002 successfully, and is now sustaining high traffic no... [14:16:28] FIRING: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate pontoon-conf-01.monitoring.eqiad.wmflabs is about to expire in 27d 23h 58m 40s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [14:29:41] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [14:54:49] 06cloud-services-team, 10Cloud-VPS, 07IPv6: horizon: enable the UI to select networks on VM creation panel - https://phabricator.wikimedia.org/T380081#10521301 (10taavi) I see the list includes a bunch of infrastructure networks, I assume that's only shown for admins? [15:01:34] (03CR) 10Majavah: [C:03+2] "bah. someone's added `labs/tools/` so we could just as well add all of `labs/` and `cloud/`. (`labs/tools/` makes this index not very usef" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1114742 (owner: 10FNegri) [15:02:41] (03Merged) 10jenkins-bot: Add missing WMCS repos [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1114742 (owner: 10FNegri) [15:18:17] FIRING: [2x] KernelErrors: Server cloudgw1002 logged kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudgw1002 - https://alerts.wikimedia.org/?q=alertname%3DKernelErrors [15:18:24] 06cloud-services-team: KernelErrors Server cloudgw1002 logged kernel errors - https://phabricator.wikimedia.org/T385595 (10phaultfinder) 03NEW [15:42:27] 06cloud-services-team: KernelErrors Server cloudgw1002 logged kernel errors - https://phabricator.wikimedia.org/T385595#10521497 (10aborrero) 05Open→03Invalid [15:48:01] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [16:23:27] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS: Cloud DNS: investigate weird graph during cloudgw replacement operation - https://phabricator.wikimedia.org/T385600 (10aborrero) 03NEW [16:24:26] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS: Cloud DNS: investigate weird graph during cloudgw replacement operation - https://phabricator.wikimedia.org/T385600#10521713 (10aborrero) [16:24:35] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS: Cloud DNS: investigate weird graph during cloudgw replacement operation - https://phabricator.wikimedia.org/T385600#10521714 (10aborrero) p:05Triage→03Low [16:25:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-76 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [16:25:49] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS: Cloud DNS: investigate weird graph during cloudgw replacement operation - https://phabricator.wikimedia.org/T385600#10521718 (10aborrero) [16:38:30] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS: Cloud DNS: investigate weird graph during cloudgw replacement operation - https://phabricator.wikimedia.org/T385600#10521753 (10cmooney) There isn't any mystery here. Anycast routing is hot-potato. If cloudgw1001 is the active one, with cloudse1-c8-eqia... [16:42:13] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS: Cloud DNS: investigate weird graph during cloudgw replacement operation - https://phabricator.wikimedia.org/T385600#10521767 (10aborrero) 05Open→03Invalid Thanks! [16:58:17] FIRING: [2x] KernelErrors: Server cloudgw1004 logged kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudgw1004 - https://alerts.wikimedia.org/?q=alertname%3DKernelErrors [16:58:22] 06cloud-services-team: KernelErrors Server cloudgw1004 logged kernel errors - https://phabricator.wikimedia.org/T385601 (10phaultfinder) 03NEW [16:59:18] FIRING: KernelErrors: Server cloudgw1004 logged kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudgw1004 - https://alerts.wikimedia.org/?q=alertname%3DKernelErrors [16:59:22] 06cloud-services-team: KernelErrors Server cloudgw1004 logged kernel errors - https://phabricator.wikimedia.org/T385601#10521848 (10phaultfinder) [17:08:40] 10Cloud Services Proposals, 06cloud-services-team, 10Cloud-VPS: Decision Request - How to openstack projects relate to tofu-infra - https://phabricator.wikimedia.org/T385604 (10aborrero) 03NEW [17:08:49] 10Cloud Services Proposals, 06cloud-services-team, 10Cloud-VPS: Decision Request - How to openstack projects relate to tofu-infra - https://phabricator.wikimedia.org/T385604#10521910 (10aborrero) p:05Triage→03Medium [17:09:04] 10Cloud Services Proposals, 06cloud-services-team, 10Cloud-VPS: Decision Request - How openstack projects relate to tofu-infra - https://phabricator.wikimedia.org/T385604#10521911 (10aborrero) [17:11:57] 06cloud-services-team, 10Data-Services, 06Data-Persistence: meta_p: Don't use utf8mb3 charset and collation - https://phabricator.wikimedia.org/T385456#10521937 (10Ladsgroup) From DBA point of view, `utf8mb4` or `binary` usually should suffice. Binary works when the clients are properly set to work with with... [17:17:25] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10521967 (10sbassett) |**Wikitech account/LDAP:**| [[ https://wikitech.wikimedia.org/wiki/User:SBassett | SBassett ]]| |**SUL account**|[[ https://meta.wikimedia.org/wiki/Special:CentralAuth/... [17:19:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:20:37] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10521981 (10Ladsgroup) @sbassett renamed `SBassett (WMF)` and force attached. [17:27:34] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10522008 (10sbassett) @Ladsgroup - works now, thanks! [17:29:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:40:36] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10522053 (10herron) |**Wikitech account/LDAP:**| Herron| |**SUL account**| KHerron (WMF)| |**Account linked on [[ https://idm.wikimedia.org/ | IDM ]]** |Y| |**I have visited [[ https://wikite... [18:10:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-76 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [18:17:10] 10VPS-project-Wikistats: Add kncwiki to wikistats - https://phabricator.wikimedia.org/T385190#10522186 (10Dzahn) 05Open→03Resolved a:03Dzahn ` MariaDB [wikistats]> insert into wikipedias (prefix, lang, loclang, method) values ("knc","Central Kanuri","Kanuri yerwa",8); .. dzahn@wikistats-bookworm:~$ /u... [18:26:14] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10522233 (10mirror-kt) >>>! In T376267#10515069, @mirror-kt wrote: >> |**Wikitech account/LDAP:**| Misato Kano| >> |**SUL account**| Mirror-kt| >> |**Account linked on [[ https://idm.wikimedi... [18:29:41] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [18:47:52] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10522303 (10Reedy) >>! In T376267#10522233, @mirror-kt wrote: >>>>! In T376267#10515069, @mirror-kt wrote: >>> |**Wikitech account/LDAP:**| Misato Kano| >>> |**SUL account**| Mirror-kt| >>> |... [18:48:55] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10522304 (10Reedy) >>! In T376267#10522053, @herron wrote: > |**Wikitech account/LDAP:**| Herron| > |**SUL account**| KHerron (WMF)| > |**Account linked on [[ https://idm.wikimedia.org/ | IDM... [18:53:45] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10522335 (10bd808) Updated timeline published on Wikitech and announced to wikitech-l and cloud-announce mailing lists: * https://wikitech.wikimedia.o... [18:59:26] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [19:01:34] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10522381 (10Giftpflanze) |**Wikitech account/LDAP:**| gifti| |**SUL account**| Giftpflanze| |**Account linked on [[ https://idm.wikimedia.org/ | IDM ]]** |Y| |**I have visited [[ https://wiki... [19:06:41] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [19:48:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:51:41] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [19:58:41] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [19:58:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:59:43] (03close) 10raymond-ndibe: test commit [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/142 [20:00:27] (03open) 10raymond-ndibe: [toolforge-weld]: add dry_run [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/69 (https://phabricator.wikimedia.org/T359804) [20:08:27] (03update) 10raymond-ndibe: [toolforge-weld]: add dry_run [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/69 (https://phabricator.wikimedia.org/T359804) [20:09:48] (03open) 10raymond-ndibe: [toolforge-weld]: add dry_run [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/70 (https://phabricator.wikimedia.org/T359804) [20:11:24] (03update) 10raymond-ndibe: [toolforge-weld] support apply_object method [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/70 (https://phabricator.wikimedia.org/T359804) [20:18:41] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [20:32:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [20:58:18] FIRING: [2x] KernelErrors: Server cloudgw1004 logged kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudgw1004 - https://alerts.wikimedia.org/?q=alertname%3DKernelErrors [20:58:42] 06cloud-services-team: KernelErrors Server cloudgw1004 logged kernel errors - https://phabricator.wikimedia.org/T385601#10522891 (10phaultfinder) [20:59:18] FIRING: KernelErrors: Server cloudgw1004 logged kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudgw1004 - https://alerts.wikimedia.org/?q=alertname%3DKernelErrors [20:59:44] 06cloud-services-team: KernelErrors Server cloudgw1004 logged kernel errors - https://phabricator.wikimedia.org/T385601#10522892 (10phaultfinder) [21:17:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [21:29:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [21:34:58] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10522967 (10Ladsgroup) Renamed and force attached your account. [21:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:57:26] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [21:59:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:16:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [23:01:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [23:18:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [23:33:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [23:40:49] (03open) 10raymond-ndibe: [jobs-api] replace load with diff_job runtime method [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T359804) [23:47:06] (03update) 10raymond-ndibe: [jobs-api] replace load with diff_job runtime method [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T359804) [23:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:50:45] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/39 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:50:48] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/39 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:50:53] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/39 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:50:59] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/43 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:50:59] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/43 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:51:02] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/43 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:51:07] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/224 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:51:14] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/224 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:51:20] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/65 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:51:21] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/65 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:51:25] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/65 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:51:30] (03update) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/38 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:51:35] (03update) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/38 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:52:07] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/118 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:52:07] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/118 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:52:10] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/118 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:52:17] (03update) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/58 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:52:17] (03approved) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/58 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:52:21] (03merge) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/58 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:52:26] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/23 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:52:27] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/23 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:52:32] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/23 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:53:32] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: builds-builder: bump to 0.0.124-20250204235137-18656c89 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/658 [23:53:44] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: maintain-harbor: bump to 0.0.20-20250204235116-d595f2ae [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/659 [23:54:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [23:55:30] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: api-gateway: bump to 0.0.61-20250204235237-f1e33e1f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/660 [23:55:48] (03update) 10raymond-ndibe: [jobs-api] replace load with diff_job runtime method [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T359804) [23:56:23] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/224 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:56:27] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/224 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:56:34] (03approved) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/38 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:56:37] (03merge) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/38 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:56:47] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/59 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:56:54] (03update) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/75 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:56:55] (03approved) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/75 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:57:02] (03merge) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/75 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:57:04] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/17 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:57:06] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: volume-admission: bump to 0.0.61-20250204235343-f67d11a3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/661 [23:57:12] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/17 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:57:13] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: volume-admission: bump to 0.0.61-20250204235343-f67d11a3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/661 [23:57:14] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/17 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:57:25] (03update) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/141 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:57:26] (03approved) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/141 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:57:30] (03merge) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/141 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:57:35] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/74 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:57:47] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/74 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:57:54] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/140 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:58:04] (03update) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/100 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:58:05] (03approved) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/100 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:58:10] (03merge) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/100 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:58:16] (03update) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/16 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:58:17] (03approved) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/16 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:58:21] (03merge) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/16 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:58:52] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/15 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:59:10] (03update) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/51 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:59:12] (03approved) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/51 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:59:16] (03merge) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/51 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:59:50] (03update) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/16 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:59:51] (03approved) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/16 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:59:54] (03merge) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/16 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:59:59] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/83 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)