[00:00:00] <wikibugs>	 (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/83 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:00:03] <wikibugs>	 (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/83 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:00:13] <wikibugs>	 (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/15 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:00:20] <wikibugs>	 (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/19 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:00:22] <wikibugs>	 (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/19 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:00:25] <wikibugs>	 (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/19 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:00:40] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: jobs-api: bump to 0.0.348-20250204235743-8cc6991d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/662
[00:00:43] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: jobs-api: bump to 0.0.348-20250204235743-8cc6991d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/662
[00:00:51] <wikibugs>	 (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/16 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:00:53] <wikibugs>	 (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/16 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:00:58] <wikibugs>	 (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/16 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:01:13] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: builds-api: bump to 0.0.178-20250204235225-204a2a86 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/663
[00:01:15] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: builds-api: bump to 0.0.178-20250204235225-204a2a86 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/663
[00:01:22] <wikibugs>	 (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/71 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:01:25] <wikibugs>	 (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/71 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:01:36] <wikibugs>	 (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/71 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:01:37] <wikibugs>	 (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/99 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:01:46] <wikibugs>	 (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/140 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:01:46] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: envvars-admission: bump to 0.0.24-20250204235727-1c4069a7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/664
[00:01:52] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: envvars-admission: bump to 0.0.24-20250204235727-1c4069a7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/664
[00:01:55] <wikibugs>	 (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/140 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:01:57] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: jobs-emailer: bump to 0.0.49-20250204235856-e9daf12d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/665
[00:01:58] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: jobs-emailer: bump to 0.0.49-20250204235856-e9daf12d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/665
[00:02:02] <wikibugs>	 (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/74 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:02:06] <wikibugs>	 (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/140 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:02:10] <wikibugs>	 (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/74 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:02:22] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.77-20250204235931-5247bf60 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/666
[00:02:24] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.77-20250204235931-5247bf60 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/666
[00:03:04] <wikibugs>	 (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/50 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:03:23] <wikibugs>	 (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/15 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:03:27] <wikibugs>	 (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/15 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:03:35] <wikibugs>	 (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/15 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:03:43] <wikibugs>	 (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/15 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:03:44] <wikibugs>	 (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/15 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:03:47] <wikibugs>	 (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/15 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:03:56] <wikibugs>	 (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/59 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:03:56] <wikibugs>	 (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/59 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:03:59] <wikibugs>	 (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/59 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:04:06] <wikibugs>	 (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/99 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:04:08] <wikibugs>	 (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/99 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:04:11] <wikibugs>	 (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/99 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[00:04:11] <jinxer-wm>	 RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[00:05:22] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: ingress-admission: bump to 0.0.56-20250205000153-8f1e7076 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/667
[00:05:51] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: jobs-emailer: bump to 0.0.49-20250204235856-e9daf12d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/665
[00:08:54] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.77-20250204235931-5247bf60 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/666
[00:08:57] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.77-20250204235931-5247bf60 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/666
[00:21:11] <jinxer-wm>	 FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[00:29:37] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] replace load with diff_job runtime method [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T359804)
[00:29:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[00:36:11] <jinxer-wm>	 RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[00:41:35] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] replace load with diff_job runtime method [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T359804)
[00:43:54] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] replace load with diff_job runtime method [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T359804)
[00:52:11] <jinxer-wm>	 FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[00:57:11] <jinxer-wm>	 RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[01:18:11] <jinxer-wm>	 FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[01:23:11] <jinxer-wm>	 RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[01:28:56] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] replace load with diff_job runtime method [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T359804)
[01:34:27] <wikibugs>	 06cloud-services-team, 10Toolforge, 10Wikimedia-Site-requests: Audit wmf-config and add toolforge.org as needed where tools.wmflabs.org is used - https://phabricator.wikimedia.org/T285364#10523781 (10Pppery) 05Open→03Resolved https://github.com/search?q=repo%3Awikimedia%2Foperations-mediawiki-config%...
[01:45:11] <jinxer-wm>	 FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[02:07:41] <jinxer-wm>	 RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[02:19:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[02:28:11] <jinxer-wm>	 FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[02:48:11] <jinxer-wm>	 RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[02:59:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[03:16:11] <jinxer-wm>	 FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[03:21:11] <jinxer-wm>	 RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[03:22:11] <jinxer-wm>	 FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[03:31:26] <jinxer-wm>	 RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[03:32:11] <jinxer-wm>	 FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[03:36:26] <jinxer-wm>	 RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[04:26:11] <jinxer-wm>	 FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[04:46:11] <jinxer-wm>	 RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[04:49:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[04:59:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[05:37:11] <jinxer-wm>	 FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[05:47:11] <jinxer-wm>	 RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[06:35:11] <jinxer-wm>	 FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[06:40:11] <jinxer-wm>	 RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[07:28:11] <jinxer-wm>	 FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[07:33:11] <jinxer-wm>	 RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[08:23:11] <jinxer-wm>	 FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[08:33:11] <jinxer-wm>	 RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[09:04:11] <jinxer-wm>	 FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[09:14:11] <jinxer-wm>	 RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[09:29:11] <jinxer-wm>	 FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[09:38:48] <wikibugs>	 06cloud-services-team: kernel error detector: have a way to ignore certain messages - https://phabricator.wikimedia.org/T380960#10524213 (10aborrero) 05Open→03Resolved a:03aborrero
[09:44:11] <jinxer-wm>	 RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[09:49:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[09:52:29] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Cloud VPS: extend tofu-infra to cover projects, users and roles - https://phabricator.wikimedia.org/T371393#10524249 (10aborrero) 05Open→03Resolved Regarding the resources mentioned: * projects: we are not sure if we want to track them - and as...
[09:54:22] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: tofu-infra: migrate default zone creation from the keystone hook - https://phabricator.wikimedia.org/T375720#10524258 (10aborrero) 05Open→03Declined This has the complication of requiring a ownership transfer dance, that it is not very convenient to implement using o...
[09:54:56] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Cloud VPS: extend tofu-infra to cover quotas - https://phabricator.wikimedia.org/T371391#10524268 (10aborrero)
[09:56:21] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Support managing Cloud VPS project membership via OpenTofu - https://phabricator.wikimedia.org/T320750#10524280 (10aborrero) I'm clarifying the scope in the ticket description.
[09:57:50] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Support managing Cloud VPS project membership via OpenTofu - https://phabricator.wikimedia.org/T320750#10524284 (10aborrero)
[09:58:13] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Support managing Cloud VPS project membership via OpenTofu - https://phabricator.wikimedia.org/T320750#10524287 (10aborrero)
[09:59:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[09:59:52] <icinga-wm>	 RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29349 bytes in 0.322 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static
[10:22:37] <wikibugs>	 10wikitech.wikimedia.org: wikitech-static has duplicated (large) files eating up a lot of space - https://phabricator.wikimedia.org/T385672 (10Reedy) 03NEW
[10:25:59] <wikibugs>	 06cloud-services-team, 10Data-Services, 06Data-Persistence: meta_p: Don't use utf8mb3 charset and collation - https://phabricator.wikimedia.org/T385456#10524401 (10fnegri) @Ladsgroup thanks! I vote for `binary` for consistency with the production tables where the same strings are held. But I'm also fine with...
[10:30:11] <jinxer-wm>	 FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[10:39:03] <wikibugs>	 10Cloud Services Proposals, 06cloud-services-team, 10Cloud-VPS: Decision Request  - How openstack projects relate to tofu-infra - https://phabricator.wikimedia.org/T385604#10524451 (10fnegri) I like option 1 //in theory//, but in practice I think option 2 is the best choice at the moment.  Implementing optio...
[10:40:11] <jinxer-wm>	 RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[10:47:28] <wmcs-alerts>	 FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-7 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[10:52:28] <wmcs-alerts>	 RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-7 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[10:55:52] <wikibugs>	 10wikitech.wikimedia.org: wikitech-static has duplicated (large) files eating up a lot of space - https://phabricator.wikimedia.org/T385672#10524579 (10Reedy) p:05Triage→03High
[10:56:39] <wikibugs>	 10wikitech.wikimedia.org: wikitech-static has duplicated (large) files eating up a lot of space - https://phabricator.wikimedia.org/T385672#10524593 (10Reedy)
[11:03:03] <wikibugs>	 10wikitech.wikimedia.org: wikitech-static has duplicated (large) files eating up a lot of space - https://phabricator.wikimedia.org/T385672#10524661 (10Reedy) ` --- /srv/mediawiki/images/wikitech/archive -------------------------------------------------------------------------------------------------------------...
[11:10:28] <wmcs-alerts>	 FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-7 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[11:15:28] <wmcs-alerts>	 RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-7 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[11:16:11] <jinxer-wm>	 FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[11:21:11] <jinxer-wm>	 RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[11:32:11] <jinxer-wm>	 FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[11:41:42] <icinga-wm>	 PROBLEM - mysqld processes on clouddb1017 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting
[11:52:42] <icinga-wm>	 RECOVERY - mysqld processes on clouddb1017 is OK: PROCS OK: 2 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting
[12:04:08] <wikibugs>	 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T385491#10525008 (10fnegri) 05Open→03Resolved a:03fnegri This was caused by {T380960} and is now resolved.
[12:39:23] <wikibugs>	 10Cloud Services Proposals, 06cloud-services-team, 10Cloud-VPS: Decision Request  - How openstack projects relate to tofu-infra - https://phabricator.wikimedia.org/T385604#10525161 (10aborrero)
[12:39:25] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07Epic: Cloud VPS: extend tofu-infra coverage - https://phabricator.wikimedia.org/T370037#10525162 (10aborrero)
[12:42:53] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: tofu-infra: refactor repo structure - https://phabricator.wikimedia.org/T375283#10525166 (10aborrero) 05Stalled→03In progress I'll restart work on this.
[12:48:28] <wmcs-alerts>	 FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-7 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[12:53:28] <wmcs-alerts>	 RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-7 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[13:02:11] <jinxer-wm>	 RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[13:21:58] <wikibugs>	 10Tool-ranker, 06translatewiki.net, 10LPL Essential (LPL Essential 2024 Nov-Jan), 13Patch-For-Review, 07Unplanned-Sprint-Work: Add Ranker to translatewiki.net - https://phabricator.wikimedia.org/T384061#10525325 (10Nikerabbit)
[13:23:11] <jinxer-wm>	 FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[13:26:32] <wikibugs>	 (03update) 10raymond-ndibe: [toolforge-weld]: add dry_run [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/69 (https://phabricator.wikimedia.org/T359804)
[13:28:58] <wikibugs>	 (03update) 10raymond-ndibe: [toolforge-weld] support apply_object method [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/70 (https://phabricator.wikimedia.org/T359804)
[13:32:39] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T383723#10525385 (10Andrew) This is flapping like crazy, I ack'd it before bed last night but have another 15 alert messages this morning.
[13:48:11] <jinxer-wm>	 RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[13:53:26] <jinxer-wm>	 FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[13:54:18] <jinxer-wm>	 FIRING: [2x] KernelErrors: Server cloudgw1004 logged kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudgw1004 - https://alerts.wikimedia.org/?q=alertname%3DKernelErrors
[13:58:26] <jinxer-wm>	 RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[14:10:20] <wikibugs>	 (03update) 10raymond-ndibe: [toolforge-weld]: add dry_run [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/69 (https://phabricator.wikimedia.org/T359804)
[14:15:43] <wikibugs>	 (03update) 10raymond-ndibe: [toolforge-weld] support apply_object method [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/70 (https://phabricator.wikimedia.org/T359804)
[14:16:29] <wmcs-alerts>	 FIRING: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate pontoon-conf-01.monitoring.eqiad.wmflabs is about to expire in 26d 23h 58m 40s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire
[14:17:45] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07IPv6: horizon: enable the UI to select networks on VM creation panel - https://phabricator.wikimedia.org/T380081#10525506 (10Andrew) >>! In T380081#10521301, @taavi wrote: > I see the list includes a bunch of infrastructure networks, I assume that's only shown for admins...
[14:35:11] <jinxer-wm>	 FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[14:45:11] <jinxer-wm>	 RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[14:48:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[14:49:48] <jinxer-wm>	 RESOLVED: KernelErrors: Server cloudgw1004 logged kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudgw1004 - https://alerts.wikimedia.org/?q=alertname%3DKernelErrors
[14:52:45] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: tofu-infra: refactor repo structure - https://phabricator.wikimedia.org/T375283#10525593 (10fnegri) @aborrero my patch https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 is in a more-or-less working sta...
[14:54:55] <wikibugs>	 (03open) 10raymond-ndibe: [toolforge-weld] make user_agent importable [repos/cloud/toolforge/toolforge-weld] (add_apply_object) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/71 (https://phabricator.wikimedia.org/T359804)
[14:55:14] <wikibugs>	 (03update) 10raymond-ndibe: [toolforge-weld] make user_agent importable [repos/cloud/toolforge/toolforge-weld] (add_apply_object) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/71 (https://phabricator.wikimedia.org/T359804)
[15:01:10] <wikibugs>	 06cloud-services-team: KernelErrors Server cloudgw1004 logged kernel errors - https://phabricator.wikimedia.org/T385601#10525645 (10Andrew) 05Open→03Invalid
[15:03:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[15:06:11] <jinxer-wm>	 FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[15:06:40] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: CloudVPSDesignateLeaks alert is flapping - https://phabricator.wikimedia.org/T384118#10525676 (10Andrew) p:05Triage→03Medium a:03Andrew
[15:08:58] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Unable to persistently set fs.inotify.max_user_instances and fs.inotify.max_user_watches - https://phabricator.wikimedia.org/T385530#10525692 (10joanna_borun) p:05Triage→03Medium
[15:09:00] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Unable to persistently set fs.inotify.max_user_instances and fs.inotify.max_user_watches - https://phabricator.wikimedia.org/T385530#10525694 (10Andrew) a:03Andrew
[15:09:21] <wikibugs>	 06cloud-services-team, 10Data-Services, 06Data-Persistence: meta_p: Don't use utf8mb3 charset and collation - https://phabricator.wikimedia.org/T385456#10525695 (10joanna_borun) p:05Triage→03Low
[15:09:58] <wikibugs>	 06cloud-services-team: SystemdUnitDown The systemd unit kiwix-mirror-update.service on node clouddumps1001 has been failing for more than two hours. - https://phabricator.wikimedia.org/T385406#10525698 (10fnegri) 05Open→03Resolved a:03fnegri This is working now, not sure what caused the error.
[15:10:29] <wikibugs>	 06cloud-services-team, 10Toolforge: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T385400#10525702 (10joanna_borun) p:05Triage→03Medium
[15:11:53] <wikibugs>	 06cloud-services-team: NetworkOutSaturated Outgoing network saturation detected on clouddumps1002:9100. - https://phabricator.wikimedia.org/T385379#10525706 (10fnegri) 05Open→03Resolved a:03fnegri This lasted for a few hours on Feb, 2nd.  @cmooney do you think we should worry about this?
[15:12:07] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Changing the IPs of cloudcephmons should not require VM reboots - https://phabricator.wikimedia.org/T385288#10525709 (10joanna_borun) p:05Triage→03Medium
[15:12:45] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: VM live migration failing for many/most VMs - https://phabricator.wikimedia.org/T385264#10525710 (10joanna_borun) p:05Triage→03High
[15:13:35] <wikibugs>	 06cloud-services-team: KernelErrors - https://phabricator.wikimedia.org/T385165#10525719 (10fnegri) 05Open→03Resolved a:03fnegri Host was rebooted
[15:13:56] <wikibugs>	 06cloud-services-team: KernelErrors Server cloudvirt1031 logged kernel errors - https://phabricator.wikimedia.org/T385163#10525724 (10fnegri) 05Open→03Resolved a:03fnegri Host was rebooted
[15:14:03] <wikibugs>	 06cloud-services-team: KernelErrors Server cloudvirt1034 logged kernel errors - https://phabricator.wikimedia.org/T385166#10525728 (10fnegri) 05Open→03Resolved a:03fnegri Host was rebooted
[15:14:40] <wikibugs>	 06cloud-services-team, 10Toolforge, 06Community-Tech, 10WS Export: Add 'Content-Length' in ws-export HTTP Response - https://phabricator.wikimedia.org/T384803#10525731 (10joanna_borun) p:05Triage→03Low
[15:15:18] <wikibugs>	 06cloud-services-team, 10Toolforge: Toolforge webservice still accepts --canonical but complains about it in service.template - https://phabricator.wikimedia.org/T384788#10525733 (10Andrew) This is fixed now, yes?
[15:15:34] <wikibugs>	 06cloud-services-team, 10Toolforge, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 07Epic: [toolforge,storage,infra,k8s] Investigate persistent volume support - https://phabricator.wikimedia.org/T384596#10525734 (10joanna_borun) p:05Triage→03Medium
[15:15:42] <wikibugs>	 06cloud-services-team, 10Toolforge, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 07Epic: [toolforge,storage,infra,k8s] Investigate persistent volume support - https://phabricator.wikimedia.org/T384596#10525735 (10fnegri)
[15:15:58] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project: [dbaas,toolsdb] Add support for management of toolsdb databases within toolforge - https://phabricator.wikimedia.org/T384591#10525737 (10joanna_borun) p:05Triage→03Medium
[15:16:11] <jinxer-wm>	 RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[15:16:19] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 07Epic: [dbaas,toolsdb] Add support for management of toolsdb databases within toolforge - https://phabricator.wikimedia.org/T384591#10525738 (10fnegri)
[15:16:40] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 07Epic: [dbaas,toolsdb] Add support for management of toolsdb databases within toolforge - https://phabricator.wikimedia.org/T384591#10525740 (10fnegri) a:03fnegri
[15:17:13] <wikibugs>	 06cloud-services-team, 10Openstack-Magnum: CSI Cinder issues causing periodic failures on Magnum cluster - https://phabricator.wikimedia.org/T383560#10525742 (10joanna_borun) p:05Triage→03Medium
[15:18:19] <wikibugs>	 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade ingress-nginx to v1.12.0+ - https://phabricator.wikimedia.org/T383516#10525746 (10joanna_borun) p:05Triage→03Low
[15:18:44] <wikibugs>	 06cloud-services-team, 10Toolforge, 10Phabricator, 10GitLab (Auth & Access): Look for ways to consolidate "we trust this human" access lists - https://phabricator.wikimedia.org/T364516#10525748 (10joanna_borun) p:05Triage→03Medium
[15:27:38] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07IPv6: horizon: enable the UI to select networks on VM creation panel - https://phabricator.wikimedia.org/T380081#10525797 (10taavi) With my admin account I can see `cloud-flat-codfw1dev`, `cloud-flat-codfw1dev-ipv4only`, `lan-flat-cloudinstances2b` and `wan-transport-cod...
[15:31:04] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: tofu-infra: refactor repo structure - https://phabricator.wikimedia.org/T375283#10525811 (10aborrero) >>! In T375283#10525592, @fnegri wrote: > @aborrero my patch https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_...
[15:33:18] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: tofu-infra: refactor repo structure - https://phabricator.wikimedia.org/T375283#10525834 (10fnegri) a:05fnegri→03aborrero Sounds good!
[15:42:30] <wikibugs>	 (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri)
[16:09:43] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] replace load with diff_job runtime method [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T359804)
[16:38:08] <wikibugs>	 10wikitech.wikimedia.org: ☂  Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10526048 (10dduvall) |**Wikitech account/LDAP:**| Dduvall| |**SUL account**| DDuvall (WMF)| |**Account linked on [[ https://idm.wikimedia.org/ | IDM ]]** |Y| |**I have visited [[ https://wiki...