[00:12:34] <jinxer-wm>	 FIRING: DiskSpace: Disk space thanos-be2006:9100:/ 0.9306% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=thanos-be2006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[00:17:34] <jinxer-wm>	 RESOLVED: DiskSpace: Disk space thanos-be2006:9100:/ 0% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=thanos-be2006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[08:26:05] <jynus>	 Expected? Last snapshot for s7 at eqiad (db1171) taken on 2026-04-12 23:09:24 is 742 GiB, but the previous one was 941 GiB, a change of -21.2 %
[08:39:21] <jynus>	 These are the files that have shrinked the most: https://phabricator.wikimedia.org/P90472
[08:40:04] <jynus>	 has there been any optimize?
[08:41:26] <jynus>	 that's between the 10th and the 12th of april
[08:48:53] <jynus>	 ok, I found it: https://wikitech.wikimedia.org/wiki/Map_of_database_maintenance
[08:48:59] <jynus>	 so all good
[10:32:03] <Emperor>	 Hi folks, could I get a +1 on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1270382 please? remove 3 drained nodes from the rings so they can be re-imaged.
[12:11:37] <Amir1>	 jynus: oh sorry. yup. That's the optimize thingy. We deleted 10M pages on viwiki, which should make the optimize even more impactful
[14:13:12] <marostegui>	 btullis: an-redacteddb is down?
[14:13:45] <btullis>	 I'm in the middle of reimaging it at the moment, so I thought it would have been silenced.
[14:14:00] <marostegui>	 ah ok, yeah, no worries, just saw it in orchestrator
[14:14:05] <marostegui>	 thank you
[14:14:35] <btullis>	 Cool. It's for T421714
[14:14:36] <stashbot>	 T421714: Data platform: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421714
[14:14:46] <marostegui>	 thanks!
[15:26:30] <marostegui>	 dhinus: is dead and will never come back
[15:28:08] <dhinus>	 marostegui: you're referring to clouddb1019?
[15:29:29] <marostegui>	 sorry, yes
[15:29:37] <marostegui>	 I will get it decommissioned https://phabricator.wikimedia.org/T423151
[15:38:39] <dhinus>	 ack thanks. let's hope that the replacement host arrives soon...
[16:28:59] <jinxer-wm>	 FIRING: SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[16:33:59] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[17:09:31] <Amir1>	 federico3: can you stop your schema change on s7? I've been running the optimize table on it and is marked in https://wikitech.wikimedia.org/wiki/Map_of_database_maintenance it is now running optimize table on a pooled host...
[20:25:22] <federico3>	 Amir1: they are acting on different DCs tho, the column drop is  only in codfw, should I stop it anyways?
[21:24:39] <Amir1>	 the optimize is on both DCs
[21:24:59] <Amir1>	 and has been
[21:45:36] <federico3>	 Amir1: the schema change completed in the meantime
[23:57:34] <jinxer-wm>	 FIRING: DiskSpace: Disk space thanos-be2006:9100:/ 3.861% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=thanos-be2006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace