[04:58:22] <marostegui>	 [06:47:12]  <jinxer-wm> RESOLVED: AlertLintProblem: Linting problems found for MySQLReplicaNotUsingGTID - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem
[04:58:27] <marostegui>	 what was this?
[04:59:15] <marostegui>	 That alert seems broken
[06:01:50] <federico3>	 starting the next switchover 
[06:27:31] <marostegui>	 federico3: did you restart the candidate master for s8 so it is already running the good kernel?
[06:32:04] <federico3>	 @marostegui the past candidate master or the current one? The past was db1193 and was already updated 
[06:32:24] <federico3>	 the current one not yet
[06:32:46] <marostegui>	 federico3: Yeah, I mean the now master
[08:03:23] <federico3>	 @marostegui I'm not sure about the linting alert... team-data-persistence/mysql-gtid.yaml has no "TODO" in it
[08:04:07] <marostegui>	 there are more alerts showing that I created a task, check -sre
[08:04:10] <federico3>	 however the timing is sus https://grafana.wikimedia.org/d/0fec1d02-1b0b-44c0-84b0-64894f3ba682/mariadb-gtid?orgId=1&from=now-6h&to=now&timezone=utc
[08:04:54] <marostegui>	 federico3: They could be valid alerts, but the text is wrong
[08:05:25] <marostegui>	 federico3: that graph is normal, it is part of the switchover
[08:05:45] <federico3>	 apart from the linting, maybe the alert is also too sensitive
[08:36:02] <jynus>	 fishing for a review https://gerrit.wikimedia.org/r/c/operations/puppet/+/1294929 with context: https://phabricator.wikimedia.org/T420506#11962117
[08:37:31] <jynus>	 thank you Manuel
[08:56:34] <jinxer-wm>	 FIRING: DiskSpace: Disk space backup1006:9100:/srv/objectstorage 0.4082% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=backup1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[09:13:21] <jynus>	 ^ handling it, those hosts will soon disappear
[09:14:43] <jynus>	 oh, I see why it alerted, the downtime disappeared because I changed the role to insetup
[09:26:35] <jinxer-wm>	 FIRING: [4x] DiskSpace: Disk space backup2004:9100:/srv/objectstorage 0.2506% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[09:30:27] <jynus>	 I acked it again, sadly alertmanager is not the most intuitive interface, because how it tries to be the most narrow ack rather than a looser one