[09:10:17] <moritzm>	 FYI, there's some inconsistency on installed wmfmariadbpy packages on cumin2002; python3-wmfmariadbpy, python3-wmfmariadbpy-remote and wmfmariadbpy-admin would be downgraded when running dist-upgrade
[09:10:39] <marostegui>	 federico3: ^
[09:17:27] <jynus>	 I'm going to do a full restart of db1171
[09:45:10] <federico3>	 moritzm: thanks, it should be fixed now
[09:54:34] <moritzm>	 ack!
[10:32:31] <jynus>	 dbprovs are all now on 10.11.14 and 6.1.0-41
[10:54:52] <moritzm>	 FYI, I'm planning to migrate cumin2002 to nftables in ~ 15 minutes (which includes a reboot). unless that is an issue for anyone, then please let me know
[10:55:09] <marostegui>	 from my side it is ok
[11:06:15] <federico3>	 marostegui: at some point we'll have to do a flip for the s8 master, I generated the task at https://phabricator.wikimedia.org/T409818
[11:06:33] <marostegui>	 federico3: want to do it tomorrow at 7:3AM?
[11:06:39] <marostegui>	 I can be present if you want
[11:07:19] <federico3>	 I've never done a primary master switchover
[11:07:27] <marostegui>	 Then it is time :)
[11:07:44] <marostegui>	 I can shadow you, but essentially following the steps there will take you to the finish line
[11:07:57] <federico3>	 ok
[11:08:03] <marostegui>	 7:30AM tomorrow then?
[11:08:22] <federico3>	 yep
[11:09:32] <marostegui>	 I'd suggest you do a couple of things today, to save time tomorrow: the first step, check the configuration differences (there will be a bunch as we are migrating between majors) and change the lines:  Merge gerrit puppet change to promote NEW primary: FIXME and  Merge DNS change: FIX ME
[11:09:32] <marostegui>	  with the links to the gerrit patches, trust me, it will be easier tomorrow if you don't have to look for them in the task
[11:09:32] <marostegui>	  
[11:10:40] <federico3>	 ok
[11:43:46] <moritzm>	 cumin2002 is on nftables and can be used again (I also updated the ancient firmware/IDRAC, so the update took a little longer)
[12:33:05] <federico3>	 2239 is warning about memory usage: https://zarcillo.wikimedia.org/ui/sections#s3
[18:15:26] <jinxer-wm>	 FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on db2230:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[22:15:41] <jinxer-wm>	 FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on db2230:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed