[12:54:15] <dhinus>	 marostegui: mysql is not running on clouddb1029, is that expected?
[12:56:01] <marostegui>	 dhinus: I've not touched that host
[12:56:39] <dhinus>	 I haven't touched it either, in phab I see you wrote on Jul, 13th "clouddb1029 is now replicating s5 s8"
[12:56:43] <dhinus>	 let me check when it stopped
[12:57:20] <dhinus>	 earlier today: Error reading relay log event: slave SQL thread was killed
[12:57:22] <marostegui>	 dhinus: Jul 24 11:33:28 clouddb1029 mysqld[49680]: 2026-07-24 11:33:28 0 [Note] InnoDB: Starting shutdown...
[12:58:06] <marostegui>	 dhinus: both instances, maybe your script?
[12:58:19] <marostegui>	 It was shutdown cleanly and at the same time
[12:58:28] <dhinus>	 checking the logs
[13:01:15] <dhinus>	 ah I see, yes it was my cookbook and it was an interesting combination of events
[13:01:28] <dhinus>	 it tried to stop all 3 instances, but one was not running
[13:01:34] <marostegui>	 dhinus: but it wasn't rebooted though
[13:01:46] <dhinus>	 yes because it doesn't handle this case, so it crashed and stopped there
[13:02:13] <dhinus>	 the script tries to stop all instances, then reboot. if one of the instances cannot be stopped, instead of skipping it, it just crashes :/
[13:02:37] <marostegui>	 and leaves it like that?
[13:03:02] <dhinus>	 yes, it reenables puppet, but does not try to restart it. I need to fix this
[13:03:11] <dhinus>	 *does not try to restart mariadb
[13:03:51] <dhinus>	 I'm restarting mariadb manually
[13:04:49] <dhinus>	 mind if I start all sections, even if x3 is depooled?
[13:06:02] <dhinus>	 s5 and s8 restarted and catching up
[13:06:56] <marostegui>	 dhinus: x3 is no more on that host
[13:07:21] <dhinus>	 you're right, then why the cookbook was detecting it? checking
[13:08:13] <dhinus>	 maybe because mysql.x3 is still present
[13:08:20] <marostegui>	 yeah, maybe a leftover
[13:08:27] <marostegui>	 we can just remove it
[13:08:59] <dhinus>	 spicerack is actually checking "/etc/mysql/mysqld.conf.d/*.cnf"
[13:09:29] <dhinus>	 I will delete both /etc/mysql/mysqld.conf.d/x3.cnf and /usr/local/bin/mysql.x3
[13:09:53] <marostegui>	 +1
[13:10:32] <dhinus>	 done
[13:10:41] <dhinus>	 s5 and s8 are back in sync, I will retry the reboot and it should work now
[13:10:44] <marostegui>	 thanks
[13:17:59] <dhinus>	 1029 rebooted successfully, I'm doing 1030, 1031 and 1033, then only 1032 is left and I'll leave that to you
[13:20:53] <marostegui>	 1032 was already done
[13:25:08] <dhinus>	 ok excellent, all clouddbs are done then!
[13:25:15] <marostegui>	 great!
[13:30:41] <dhinus>	 1029 also has "wmf-pt-kill@x3" that needs removing, doing it now
[13:30:53] <marostegui>	 thank you
[13:34:00] <dhinus>	 done