[13:08:43] <hoo>	 jynus: Think you forgot to ack the icinga alert re es1015 (or schedule downtime)
[13:09:41] <jynus>	 yes, see ops-
[13:10:07] <hoo>	 Wasn't on IRC at that time, but got it on my phone
[13:10:13] <hoo>	 SAL has it, though
[13:10:44] <jynus>	 do not worry, everything is fine. I have to do 20 manual steps to just restart mysql
[13:11:04] <jynus>	 it is easy for me to miss one
[13:11:11] <hoo>	 Yeah, I see
[13:11:27] <hoo>	 in a perfect world systemd would also take care of telling icinga
[13:11:31] <jynus>	 including a couple of mediawiki commits
[13:11:32] <hoo>	 I guess
[13:11:48] <jynus>	 in a perfect world I would have a load balancer
[13:12:04] <jynus>	 and checks would first check if the server is pooled
[13:12:19] <jynus>	 but that requires rearchitecture
[13:13:02] <hoo>	 true
[13:13:19] <hoo>	 even getting an exhaustive list of servers in use by MW in some way sounds quite hard
[13:13:36] <jynus>	 that would fix bot the checks and the commits
[13:14:01] <jynus>	 but do not worry, the ticket is there
[13:14:22] <jynus>	 https://phabricator.wikimedia.org/T119626
[13:14:59] <jynus>	 also needed for other services to query the database reliably
[13:17:27] <hoo>	 Sounds interesting
[13:18:17] <jynus>	 oh, very, if dba team had the time
[13:18:20] <jynus>	 all of us
[13:18:46] <jynus>	 :-)
[13:28:28] <jynus>	 es1015 downtime has finished
[13:28:32] <jynus>	 will return later
[13:29:06] <Reedy>	 it will, or you will? :P
[13:29:25] <hoo>	 Both, eventually
[13:29:26] <jynus>	 hopefully me
[13:29:29] <jynus>	 yes
[19:59:01] <jynus>	 I've logged it, but just in case
[20:10:45] <jynus>	 db1019 is down
[20:10:48] <jynus>	 *sorry
[20:10:53] <jynus>	 I meant es1019
[20:11:13] <jynus>	 it should not cause any impact on production, as we have enough capacity