[07:46:31] <moritzm>	 jynus: good morning, Jaime! ok to go ahead with https://gerrit.wikimedia.org/r/#/c/233671/ ?
[07:46:51] <jynus>	 let me open my monitoring, and lets do it
[07:47:25] <moritzm>	 ok, didn't notice you wrote that five minutes ago :-)
[07:47:49] <jynus>	 oh, didn't want to bother you, as it is not time-critical
[07:48:03] <jynus>	 would have pinged you before office hours
[07:49:21] <jynus>	 moritzm, ready
[07:49:32] <jynus>	 do I merge or do you?
[07:49:38] <moritzm>	 I'll merge
[07:49:54] <jynus>	 let's log it first
[07:50:29] <jynus>	 technically there is probably going to be a disruption, even if only for some seconds
[07:50:52] <jynus>	 thanks
[07:50:53] <moritzm>	 sure, done
[07:51:00] <moritzm>	 making a puppet run on 1043 now
[07:51:56] <jynus>	 and we are back
[07:52:33] <moritzm>	 rules are up, nothing dropped so far
[07:52:40] <jynus>	 hardly disrupting
[07:52:54] <moritzm>	 phabricator search working for me as well
[07:53:13] <jynus>	 low traffic apps only see a spike of latency
[07:53:19] <jynus>	 so not an issue
[07:53:38] <moritzm>	 are the codfw phab dbs in production?
[07:53:47] <jynus>	 only thing that it could be is some long term things, like the bot for statistics or dumps
[07:54:03] <jynus>	 but that should be inside the internal network in any case
[07:54:06] <moritzm>	 otherwise I'd just have them have their cronned puppet runs
[07:54:36] <moritzm>	 yeah, I'll leave the logging enabled on 1043 and check up on it for a while
[07:54:39] <jynus>	 I think there is only 1 node, in a passive way (replicating, but not receiving user traffic)
[07:54:59] <moritzm>	 on labsdb100[1-3] nothing has been dropped yesterday or today, so we should be good there as well
[07:55:30] <jynus>	 yes
[07:55:59] <jynus>	 as I said, my main concern right now would be the heavy-connection datbases and the masters for the wikis
[07:57:13] <moritzm>	 true, these will be a bigger challenge, shall we go ahead with db1069/sanitarium as well?
[07:57:29] <jynus>	 yes
[07:57:38] <moritzm>	 ok, preparing a patch
[07:57:52] <jynus>	 those are important hosts but receive no app traffic
[07:58:14] <jynus>	 so network-wise are very easy to check, with a ver specific role
[07:59:24] <moritzm>	 https://gerrit.wikimedia.org/r/235417 for db1069/sanitarium
[08:00:06] <jynus>	 +1
[08:01:43] <moritzm>	 merging
[08:01:57] <moritzm>	 and logged
[08:03:30] <moritzm>	 making a puppet run
[08:06:04] <jynus>	 lost contact from the monitoring
[08:06:19] <jynus>	 oh, here it is again
[08:06:24] <jynus>	 it took more time than usual
[08:06:34] <jynus>	 probably because it is more loaded and more ports open
[08:08:39] <jynus>	 monitoring is back to normal, replication is working
[08:08:43] <moritzm>	 one dropped connection from iron, was that you?
[08:09:02] <jynus>	 yep
[08:09:10] <jynus>	 cannot connect from iron, true
[08:09:36] <moritzm>	 so aside from the six connection attempts by you all looks good now
[08:09:50] <jynus>	 yes, lets add the iron rule
[08:10:05] <moritzm>	 ok, let me prepare a patch
[08:10:15] <jynus>	 not a big issue
[08:10:31] <jynus>	 the iron thing is for us to administrate from a central place
[08:10:47] <moritzm>	 sure, you need that for other mariadb classes as well or specifically for sanitarium?
[08:11:22] <jynus>	 we generally run mysql queries from iron to all db, es, pc, etc. hosts
[08:12:09] <jynus>	 terbium and tin access is not needed for sanitarium
[08:13:38] <jynus>	 for example, I was running a "watch "mysql -e \"SHOW PROCESSLIST\" remotely
[08:13:47] <moritzm>	 ok, will prepare a patch
[08:14:19] <jynus>	 I think we added it to the main class, I forgot on other less important classes
[08:14:26] <jynus>	 so my fault
[08:15:34] <moritzm>	 the rule to access from iron is already in the common ferm mariadb::ferm class (but limited to the standard 3306/3307, we only need to amend the additional ports sanitarium uses
[08:19:11] <jynus>	 ah
[08:19:13] <jynus>	 of course
[08:19:18] <jynus>	 :-)
[08:19:38] <moritzm>	 jynus: https://gerrit.wikimedia.org/r/235419
[08:20:01] <jynus>	 in the future, I would like to get rid of the 7 instances
[08:20:08] <jynus>	 and with it, the ports
[08:20:14] <jynus>	 but that is not going to be soon
[08:21:00] <jynus>	 +1
[08:21:17] <moritzm>	 sounds good, I'll merge the patch now
[08:23:24] <jynus>	 I can access now
[08:24:09] <moritzm>	 great, also no further dropped traffic, I'll keep an eye on it for the next days, but I think we're all good here
[08:25:29] <jynus>	 for sanitarium do not worry
[14:30:28] <moritzm>	 jynus: I checked the ports for role::mariadb::analytics, do you think it needs additional accessors outside of $INTERNAL? i can run tcpdump on it for a bit to check for current clients
[14:56:10] <jynus>	 if you can, yes, but I was merely -1ing to leave them for later, because they are less straigthforward, and I actually have a meeting with those users on Tuesday
[14:56:35] <jynus>	 I do not want you to waste time if I can solve it myself
[14:58:58] <moritzm>	 sure, I'll do that tomorrow so gather some data