[08:55:36] <XioNoX>	 In ~5min I'm going to do some changes on the cloud-hosts vlan see https://phabricator.wikimedia.org/T261866
[08:59:45] <XioNoX>	 cc arturo
[09:00:00] <arturo>	 👍
[09:00:23] <arturo>	 I'm around to assist you in whatever you may need
[09:01:05] <XioNoX>	 thanks!
[09:01:13] <XioNoX>	 I'll be !log stuff on -operations
[09:01:22] <arturo>	 ack
[09:23:07] <elukey>	 the Apachecon NA this year is virtual, and the cost is really low - https://apachecon.com/acah2020/tracks/
[09:23:32] <elukey>	 the big data tracks are full of interesting things for us (as Analytics), but there might be also something interesting for the whole SRE team
[09:24:17] * moritzm sighs at the https://apachecon.com/acah2020/tracks/openoffice.html track
[09:29:10] * elukey knows that moritzm wants to secretly attend that track
[12:35:38] <kormat>	 godog: is there anyway to disable notifications for a host which is down (and hence can't run puppet)?
[12:36:04] <marostegui>	 kormat: good point...should we start it back even with mysql stopped?
[12:36:12] <_joe_>	 go modify the record in puppetdb
[12:36:24] <_joe_>	 simple enough!
[12:37:00] <kormat>	 marostegui: the host failed to reboot. the power is on, but it's not responsive on console or by network.
[12:37:08] <godog>	 kormat: mhhh which host? if it is DOWN according to icinga then you won't get notifications no, which host is it ?
[12:37:17] <volans>	 kormat: icinga UI
[12:37:18] <kormat>	 godog: db2125
[12:37:34] <volans>	 but make a note to reset the override after the fact
[12:37:39] <marostegui>	 kormat: Oh, I think that happened to me with that same one, have you tried the hard reset option?
[12:37:57] <kormat>	 marostegui: i can try that. i was just curious about icinga
[12:38:19] <godog>	 kormat: yeah propbably easiest to either downtime or disable notifications in icinga
[12:38:22] <kormat>	 volans: ahh, i see. 'disable notifications for checked host'. not a trap at all :)
[12:38:23] <marostegui>	 kormat: sure no rush, it is interesting it is behaving exactly the same way: https://phabricator.wikimedia.org/T260670#6392732
[12:38:38] <volans>	 :)
[12:41:04] <kormat>	 using puppetdb as a source for config tends to strike me as an anti-pattern. but it might be that's the only pattern puppet allows
[12:41:34] <volans>	 puppetdb it's a byproduct, exported resources is the actual pattern :)
[12:43:00] <kormat>	 volans: same difference really
[12:43:47] <godog>	 yeah for use cases like monitoring that require info from all hosts by default that's essentially it
[12:45:02] <kormat>	 "puppet is an anti-pattern" has a nice ring of truth to it :)
[12:47:28] <godog>	 haha I think I made peace with it, though still irritating
[12:48:17] <volans>	 kormat: should be it's tagline :-P
[13:34:59] <marostegui>	 In around 25 minutes I will put wikitech on read only for a around 1 minute to failover its master
[13:52:45] <marostegui>	 If someone needs to merge a DNS change, please talk to me first, I am about to merge but not deploy the m5 DNS change for the failover in around 3 minutes
[14:02:01] <volans>	 ack
[14:02:48] <volans>	 marostegui: an icinga alert might fire, I don't recall if we check that one too
[14:07:16] <marostegui>	 dns changes can flow now normally
[14:07:39] <marostegui>	 volans: not sure either, as wikitech remained up at all times (but read only)
[14:07:56] <volans>	 I meant for merged-not-deployed dns changes :)
[14:08:01] <marostegui>	 aaah :)
[14:08:07] <marostegui>	 it was just 2-3 minutes only :)
[14:08:33] <volans>	 nothing icinga can catch then :D