[08:29:13] <kormat>	 anyone have a trick for mass removal of downtimes?
[08:30:31] <moritzm>	 write a cookbook :-)
[08:30:42] <kormat>	 your definition of "trick" needs.. improving. :)
[08:30:43] <moritzm>	 I think the only way is the GUI, where you can tick off multiple ones
[08:31:22] <marostegui>	 kormat: I do!....let them expire
[08:32:37] <moritzm>	 like if you navigate to the full host list, then tick the affected servers and select "Remove Downtimes (..)" under "Commands for checked hosts(s)"
[08:34:20] <kormat>	 hmm. i'll have a look.
[08:35:08] <kormat>	 ahh, this is much better. thanks moritzm :)
[08:43:42] <kormat>	 victory \o/ https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=all&style=hostdetail&hostprops=1 will show all hosts that are downtimed. bookmarking this
[12:26:46] <cdanis>	 I have a shell one liner as well kormat
[12:30:07] <kormat>	 cost: 0.25 souls
[12:31:20] <cdanis>	 it might have been lost to the sands of time, actually, if icinga1001 has been decommed (haven’t checked yet)
[12:39:04] <godog>	 still alive and kicking
[12:53:32] <kormat>	 godog: poor prometheus2003 sent a mail 13mins ago saying it was unable to fork
[12:55:26] <godog>	 :( indeed poor prometheus2003
[12:55:54] <godog>	 looks like it was due to a heavy query, mitigations are part of o11y OKRs this quarter tho
[12:58:29] <kormat>	 godog: out of interest - do you have a dashboard that shows this clearly?
[13:00:17] <godog>	 kormat: yeah e.g. cluster overview for prometheus/codfw, https://grafana.wikimedia.org/d/000000607/cluster-overview?orgId=1&from=now-1h&to=now&var-site=codfw&var-cluster=prometheus&var-instance=All&var-datasource=thanos
[13:00:27] <godog>	 kormat: also 'prometheus server' https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?orgId=1&refresh=1m&var-Prometheus=%5E.*$&var-RuleGroup=All&var-datasource=codfw%20prometheus%2Fops&from=now-1h&to=now
[13:03:46] <kormat>	 godog: mm. i'm not seeing a graph there that makes it clear that the machine hit max procs
[13:04:03] <kormat>	 oh, or maybe it just ran out of ram?
[13:05:03] <kormat>	 yeah ok, that's what it was
[13:11:31] <kormat>	 it looks like prom node exporter 0.17.0 _just_ missed out on having metrics for number of processes: https://github.com/prometheus/node_exporter/pull/950/files
[13:12:37] <godog>	 :(
[18:02:31] <andrewbogott>	 mutante: I'm looking at bd5ce0aceaeba2a3dece91c9d36a0c9c1492928a and I see a renamed (or replaced?) argument:
[18:02:31] <andrewbogott>	 $puppetmasters = hiera('puppetmaster::servers'),
[18:02:40] <andrewbogott>	 replaced with
[18:02:40] <andrewbogott>	 +    Hash[String, Puppetmaster::Backends] $servers = lookup(puppetmaster::servers),
[18:02:44] <andrewbogott>	 is that on purpose or a typo?
[18:03:48] <rzl>	 do you mean hiera() vs. lookup(), or quotes vs. no quotes?
[18:04:07] <rzl>	 oh wait, neither, you probably mean $puppetmasters vs. $servers, sorry
[18:04:28] <andrewbogott>	 yeah
[18:04:43] <andrewbogott>	 It looks like it was on purpose but I have a spot where the change wasn't tracked which made me wonder
[18:05:01] <andrewbogott>	 I guess I'll hope that it didn't change format as well
[18:08:42] <mutante>	 andrewbogott: it looks like it's my change but it's actually not. but yea, it was on purpose
[18:08:57] <andrewbogott>	 ok, I'll try to fix in the class that calls it
[18:09:08] <andrewbogott>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/633215
[18:09:30] <mutante>	 "we can infer the workers with `$servers[$facts['fqdn']]`"
[18:12:34] <mutante>	 andrewbogott: yes, that looks correct. and no, format has not changed
[18:12:49] <andrewbogott>	 thx
[18:12:54] <mutante>	 it's still getting it from the same key in hiera
[19:36:11] <mutante>	 jbond42: re: nmap (too many channels, not sure where the backlog was). but looks like it was already installed manually in multiple places:
[19:36:15] <mutante>	 https://debmonitor.wikimedia.org/packages/nmap
[19:37:01] <mutante>	 we should probably avoid doing that and make them official
[19:38:19] <mutante>	 the only actual puppet classes installing it are a role pentest::tools which doesn't seem to be used anywhere and diffscan itself
[19:39:13] <mutante>	 somewhat tempted to removed unpuppetized packages
[23:19:32] <cdanis>	 no page at 22:00 today :)