[08:43:18] <godog>	 started tomorrow's pad https://etherpad.wikimedia.org/p/SRE-Foundations-2019-07-31
[16:52:56] <paravoid>	 hey folks
[16:53:05] <paravoid>	 where are we with the logstash postmortem?
[16:53:10] <paravoid>	 I see nothing on https://wikitech.wikimedia.org/wiki/Incident_documentation still?
[16:53:57] <cdanis>	 https://wikitech.wikimedia.org/wiki/Incident_documentation/20190723-logstash
[16:54:22] <paravoid>	 oh I see it now
[16:54:22] <paravoid>	 doh
[20:14:13] <XioNoX>	 godog: https://github.com/Selfnet/prometheus-network-exporter
[20:14:39] <cdanis>	 :O
[20:17:13] <XioNoX>	 not sure how efficient it is, as it uses netconf over ssh
[20:17:28] <XioNoX>	 but it's better than snmp
[20:17:30] <cdanis>	 lol, 'utitlities.py'
[20:17:59] <XioNoX>	 but not as good a the junos streaming telemetry stuff
[20:20:54] <XioNoX>	 where to start looking when promehtheus doesn't want to show me data? https://grafana.wikimedia.org/d/dxbfeGDZk/internal-anycast?orgId=1
[20:21:23] <XioNoX>	 nothing in the logs, the prometheus hosts can query the exporter, the dashboard queries look good
[20:24:44] <cdanis>	 ✔️ root@prometheus1003.eqiad.wmnet ~ 🕟⁉️  cat /srv/prometheus/ops/targets/bird_eqiad.yaml 
[20:24:47] <cdanis>	 # This file is managed by puppet
[20:24:49] <cdanis>	 ✔️ root@prometheus1003.eqiad.wmnet ~ 🕟⁉️  
[20:25:02] <cdanis>	 so the file is being generated empty, which seems wrong
[20:28:53] <XioNoX>	 yeah for sure
[20:30:10] <XioNoX>	 maybe because of that line https://github.com/wikimedia/puppet/blob/production/modules/profile/manifests/prometheus/ops.pp#L646
[20:30:38] <cdanis>	 does anything in the fleet actually have profile::bird?
[20:31:27] <XioNoX>	 nop
[20:31:30] <XioNoX>	 only profile::bird::anycast
[20:31:40] <XioNoX>	 cdanis: thanks!
[20:33:01] <cdanis>	 one of these days i will understand how to use puppetboard's query page
[20:34:37] <XioNoX>	 cdanis: https://gerrit.wikimedia.org/r/c/operations/puppet/+/526521
[20:42:17] <XioNoX>	 it works!
[20:42:47] <cdanis>	 \o/
[21:50:54] <volans>	 cdanis: $ sudo cumin 'P:bird'
[21:50:55] <volans>	 No hosts found that matches the query
[21:50:56] <volans>	 :-P
[21:54:23] <volans>	 jokes apart, the resources endpoint is not enabled in config, I'm trying to recall why. We disabled a bunch of them because potentially could expose private data
[21:55:33] <volans>	 and resources is one of them IIRC, and that's the one you need to do that kind of query
[21:55:48] <volans>	 (is the same endpoint that cumin uses fwiw)