[08:40:01] * volans back FYI, reading backlogs and emails
[08:44:08] <godog>	 welcome back volans !
[14:16:52] <cdanis>	 btw godog -- re https://wikitech.wikimedia.org/wiki/Incident_documentation/20190425-prometheus -- I think I have found some of the queries responsible
[14:18:14] <cdanis>	 there are some prometheus-side settings to limit max # of samples read for a query (which has a good but not perfect correspondence to RAM consumption)
[14:21:16] <godog>	 cdanis: awesome (re: problematic queries), seems like the default limits are a big generous/optimistic
[14:21:29] <cdanis>	 in 2.something they added a limit for the # of samples
[14:21:48] <cdanis>	 let me find the post
[14:22:10] <cdanis>	 https://www.robustperception.io/limiting-promql-resource-usage
[14:22:19] <cdanis>	 --query.max_samples in 2.5.0
[14:22:36] <cdanis>	 in theory we were under this limit at the defaults!
[14:27:21] <godog>	 sigh, mmh perhaps we could start with 30/40% reduction in max samples
[14:29:20] <cdanis>	 also maybe I should try asking in #prometheus on freenode again
[14:33:30] <godog>	 why not, can't hurt! I'm lurking there and folks seem to be generally helpful
[14:51:48] <cdanis>	 yeah, last time I asked no one seemed to be around/interested; I'll try again
[17:09:19] <cdanis>	 godog: these swift replicates are taking unreasonably long
[17:09:29] <cdanis>	 it will be days heh