[17:51:22] does anyone understand how, in grafana for a prometheus query, you make a multi-select for $server work? [17:51:43] yes [17:52:10] I keep getting stuck on something about both the :port part of "instance", and the comma-separated lists [17:52:15] it's a horrid kludge, and actually one of the things that new-grafana was supposed to fix (let me look at the docs on that real quick before I give you the old way) [17:52:34] I'm looking at our "Host Overview" dashboard as an example [17:52:49] the CPU graph there uses: instance=~"$server" [17:53:10] I've defined in a new dashboard what seem to be an identical $server variable with the same regex-stripping of the port number in the variable definition [17:53:28] but it only works if I do instance=~"$server.*" because of the port-matching issue... [17:53:44] (which is I guess a separate issue from the comma-separated multi-select issue) [17:54:11] heh [17:54:45] the first query on the "CPU: utilization" graph is disabled, and the remainder do the instance=~"$server.*" thing [17:54:58] oh, duh, bad example then [17:55:39] that solves that mystery, I didn't realize I was copying the disabled bad example :) [17:55:52] anyway I think you should be able to do instance=~"($server).*" [17:56:52] I don't understand why that works, but it seems to work! [17:56:56] thanks! [17:57:29] it's because Grafana generates $server as `foo1001|foo1002|foo1003` when you ticky multiple boxes [17:58:34] if, in Grafana edit mode, you press the "Query Inspector" button, you get to see the GET request it sends (to Grafana, which contains the expanded-and-then-URL-encoded Prometheus query as an HTTP query arg) [17:58:52] yeah [17:58:56] it does make a | now [17:59:06] but it was a , when it was "$server.*" [17:59:13] I guess context-sensitive? [17:59:22] anyways [18:00:18] yeah, not sure about that :) [18:34:23] is there any good example dash that manages to abstract away summing-over-servers vs breaking out servers? [18:34:59] https://grafana.wikimedia.org/d/Jj8MztfZz/authoritative-dns [18:35:08] ^ I'm working on this one to replace the other DNS panel [18:35:28] those two panels show the same basic thing, but one aggregates the currently-selected servers, and one breaks them out [18:35:42] both are useful views of the data, but it seems a shame to double up every graph, too :) [18:36:19] I was trying to think of a way I could define a $variable that effectively toggles in the the effect of the sum(), somehow [18:42:58] but that seems like a pretty likely thing to want to do, there must be a simpler way [18:44:57] one such compromise I've seen is on the cluster dashboard, with the toggles to open-up a per-host breakdown [18:46:08] right, I think that's the repeating-panel pattern [18:46:14] yeah [19:45:12] getting better now [19:45:14] https://grafana.wikimedia.org/d/Jj8MztfZz/authoritative-dns?orgId=1 [19:45:37] to replace https://grafana.wikimedia.org/d/000000341/dns?orgId=1 [19:46:18] thanks for the help! :) [19:47:29] now to get them aggregate to the global view too..