[06:47:27] 10netops, 10SRE: Higher latency on Lumen eqiad/esams link - https://phabricator.wikimedia.org/T277654 (10ayounsi) Seems expected: > Lumen Maintenance 20795082 is currently ongoing on your service. You may experience a full service interruption or degradation from Wed, 2021/03/17 04:00:00 GMT to Wed, 2021/03/2... [10:12:59] ema: Welcome back [10:17:31] Amir1: thanks! :) [10:55:42] o/ [14:36:43] gilles: hi, I've seen going through my phab backlog that you added cache_status to response-time-by-host, nice [14:36:55] the dashboard seems broken now though? https://grafana.wikimedia.org/d/M7xQ_BeWk/response-time-by-host?orgId=1 [14:37:50] I don't see eg. webperf_navtiming_responsestart_by_host_seconds_bucket on https://grafana-rw.wikimedia.org/explore -- or any other metric starting with webperf_ [15:26:32] Ema: yes, because the migration to EventGate removed the field they told us which cache host served the request. I sent a VCL patch your way last week to get that data another way, using Server Timing [15:26:40] That told [15:27:29] Varnish will emit the host name in the Server-Timing header, which JS can read and then send back along with the client side perf data [15:27:36] hmm gilles is that info available in a header to eventgate? [15:27:47] oh that is a good way [15:27:49] if the client can send it [15:28:01] Yes I’ll add a field to the schema [15:28:17] gilles: did you want to know wwhich cache host served a page, or which frontend server handled the event POST? [15:28:29] Which one served the page [15:29:34] ok that makes sense [15:29:38] client has to send it then [15:29:52] ema: https://gerrit.wikimedia.org/r/c/operations/puppet/+/673295 [15:29:53] gilles: iiuc though, that was not what you were charting before [15:30:36] It sort of was because the cache host serving the EventLogging was usually the same as the one serving the page [15:30:41] oh [15:30:43] did not know that [15:30:44] ok [15:30:54] i guess that would be true of eventgate posts then too? [15:30:55] The new method will be more accurate and less hacky for sure [15:31:06] if so, you could maybe automatically get this in the http.request_headers map [15:31:10] without adding a new field to the schema [15:31:12] I’d rather get the real info that infer it [15:31:17] Than [15:31:42] I’m not sure that there’s a guarantee per se that they should be the same [15:57:13] 10netops, 10SRE, 10Patch-For-Review: Rename cloud-hosts1-b-eqiad to cloud-hosts1-eqiad - https://phabricator.wikimedia.org/T277771 (10ayounsi) 05Open→03Resolved All done! [15:58:56] 10netops, 10SRE, 10Patch-For-Review: Rename cloud-hosts1-b-eqiad to cloud-hosts1-eqiad - https://phabricator.wikimedia.org/T277771 (10aborrero) thanks! [16:56:10] gilles: excellent, I'll merge tomorrow [17:37:41] gilles: if you like, you could stick that value in the http.request_headers map fiield without changing the schema [17:38:17] but, the semanitcs for that field is not really clear...it contanis informationi about an http request...but which one? in most events atm it is about the http request that posts the event [17:38:31] ottomata: if I make it a request header on the event beacon request? [17:38:40] not sure that's possible [17:38:48] gilles: no, i mean, the client can put whatever it wants there [17:38:53] its already a field in the schema [17:38:59] ah, ok [17:39:22] https://schema.wikimedia.org/repositories//secondary/jsonschema/analytics/legacy/navigationtiming/latest.yaml [17:39:29] request_headers [17:39:30] is a map type [17:39:41] https://wikitech.wikimedia.org/wiki/Event_Platform/Schemas/Guidelines#map_types [17:39:54] so, i dunno if that is quite the right thing to do [17:39:59] but it would work now without a schema change