[16:01:33] Krinkle: Found it, indeed removed last month: https://phabricator.wikimedia.org/T359345 [16:02:34] James_F: 2 years + 1 month :) [16:02:50] Counting is hard. [16:02:58] Presumably the dashboard is still trying to load it? [16:03:24] indeed https://grafana-rw.wikimedia.org/d/000000034/media?orgId=1&refresh=5m [16:03:37] most of that dash is fine, but the MMV bits are redundant indeed [16:08:49] Krinkle: I've dropped the old panels. [16:25:31] ack [16:32:24] Krinkle: do you know how to update field types (mappings) in Kibana? It has an IP type, would be nice if clientip was that and not string [16:32:56] (trying to filter out the autologin flood we are still getting from Tencent Cloud) [16:58:46] James_F: just for my edification, what was the process you used to drop those panels? basically did you edit via the ui or remove some grizzly entries someplace? [17:01:57] apergos: Via the UI. "Sign in", click the rubbish bin icon, click agree. [17:02:07] (Then Save the dashboard state for other users.) [17:04:23] tgr|away: is this about a freeform context key-value pair or one of the "standard" fields that all messages emit? [17:04:48] James_F: ok, I wonder if we might not want these things in grizzly in a repo but that's a conversation for another time. [19:06:49] Krinkle: freeform, but it has a distinctive enough name that we can be sure it's always an IP address [19:07:56] hm.. yeah, probably. though I wouldn't be surprised if someone used it for a list of IP adddresses that include XFF or something else concatenated after an IP. Anyway, this is controlled in Puppet afaik where we hand-tune a number of fields. [19:10:23] example: https://gerrit.wikimedia.org/r/c/operations/puppet/+/576910/1/modules/profile/files/logstash/filter-mediawiki.conf [19:10:58] which is now https://gerrit.wikimedia.org/g/operations/puppet/+/HEAD/modules/profile/files/logstash/filters/15-filter_mediawiki_ecs.conf [19:12:05] althoguh the suspicious absense of any converts today make me thing we prefer them elsewhere [19:12:17] cwhite wrote "Abandoned: handled by es7 template" [19:12:20] not sure what or where that is. [19:16:01] template is here https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/files/logstash/templates/logstash_7.0-1.json [19:21:12] We recommend migrating to ECS to take advantage of strongly-typed fields because they have documentation. Changing types on existing fields risks breaking the ability to query the field for 90 days and introduces the possibility of preventing other log streams from being ingested due to mapper parsing exceptions. [19:23:23] Fun fact, we have no less than 6 clientip-like fields in various permutations (hyphens, underscores, upper/lowercase, etc). [19:24:28] we don't use ECS for MediaWiki logs though [19:26:13] and there is an extra amount of historical baggage for IPs where some Monolog plugin used to add an 'ip' field to everything, but it wasn't actually the IP, so when application code wanted to log the IP it had to come up with new names [19:26:36] (or use 'ip' and then that got conflict-resolved to 'c_ip' or something like that) [19:29:06] (also using ECS in a legacy codebase is kinda scary because it loves deeply nested structures and if two different pieces of code log to the same field but one sends a string and the other sends an array that will break logging entirely) [19:31:39] Using nested structures is one possibility. We also support dot-delimited flat keys, e.g. `{"http.request.method": "GET", "http.response.status_code": 200}`. These get expanded into the nested structures by the pipeline. [19:34:43] but the end result will be the same, no? you log http.request.method: GET, some other code logs http: 1, elasticsearch auto-guesses the type of the field based on whichever log entry it sees first and silently drops the other one [20:03:01] The drop isn't silent, and `{"http": 1}` throws a mapper parsing exception. The race condition is eliminated in ECS. [20:04:08] I should say, the race condition is eliminated with one caveat: the labels field. It's still possible with unmanaged fields like the labels field, but we cast labels to string. [20:20:51] I wonder how many human years of effort it would be to audit and improve every log message in core + wmf deployed extensions... [20:21:16] I really did drop the ball on cleanup after I got monolog working [20:44:25] cwhite: that /logstash_7.0-1.json file, is that for mediawiki general or mediawiki ECS? [20:45:07] that `"properties" ` section at the end looks like it sets types for several MW properties that we use today, so presumably tgr|away could add an overrride there for a `client_ip` field of sorts. [20:47:55] That's the mapping template for the legacy indexes and doesn't affect ECS. [20:49:31] Yes, a field type defined there will change the field type with all the risks I listed above. [22:05:36] we seem to use 'clientip' pretty consistently FWIW: https://codesearch.wmcloud.org/search/?q=%5B%22%27%5D%28clientip%7Cclient_ip%7Cclient-ip%7CclientIp%29%5B%27%22%5D&files=&excludeFiles=&repos= [22:05:50] client_ip is only used in EventGate logging [23:35:28] cwhite: what impact does a mapper exception have? Are you saying echo wfDebugLog('Ad-Hoc', 'hello', [ maxSeconds => notafloat ]) will do something other than drop that field or drop that message? [23:37:09] Krinkle: it redirects the event to a dead-letter queue and a short-lived index for triage [23:38:24] ok, so it doesn't crash the logstash intake while it reboots or cause some other kind of operational fault. [23:54:12] I went back 90d and the field `clientip` is consistently IPs - I doubt we'll encounter mapper parsing exceptions on it.