[13:14:40] <jinxer-wm>	 FIRING: [2x] LogstashIndexingFailures: Logstash Elasticsearch indexing errors - https://wikitech.wikimedia.org/wiki/Logstash#Indexing_errors  - https://alerts.wikimedia.org/?q=alertname%3DLogstashIndexingFailures
[13:29:40] <jinxer-wm>	 RESOLVED: [2x] LogstashIndexingFailures: Logstash Elasticsearch indexing errors - https://wikitech.wikimedia.org/wiki/Logstash#Indexing_errors  - https://alerts.wikimedia.org/?q=alertname%3DLogstashIndexingFailures
[13:34:55] <jinxer-wm>	 FIRING: [2x] LogstashIndexingFailures: Logstash Elasticsearch indexing errors - https://wikitech.wikimedia.org/wiki/Logstash#Indexing_errors  - https://alerts.wikimedia.org/?q=alertname%3DLogstashIndexingFailures
[13:39:55] <jinxer-wm>	 RESOLVED: [2x] LogstashIndexingFailures: Logstash Elasticsearch indexing errors - https://wikitech.wikimedia.org/wiki/Logstash#Indexing_errors  - https://alerts.wikimedia.org/?q=alertname%3DLogstashIndexingFailures
[13:41:00] <cdanis>	 cwhite: thanks very much for your helpful writeup :)
[13:44:55] <jinxer-wm>	 FIRING: [2x] LogstashIndexingFailures: Logstash Elasticsearch indexing errors - https://wikitech.wikimedia.org/wiki/Logstash#Indexing_errors  - https://alerts.wikimedia.org/?q=alertname%3DLogstashIndexingFailures
[13:45:10] <jinxer-wm>	 FIRING: [2x] LogstashIndexingFailures: Logstash Elasticsearch indexing errors - https://wikitech.wikimedia.org/wiki/Logstash#Indexing_errors  - https://alerts.wikimedia.org/?q=alertname%3DLogstashIndexingFailures
[13:49:55] <jinxer-wm>	 RESOLVED: [2x] LogstashIndexingFailures: Logstash Elasticsearch indexing errors - https://wikitech.wikimedia.org/wiki/Logstash#Indexing_errors  - https://alerts.wikimedia.org/?q=alertname%3DLogstashIndexingFailures
[21:06:12] <cdanis>	 hey cwhite, random question if you have a minute -- is this sort of thing allowed? {"ecs.version":"1.11.0","http.response": {"body.content": "Ask Again Later", "status_code": 429}}
[21:06:31] <cdanis>	 or does the de-dotting only apply to top-level fields
[21:43:36] <cwhite>	 Dot expansion isn't recursive right now, so I don't think that would work.  Should be easy to test with logstash-filter-verifier if you wanted to verify as opensearch isn't the software doing the dot expansion work.
[21:51:28] <cdanis>	 okay cool, was wodnering if it was logstash-filter-verifier or not
[21:51:49] <cdanis>	 er sorry, I mean I was wondering if logstash filters was where dot-expansion happened
[21:52:36] <cdanis>	 I hacked up an utter mess of a tool to check for ECS conformance given JSONLines input like k8s apps produce, I will try to clean it up and turn it into a proper thing eventually
[21:54:10] <cwhite>	 Nice!  That's a helpful tool that we don't have yet.
[21:54:22] * cwhite ran logstash-filter-verifier
[21:55:16] <cwhite>	 `"http.response":{ "body.content": "foo" }` -> `{ "http": "response": { "body.content": "foo" } }`
[21:55:41] <cwhite>	 missed some brackets, but you get the gist :)
[21:56:19] <cdanis>	 yeah :D not going to space today
[21:56:24] <cdanis>	 thanks
[21:58:27] <cwhite>	 Do you think dot expansion should do deep inspection?  The concern I had when I implemented it was that inspecting every key for every message could slow things down.  I don't know by how much, though.
[22:00:52] <cdanis>	 I think it absolutely shouldn't :D
[22:03:45] <cwhite>	 Fair enough :)