[13:03:53] <ebernhardson>	 \o
[13:08:23] <inflatador>	 <o/
[14:53:07] <atsukoito>	 hiya!
[17:40:36] <ebernhardson>	 inflatador: just checking, we switched the opensearch admin stuff from text passwords to pki stuff? Or am i just not finding the passwords
[17:40:46] <ebernhardson>	 in the dse-k8s
[17:41:05] <inflatador>	 ebernhardson we're still using basic auth AFAIK
[17:41:37] <inflatador>	 if you become `opensearch-semantic-search-deploy` user you should be able to read the kubeconfigs with the pw, let me check the docs real quick
[17:42:28] <inflatador>	 oops, looks like I didn't do a great job of documenting that. Will check the deploy host and update docs afterwards
[17:42:35] <ebernhardson>	 inflatador: any ideas where the creentials are? I was looking in /etc/kubernetes, our thing takes a credentials file in yaml that has the password at config.private.users.operator
[17:42:50] <ebernhardson>	 i should have better documented the path...
[17:45:13] <inflatador>	 ebernhardson ah, it's not in a file, you have to read it with kubectl. ` kubectl get secret opensearch-semantic-search-admin-password -o jsonpath={.data.password} | base64 -d`
[17:45:27] <ebernhardson>	 oh, it's in the helmfile dirs
[17:46:06] <ebernhardson>	 inflatador: turns out it's in /etc/helmfile-defaults/private/dse-k8s_services/opensearch-semantic-search/dse-k8s-eqiad.yaml
[17:46:18] <inflatador>	 Ah, so it **is** in files too
[17:46:28] <inflatador>	 Docs do mention the 1-liner https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/OpenSearch-on-K8s#Access
[17:46:31] <ebernhardson>	 although i suppose it's probably that it's sourced from those files then applied to k8s, so k8s would be authoritative
[17:47:48] <inflatador>	 I had an unrelated question to ask re: ECS log output from opensearch. gehel suggested https://github.com/elastic/ecs-logging-java , does that seem like a viable option?
[17:49:18] <inflatador>	 And if so, what does a deployment look like? Maybe a puppet patch for log4j and then we bundle that in our opensearch plugins deb pkg or something?
[17:55:44] <inflatador>	 Per Andrew's comment in https://phabricator.wikimedia.org/T324335#12101449 , could it really be as easy as just changing the log4j config?
[17:57:10] <ebernhardson>	 hmm, looking
[18:02:46] <ebernhardson>	 inflatador: looks viable and like an intended use case
[18:03:31] <ebernhardson>	 in theory it should be pretty simple, deploy a couple jars to the expected paths and change the log4j config
[18:04:18] <inflatador>	 ebernhardson ACK, thanks for checking. I guess the next step would be to try the log4j config Andrew linked on cloudelastic and see what we get? I'd say relforge but I think it's still empty ATM
[18:05:55] <ebernhardson>	 inflatador: relforge is up, localhost:9200 returns opensearch 2.19.5
[18:06:31] <ebernhardson>	 sure the indexes are empty but it should still be able to generate logs
[18:06:49] <inflatador>	 ebernhardson touché, I'll try relforge 1st then
[18:17:03] <ebernhardson>	 inflatador: this will trigger exception logging in opensearch: https://phabricator.wikimedia.org/P94844
[18:44:37] <inflatador>	 sorry, got distracted looking at RDF streaming updater S3 usage, something happened starting a week ago and it's up to 3 TB
[18:45:39] <ebernhardson>	 :S
[20:22:47] <inflatador>	 FYI, here's the latest on the streaming updater stuff, sounds like search update pipeline is affected as well https://wikimedia.slack.com/archives/C01LE7VPU5A/p1784054335079429