[12:05:27] <dcausse>	 lunch
[14:45:45] <ebernhardson>	 \o
[14:47:15] <dcausse>	 o/
[14:49:16] <inflatador>	 <o/
[14:57:05] <dcausse>	 heading out early, have a nice week-end
[14:57:10] <ebernhardson>	 enjoy!
[15:11:49] <inflatador>	 latersville!
[15:22:36] <inflatador>	 ran into a bit of static with the relforge reimages: T410751
[15:22:37] <stashbot>	 T410751: Reimage cookbook: Warn/set defaults for hosts connected to Nokia switches - https://phabricator.wikimedia.org/T410751
[15:31:29] <ebernhardson>	 hardware can never be easy :(
[16:23:37] <inflatador>	 kids' thanksgiving at the school, back in ~1h
[17:08:49] <ebernhardson>	 hrm i hadn't realized lucene puts the knn graph indexes in off-heap memory: OpenSearch uses a circuit breaker, with a default limit of 50% of the non-JVM memory for the native k-NN index structures.
[17:08:57] <ebernhardson>	 i wonder how that will effect disk caching
[17:09:30] <ebernhardson>	 on the one hand, it means we aren't limited by the ~30G heap. on the other hand that memory had other uses :P
[17:12:50] <ebernhardson>	 although there does look to be a switch to go disk-based, guess will have to find out how that perf tradeoff is
[18:11:04] <inflatador>	 sorry, been back awhile. Relforge should be reimaged soon
[18:13:22] <ebernhardson>	 awesome! looks like 1008 is already up
[18:14:09] <inflatador>	 ebernhardson do you want to try and stand up a cluster, or are you just gonna run standalone? I ask b/c I have an ansible playbook I use in my homelab that would probably work (although I haven't tried it on 3 yet)
[18:14:29] <ebernhardson>	 inflatador: was intending to stand up docker-compose on each instance and join them to a cluster
[18:14:50] <inflatador>	 ebernhardson OK, that works
[18:15:36] <ebernhardson>	 i'll probably have to do some puppet patch for firewall between the instances and analytics network at some point, but still figuring out what exactly. I suspect (without verifying yet) that puppet wont let me hack the firewall settings manually
[18:16:42] <inflatador>	 yeah, I haven't looked too closely at the nftables role
[18:17:17] <inflatador>	 if it's like ferm, you might be able to manually add a rule file in /etc/nftables.d/ or something 
[18:17:28] <ebernhardson>	 yea perhaps, expect to find out today :)
[18:29:06] <inflatador>	 are you working on an OpenSearch 3 blubber image?
[18:29:57] <ebernhardson>	 for now using `opensearchproject/opensearch:3` but will see
[18:30:28] <inflatador>	 ACK
[18:30:28] <ebernhardson>	 i don't think we need extra plugins or other container changes, since this is mostly about testing knn
[19:03:40] <ebernhardson>	 yup, if i create a config in /etc/ferm/conf.d puppet purges it and reloads, will put together a patch
[20:23:07] <ebernhardson>	 no clue how broken it is, but thats makes a formed 3-node cluster.  Currently puppet is disabled for the firewall though, I think https://gerrit.wikimedia.org/r/c/operations/puppet/+/1208432 will make the firewall work (i'll turn puppet back on today, just getting the testing done)  
[20:28:45] <inflatador>	 👀
[20:35:30] <inflatador>	 Just merged/puppet-merged that above
[20:45:40] <ebernhardson1>	 excellent, worked as expected.  puppet is re-enabled and run on them, and everything still seems to be able to boot up and talk. Can access api from stats machines.
[20:46:03] <inflatador>	 🍾