[07:27:50] <elukey>	 btullis, brouberol o/ did you see https://alerts.wikimedia.org/?q=%40state%3Dactive&q=%40cluster%3Dwikimedia.org&q=alertname%3DOpenSearchCertAlmostExpiredCritical ?
[07:28:28] <brouberol>	 Yep, I was planning to run the rolling-restart script this morning
[07:28:41] <elukey>	 gooood thanks
[07:29:01] <brouberol>	 long story short, the opensearch operator is the equivalent of a ponzi scheme. It does nothing for you except causing you anger and work
[07:35:29] <elukey>	 brouberol: after all this time it is your fault, you have faith in software
[07:35:52] <brouberol>	 I mean. I has "operator" in the name. I'd expect it to operate, even badly
[08:33:38] <atsukoito>	 jayme: good morning! i wanted to ask you what do you think about this approach for bringing the admin_services secrets? https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1298859/1/Rakefile#565
[08:38:00] <jayme>	 atsukoito: the problem with this is that you're injecting the admin_services secrets into the general fixtures. IIRC these are used by service deployments as well which means the service deployments in CI will have values defined that they won't have in prod
[08:39:28] <atsukoito>	 yeah.. figured.. i guess proper solution will be creating every small file with secret part, and then replacing it one-by-one in `.rake_modules/tester/asset.rb`. thanks
[08:39:49] <atsukoito>	 do you think we can proceed with https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1298327 for now?
[08:58:46] <jayme>	 yeah, sure. let me comment on the CR
[09:08:24] <atsukoito>	 jayme: thanks, i'll update the diff
[09:23:06] <jayme>	 <3
[09:30:27] <bjensen>	 hey folks, a heads up - we'll be doing a point upgrade of k8s today, from 1.31.4 to 1.31.14. this has been tested on the staging clusters, and we're now proceeding to everything else. please let me know if the timing is inconvenient, or if there are other objections :)
[09:33:15] <elukey>	 bjensen: o/ nice! On Wikikube prod you mean?
[09:34:22] <bjensen>	 elukey: indeed, and also the ancillary clusters (aux, dse-k8s, ml-serve)
[09:37:53] <elukey>	 ah wow all?
[09:37:58] <elukey>	 you folks rock, all in
[09:38:43] <elukey>	 for Aux it is fine, I'd also alert other folks for DSE and ML (cc: dpogorzelski, brouberol, btullis, klausman)
[09:42:21] <bjensen>	 good idea, thanks :D i've added such a notification to the process for point release upgrades
[09:43:02] <jayme>	 we discussed/announced that during the last SIG meeting - which you've developed a habit of missing elukey :p
[09:44:47] <elukey>	 jayme: people usually don't listen to k8s things because they are booooring so they may have missed it! :P :P
[09:45:28] <jayme>	 your point :)
[09:46:55] <atsukoito>	 jayme: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1298327/5..6/.rake_modules/tester/asset.rb
[09:48:21] * atsukoito is so sorry for slowing down the CI for ~30 minutes because my CR is re-testing whole repo
[10:17:20] <jayme>	 no worries...we're competing for runtime records (https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1298799) :)
[10:37:47] <atsukoito>	 can i have a review please?
[11:30:25] * atsukoito rolling out admin_ng diff
[11:41:08] <atsukoito>	 note that because of https://gerrit.wikimedia.org/r/q/topic:%22T427517%22, admin_ng/dse-k8s will not pass until i finish merging/applying secrets.. puppet is still running
[11:44:31] <atsukoito>	 applied, https://phabricator.wikimedia.org/P93956
[12:27:57] <bjensen>	 1.31.14 is now deployed everywhere :)
[12:38:42] <elukey>	 nice work folks!
[13:07:05] <jayme>	 good job!
[13:08:06] <bjensen>	 thanks, y'all :)
[13:16:46] <brouberol>	 thanks!