[11:17:53] hi, i have a k8s/helmfile question: what is adding `.fixtures.yaml` in the lint run, and is there a way to have a fixture per environment? I want to have the values in https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1298327/2/helmfile.d/admin_ng/.fixtures.yaml available only for dse-k8s. [11:18:53] brouberol mentioned labs/private as well, I've tried bringing those values there, but I'm not sure how to update helm-lint environment with new labs/private, https://gerrit.wikimedia.org/r/c/labs/private/+/1298722 [11:19:08] atsukoito: it's downloaded by the ruby CI code from puppet [11:19:35] if you mean the stuff that's not checked into git [11:20:52] generally speaking, the answers are in the CI code in .rake_modules/tester/ , and the CI code is terrifying [11:21:17] Raine: thanks, I'll try figuring out [11:21:37] good luck :'D [11:22:30] anyway, second part of the question is still stands, does the labs/private executed in any form on helm-lint? [11:29:27] no, I don't think we pull anything from there [12:53:47] jayme: thanks!! [13:55:47] jayme: something like this maybe? https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1298327/5/.rake_modules/tester/asset.rb [14:00:16] atsukoito: would it be an option to just add that to the "global" fixtures file? [14:03:31] jayme: the change is split between https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1298327/5/helmfile.d/admin_ng/.fixtures/dse-k8s-eqiad.yaml and https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1298327/5/helmfile.d/admin_ng/values/dse-k8s-eqiad/cfssl-issuer-values.yaml, having only one of it will make the `cfssl-issuer/templates/secret.yaml` raise error. [14:03:58] however, we don't have `cfssl-issuer` outside of admin_ng, so it is probably safe [14:06:16] Not sure I fully understand what risk you see. As I understand it you're adding ad second clusterissuer, so you need a fixture for it's auth key (in addition to the existing one). [14:06:29] jayme: did you mean `.fixtures/general-#{env}.yaml` or `helmfile.d/admin_ng/.fixtures.yaml` (if second, then it won't work: it will start breaking test for other clusters, I tried it in second patchset) [14:06:57] adding that to admin_ng/.fixtures.yaml will inject that into admin_ng deployments for all environments (clusters), but that's about it [14:07:44] ah, interesting...guess that leads to a half broken issuer config then [14:09:17] do you have logs of the error you mentioned? [14:42:08] jayme: yes, it is https://integration.wikimedia.org/ci/job/helm-lint/32909/console [14:45:51] it is either charts/cfssl-issuer/templates/issuer.yaml fails with missing values, or charts/cfssl-issuer/templates/secret.yaml [14:46:54] yeah, makes sense [14:48:45] i can try `.fixtures/general-dse-k8s-{eqiad,codfw}.yaml` with the caveat that we won't be using cfssl-issuer outside of admin_ng. otherwise it will lead to the same error [14:49:10] the general fixtures are auto-generated by CI [14:49:39] i see.. (for some reason one file is checked in the repo) [14:49:52] yeah, I think that's an oversight [14:50:52] your CI change does address the problem correctly I guess. Although ultimately correct would be to pull the data from labs/private and compute the fixtues file for that env on the fly [14:51:42] there is a bunch of ugly code that does so for non-secret values in 'task :refresh_fixtures' [14:52:25] if you feel like it you could replicate that for labs/private [14:53:13] I did commit the corresponding change to labs/private, https://gerrit.wikimedia.org/r/c/labs/private/+/1298722 [14:53:30] but the only thing I'll need to figure out, how to populate it then [14:53:31] yes. but there is no code in deployment-charts CI that uses it [14:53:35] yeah.. [14:54:15] for the public hieradata values the refresh_fixtures tasks pulls the data from gerrit and mangles it into the expected format [14:54:25] to generate the general-* files for example [14:55:11] same could be done for 'profile::kubernetes::deployment_server_secrets::admin_services' from labs/private I suppose [14:56:25] but I'm also not opposed to your initial CI patch to support per environment fixture files [14:57:55] I would just ask you to open a task with a quick description of the problem and the options to solve it and to split out the CI change from your config change in https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1298327 so we can have that merged seperately [14:58:09] lemme see how hard it will be to implement refresh_fixtures support for it [15:12:28] yeah, lemme create another CR for this, i don't expect it to take much time.. [15:13:45] tho i've been wondering, was there any attempts to run whole puppet to generate all k8s yaml from labs/private and prod? [15:14:21] s,prod,operations/puppet, [15:15:15] run puppet in CI to generate all those files you mean? That was never tried to my knowledge [15:16:52] maybe not in CI time, but along those lines.. humu.. i wonder.. would it be interesting if I spend some time trying it? gonna ask gehel for it later [15:20:15] I'm not sure that's viable since it would need to work in CI (docker) and local (docker) environments as well. But I might be missing some clever solution here. So if you would like to explore that I'm very open to support that