[11:42:56] in our tofu-infra meeting I mentioned that .tfvars files are discouraged. Well, I'm reading now that they are discouraged for _keeping secrets_. Some guides even recommend you gitignore .tfvars files because of this [11:43:23] other usages seem perfectly fine, including its main purpose of decoupling data from logic [11:48:22] arturo: I was reading something similar at https://www.terraform-best-practices.com/code-structure [11:48:35] though it was not very clear, and also it's just a single datapoint [14:26:39] I'm syncing the components versions in tools, as I did yesterday in toolsbeta. no impact expected. [14:27:57] only for the ones not in sync with "main": builds-builder, calico, components-api, wmcs-k8s-metrics [14:29:03] oh there's one error that didn't happen in toolsbeta: Error: UPGRADE FAILED: cannot patch "toolforge-buildpacks-phases" with kind Task: admission webhook "webhook.pipeline.tekton.dev" denied the request: mutation failed: cannot decode incoming new object: json: unknown field "name" [14:29:49] dhinus: I found an issue with the script, the diff command does not accept '--wait' and the deploy cookbook passes it [14:30:12] ouch, thanks [14:30:22] oh, interesting error [14:30:30] might be related to crd versions [14:30:35] I will fix the --wait error [14:33:21] Hmm, the tekton webhook was deployed 5 min ago (the pods restarted), so I don't see that log anymore [14:35:46] I can reproduce dumping the current task, and trying to apply it (kubectl apply -f) [14:39:01] hmm, the version that tools shows is the old one [14:39:14] (using v1beta format for the task resource) [14:40:11] dhinus: should I try redeploying the helm chart? [14:40:19] yes please, I'm pushing the fix for --wait [14:41:05] ack, thanks [14:41:16] https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/714 [14:48:04] I think that to get builds-builder going I have to i) disable webhook for tasks, ii) update the task resource to the latest version (v1 format), iii) enable the webhook again [14:48:27] sounds ok? [14:49:07] (it's what I did for toolsbeta, and I though I did for tools, but it seems the task was not upgraded somehow :S ) [14:49:41] I opened T388797 to have a trace [14:49:42] T388797: Some toolforge components are running an old version - https://phabricator.wikimedia.org/T388797 [14:50:17] your plan looks good [14:50:38] similar to https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/blob/main/components/builds-builder/0.0.121-upgrade_tekton_to_0.59.3.py?ref_type=heads#L50 [14:53:57] nice, I didn't see those py files before [15:32:33] dcaro: anything else to do for builds-builder? I see the correct version now in get_versions.sh [15:42:03] everything ok there, all fixed, I restored the hooks and all [15:42:10] tested a couple builds too [15:46:48] I'll deploy the other components (calico, components-api, wmcs-k8s-metrics) [15:48:41] calico worked fine [15:49:02] components-api failed but it's maybe expected? Failed to render chart: exit status 1: Error: repo donotdeployyet not found [15:49:25] get_versions.sh says it IS deployed though [15:49:48] in tools yep [15:49:52] I have to look into that [15:50:10] we are not deploying in tools (we did a first version) [15:50:16] I might want to start changing that [15:51:32] so we're fine with having the old version 0.0.19 deployed, but we don't want to push newer ones for now? [15:52:59] not to tools (though I might start) [15:53:09] the idea is that it's still changing very fast [15:53:33] so if there's any big changes needed, doing those on tools might be more work than just deploying the latest version [15:53:55] yep makes sense [15:54:08] I wanted to double check if we should remove 0.0.19 or if we can leave it there [15:54:35] we can leave it there for now, I might start deploying the newer versions soon-ish [15:54:38] ack! [15:56:21] dhinus: can I merge https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/714 ? I want to deploy components-api [16:01:29] yes sorry didn't see the approve [16:01:32] I'll merge it [16:02:13] ack thanks! [16:04:16] I'll start running the deploy cookbook on tools for the other open MRs [16:04:44] I have a question about the "dev flow" at https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy#expected-development-flow [16:04:48] fyi. it work for toolsbeta components-api (running tests now) [16:04:52] (btw thanks for making that diagram, it's been very useful this week!) [16:05:22] in the final section, I see there's "test" both in the developer and the reviewer columns [16:05:33] after "manually run cookbook" [16:06:12] yep, just saw that, usually we don't wait for review the 'bump_version' MRs, unless there's more than just bumping the version (ex. if there's migration scripts, or extra changes in the config) [16:06:58] ok so I can just self-merge. also manual "test" is probably unnecessary if the functional tests pass? [16:07:05] (unless it's a new feature or something) [16:07:11] yep [16:07:19] ok! [16:07:26] if it's a new feature of sorts, you might also want to add a functional tests [16:07:56] I guess that the flow is in it's "safest" version xd [16:15:22] hmm I just realized that I need rebasing all the MRs... but the cookbook did not fail, maybe because it's not using --wait [16:15:38] maybe the cookbook should stop if the branch is not rebased on top of main? [16:15:39] maybe? [16:15:44] yep, that's a good point [16:15:52] or rebase it by default [16:27:09] arturo: if you are around could you maybe do a quick review for me? [16:27:15] https://gerrit.wikimedia.org/r/c/operations/homer/public/+/1127526 [16:27:16] topranks: sure [16:27:23] thanks <3 [16:28:35] topranks: +1 [16:28:48] thanks dude, I'll merge it now and see does that fix it [16:29:36] thanks! [16:57:52] * arturo offline [18:16:36] all MRs in toolforge-deploy are deployed and merged. all tests are passing! [18:19:58] \o/ [18:20:32] I deployed a bunch of components-api stuff too without issues (well, except my own tangling of the cli-api and cleaning up old tooldeployments xd, but that was expected) [18:20:49] thanks for that fix, I think that having the right versions will help a lot debugging [18:23:31] you're welcome! [18:24:41] * dhinus offline [18:26:18] (still here: I forgot to merge the last MR!) [18:26:31] done :) [18:34:02] * dcaro off [18:34:05] cya next week!