[07:50:29] <arturo>	 I'm online! back from vacations
[07:51:14] <dcaro>	 Welcome back!
[08:00:22] <arturo>	 thanks
[08:00:25] <blancadesal>	 morning :)
[08:00:31] <arturo>	 o/
[08:00:33] <blancadesal>	 welcome back arturo
[08:00:46] <blancadesal>	 we've tried not to burn the place down
[08:01:15] <arturo>	 :-P
[08:17:40] <dcaro>	 emphasis on "tried"
[08:24:54] <dhinus>	 morning! I'm also back from holidays :)
[08:36:50] <blancadesal>	 welcome back dhinus!
[08:40:16] <dcaro>	 Full house today xd
[09:32:17] * dcaro happy everyone is around
[09:37:43] <arturo>	 <3
[09:38:08] <arturo>	 dhinus: do you remember what was left for this MR to be merged? https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
[09:56:26] <dhinus>	 arturo: looking
[09:58:55] * dcaro lunch
[09:59:55] <dhinus>	 arturo: I think it can be merged, the only concern I have is that it's easy to forget projects are managed by tofu and create a new project manually, but that's probably ok
[10:00:13] <dhinus>	 I would suggest updating the docs here https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Projects_lifecycle
[10:00:18] <arturo>	 yeah
[10:00:48] <arturo>	 I can write a patch for the wmcs-openstack wrapper print a warning
[10:16:59] <dhinus>	 the wiki actually recommends using the "wmcs.vps.create_project" cookbook, so maybe you can add a warning to the cookbook instead?
[10:17:14] <arturo>	 sure
[10:17:40] <arturo>	 or, even better, drop the cookbook, we no longer need it
[10:17:48] <arturo>	 and point the wiki to the tofu repo
[10:20:34] <dhinus>	 sgtm
[10:52:55] <arturo>	 dcaro: I had to revert a previous patch by yours, the tofu cookbook was broken :-( https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/1069987
[11:21:34] <dcaro>	 arturo: that breaks toolforge deployment
[11:21:57] <dcaro>	 we should fix the tofu cookbook
[11:22:29] <dcaro>	 what's the error that you are seeing?
[11:22:41] <arturo>	 how is this related to toolforge?
[11:22:55] <dcaro>	 toolforge uses the gitlab code
[11:22:59] <dcaro>	 *reuses
[11:23:34] <dcaro>	 ex. https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/497#note_102979
[11:24:17] <arturo>	 mmm I see. Then the gitlab code change should have been originally in a different patch :-(
[11:24:40] <arturo>	 anyway, the toolforge deploy should work via cloudcumin servers, no?
[11:24:58] <dcaro>	 and laptopts
[11:25:11] <arturo>	 the tofu cookbook error is not clear and I did not do any further investigation
[11:25:24] <dcaro>	 revert then only the tofu side of the patch
[11:26:05] <arturo>	 ok
[11:27:00] <dcaro>	 👍 we should remove the tech debt as soon as possible though
[11:32:28] <blancadesal>	 dcaro: tools k8s upgrade rocket about to launch
[11:32:43] <dcaro>	 I'm in the coworking space meet
[11:33:30] <blancadesal>	 i'll be there in 5 min
[11:33:32] <dcaro>	 ack
[11:35:43] <dcaro>	 arturo: if I was to test the patch, how would I test it?
[11:36:26] <dcaro>	 I'd like to debug the error
[11:37:57] <dcaro>	 btw. it does not work from the laptop (it uses puppetdb and that breaks somehow)
[11:38:11] <dcaro>	 this works
[11:38:15] <dcaro>	 https://www.irccloud.com/pastebin/ABkTpG4e/
[11:39:56] * andrewbogott waves to arturo
[11:40:08] <andrewbogott>	 I'm not really  here today but also Simon and I are going to break labtesthorizon for a while :)
[11:43:09] <dcaro>	 https://www.irccloud.com/pastebin/kM8MwMmf/
[11:43:25] <dcaro>	 (probably not doing much though)
[11:48:04] <dcaro>	 this also worked :/
[11:48:12] <dcaro>	 https://www.irccloud.com/pastebin/Hxxs2sJl/
[12:14:23] <arturo>	 this is the log
[12:14:25] <arturo>	 dcaro: 
[12:14:39] <arturo>	 https://www.irccloud.com/pastebin/Rx7MQuPo/
[12:18:18] <dcaro>	 that looks like what failed is the cleanup `rm -v`, how come that's not `-r -f`?
[12:20:56] <dcaro>	 arturo: I don't see any `rm -v` anywhere in the code :/, were you using a clean code?
[12:21:39] <arturo>	 I guess is from `with_temporary_file`
[12:23:20] <dcaro>	 my guess is that the temporary file did not exist anymore
[12:23:26] <dcaro>	 probably that should have a '-f' also there
[12:24:20] <arturo>	 the failure I got this morning was from the cookbook code as present in the cloudcumin server
[12:26:00] <dcaro>	 arturo: this should fix that error and raise any underlying error if there's any https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/1070018
[12:33:43] <dcaro>	 and re-revert of that one https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/1070020
[12:34:31] <dcaro>	 (as the error was not there)
[12:41:46] <arturo>	 yes, this patch still introduces a problem
[12:42:10] <dcaro>	 ack, got the traceback?
[12:42:31] <arturo>	 see the MR note it creates https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28#note_103030
[12:43:42] <arturo>	 also, the fact that this function hides all commands behind base64 makes it unnecessarily complex to debug what is happening
[12:49:23] <arturo>	 dcaro: if you want spend time fixing it (I would suggest you dont), you can compare with this MR note that the previous code creates:
[12:49:24] <arturo>	 https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28#note_103036
[12:56:53] <dcaro>	 I might between tasks, the advantages of reusing the code in the cookbooks is worth enough
[12:56:57] <dcaro>	 (imo)
[13:03:27] <arturo>	 👍
[13:03:38] * arturo food time
[13:32:52] <blancadesal>	 dcaro: i'll be afk for 1 hour or 2. I have tools-k8s-worker-nfs-{29-31} left. let me know if you get to them, or I'll do them later
[13:33:00] <dcaro>	 blancadesal: ac
[13:33:01] <dcaro>	 ack
[14:12:19] <dcaro>	 I added everyone in the k8s upgrade workgroup to the syncs for the 1.27 upgrade this week as optional, Raymond_Ndibe and I will be working on the next upgrade, everyone is welcome to join
[14:17:51] <dcaro>	 arturo: btw. https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/1070020 is ready, can you please try it the next time you have to apply anything with tofu? the plan works as expected
[14:24:57] <arturo>	 sure!
[14:49:14] <dcaro>	 @blancadesal all the nodes are now upgraded, and tests are passing :), I'll leave to you to announce the upgrade is over (after you double check that I did not mess up xd)
[15:36:40] <blancadesal>	 dcaro: thanks!
[16:02:46] <arturo>	 I just created T373816 dcaro dhinus 
[16:02:46] <stashbot>	 T373816: Cloud VPS: investigate conntrack table usage on cloudvirt1050 - https://phabricator.wikimedia.org/T373816
[16:08:32] <blancadesal>	 dcaro: all done, over to you & Raymond_Ndibe :)
[16:13:28] <dcaro>	 \o/
[16:13:34] <claime>	 anyone available to restart wikibugs? I don't have access to login.toolforge.org
[16:13:40] <claime>	 (hello o/)
[16:16:51] <dhinus>	 claime: restarted, but not sure it worked (there are multiple processes)
[16:18:00] <claime>	 dhinus: I think it's the irc job that's gone wrong, wikibugs isn't in -operations anymore
[16:18:01] <dhinus>	 still offline, let's try the znc process
[16:22:40] <dhinus>	 I restarted both "znc" and "irc" but it's not joining all the channels
[16:23:16] <dhinus>	 ah I think it actually only joins them when required
[16:23:51] <dhinus>	 it is logging things to #wikimedia-dev
[16:23:57] <dhinus>	 so hopefully it should be fine
[16:25:35] <dhinus>	 and it's back in -operations too!
[16:25:56] * dhinus offline
[16:39:37] <claime>	 yay, thanks
[17:28:47] * dcaro off