[07:49:17] morning! [07:50:02] probably unrelated, but there's also some network weirdness happening here T432338 [07:50:02] T432338: [toolsbeta,infra,admin] toolsbeta-test-k8s-haproxy7 is flaky on ip6 when replying to admin - https://phabricator.wikimedia.org/T432338 [08:32:08] morning [08:35:56] ^ that probe has been flaky for a while, see T426584 [08:35:57] T426584: [toolsbeta] probe flapping on ipv6 only - https://phabricator.wikimedia.org/T426584 [08:53:27] 👍 [09:18:29] I'll resume reboots for cloud hosts, that's T431659 and T431658 [09:19:09] stashbot: :( [09:19:09] See https://wikitech.wikimedia.org/wiki/Tool:Stashbot for help. [09:19:27] ah nevermind the tasks are not public [10:02:17] godog: fyi. I'm ignoring the alerts until you say you are done with reboots [10:02:40] dcaro: ack thank you, yes [10:03:24] done just now with cloudcontrol, doing cloudgw now then lunch [10:06:34] 👍 [10:08:41] I said cloudgw, what I meant was cloudlb [10:56:26] bah, of course the elastic cluster only exists in the tools project [10:56:42] (which I guess is nice as there's less to upgrade, but also no staging cluster) [10:57:09] although if we're (read: I am) just building a new cluster in parallel and then have people migrate, that's not that big of a deal [11:18:09] indeed [12:18:54] ok proceeding with 1001 [12:45:28] dcaro: did you intend to change the status of T432450? [12:45:29] T432450: Request tool account: football-kit-sync - https://phabricator.wikimedia.org/T432450 [12:49:15] Hmpf.... the triggers in columns are good most of the time, but some times they get in the way [12:49:16] I did not [12:50:31] restored [12:50:37] thanks [12:59:21] quick review, just copying the coverage setting to the rest of tox jobs https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/91 [13:02:47] lgtm [13:05:42] thanks! [13:22:36] I guess triggers in columns are not as flexible as herald rules right? I was checking the config and doesn't seem like we can add a condition to not change to "Resolved" if it was in another closed state [13:24:52] yep, I think there's no "conditions" for them [15:25:30] andrewbogott: is T432150 something we can do storage-wise? (or for the other resources, although I am less worried about them) [15:25:31] T432150: Quota increase request for project wikiwho - https://phabricator.wikimedia.org/T432150 [15:27:28] We can, but I don't think they answered your question about the storage change. If they're duplicating 20GB VMs... why so much space? [15:34:54] andrewbogott: I read that as requesting a new VM and enough volume space 3*5T volumes [15:35:14] from the reply I understand that they don't need the volumes no? [15:35:17] yeah, but then don't they say in their last comment that they aren't actually making new volumes? [15:35:19] yeah [15:35:38] I think they just wrote "double all the things!" in the initial request but don't actually want that [15:35:39] huh [15:35:41] right [15:36:06] anyway, I asked [15:36:14] I just read to [another] "identical VM" and imagined that meant with identically sized volumes [15:37:46] it's unclear! We'll see what they say [15:38:36] +1 for asking xd [15:39:42] btw taavi, you wanted me to nudge you so here's the nudge https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/333 [15:40:49] andrewbogott: tofu-wise that all seems fine.. but what happens when there's a new magnum release that upgrades the kubernetes version as well? do we just update the existing template? [15:41:48] I think we either create a new one, or replace the existing template with a new one. As far as I know magnum doesn't refer back to the template after creation so it's ok to leave an orphan. [15:42:13] But I wouldn't want to modify the existing template because then a given cluster would lie about it's origins. [15:42:38] none of these options are entirely great [15:43:00] fwiw the real question in my mind is "should the template name have a version number in them?" [15:43:38] oh... yes it should. Will fix. [15:44:47] * andrewbogott looks up how to do variable interpolation [15:47:12] https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/333/diffs?file=84c8e6c044613551e5f98a2e9c7069e21e36c3f8#84c8e6c044613551e5f98a2e9c7069e21e36c3f8_0_30 [15:49:59] godog: if you're still around, can we talk about https://phabricator.wikimedia.org/T430933? I think I'm still missing something. [15:50:19] You link to a change that adds the IDs and cite it as a reason to no longer track the IDs -- has something changed so we don't need them anymore? [15:52:11] * andrewbogott reading more... [15:53:08] ah, ok, so we can generate a predictable uuid and use that instead! [15:53:18] I think I'm caught up now :) [15:57:07] ty taavi [16:02:54] T432150 has clarification that it's just about the CPU/RAM, which seems all fine to me [16:02:55] T432150: Quota increase request for project wikiwho - https://phabricator.wikimedia.org/T432150 [16:03:04] so, can I have (two) +1's on that please? [16:03:22] done [16:03:26] well, half [17:06:03] i added the other half (if my vote counts) since user is itching to move on it [17:36:36] * dcaro off [17:36:39] cya tomorrow! [17:54:12] * andrewbogott waves [21:58:43] Tragically, those eqiad1 and codfw1dev tofu definitions do not do what we expected :( [22:02:20] it adds all four templates to each deployment, so we get two broken codfw1dev templates in eqiad and two broken eqiad deployments in codfw1dev [22:26:19] andrewbogott: did you try some debug mode (TF_LOG=DEBUG tofu plan 2>&1 | tee tofu-debug.log) or trying some of the statements inside something like "tofu console"? [22:27:15] Not yet! Soon [22:31:27] Although to be honest I'm more surprised by why that doesn't happen for everything than why it does happen for these resources in particular. [22:31:41] But first I must heed the dinner gong!