[09:00:34] bd808: I don't think I want to see a general "everything hosted in Cloud VPS" zone in the puppet repo, so I agree with Andrew that you maybe should do profile::zuul::something, with a possible subtree of that if you do need that separation. [09:36:39] andrewbogott: I'm not sure what happened in https://netbox.wikimedia.org/extras/changelog/230387/ and https://phabricator.wikimedia.org/rONED258ea7aae84a4567527c85b73c61eb12d7bc226a but you seem to have removed the cloud-private address of cloudvirt2004-dev [13:45:47] taavi: I guess that was me trying to https://phabricator.wikimedia.org/T396396#10951792 [13:46:14] have you already fixed my mistake or shall I do that? [13:46:45] yes(ish). I added it back to netbox, but https://phabricator.wikimedia.org/T397868#10952973 is causing a mess and blocking the dns update for the moment [13:47:52] ok. I guess it's fine if that host remains broken for a while so we can wait for frtech to respond on that. [13:51:12] taavi: so that I don't make things even worse... I am now going to mash the 'delete' button on https://netbox.wikimedia.org/dcim/interfaces/29742/ <- sound right to you? [13:52:16] andrewbogott: uh, sure? [13:52:28] is there a task to fix the bug that https://phabricator.wikimedia.org/T396396#10951416 came up with? [13:54:01] I think that's a subcase of "someday topranks will automate handling of the .private network for cloud-vps but in the meantime every server install and decom requires Andrew to make a bunch of netbox mistakes" [13:54:10] I'm not sure if cathal has a bug open for that. [13:54:53] clearly we should blame arturo for the setup here guys [13:55:00] :P [13:56:03] yeah at some point we can get back to the script to set up hosts with more complex network setup at the provision stage and not need manual changes. [13:56:12] there are a handful of others out there that also need it [13:57:00] At the moment the only real problem is that I seem incapable of remembering to cleanup before I send a server for decom. [13:57:32] that problem actually won't be fixed by us provisioning things right for new installs [13:57:39] not sure what the issue exactly is here tbh [13:58:10] on the face of it we need to look at sre.hosts.decommission [13:58:17] and make sure that is deleting all interfaces/IPs [13:58:20] oh? I was just going to say something like "the decom case seems easier to automate since we can just remove everything associated with a host and don't need to actually understand what's happening" [13:58:22] shouldn't be hard to do I think [13:58:30] yeah exactly [13:58:43] I think it's probably a bit of a race condition [13:58:48] I'll make a task for that [13:59:02] please do, I'm tied up today but can look next week, I don't think it should be hard [13:59:14] also it can be a general thing without being a "cloud-private" snowflake [13:59:33] add me as a subscriber if you can thanks [14:02:39] T398052 [14:02:40] T398052: sre.hosts.decommission often leaves dangling things in netbox - https://phabricator.wikimedia.org/T398052 [15:59:36] taavi: can you tell me more about how you envision Cloud VPS projects managing server lifecycle automation without a space in the ops/puppet.git tree? I have options here, but I am curious about your informed opinion. [16:01:21] I'm trying to think about how to add manifests that can be used in multiple Cloud VPS projects to augment Magnum as part of this, but I'm trying not to worry about designing a reusable system as step 0. [16:02:55] bd808: I may have misworded this, but what I meant that I think you should be able to put your manifests in the general role::* / profile::* namespace instead of having a specific corner for roles / profiles intended to be used in cloud vps projects. [16:13:30] taavi: thanks for that clarification. I think I read things in your words that weren't actually there. :) [18:29:29] slyngs: if you're around, can you help me get a dev account unstuck? I created an account named 'designate' with a bogus email address thinking that I'd just fix it in ldap later, but I think it's now stuck in a pending state, not yet in ldap but I also can't confirm it on account of the bogus email. [18:29:40] (this is related to T398074 but that task isn't blocking me) [18:29:42] T398074: developer service accounts and email - https://phabricator.wikimedia.org/T398074 [18:30:18] let's see, iirc jhathaway also probably knows how to get me unstuck [18:30:39] o/ [18:32:52] andrewbogott: I don't know off hand, I would need to dig into bitu to try and figure out how to unblock [18:33:10] ok, no need to dig I can just wait for Simon [18:33:40] ok, if it becomes urgent I am happy to do some digging [18:34:18] thanks! It's unlikely to be urgent since the task I'm working on has been open for years [22:48:45] andrewbogott: is there an "easy" way to use curl to talk to an OpenStack endpoint? Use case is I want to make some calls to https://puppet-enc.cloudinfra.wmcloud.org to debug something. [22:50:33] I have all the OS_* envvars to auth using the python `openstack` cli client, but I don't know how to turn that into whatever barer token I need to talk to an endpoint. [22:50:49] You can use the openstack client to get a token. I think it's just 'token issue' [22:50:50] * andrewbogott checks [22:51:18] yeah, " openstack token issue" [22:51:32] and if you make a clouds.yaml that's probably easier than using environment vars [22:52:02] for an example look at ~root/.config/openstack/clouds.yaml on any VM [22:52:13] you can do the same in a regular user homedir [22:52:38] ~/.config/openstack or /etc/openstack [22:52:57] then you would do "openstack --os-cloud token issue" [22:53:06] oh, nice. there's the token. Thanks! [22:53:44] then just -H "X-Auth-Token: $OS_TOKEN" in your next request [22:54:00] if you're being extra fancy you can also use the openstack client to discover the enc endpoint [22:55:13] openstack endpoint list --service puppet-enc --interface public [22:55:57] or really openstack endpoint list --service puppet-enc --interface public --format value -c url [22:57:20] * andrewbogott wonders if this is documented anyplace [22:58:00] not really [23:11:38] `OS_TOKEN=$(openstack token issue -f yaml | yq .id)` is a nice hack in the environment I'm working in [23:12:02] then `curl -H "Accept: application/json" -H "X-Auth-Token: $OS_TOKEN" https://puppet-enc.cloudinfra.wmcloud.org/v1/c26d9d326bdf464fa1025939ded7e5a2/prefix` [23:12:33] and bam! That's what I wanted to call. I turns out the data I want isn't there, but that's a different problem. [23:12:35] Cool [23:18:07] andrewbogott: https://phabricator.wikimedia.org/T397994#10955587 is what you just helped me get past :)