[07:57:06] Hi, I am trying to ssh into my petscan5 VM from bastion. This always worked, but now I get "petscan5.petscan.eqiad1.wikimedia.cloud: Permission denied (publickey)". I created new keys on horizon but no joy. Any ideas? [07:59:55] MagnusManske: I see errors when authing the ssh key (there's a few tries) [08:00:30] root@petscan5:~# ssh-key-ldap-lookup magnus [08:00:33] returns two keys [08:00:54] https://www.irccloud.com/pastebin/DFr74RFY/ [08:00:56] there may be some old key but I just created a new pair in horizon, that should work, no? [08:02:05] those come from ldap, is any of those the new one? [08:02:12] I also tried the two you put in pastebin, they dont work either [08:02:26] no the two are manual "old" ones [08:03:18] in the logs it says you failed using `RSA SHA256:F3OMj/3VUzAI5hsDeVnGY322PJ1nx/GPVJCoNkMm5Bs` [08:05:21] that doesn't help me, sorry [08:05:44] why won't the ssh key I created in horizon work? [08:05:56] do I need to reboot the instance? [08:07:20] your ssh keys are managed in ldap, by toolsadmin or idp afaik [08:07:35] the ones in horizon I think are for instance creation and such [08:07:43] (when a new VM is created) [08:07:51] I'm not sure though, let me investigate [08:08:17] (about horizon, about toolsadmin and idp I'm sure) [08:09:32] here are the link to upload keys to ssh into the VMs https://wikitech.wikimedia.org/wiki/Help:Accessing_Cloud_VPS_instances#Set_up_and_upload_SSH_keys [08:12:19] I'm not sure if the horizon keys are used for regular instances at all, I think they are used for trove/magnum [08:13:02] they might actually: [08:13:05] https://usercontent.irccloud-cdn.com/file/V0rk3vPg/image.png [08:13:26] "In most cases Puppet will manage VM access and you can ignore this panel. Assigning a keypair is only useful for VMs running without Puppet." [08:13:32] they are used for https://wikitech.wikimedia.org/wiki/Help:Unmanaged_Cloud_VPS_instances, we should add a warning to the ssh key management page to that effect as well [08:13:50] taavi: ack, I'll add a note [08:14:50] I was thinking about just filing a task and hoping a.ndrewbogott will deal with it, but if you have the horizon source handy and know what to modify then sure :D [08:19:35] to be fair, I have the code locally, but I don't know how to do the modifications just yet xd [08:29:19] hmm, there's two versions in the codebase, an angular one, and a django one, and it seems the django one is deprecated, though not sure which one we use [08:40:16] for now I added it to the wiki [08:45:12] Thanks, adding to idm and gerrit worked! [13:46:08] andrewbogott: kindly pinging about my message last Friday [13:46:27] alternatively, if someone else can help with Ceph backups, please give me a shout [13:47:42] jnuche: noted, I will follow up but I have meetings for a bit [13:47:53] ty! [13:49:54] jnuche: how much data, and is this a one-off thing or would you need the ability to make ongoing copies? [13:50:05] we're talking about duplicating a cinder volume, right? [13:52:30] yeah, three volumes for a total size of 530GB. We will need this done twice: first time to test the migration and second for the actual migration itself. No need for an ongoing process [13:52:39] ok, got it [15:48:16] jnuche: do you have an openstack cli set up someplace, or are you only using horizon? There's a simple command to copy, I'm not clear yet on why it's not available in horizon. [15:48:47] oh wait, I take it back -- it /is/ available on Horizon. [15:49:22] Just create a new volume and select 'volume' as the 'volume source'. Then it will give you a menu to select the volume you want to clone into the new one. [15:49:31] Is that what you needed, or did I misunderstand the question? [16:16:35] andrewbogott: we need the volumes to be copied from project `catalyst` to `catalyst-dev`, which I imagine admin rights are needed for [16:16:40] but even when just trying to create a fresh volume, the only options I get offered are "empty source" or "image": [16:16:46] https://usercontent.irccloud-cdn.com/file/KO5O5TG4/image.png [16:17:35] ok, obviously I'm seeing different things than you're seeing due to different permissions. Stay tuned... [16:17:50] ack [16:18:19] there're volumes in `catalyst-dev` right now taking up space, I'm gonna wipe those out [16:20:29] jnuche: when you tried 'create volume' which project did you do that in? [16:20:42] catalyst-dev [16:21:44] in `catalyst` I get the extra "snapshot" option as source, which makes sense because we do have snapshots there [16:21:45] but that's it [16:22:06] I don't know if this will all work for you, but the process should be: 1) Create a clone volume in the 'from' project (catalyst), then initiate a volume transfer in the 'from' project, then accept the transfer in the 'to' project. [16:22:53] but yeah, sometimes it offers 'volume' as a source option and sometimes not. When/why? I'm investigating. [16:23:07] yeah, that's essentially the steps we need [16:23:47] https://usercontent.irccloud-cdn.com/file/9LoeO8Ng/Screenshot%202025-09-22%20at%2011.23.20%E2%80%AFAM.png [16:24:00] it sometimes looks like that for me, even when I'm not an admin user [16:24:03] but not always! [16:26:07] oh fun [16:30:12] jnuche: I think that in order to copy a volume it has to have state 'available' which means not attached to a VM. Is that something you can try? It should be quick if it works. [16:31:03] Of course it will cause an outage in the service which is not great. [16:32:06] that's one of the reasons we wanted to do this via cinder ops and not horizon snapshots, we can't just unplug those volumes [16:35:02] ok, so... what if you snapshot, build a new volume from the snap, then /copy/ that volume, and then delete the snap? [16:35:26] (that extra copy is because otherwise there will be an eternal dependency on the original volume due to the way snaps work) [16:35:33] I'm sorry if I keep suggesting things you've already tried [16:37:53] andrewbogott: yeah, we did try: [16:37:53] 1. Trying to create a snaphost of an attached snapshot via horizon gives a scary warning about the data potentially getting corrupted [16:37:53] 2. The snapshot counts towards the project's quota, and we don't have enough quota left at the moment [16:37:53] 3. Event if we could still create the snapshot safely, it's not possible to create the volume cross-project [16:39:10] so, the scary warning shouldn't matter since you aren't going to mount the snap anyplace [16:39:17] the quota issue will happen no matter what process we follow [16:39:25] "it's not possible" as in we're not given the option in the UI [16:40:24] ok, I'm going to give my 'moral' user some normal human rights in the catalyst project and try this :) [16:42:17] * andrewbogott wishes that snapshots did not work the way they work [16:42:33] jnuche: how recent of a copy do you need? If the new volume reflects the volume from a couple days ago is that Ok? [16:42:57] yeah, that would work [16:43:02] and, we're talking about catalyst-k3s-data ? [16:43:29] k3s-worker01-data, catalyst-k3s-data-DB and catalyst-k3s-data [16:43:32] we need those three [16:55:22] andrewbogott: I need to stop for the day soon [16:55:29] let's continue tomorrow if you have the time. Ty for your help! [16:55:46] ok. I'll see what I can figure out. This ought to be self-serviceable but I don't quite understand the error I'm hitting now. [17:57:45] jnuche: I have a process that seems to work fine without admin privs. I took a snapshot, built a new image off the snapshot, deleted the snapshot, transferred the image. To do that you'll need extra quota in the 'catalyst' project for room to maneuver, you can make that request here: https://phabricator.wikimedia.org/project/view/2880/ [18:08:58] !log damian-scripts@tools-bastion-15 tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17924276736 [18:09:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [18:30:44] https://cluebotng-trainer.toolforge.org currently is currently backed by NFS, but would fit perfectly into object storage... it's not so big, around 45G and 7k files, growing daily. https://wikitech.wikimedia.org/wiki/Help:Object_storage_user_guide currently states this is not supported from Toolforge, does this just mean the dance of project + provision manually + then use from anywhere has to be done similar to trove? [18:50:57] !log damian-scripts@tools-bastion-15 tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925326118 [18:50:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [19:01:20] !log damian-scripts@tools-bastion-15 tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925551560 [19:01:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [19:02:04] !log damian-scripts@tools-bastion-15 tools.cluebotng report deployed @ refs/tags/1.2.1 [19:02:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [19:02:13] !log damian-scripts@tools-bastion-15 tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925536572 [19:02:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [19:04:08] !log damian-scripts@tools-bastion-15 tools.cluebotng report deployed @ refs/tags/1.2.2 [19:04:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [19:08:29] !log damian-scripts@tools-bastion-15 tools.cluebotng report deployed @ refs/tags/1.2.3 [19:08:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [19:10:11] !log damian-scripts@tools-bastion-15 tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925753138 [19:10:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [19:12:04] !log damian-scripts@tools-bastion-15 tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925793114 [19:12:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [20:30:15] !log damian-scripts@tools-bastion-15 tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17927644810 [20:30:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL