[00:00:18] don't forget to disable them on pmtpa's tools-dev, for that matter. [00:05:29] Hmm, I found one. Not sure about two others I'm somewhat confident I had. [00:08:51] If only SGE had a scheduled job type [00:18:23] Betacommand has there been progress on that bug? [00:18:38] ToAruShiroiNeko: I havent checked [00:18:48] Coren: is the filearchive table available yet? [00:28:01] Betacommand: I've been a bit busy to keep an eye on the bug. I'll take a look early next week as I do bug wrangling to sort what's still relevant and not. [00:33:15] ToAruShiroiNeko: ^ [00:42:23] thats a shame [00:42:33] I needed the information quicker than that for the submission [00:42:38] I'll post what I have got [00:42:46] I'll note the bug though, which one was it? [01:01:14] Coren could you tell me the bug? [01:08:01] the one Betacommand was refering to? [01:53:52] would anyone else know? [01:54:28] UTFS? [01:55:07] https://bugzilla.wikimedia.org/buglist.cgi?quicksearch=filearchive&list_id=283142 ctrl-f labs [02:31:54] Change on 12mediawiki a page Developer access was modified, changed by PiRSquared17 link https://www.mediawiki.org/w/index.php?diff=921038 edit summary: [+138] mark for translation :-) [03:27:55] Is eqiad up and running for new labs instances, or are there known issues? [04:04:06] tried creating my first new instance in eqiad, i see it being done in console output, but channel 0: open failed: administratively prohibited: open failed [04:04:16] while i can connect fine to existing pmtpa instance with that key [04:04:35] in the same project [04:08:06] puppet status at 'unknown' [04:23:25] mutante|away, FWIW, the one I created did eventually come up (http://mwui.wmflabs.org/wiki/Main_Page), but it took longer than I expected. [04:29:51] Coren: wikitech losing sessions was due to a mistake I made when forcibly logging people out [04:30:29] superm401: ok, thanks, i'll try again in a little while, can still use the pmpta instance for the moment [04:31:20] Coren: I fixed it last night [04:38:46] hmm .. Error 400 on SERVER: Could not find class role::labs::instance for i-000000ea.pmtpa.wmflabs [04:39:46] configured puppet::self on existing pmtpa instance right before this [04:47:15] eh, other instance in same project, same config, doesn't happen, a bit weird but who knows what else happened there for testig [04:47:50] and root's bash_history is global across instances? [04:55:21] Ryan_Lane, mwui is now down again, and when I try to run puppetd --test, I get: [04:55:25] err: Could not request certificate: getaddrinfo: Name or service not known [04:55:39] Never mind, forgot to sudo [05:04:29] mutante|away: Re "channel 0: open failed: administratively prohibited: open failed", a) did you try logging into bastion-eqiad and ssh'ing there to see if you are using the right hostname? b) If it just times out, you might need to check the security groups to see if they allow ssh from bastion-eqiad (10.68.*?) [05:09:16] scfc_de: it doesn't time out, it really closes on me, i'm using bastion-restricted for all of them [05:09:35] i guess that is the problem and i need bastion-restricted-eqiad [05:10:08] the ssh config is identical to the pmtpa instances i can connect to and which are in same project [05:10:13] mutante|away: Can bastion-restricted resolve the hostname? [05:10:46] no, it can't [05:11:00] only the pmtpa instances, not the eqiad instance [05:11:53] That's odd and a bug :-). [05:16:13] what times out is bastion-eqiad, because i'm not supposed to be on it [05:16:25] ok.. hmm. [05:23:26] hello, i know it sucks to be greeted with bugs :) [05:23:42] but there may be one for ops people trying to get to eqiad instances [05:23:54] is there something like bastion-restricted-eqiad? [05:27:42] mutante|away: I'm behind… what's happening? [05:28:10] andrewbogott: which bastion should i use to get to eqiad instances? [05:28:21] because bastion-restricted doesnt work for them [05:28:25] mutante|away: just use bastion-eqiad.wmflabs.org for now [05:28:31] and bastion-eqiad doesnt like me [05:28:34] times out [05:28:37] I haven't set up a proper restricted bastion yet. [05:28:39] Really? [05:28:40] Hm... [05:28:44] can you ping it? [05:28:59] it should be 208.80.155.129 [05:29:35] i can, just time 10498ms [05:29:40] 3 packets transmitted, 3 received, 0% packet loss, time 10232ms [05:29:41] (I think it must be working for me because otherwise I wouldn't be able to use IRC) [05:29:58] let me try proxy'ing through it again [05:31:09] ok, so i can connect to it directly, just not to the new eqiad instance behind it [05:31:26] which one, for example? [05:31:37] eris.eqiad.wmflabs has address 10.68.16.62 [05:31:37] dzahn@bastion1:~$ ssh eris [05:31:45] and then nothing happens [05:32:00] but i see it being done in console log [05:32:14] project "planet" [05:32:16] ssh: connect to host eris port 22: Connection timed out [05:32:42] it's the first eqiad instance i created but the project existed before [05:32:57] connecting to the pmtpa instances isnt an issue [05:33:28] Is the problem maybe this? https://wikitech.wikimedia.org/wiki/Labs_Eqiad_Migration_Howto#Security_Groups [05:33:31] (I just wrote that last night) [05:34:05] ah! checks [05:34:05] also, btw, if you're accessing via bastion-restricted (in pmtpa) you need to specify domain. So it would be 'eris.eqiad.wmflabs' rather than just 'eris' [05:34:16] but from bastion-eqiad it's just 'eris' since you're in the same region [05:34:18] dzahn@bastion1:~$ host eris [05:34:19] eris.eqiad.wmflabs has address 10.68.16.62 [05:34:27] ok, yea [05:35:49] trying to proxy via bastion-restricted and using FQDN [05:35:51] =channel 0: open failed: administratively prohibited: open failed [05:36:03] looks at security groups [05:36:53] yes, 10.4.0.0/21 for SSH [05:37:04] but that is simply the default security group [05:37:04] So that's tampa only [05:37:10] that's not custom [05:37:19] yea, changing rule [05:37:45] ah, i have 2 security groups, one for pmtpa and one for eqiad [05:37:57] but both use 10.4.0.0/21 [05:39:13] Failed to add rule. [05:39:17] tries again [05:39:39] should i delete the entire default security group for pmtpa and the instances there if i dont care? [05:39:45] to keep them [05:41:11] Successfully added rule. (when _not_ selecting default in source group) [05:42:49] ok, that changed it from timeout to channel 0: open failed: administratively prohibited: open failed [05:43:08] You should delete the instance if you don't want it. No need to delete the security group. [05:43:21] which project is this, again? [05:43:35] ok, i'm just editing the one for eqiad [05:43:39] project planet [05:47:18] ssh eris.eqiad.wmflabs.org [05:47:18] ssh: Could not resolve hostname eris.eqiad.wmflabs.org: Name or service not known [05:47:22] uhm [05:47:27] still trying different ways [05:47:52] 64 bytes from eris.eqiad.wmflabs (10.68.16.62): icmp_req=1 ttl=64 time=0.618 ms [05:47:56] on the same bastion1 [05:48:00] wth [05:48:13] mutante|away: I think that the instance may never have come up. Unless you know otherwise. [05:48:27] It looks like you checked role::puppet::self when creating the instance, is that right? [05:48:28] the console log showed "Login:" [05:48:32] no [05:48:38] i did on venus and mars [05:49:02] it is definitely checked [05:49:19] puppet status is at unknown [05:49:38] and it never had a different status [05:50:08] maybe i did after it was up but i know it doesnt work when checking anything during creation [05:50:13] Mind if I delete eris and build a new one? It's just a fresh empty instance right? [05:50:15] and i also watched the console log [05:50:17] until it was done [05:50:23] no, don't mind [05:50:27] yes [05:50:49] ok, just a moment... [05:53:56] thanks [05:56:42] mutante|away: try now? [05:56:53] I changed the security rule again. I don't know what your previous change didn't work... [05:57:21] *why [05:58:53] andrewbogott: it works! [05:59:11] eh, yea, why though:) [05:59:18] but cool, ty [05:59:33] i just copy/pasted the CIDR mask and 22,22 tcp [06:00:28] oh, well, works, via bastion-eqiad [06:00:50] via bastion-restricted it doesn't, but it's ok, i'm unloading the prod key anyways [06:05:36] should the puppet status not be unknown anymore? [06:05:59] tries to apply puppet::self again [06:06:18] notice: Finished catalog run in 24.35 seconds [06:08:31] that looks like i works, finished run again after getting ::self [06:09:39] andrewbogott: yea, it looks all ok, minus this one part that it claims puppet status says Unknown in the web ui, while it actually finishes fine [06:09:49] s/says/is [09:04:44] Once migration to eqiad is complete, do we let a root know to purge the pmpta side of things? [09:05:47] TheLetterE what I have seen is that this will be done eventually ... no rush [09:06:11] Thanks GerardM- I will sign on wiki to say I'm done :) [09:07:33] !log deployment-prep restarted varnish and varnish-frontend on deployment-cache-text1 [09:07:34] Logged the message, Master [09:08:47] TheLetterE, if you're migrating a whole project then make a note here: https://wikitech.wikimedia.org/wiki/Labs_Eqiad_Migration_Progress [09:08:58] If you're talking about tools, I think Coren is keeping track of things some other way. [09:09:28] Just a tool - it just said if we have purged pmtpa to sign on wiki, but I've just signed anyway. I'm all done. [09:41:54] help!!!!!! [09:42:00] according to https://wikitech.wikimedia.org/wiki/Labs_Eqiad_Tools_Migration [09:42:10] first migrate-tool gabrielchihonglee-bot [09:42:22] then finish-migration gabrielchihonglee [09:42:43] while I do the second step, it shows finish-migration: command not found [09:42:51] what should I do? [09:43:23] gabrielchihonglee@tools-login:~$ migrate-tool gabrielchihonglee-bot Copy started Thu Mar 6 09:38:22 UTC 2014 [09:47:16] Copy complete Thu Mar 6 09:43:12 UTC 2014 [09:52:40] Gabrielchihongle: you may need for Coren to show up here… he's in Montreal so it will be a while. [09:52:52] okey, thanks! [10:38:51] (03CR) 10Hashar: "IIRC Carl told me he was not too confortable with nodejs and regex mixed together. Yuvi you might want to take over." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/116996 (owner: 10AzaToth) [13:56:56] "finish-migration" says: "User databases owned by this tool on the replicas are not renamed but the new tool credentials have been granted access.". But my database was renamed from pXXXgYYY_abc to sZZZ_abc. Was this intended, that I have to change all programs? [13:59:08] the merge script is merging cronjobs too? [14:00:29] Coren: ping O_O [14:00:55] Ping. [14:01:24] apper: That may have been made clearer; databases /on the replicas/ aren't renamed, but those on tools-db are. [14:01:56] Coren: mabye i go to move delinker too equiad... will all cronjobs etc by copyed and is the "sys" the completly the same? [14:01:59] Coren: ahhh, okay, thanks. The replicas are the replicated wiki databases... [14:02:02] Steinsplitter: Yeah, but you can split them back up again if you want. It just was unreliable to try and make two sets of migration from "stuff-from-login" and "stuff-from-dev" [14:02:11] Coren: than everything is right ;) [14:03:05] just migrated my first tool, everything went right [14:03:13] And yes, the change of DB username is a major annoyance, but this was the only right time to correct those (the current scheme had issues that prevented a number of requested features) [14:03:51] Coren: yes, it's a bit annoying, but it's okay [14:04:29] Coren: i see the db for delinker is on equiad but the tool is still on the *old*, is this possible? [14:04:35] apper: Although, if you wanted your code to be /really/ fance you could have it use table names derived from the username at runtime. :-) [14:05:02] Steinsplitter: No; but the migration script only /copies/ it does not delete in any way. [14:05:26] Coren: yes, the scipt dos not change database names etc i guess? [14:05:35] Steinsplitter: Not in pmtpa no. [14:05:49] Coren: if i break somthing, you can revert from backup? :O [14:05:56] Coren: hehe ;). I normally only have one php include file which has functions to give a database connection, so I only have to change one line in one file, this is okay [14:06:07] Steinsplitter: Not really; but not files I harmed by the process. It's just a bit manual. [14:07:00] k, thx [14:09:28] what should I do on pmtpa? delete all files? destroy database? or just leave everything as it is? [14:11:24] apper: You can do any/all of that, or keep the data around as a safeguard. At the end of the migration period, tools marked as having been migrated will just go away with pmtpa. [14:12:37] Coren: "marked as having been migrated" where? Just using finish-migration or in the table at https://wikitech.wikimedia.org/wiki/Tool_Labs/Migration_to_eqiad ? [14:13:25] apper: finish-migration having ran. Marking it in the table is appreciated though, as it means there is no question. [14:13:47] okay [14:14:19] Also, the wiki page gives a good overview but many people haven't marked it. [14:22:21] Coren: Any news about the mail thingy? [14:22:39] hedonil: Mail is on todo today. [14:22:51] You shouldn't get the error anymore, at least, but also no mail. [14:23:06] There are issues atm with the mail going to the wrong place I'm working out. [14:24:31] Coren: I still get the error (but it's also the last chance to run ultd. scripts on tools-logon w/o being email'd ;) ) [14:25:10] Ah, right, the destination exists but since you never got mail, the mailbox doesn't. [14:27:03] Coren: yep yep. mutt is looking for a matching folder in /var/mail, but of course there isn't any yet [14:50:38] Coren: I'm thinking about a wikipage with some facts & figures ( Volumes, shared Resources, user based api's etc.) as an overview [14:51:14] hedonil: It's a Wiki™ :-) [14:51:17] Coren: Would you mind providing a blank page Tool Labs\Resources overview or similar ? [14:51:49] Coren: ha hello :-] I added you as a reviewer to a bunch of beta eqiad changes :] [14:52:16] hedonil: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Overview [14:52:22] hashar: Lemme go see those. [14:52:47] Coren: fine. there we go [15:02:27] hashar: All three should be merged for you now. [15:02:43] !!! [15:02:56] Coren: and while you are in an interrupted state I could use https://gerrit.wikimedia.org/r/117193 for contint :] [15:03:09] it git::clone a bunch of repo on the labs jenkins slaves [15:05:49] err: /Stage[main]/Role::Applicationserver::Common/Labs_lvm::Volume[second-local-disk]/Exec[create-vd-second-local-disk]/returns: change from notrun to 0 failed: /usr/local/sbin/make-instance-vol 'second-local-disk' '100%FREE' 'ext4' returned 1 instead of one of [0] at /etc/puppet/modules/labs_lvm/manifests/volume.pp:52 [15:05:52] I love puppet [15:05:55] lack of verbosity :] [15:07:27] for some reason our puppet as exec { logoutput => False } [15:07:28] (03CR) 10coren: [C: 032] "ACHIEVEMENT UNLOCKED" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/113335 (owner: 10Tim Landscheidt) [15:11:00] hashar: That error message is completely incomprehensible. [15:11:31] hashar: Can you tell me what instance this is running on so I can debug? [15:12:26] Coren: https://gerrit.wikimedia.org/r/117199 would make the labs_vm exec calls to log output on failure [15:12:35] the instance is deployment-apache02.eqiad.wmflabs [15:12:44] puppet fails for other reasons as well [15:13:07] and I forgot to says to you "good morning" [15:16:50] hashar: Ah, hell. There's something funky going on when the volume name contains dashes. [15:17:10] ahh [15:17:13] * Coren did not know that. [15:17:13] would underscore bellowed ? [15:17:28] Yes. But you can't change it yet; the volume was already created. [15:17:32] Hang on. [15:17:33] I suspected dash would cause an issue but tried nonetheless [15:17:52] It just causes the device name to become 'vd-second--local--disk' [15:17:57] Which confuses the mount. [15:18:36] yeah I rebooted the other instance deployment-apache01.eqiad.wmflabs and it is not happy :D [15:18:37] I'd rather fix the issue than change the volume name -- you are /not/ going to be the only one who does that. [15:19:16] ... that's not related; the worse this can to is fail to mount /srv [15:22:00] Coren, when you have a minute can you send those two lists of self-hosted puppet instances? (I'm assuming you already have a script handy to produce the lists, otherwise I can sort it out myself) [15:22:59] andrewbogott: Better yet, I already /have/ the list if only I can remember where it is. :-) [15:23:07] that'll do [15:25:13] Coren, indeed /srv can't be mounted :D The disk drive for /srv is not ready yet or not present. Continue to wait, or Press S to skip mounting or M for manual recovery [15:25:42] hashar: ... stupid ubuntu. [15:25:46] :-] [15:26:03] there must be some option in fstab to skip that [15:26:49] hashar: No, that's part of rc.something rather. It's also comletely insane; not havin /usr is reasonable to stop boot but some random /srv? [15:28:11] yup that is a bit insane [15:30:59] hashar: -02 is now correct with the fixes to the class. [15:31:18] good! [15:31:23] mind fixing up deployment-apache01.eqiad.wmflabs while at it ? [15:31:50] hashar: I don't think I can; that idiocy seems to happen before puppet gets a chance to run. [15:32:21] well puppet did run [15:32:31] before the boot, no? [15:32:39] the instance crashed when the labs_lvm got applied and I attempted to reboot it [15:32:58] Right. But now puppet can no longer run on it; it stops booting /before/ it tries. [15:33:07] yup [15:33:19] cant we access to the console to press S (skip) ? [15:33:36] hashar: No. There is no console to access. [15:33:52] Hmmm. [15:34:02] I *might* be able to send it a fake keypress. [15:34:18] But it's almost certainly easier to delete and recreate. [15:34:53] Coren: ok deleting :] [15:35:05] ah [15:35:17] can you try the fakekeypress or console access? [15:35:28] cause if I recreate it I got to update a bunch of puppet/mw code to change the IP address [15:37:42] hashar: What's the kvm instance number? (I-XXXXXX) [15:42:49] Coren: I-0000007d.eqiad.wmflabs [15:43:05] sorry for the trouble :-( [15:45:43] I just hit 's' on its "keyboard". No help, I'm afraid. If you are quick in delete/recreate you'll probably get the same IP. :-) [15:47:14] hehe [15:47:22] deleting it [15:48:07] Coren, fingers are being pointed at labs for bots editing logged out. :p [15:48:41] Cyberpower678: And why would labs be pointed at for bugs in bots? [15:49:00] Coren, because people don't know what they're talking about. [15:49:24] Because the issue happened simultaneously in 3 different bots/ [15:49:46] No bot should edit without &assert=bot anyways. [15:49:58] Where is this discussion? [15:51:42] Coren, https://en.wikipedia.org/wiki/Wikipedia:AN#Possible_bot_malfunctioning.3F [15:53:30] $ ssh dumps-1.eqiad.wmflabs [15:53:30] The authenticity of host 'bastion2.wmflabs.org (208.80.153.202)' can't be established. [15:53:33] RSA key fingerprint is 45:e3:61:b6:e4:0a:69:fd:95:31:89:2c:0b:db:47:3b. [15:53:38] Someone please update V [15:53:40] https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints [15:59:08] Cyberpower678: https://bugzilla.wikimedia.org/show_bug.cgi?id=62288 [16:05:57] Hi all. I'm having trouble looging into my newly created eqiad instances [16:06:15] I have set up my ssh config with the ProxyCommand directive (a long time ago) [16:06:34] my config includes statements for pmtpa and eqiad [16:06:52] the former always worked well, the latter not at all [16:07:13] it seems that I get onto the eqiad bastion but then it disconnects [16:07:14] dschwen: You likely will have to log in (and/or proxycommand) though bastion-eqiad.wmflabs.org [16:07:20] Ah. [16:07:26] Hm. Need dpaste. [16:08:12] and how do I ssh from a pmtpa instance to an eqiad instance [16:08:20] I'm sure there is an FM to R [16:08:33] but I don't see it [16:09:58] dschwen: Use fqdn; 'instancename.eqiad.wmflabs' [16:10:00] ok, looks like I've got to update my ssh config [16:11:51] Coren: thanks for your consistent communication and help throughout this migration (of course) [16:12:54] :) [16:12:58] sumanah: All in a day's work, ma'am. [16:13:10] * sumanah curtsies [16:16:24] ssh: Could not resolve hostname fastcci-master.eqiad.wmflabs: Name or service not known [16:17:01] but the instance shows up in my instance list [16:17:10] created about 30mins ago [16:26:41] host tools-login.eqiad.wmflabs does not return any result [16:26:53] host tools-login.pmtpa.wmflabs works as expected [16:26:54] ... what? [16:27:15] Ah. It's entirely possible that eqiad names are only visible from eqiad bastions. [16:27:17] this seems to be the most basic thing needed to migrate [16:27:31] how am I supposed to rsync my stuff? [16:27:33] dschwen: login(-)eqiad [16:27:53] or is /data/projects synced automatically [16:27:53] hedonil: That's public IP. [16:28:19] dschwen: Wait, you want to migrate tools from the tools project or a non-tools project? [16:28:55] no, my instances [16:29:22] I appreciate your responsiveness, but this is not specifically tools related ;-) [16:29:57] I would just like to rsync stuff from my old pmtpa instances to my newly created eqiad instances [16:30:08] Aaah. [16:30:35] and I would like to be able to log into my eqiad instances [16:30:50] (but that's a bonus ;-) ) [16:30:56] Logging into your eqiad instances is easy, but you have to do it from the eqiad bastions. :-) [16:31:11] I have the ProxyCommand setup [16:31:50] The rsync might need to use IPs; but you might also want to talk to Andrew first. My understanding is that he's doing a rsync of most /data/project things in the background and you might just be a rename away. :-) [16:31:53] as I said it does login to the bastion but not one step further [16:32:27] Before you proxycommand, have you tried doing the steps yourself? [16:32:35] I really don't get it. Am I exceptionally stupid, or why do these super basic steps fail for me?! [16:32:53] one sec [16:32:59] dschwen: I don't know; you're the first to report issues with logging into their eqiad instances. [16:33:32] ssh bastion-eqiad.wmflabs.org [16:33:32] If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [16:33:32] Permission denied (publickey). [16:33:35] ugh [16:33:37] what? [16:34:31] You have the same username in labs as you do on your own box? [16:36:16] of course not ... :-( [16:36:40] So you're likely to have better luck with ssh labsusername@bastion-eqiad.wmflabs.org :-) [16:36:57] yeah [16:37:19] anyhow from my box "ssh tools-login.pmtpa.wmflabs" and "ssh tools-login.eqiad.wmflabs" both work [16:37:22] so far so good [16:38:08] "ssh fastcci-master.eqiad.wmflabs" does hang [16:38:23] (1h old eqiad instance) [16:39:13] Just hangs there? Have you checked your security groups? IIRC, they are copied verbatim from pmtpa so if you had ssh restricted to pmtpa there, it'll also be the case here. [16:39:17] "tools-login:~$ ssh fastcci-master" hangs, too [16:39:29] oooooooooh [16:39:42] I need a new network in there? [16:40:01] 10.0.0.0/8 should be okay in practice unless you fear opsen logging into your instances from prod. :-) [16:40:24] ok [16:40:31] it was 10.4.0.0/16 [16:40:39] actually /21 [16:41:00] what would be the corresponding subnet on eqiad? [16:41:08] hiyaa [16:41:20] couldn't that have been changed automatically? [16:41:22] andrewbogott: how does an opsen access an eqiad labs instance? [16:41:26] through bastion-restricted? [16:41:38] and why am I the only one stumbling over this :-/ [16:42:08] ottomata: bastion-restricted or bastion-restricted-eqiad [16:42:14] ottomata: andrewbogott just created bastion-restricted-eqiad [16:42:15] that:) [16:42:21] If the former you'll have to specific .eqiad.wmflabs as part of the instance name [16:42:37] yeah having trouble at the moment, will try -eqiad [16:42:42] dschwen: you're not [16:43:27] dschwen: I think that was basically impossible without doing it all by hand. IIRC, andrewbogott meant to write an email about it but may have forgotten. [16:43:38] Unable to create and initialize directory '/home/otto'. [16:43:56] ok, so what is the corresponding subnet then? [16:43:59] Coren: I wrote https://wikitech.wikimedia.org/wiki/Labs_Eqiad_Migration_Howto#Security_Groups [16:44:14] dschwen: 10.68.0.0/20 [16:44:19] dschwen: 10.68.0.0/20 [16:44:22] hmm, andrewbogott, i get that when trying to ssh to bastion-restricted-eqiad.wmflabs.org [16:44:25] ok, thx [16:44:38] looks like it kinda works though [16:44:39] but I guess there os really no harm in expanding to 10.0.0.0/8 [16:44:40] but then it logs me out [16:44:45] Although I've seens some weird behavior when there are two different rules for the same port. So you may have to use 10.0.0.0/8 -- that can be your fallback [16:44:52] ottomata, just a minute, I will look. [16:45:20] since we have 2 separate security group groups, one for pmtpa and oen for eqiad [16:45:27] ah ok, no probs, i just got in through usual bastion-restricted [16:45:32] i did not use the 10.0.0.0/8 [16:45:45] but the actual IP range for eqiad labs is 10.68.0.0/20. [16:45:49] mutante, does bastion-restricted-eqiad.wmflabs.org let you in? [16:46:18] hold on, gotta disconnect from prod [16:46:50] Hi All [16:47:28] Please I have problem in access tool labs [16:47:34] any one here [16:47:35] andrewbogott: yes, that works [16:47:44] changed ssh config [16:47:52] thanks! [16:48:04] Hm, so why not for ottomata... [16:48:21] ottomata: does bastion-eqiad.wmflabs.org work for you, or does it hang up when you connect? [16:48:26] because of his home dir being gone? [16:48:44] Sure, but it should create a homedir when he logs in... [16:48:59] same thing last night for me [16:49:06] you had to add the security group for me [16:49:12] for some reason we didnt quite get [16:49:26] when i tried to add it, something didnt quite work [16:49:45] then you do seemingly same thing, worked.. hmmm [16:50:12] Nah, last night you accessed bastion fine, it was the next hop that was the problem I think [16:51:05] ottomata: can you connect directly to bastion without proxying anything? [16:51:21] andrewbogott: wasnt sure which error he gets, ok, yea [16:51:54] re: 2 different rules for same port. i deleted the old one after adding the new one [16:53:21] andrewbogott: no i cannot connect to bastion-eqiad [16:53:24] it disconnects [16:53:30] ottomata: ok, stay tuned... [16:53:33] debug1: Server accepts key: pkalg ssh-rsa blen 279 [16:53:33] Connection closed by 208.80.155.129 [16:53:33] well now I get a step farther [16:53:37] please can anyone help me [16:53:41] I get logged into my new instances [16:53:45] I still don't see anything in https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints [16:53:47] I get the MOTD [16:53:49] Ouda, please tell us your actual question [16:53:51] bastion-restricted-eqiad gets farther [16:53:55] but then I get "Connection to fastcci-worker1.eqiad.wmflabs closed." [16:53:59] i see the welcome motd or whtaever [16:54:01] How am I supposed to ssh in if I don't know what fingerprints to trust? :< [16:54:04] The last Puppet run was at Thu Mar 6 16:41:06 UTC 2014 (12 minutes ago). [16:54:05]