[01:45:47] (03PS3) 10Diederik: First release. [labs/migration-assistant] - 10https://gerrit.wikimedia.org/r/108340 [01:45:49] (03CR) 10jenkins-bot: [V: 04-1] First release. [labs/migration-assistant] - 10https://gerrit.wikimedia.org/r/108340 (owner: 10Diederik) [01:46:49] (03PS4) 10Diederik: First release. [labs/migration-assistant] - 10https://gerrit.wikimedia.org/r/108340 [01:47:04] (03CR) 10Diederik: [C: 032] "Ok." [labs/migration-assistant] - 10https://gerrit.wikimedia.org/r/108340 (owner: 10Diederik) [03:29:03] 01/21/2014 - 03:29:03 - Creating a mountpoint [03:29:03] 01/21/2014 - 03:29:03 - Failed to mount the key volume /mnt/keys/ [03:30:03] 01/21/2014 - 03:30:03 - Failed to mount the key volume /mnt/keys/ [03:31:03] 01/21/2014 - 03:31:03 - Failed to mount the key volume /mnt/keys/ [03:32:02] 01/21/2014 - 03:32:02 - Failed to mount the key volume /mnt/keys/ [03:33:03] 01/21/2014 - 03:33:03 - Failed to mount the key volume /mnt/keys/ [03:34:02] 01/21/2014 - 03:34:02 - Failed to mount the key volume /mnt/keys/ [03:35:07] 01/21/2014 - 03:35:03 - Failed to mount the key volume /mnt/keys/ [03:36:03] 01/21/2014 - 03:36:03 - Failed to mount the key volume /mnt/keys/ [03:37:02] 01/21/2014 - 03:37:02 - Failed to mount the key volume /mnt/keys/ [03:38:03] 01/21/2014 - 03:38:03 - Failed to mount the key volume /mnt/keys/ [03:39:03] 01/21/2014 - 03:39:02 - Failed to mount the key volume /mnt/keys/ [03:40:03] 01/21/2014 - 03:40:03 - Failed to mount the key volume /mnt/keys/ [03:41:03] 01/21/2014 - 03:41:03 - Failed to mount the key volume /mnt/keys/ [03:41:49] That's going to get old fast... [03:42:03] 01/21/2014 - 03:42:02 - Failed to mount the key volume /mnt/keys/ [03:42:42] An instance of wm-bot? [03:43:03] 01/21/2014 - 03:43:02 - Failed to mount the key volume /mnt/keys/ [03:43:35] labs-home-wm: Who are you? [03:44:02] 01/21/2014 - 03:44:02 - Failed to mount the key volume /mnt/keys/ [03:44:29] labs-home-wm, can you take it to wikimedia-labs-nagios please? [03:45:03] 01/21/2014 - 03:45:03 - Failed to mount the key volume /mnt/keys/ [03:46:03] 01/21/2014 - 03:46:02 - Failed to mount the key volume /mnt/keys/ [03:46:41] andrewbogott: Is 208.80.153.192 a Labs instance? Is there a global query to find the corresponding project? [03:47:04] 01/21/2014 - 03:47:03 - Failed to mount the key volume /mnt/keys/ [03:47:16] labs-home-wm: quit [03:48:03] 01/21/2014 - 03:48:03 - Failed to mount the key volume /mnt/keys/ [03:49:03] 01/21/2014 - 03:49:03 - Failed to mount the key volume /mnt/keys/ [03:49:33] scfc_de: Or, related, how can I become Op so I can kick? [03:50:02] 01/21/2014 - 03:50:02 - Failed to mount the key volume /mnt/keys/ [03:50:51] andrewbogott: Dunno. [03:50:53] @ops [03:50:55] !ops [03:51:03] 01/21/2014 - 03:51:03 - Failed to mount the key volume /mnt/keys/ [03:52:03] 01/21/2014 - 03:52:03 - Failed to mount the key volume /mnt/keys/ [03:52:54] Hello. [03:53:02] 01/21/2014 - 03:53:02 - Failed to mount the key volume /mnt/keys/ [03:54:03] 01/21/2014 - 03:54:03 - Failed to mount the key volume /mnt/keys/ [03:54:04] Gloria, happen to know how I can /op myself? [03:54:16] /cs access #wikimedia-labs list [03:54:21] Probably Marc. [03:54:45] Ryan L. or Barras, actually. [03:54:47] Who knew. [03:54:53] Probably should add Marc when adding you. [03:55:02] 01/21/2014 - 03:55:02 - Failed to mount the key volume /mnt/keys/ [03:55:16] Hmmm. 208.80.153.192 definitely looks like a Public IP of a Labs instance, but isn't listed. [03:56:03] 01/21/2014 - 03:56:03 - Failed to mount the key volume /mnt/keys/ [03:57:03] 01/21/2014 - 03:57:02 - Failed to mount the key volume /mnt/keys/ [03:58:03] 01/21/2014 - 03:58:02 - Failed to mount the key volume /mnt/keys/ [03:59:03] 01/21/2014 - 03:59:03 - Failed to mount the key volume /mnt/keys/ [03:59:05] andrewbogott: Is 208.80.153.192 part of OpenStack? Listed as part of role::nova::network & Co. [03:59:56] petan: You're not really here, are you? [04:00:03] 01/21/2014 - 04:00:02 - Failed to mount the key volume /mnt/keys/ [04:01:03] 01/21/2014 - 04:01:03 - Failed to mount the key volume /mnt/keys/ [04:01:29] scfc_de: not that I can see. [04:01:54] andrewbogott: So that's some NAT? [04:02:03] 01/21/2014 - 04:02:03 - Failed to mount the key volume /mnt/keys/ [04:02:24] scfc_de: I would think that all public labs instances look like that [04:03:03] 01/21/2014 - 04:03:02 - Failed to mount the key volume /mnt/keys/ [04:04:02] 01/21/2014 - 04:04:02 - Failed to mount the key volume /mnt/keys/ [04:05:03] 01/21/2014 - 04:05:03 - Failed to mount the key volume /mnt/keys/ [04:05:34] andrewbogott: I looked around on Tools and didn't find anything obvious with labs-home-wm. I would suspect Bots, but I can't log into any host there and I assume I don't have root there either. [04:05:49] Looked on bots, doesn't seem to be a bots host either. [04:06:03] 01/21/2014 - 04:06:02 - Failed to mount the key volume /mnt/keys/ [04:07:03] 01/21/2014 - 04:07:03 - Failed to mount the key volume /mnt/keys/ [04:07:35] I'm not cool enough to op here either. :-( [04:07:54] Coren, yeah, I just emailed Ryan asking to add us. [04:08:02] 01/21/2014 - 04:08:02 - Failed to mount the key volume /mnt/keys/ [04:09:03] 01/21/2014 - 04:09:03 - Failed to mount the key volume /mnt/keys/ [04:09:34] Well, there's always /ignore and /quit. See you later. [04:10:03] 01/21/2014 - 04:10:03 - Failed to mount the key volume /mnt/keys/ [04:11:03] 01/21/2014 - 04:11:03 - Failed to mount the key volume /mnt/keys/ [04:11:31] according to nova, 208.80.153.192 is not a labs IP [04:12:03] 01/21/2014 - 04:12:03 - Failed to mount the key volume /mnt/keys/ [04:12:19] Ah, it's just virt2. So that teaches us nothing. [04:13:03] 01/21/2014 - 04:13:02 - Failed to mount the key volume /mnt/keys/ [04:14:03] 01/21/2014 - 04:14:03 - Failed to mount the key volume /mnt/keys/ [04:15:03] 01/21/2014 - 04:15:03 - Failed to mount the key volume /mnt/keys/ [04:16:03] 01/21/2014 - 04:16:03 - Failed to mount the key volume /mnt/keys/ [04:17:00] someone had to start it... [04:17:03] 01/21/2014 - 04:17:02 - Failed to mount the key volume /mnt/keys/ [04:17:26] andrewbogott: it seems to be in puppet [04:17:33] https://gerrit.wikimedia.org/r/#/c/105137/1/modules/ldap/manifests/client.pp [04:17:43] (unrelated change, but thats the code) [04:18:03] 01/21/2014 - 04:18:03 - Failed to mount the key volume /mnt/keys/ [04:18:50] https://github.com/wikimedia/operations-puppet/blob/production/modules/ldap/manifests/client.pp#L357 [04:19:03] 01/21/2014 - 04:19:03 - Failed to mount the key volume /mnt/keys/ [04:20:02] 01/21/2014 - 04:20:02 - Failed to mount the key volume /mnt/keys/ [04:21:03] 01/21/2014 - 04:21:03 - Failed to mount the key volume /mnt/keys/ [04:21:30] legoktm: I've lost the context for what you're talking about... [04:21:45] andrewbogott: labs-home-wm? [04:22:03] 01/21/2014 - 04:22:03 - Failed to mount the key volume /mnt/keys/ [04:22:17] oh! So I see, I wonder what turned that on suddenly [04:23:03] 01/21/2014 - 04:23:02 - Failed to mount the key volume /mnt/keys/ [04:24:03] 01/21/2014 - 04:24:03 - Failed to mount the key volume /mnt/keys/ [04:25:03] 01/21/2014 - 04:25:03 - Failed to mount the key volume /mnt/keys/ [04:26:03] 01/21/2014 - 04:26:02 - Failed to mount the key volume /mnt/keys/ [04:26:51] * andrewbogott tries to fix the reported problem rather than silence the bot [04:27:03] 01/21/2014 - 04:27:03 - Failed to mount the key volume /mnt/keys/ [04:28:03] 01/21/2014 - 04:28:03 - Failed to mount the key volume /mnt/keys/ [04:29:02] 01/21/2014 - 04:29:02 - Failed to mount the key volume /mnt/keys/ [04:30:03] 01/21/2014 - 04:30:03 - Failed to mount the key volume /mnt/keys/ [04:31:03] 01/21/2014 - 04:31:03 - Failed to mount the key volume /mnt/keys/ [04:32:03] 01/21/2014 - 04:32:02 - Failed to mount the key volume /mnt/keys/ [04:33:03] 01/21/2014 - 04:33:03 - Failed to mount the key volume /mnt/keys/ [04:33:16] Coren: you still around? [04:34:03] 01/21/2014 - 04:34:03 - Failed to mount the key volume /mnt/keys/ [04:35:03] 01/21/2014 - 04:35:03 - Failed to mount the key volume /mnt/keys/ [04:36:04] 01/21/2014 - 04:36:02 - Failed to mount the key volume /mnt/keys/ [04:37:03] 01/21/2014 - 04:37:03 - Failed to mount the key volume /mnt/keys/ [04:38:02] 01/21/2014 - 04:38:02 - Failed to mount the key volume /mnt/keys/ [04:39:03] 01/21/2014 - 04:39:03 - Failed to mount the key volume /mnt/keys/ [04:40:03] 01/21/2014 - 04:40:03 - Failed to mount the key volume /mnt/keys/ [04:41:03] 01/21/2014 - 04:41:02 - Failed to mount the key volume /mnt/keys/ [04:42:03] 01/21/2014 - 04:42:02 - Failed to mount the key volume /mnt/keys/ [04:43:03] 01/21/2014 - 04:43:03 - Failed to mount the key volume /mnt/keys/ [04:44:02] 01/21/2014 - 04:44:02 - Failed to mount the key volume /mnt/keys/ [04:45:03] 01/21/2014 - 04:45:03 - Failed to mount the key volume /mnt/keys/ [04:46:03] 01/21/2014 - 04:46:02 - Failed to mount the key volume /mnt/keys/ [04:46:22] Ryan_Lane, any thoughts? Labs instances are saying "[afr-common.c:3665:afr_notify] 0-keys-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up." [04:47:03] 01/21/2014 - 04:47:03 - Failed to mount the key volume /mnt/keys/ [04:48:03] 01/21/2014 - 04:48:03 - Failed to mount the key volume /mnt/keys/ [04:49:02] 01/21/2014 - 04:49:02 - Failed to mount the key volume /mnt/keys/ [04:50:03] 01/21/2014 - 04:50:02 - Failed to mount the key volume /mnt/keys/ [04:51:03] 01/21/2014 - 04:51:03 - Failed to mount the key volume /mnt/keys/ [04:52:02] 01/21/2014 - 04:52:02 - Failed to mount the key volume /mnt/keys/ [04:53:03] 01/21/2014 - 04:53:03 - Failed to mount the key volume /mnt/keys/ [04:54:03] 01/21/2014 - 04:54:02 - Failed to mount the key volume /mnt/keys/ [04:55:03] 01/21/2014 - 04:55:03 - Failed to mount the key volume /mnt/keys/ [04:56:03] 01/21/2014 - 04:56:03 - Failed to mount the key volume /mnt/keys/ [04:57:02] 01/21/2014 - 04:57:02 - Failed to mount the key volume /mnt/keys/ [04:58:02] 01/21/2014 - 04:58:02 - Failed to mount the key volume /mnt/keys/ [04:59:03] 01/21/2014 - 04:59:03 - Failed to mount the key volume /mnt/keys/ [05:00:03] 01/21/2014 - 05:00:02 - Failed to mount the key volume /mnt/keys/ [05:01:03] 01/21/2014 - 05:01:03 - Failed to mount the key volume /mnt/keys/ [05:02:03] 01/21/2014 - 05:02:03 - Failed to mount the key volume /mnt/keys/ [05:02:45] andrewbogott: hm [05:03:03] 01/21/2014 - 05:03:02 - Failed to mount the key volume /mnt/keys/ [05:03:08] Things on labstore2 look right… I'm not clear who is sending that message. [05:03:14] me neither [05:03:48] I guess the first thing to do is to fix that chatbot :) [05:03:49] it would have to be one of the labstore boxes [05:04:03] 01/21/2014 - 05:04:03 - Failed to mount the key volume /mnt/keys/ [05:04:10] coming from 208.80.153.192 [05:04:29] that's virt2, I figured that could mean 'any labs instance' [05:04:54] yeah [05:04:56] alas [05:05:03] 01/21/2014 - 05:05:02 - Failed to mount the key volume /mnt/keys/ [05:05:10] hm [05:06:03] 01/21/2014 - 05:06:03 - Failed to mount the key volume /mnt/keys/ [05:06:21] labs-home-wm: ... [05:07:03] 01/21/2014 - 05:07:02 - Failed to mount the key volume /mnt/keys/ [05:07:05] well, it's either manage-exports or manage-keys [05:08:03] 01/21/2014 - 05:08:03 - Failed to mount the key volume /mnt/keys/ [05:08:12] let me see if I can find which instances are running ircecho [05:08:25] * andrewbogott thinks ircecho should include $hostname in every message [05:08:43] yeah [05:09:03] 01/21/2014 - 05:09:03 - Failed to mount the key volume /mnt/keys/ [05:10:02] 01/21/2014 - 05:10:02 - Failed to mount the key volume /mnt/keys/ [05:10:37] go tit [05:10:39] got it [05:10:59] it's weird. puppet should have disabled that cron [05:12:02] 01/21/2014 - 05:12:02 - Failed to mount the key volume /mnt/keys/ [05:12:30] andrewbogott: it's this instance: https://wikitech.wikimedia.org/wiki/Nova_Resource:I-00000a26.pmtpa.wmflabs [05:12:41] the only reason this should happen is if $managehome is true [05:13:03] 01/21/2014 - 05:13:03 - Failed to mount the key volume /mnt/keys/ [05:13:13] and of course it is [05:14:03] 01/21/2014 - 05:14:02 - Failed to mount the key volume /mnt/keys/ [05:14:26] Oh, a brand new instance. [05:14:50] fixed [05:15:03] 01/21/2014 - 05:15:03 - Failed to mount the key volume /mnt/keys/ [07:25:50] I'm doing some performance testing on wikitech so the labs gui may be extra-slow for a bit. [11:28:04] What's happening? "database lag: 2 days, 15 hours, 14 minute, 32 seconds" [11:31:38] !replag [11:31:39] :O [11:31:59] !replag [12:09:26] Coren: I think commonswiki is busted Im seeing a lag of over 2 dats [12:09:29] *days [12:38:05] :O [13:35:57] Hi, I'm trying to connect to a labs instance via bastion, but I'm getting "permission denied (publickey)" problem. As far as I can see, I've setup all the prerequisites: I have shell access (I received a message from Tim Landscheidt on wikitech indicating I have shell access), and I've uploaded my public key to both wikitech and gerret. What am I missing? [13:41:22] I've also added the tunneling configuration lines to my .ssh/config file [13:43:15] csalvia: did you load that key into an "agent"? [13:43:46] you need to do one of 2 things, either have it loaded in agent (ssh-add), or specify the key file with ssh -i /path/to/key when you connect [13:44:14] the private key on your local computer that is [13:46:28] Hmm... okay, just did both and I still get permission denied (publickey). Which host should I connect to? bastion.wmflabs.org ? [13:48:08] mutante, can you clarify what you mean by load the key into an "agent" ? [13:49:42] csalvia: so you have a private key part and you need to specific it when you connect with ssh, and for convenience most people use an agent [13:50:16] so that the key is already loaded in background and you don't have to keep loading it on every connection [13:50:23] csalvia: if you did that correctly the key is listed when you run ssh-add -L [13:50:29] it's not a requirement for this to work, it's convenience [13:50:39] the alternative is ssh -i [13:51:04] s/specific/specify [13:51:28] jzerebecki, yes when I run ssh-add -L it lists the key [13:51:42] are you using ProxyCommand in ssh config? [13:52:13] Yes: [13:52:21] Host *.labs *.wmflabs\nProxyCommand ssh -a -W %h:%p bastion.wmflabs.org\n [13:53:46] csalvia: try replacing "bastion.wmflabs.org" with "csalvia@bastion.wmflabs.org" [13:54:10] csalvia: and what do you type to connect? [13:54:19] okay, still getting permission denied [13:54:21] to connect I type: [13:54:23] and where do you want to end up? [13:54:32] ssh bastion.wmflabs.org -i .ssh/id_rs [13:54:42] from my home dir [13:55:08] but you actually want to connect to a labs VM _behind_ the proxy, right [13:55:13] yes [13:55:14] you need to connect to the instance not the bastion [13:55:25] csalvia: what is the name of that labs instance? [13:56:46] Host labsinstance [13:56:54] ProxyCommand .. as you said it above [13:57:10] to connect: simply "ssh labsinstance" [13:59:34] Well, I'm trying to connect to any instance ... where can I get a list of instance names? [14:01:59] mutante: i wanted to test apply some stuff from operations/puppet.git by use of an labs instance, can you add me as an admin to e.g. https://wikitech.wikimedia.org/wiki/Nova_Resource:Testlabs or a similar labs projects I can use for that? [14:02:26] csalvia: on the labs project page [14:04:56] csalvia: on https://wikitech.wikimedia.org/wiki/Main_Page on the right under usage there is one link for the labs projects, find the one you need and on its page is a list of instances [14:05:32] csalvia: using wildcards in Host should work, but *.wmflabs.org will not work as expected [14:05:43] try *.wmflabs [14:05:52] then connect to instance.pmtpa.wmflabs [14:06:35] that's internal, really just .wmflabs. example PING wikistats-01.pmtpa.wmflabs [14:06:44] 64 bytes from i-00000042.pmtpa.wmflabs (10.4.0.9) [14:07:02] jzerebecki: yea, let me take a look which project makes sense [14:08:21] "pmtpa" means Tampa and will change to eqiad after migration [14:08:49] I'm able to ping it: [14:08:55] ping wikistats-01.pmtpa.wmflabs [14:09:02] PING wikistats-01.pmtpa.wmflabs (67.63.55.3) 56(84) bytes of data. [14:09:08] 64 bytes from 67.63.55.3: icmp_req=1 ttl=242 time=90.0 ms [14:09:11] csalvia: from where? [14:09:20] although the ip doesn't look internal [14:09:29] that was just an example, replace wikistats-01 with the instance you want [14:09:47] csalvia: that's a special case, because that instance ALSO has a public IP [14:09:54] don't let it confuse you :p [14:10:04] I'm just trying to connect to any instance. wikistats-01 is as good as any just to test. [14:10:08] that is because it is a webserver for the public [14:10:12] I see [14:10:43] ssh wikistats-01.pmtpa.wmflabs [14:10:48] Permission denied (publickey). [14:10:49] csalvia: but you need to be a member of the labs project for you to have ssh access [14:10:55] ssh_exchange_identification: Connection closed by remote host [14:11:16] which labs project are you a member of? [14:11:21] csalvia: i did now mean to actually connect to that one, i just wanted to show you how for your project [14:11:33] as jzerebecki said, you need to be a project member [14:11:51] s/now/not .. damn can't type [14:12:57] you can just be a global shell user to any instance, even the labs admins need to add themselves to projects separately it they need shell [14:12:57] I'm pretty sure I'm a project member: https://wikitech.wikimedia.org/wiki/User:Csalvia [14:13:00] cant [14:13:26] of which project though? there are so many of them [14:13:47] what do you want to work on? [14:14:58] analytics [14:15:12] right now I'm not really trying to work on anything in particular, just trying to setup access [14:16:06] Regardless, I'm pretty sure I have shell access to any wikilabs instance: [14:16:07] https://wikitech.wikimedia.org/w/index.php?title=User_talk:Csalvia&oldid=prev&diff=96623 [14:16:11] csalvia: your setup sounds ok so far then, you just need to pick a project at this point [14:16:22] and request membership in it [14:16:46] that's an AND thing [14:16:58] you need global shell right and also be in a project [14:17:04] to be able to ssh into an instance [14:18:20] because the instances are separated in projects, with project-wide permissions [14:19:11] you could be a project member without shell or a shell user without any project (that's what you have right now) [14:21:02] csalvia: https://wikitech.wikimedia.org/w/index.php?search=analytics&title=Special%3ASearch [14:22:47] okay, I see, I will request membership and then try to SSH. [14:22:50] Thanks very much [14:23:02] welcome to labs [14:30:45] Coren: you around? [14:31:14] Am now, but on first coffee so in first gear. :-) What's up? [14:32:06] some projects exist but don't show up in search apparently [14:32:10] reported by jzerebecki [14:32:33] Coren: Im seeing commons with a 238748 second lag [14:32:51] Betacommand: Probably some stuck query. Lemme go check. [14:33:00] confirmed, i am member and use it, but it's not shown in the list of projects i'm member of [14:33:35] FYI: having shell access gets you to the bastions if nowhere else. [14:34:02] ---TRANSACTION 2EF192B8, ACTIVE 238987 sec [14:34:23] whats that transaction ? [14:35:27] Betacommand: Ah, nevermind, that's the /replication/ thread. I see the problem: Got a packet bigger than 'slave_max_allowed_packet' bytes [14:35:38] How odd. [14:37:26] That's a new global variable; I need to read up on it first. [14:45:14] Betacommand: That'll need some TLC from Sean; it doesn't look like I can simply increase this. I'll jump on him the moment he wakes up. [14:45:54] Coren:.... that just sounds wrong [14:46:18] Heh. Thankfully, we are insulated by IRC so no impropriety will take place. :-P [14:47:35] !log puppet-cleanup adding jzerebecki incl. project admin (volunteer ops/puppet work) [14:50:37] Coren: Im surprised the monitoring software isnt screaming bloody murder about commons replication [14:51:47] Betacommand: Our labs monitoring is subpar atm; most of our bandwidth has gone towards the new datacenter and migration towards eqiad. On my todo in eqiad is a proper icinga and ganglia setup for Labs in general, tool labs in particular. [14:51:57] no logs? [14:52:09] bot i mean [14:52:18] mutante: The bot went braindead last night and we had to kick it from the channel. [14:52:26] Coren: gotha, ok [16:08:39] hey guys [16:08:44] anybody up? [16:08:49] i'm trying to help a new analytics team member get access to labs [16:08:56] he's been told he's got a shell account on bastion.wmflabs.org [16:09:06] but, since I am an ops member, I have an account on a different bastion [16:09:20] so i can't log into bastion.wmflabs to check if he actually does [16:09:54] ottomata: what is the username? [16:10:14] csalvia [16:10:37] ottomata: I see 'Charles-salvia' is part of the bastion project [16:10:41] so has been added [16:11:35] ok [16:11:48] can you cat his public key there? [16:11:53] maybe you don't have perms to do that? [16:12:00] ottomata: no, I don't, sadly [16:12:02] ok [16:12:08] what's his shell username [16:12:13] there? [16:12:22] ottomata: I can't see that either :) [16:12:26] ls /home ? [16:12:28] ottomata: in fact, everyone is a part of the bastion project [16:12:28] :p [16:12:29] https://wikitech.wikimedia.org/wiki/Special:NovaProject [16:12:37] ottomata: including you :P [16:12:39] can you ls -ld /home/csalvia [16:12:42] so you can indeed look at it [16:12:47] ottomata: I don't have admin rights :P [16:12:50] you need Coren or andrewbogott_afk [16:13:01] you can log into bastion.wmflabs.org though, right? [16:13:47] Only non-opsen can log in the "normal" bastions. [16:13:54] opsen get bastion-restricted. [16:14:41] ottomata: But as for his public key, that'd be the same he can see from the Preferences tab on Wikitech (also the same for gerrit) [16:15:36] Coren, why can't we also log into the normal bastions? [16:15:44] it would be helpful if we are trying to help people get access [16:15:47] i'd like to tail -f auth.log [16:15:53] we don't know why he can't log in righit now [16:15:56] ottomata: Only with root key. [16:16:09] ottomata: The restriction is to avoid exposure of agents. [16:16:28] (because the bastions are untrusted) [16:20:15] csalvia: Coren just said he sees [16:20:15] key_read: uudecode [...] charles@hailoo.com\n failed [16:20:18] in bastion auth.log [16:20:19] ? [16:20:29] Coren, his public key has line breaks in it? [16:20:33] that's what that means? [16:20:42] in wikitech/labsconsole you mean? [16:20:53] ottomata: Cut and paste failiure when he put in on wikitech (like word wrap, etc) [16:21:20] aye (moving this part of discussion back to #-labs) [16:21:21] so he can see [16:23:25] Coren: csalvia says no line breaks [16:23:56] Yeah, I see now. [16:24:29] Yet sshd complains that it can't uudecode the key. [16:26:25] Huh, I see two slightly different keys now on his account. [16:26:32] oh weird [16:27:01] AAAAB3NzaC1yc2EAAAADAQABAAABAQCjGSS/3k++mudvR5wrGYhLPYa/RQbtURNRhgjYA7lx9dPg6lKilQliaOMhKOxlwKDRCzNZNSz5CGG9Ype6dFrZf/c1nxF6D3YwD07eOITIy85qD7nGmFKTG2+olbwbvl0liC85AQv1Xi6C4QxL2za/t3iXb10XA7EJ0GXYKLp2nfFQ4FkR5+teFqgKpH+gKpH+gd9SswiefW97HcTvuURc4n4YIC+WZJhD5majNV9Ben5QJe/qz+GnAfGSsKxMWlRRnevx7VP3KfFy9+6Lzj4Gspq+V+34UTlvY6VK4cL37UG/53dfBLgpMVgWQ0G7sooHw2xj//7fVMpAd+ipMQzDabEJ/ [16:27:02] vs [16:27:06] AAAAB3NzaC1yc2EAAAADAQABAAABAQCjGSS/3k++mudvR5wrGYhLPYa/RQbtURNRhgjYA7lx9dPg6lKilQliaOMhKOxlwKDRCzNZNSz5CGG9Ype6dFrZf/c1nxF6D3YwD07eOITIy85qD7nGmFKTG2+olbwbvl0liC85AQv1Xi6C4QxL2za/t3iXb10XA7EJ0GXYKLp2nfFQ4FkR5+teFqgKpH+gd9SswiefW97HcTvuURc4n4YIC+WZJhD5majNV9Ben5QJe/qz+GnAfGSsKxMWlRRnevx7VP3KfFy9+6Lzj4Gspq+V+34UTlvY6VK4cL37UG/53dfBLgpMVgWQ0G7sooHw2xj//7fVMpAd+ipMQzDabEJ/ [16:28:40] Those /can't/ be two really different keys, so something funky is going on at C&P [16:29:33] C&P? [16:30:00] those are different keys [16:37:01] Copy and Paste. [16:37:30] ottomata: They can't be /truly/ different keys with 95% identical bits! [16:38:01] ottomata: we tried to support csalvia earlier, the status was "has shell" but also "isn't in any projects" [16:38:04] The only difference between them is 'gKpH+' in the middle of one. [16:38:08] ottomata: so get him some projects?:) [16:38:12] he should be in ananlytics [16:38:16] will double check [16:38:16] ok,cool [16:38:24] mutante: That's not it. Broken key. [16:38:25] he's def. in analytics as an admin [16:38:26] that key issue now is extra i didn't know [16:38:29] yea, saw [16:39:18] is that Charles' fault or something weird with ldap? [16:39:26] Jan 21 16:38:00 bastion1 sshd[19961]: Accepted publickey for csalvia from 24.191.70.48 port 39879 ssh2 [16:39:26] Jan 21 16:38:00 bastion1 sshd[19961]: pam_unix(sshd:session): session opened for user csalvia by (uid=0) [16:39:41] D'oh! Sorry about the IP. [16:40:41] ottomata: Well, both the broken key and the correct ones are in LDAP, and I expect both are visible from the wikitech interface. [16:41:01] ottomata: Blame his OS's paste function, remove the broken key, and call it a day. :-) [16:41:12] he's looking [16:45:21] weird, Coren, he can get into bastion now, but i get the uudecode error on the labs instance we're trying [16:46:54] Well, now there's only one key left in there; I hope he removed the right one. :-) Have him log off and back on to bastion? [16:49:43] Coren, if you have a minute could you look at https://gerrit.wikimedia.org/r/#/c/108664/? That patch is so simple and works so well that it makes me very suspicious. [16:50:33] andrewbogott: Heh. "Wait a minute, that can't be right -- it's working!" :-) [16:51:02] Yeah. Also, caching is scary. I keep thinking, is there some reason why it should refresh this object during the course of a page load? [16:51:21] I'm really not used to web programming where the whole program runs and exits each time the user interacts with it... [16:51:27] ok Coren, uhhh [16:51:51] totally [16:52:01] in auth.log on this labs instance (limn0) [16:52:02] we see [16:52:37] andrewbogott: This has liftetime of a single PHP run. Honestly, I'd be more worried about what could break if the project changed mid-code. [16:53:02] ok, well… I will merge it sometime when I'm not about to go to sleep. [16:53:11] It makes some wikitech pages much faster! [16:54:18] No doubt. There are probably some other opportunities to cache lurking in there too. [16:54:40] ottomata: Okay, the key I see from the bastion is the good one that is full of happy. [16:55:10] ottomata: What instance is he having trouble on, then? [16:57:50] Coren, hang on, this might be a multiple account/username problem [17:03:15] Coren: if people have line breaks in the keys they'll stick in ldap. someone needs to fix that code in OSM [17:03:33] would also be nice to add some more validation over what's currently being done [17:04:03] it's likely a really simple fix. someone just needs to do it. [17:05:25] yeah Coren [17:05:27] so, ha [17:05:40] apparently someone created a wikitech account for him called charles-salvia [17:05:45] which was not added to the project [17:05:47] so, working onw [17:05:48] now [17:05:55] there was a weird ssh paste error originally we think [17:05:56] but we got it [17:37:29] Coren: scfc_de: tools-exec-09 seems to have a problem to connect to replicas, other nodes (so far) seem to be ok [17:40:54] hedonil: The NAT table is gone; one moment, please. [17:41:37] hedonil: Should work now. [17:41:54] !log tools tools-exec-09: iptables-restore /data/project/.system/iptables.conf [17:42:33] scfc_de: yeah. fine. thx. [17:42:49] petan: wm-bot is down? [17:43:03] !help [17:43:03] !documentation for labs !wm-bot for bot [17:43:14]