[00:07:17] PROBLEM - Puppet run on tools-k8s-master-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:12:02] Krenair: do you know, how long it will take, if I setup a http-proxy, till I can reach the instance with it? [00:12:13] should be pretty quick [00:12:17] I mean when is it reachable [00:12:19] why did you set one up but it's not working? [00:12:53] I set up xenon.wmflabs.org, and my FF does not finding something [00:12:59] but if I curl from that instance, it works [00:13:56] Luke081515, did you try browsing to the domain before creating it? [00:14:17] I think once [00:14:21] hm. [00:14:25] how long ago did you create it? [00:15:06] about 10 minutes ago [00:15:15] okay, in that case try again in 50-60 minutes [00:15:31] I ran dig on that domain against both labs-ns0.wm.o and labs-ns1.wm.o [00:15:39] NS1 returns NXDOMAIN [00:15:57] hm, ok [00:16:56] in future I wouldn't attempt to browse to or otherwise resolve a domain until after you create it [00:17:11] :( [00:17:25] I wanted to take a look first, if it is free [00:18:16] ah, got it, I created a new domain for that 60 minutes [00:18:20] so I can work :) [00:18:36] you know you could just have use /etc/hosts or equivalent [00:18:39] used* [00:18:58] hm, ok [00:19:19] point the domain at the proxy ip [00:37:39] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Cauchy7 was created, changed by Cauchy7 link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Cauchy7 edit summary: Created page with "{{Tools Access Request |Justification=Improve the captcha.Make it easier for human and difficlut for robot. |Completed=false |User Name=Cauchy7 }}" [01:00:06] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Jérémie Roquet was created, changed by Jérémie Roquet link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/J%c3%a9r%c3%a9mie_Roquet edit summary: Created page with "{{Tools Access Request |Justification=1) Feed projects for the French Wikipedia (and maybe Wikidata) with dynamic web pages updated everyday instead of wiki pages updated ever..." [01:41:38] 06Labs, 06Collaboration-Team-Triage: Investigate and remove NFS from editor-engagement project - https://phabricator.wikimedia.org/T102663#2550027 (10Mattflaschen-WMF) a:05Mattflaschen-WMF>03None [04:00:37] hi mratrix [04:01:03] how's it goin yuvi [04:01:09] :D [04:02:56] mratrix: http://paws-public.wmflabs.org/paws-public/User:YuviPanda/wlm/Geocoding.ipynb [04:04:04] https://fa.wikipedia.org/wiki/%D9%88%DB%8C%DA%A9%DB%8C%E2%80%8C%D9%BE%D8%AF%DB%8C%D8%A7:%D9%88%DB%8C%DA%A9%DB%8C_%D8%AF%D9%88%D8%B3%D8%AA%D8%AF%D8%A7%D8%B1_%DB%8C%D8%A7%D8%AF%D9%85%D8%A7%D9%86%E2%80%8C%D9%87%D8%A7_%DB%B2%DB%B0%DB%B1%DB%B5_%D8%A7%DB%8C%D8%B1%D8%A7%D9%86/%D8%A7%D8%B1%D8%AF%D8%A8%DB%8C%D9%84 [04:07:46] ایران‎‎ [05:33:30] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Trofimovamw was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=817240 edit summary: [05:33:46] mratrix: http://tools.wmflabs.org/heritage/ [05:35:58] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Jérémie Roquet was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=817243 edit summary: [05:36:38] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Cauchy7 was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=817246 edit summary: [05:44:23] PROBLEM - Puppet staleness on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [43200.0] [05:54:54] PROBLEM - Puppet staleness on tools-exec-1211 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [43200.0] [05:58:58] PROBLEM - Puppet staleness on tools-exec-1213 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [43200.0] [05:59:10] PROBLEM - Puppet staleness on tools-exec-1204 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [43200.0] [06:13:15] 10Labs-Kubernetes: Install Helm on Kubernetes - https://phabricator.wikimedia.org/T142743#2550144 (10Ebraminio) Honestly, I didn't know that when I filed this bug. If the situation is like that no one is defining any custom RS/RC/deployment/service/pods, putting helm certainly adds no value but if anyone is inte... [07:58:12] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 0.76 ms [08:09:25] RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 0.70 ms [08:11:15] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [08:12:27] RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 1.22 ms [08:13:15] PROBLEM - Puppet staleness on tools-webgrid-lighttpd-1207 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [43200.0] [08:14:23] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [08:51:34] PROBLEM - Puppet staleness on tools-k8s-master-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [43200.0] [12:08:46] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [13:45:56] 06Labs, 15User-Luke081515: Revert: Request increased quota for rcm labs project - https://phabricator.wikimedia.org/T142311#2550415 (10Luke081515) a:03Luke081515 (Claiming, that I don't forgot this task). Actually I set up a new instance, and will now clone from the old. When I'm finished, I will at least wa... [15:41:52] ah, Jensbest der Super-Volldepp auf der vm ^^ [17:51:23] is an labsadmin here? [17:51:28] I need help for one of my instances [17:55:28] hi [17:55:37] hi :) [17:55:51] I get Permission denied (publickey,keyboard-interactive). when I try to ssh [17:56:01] before the first reboot today, everything was fine [17:56:14] what instance? [17:56:19] now I can use a normal reboot or a hard one, but after the instance is up, I can't ssh [17:56:24] it's rcm-2.rcm.eqiad.wmflabs [17:56:57] ssh as root@? [17:57:02] how to? [17:57:11] *how [17:57:22] instead of `ssh rcm-2.rcm.eqiad.wmflabs`, `ssh root@rcm-2.rcm.eqiad.wmflabs` [17:57:34] luke081515@bastion-02:~$ ssh root@rcm-2.rcm.eqiad.wmflabs [17:57:34] Permission denied (publickey,keyboard-interactive). [17:58:03] you can't ssh from bastions without agent forwarding enabled, which you shouldn't be using [17:58:46] does that work for other instances? [17:58:51] I belive this is happening for users creating new instances. [17:58:52] https://phabricator.wikimedia.org/T142186#2549243 [17:59:11] but it is an old instance [17:59:15] Oh [17:59:30] Maybe your ssh key, but even then it should ask for a password if ssh key failed [17:59:38] hm, if I use ssh root@cac.rcm.eqiad.wmflabs, ssh requests a password [17:59:39] * paladox goes back to ordering dominos LOL [17:59:58] I didn't change anything, the only thing that I've did was rebooting that instace [18:00:23] and sshing to other instances works anyway [18:00:41] can you ssh as root to other instances in that project? [18:01:11] luke081515@bastion-01:~$ ssh root@cac.rcm.eqiad.wmflabs [18:01:12] Password: [18:01:22] but as normal user it works [18:01:34] so your root key in hiera isn't up to date? [18:02:02] lemme check [18:02:21] https://wikitech.wikimedia.org/wiki/Hiera:Rcm [18:03:15] so I need to paste my actual public key there, I guess? [18:03:23] without ---- END SSH2 PUBLIC KEY ---- etc? [18:05:38] yes [18:06:18] see how we do it on Hiera:Deployment-prep [18:06:20] hi valhallasw`cloud [18:06:32] hi Krenair [18:07:11] Can you ssh to root@rcm-2.rcm.eqiad.wmflabs ? [18:07:30] better now, or did I missed something? https://wikitech.wikimedia.org/wiki/Hiera:Rcm [18:07:39] * valhallasw`cloud tries [18:07:50] no, it's not accepting my labs root key [18:07:55] o.O [18:08:05] but if puppet wasn't working in the last few days, it might not work [18:08:10] what about cac.rcm.eqiad.wmflabs valhallasw`cloud? [18:08:41] that one works for me [18:09:06] my guess is that rcm-2 did get the puppet change that removed all root keys but not the one that re-added them [18:09:18] :-/ [18:09:56] Luke081515: I fixed your pubkey formatting, the newlines break yaml [18:10:02] thx [18:10:25] Luke081515: let me check if this correctly adds your root key to cac.rcm [18:10:26] hm, but sshing as root still wants a pwd [18:10:34] yeah needs a puppet run [18:10:41] ah [18:10:58] what's the command to trigger one? [18:11:06] sudo puppet agent -tv [18:11:25] ok, I triggered one for cac to test [18:11:31] but sadly this doesn't helps at rcm-2 [18:11:43] maybe somebody can access via salt or something? [18:11:43] Luke081515: cac should work now [18:12:01] I think the only people who can *attempt* to get into that rcm-2 instance now are full labs ops using salt (since you have no project saltmaster), or *maybe* (not sure if it's set up yet) interactive console [18:12:14] luke081515@bastion-01:~$ ssh root@cac.rcm.eqiad.wmflabs [18:12:14] Password: [18:12:17] :-/ [18:12:28] Luke081515, are you using agent forwarding? [18:12:39] I think yes [18:12:56] why? [18:12:57] yep, it's active [18:13:07] I can try without [18:13:55] hm, when I disable it, I can't still ssh with root [18:14:29] I wouldn't expect you to be able to [18:14:34] You just shouldn't use agent forwarding [18:15:24] Luke081515: can I reboot cac-2 again? [18:15:28] eh [18:15:29] rcm-2 [18:15:35] valhallasw`cloud: sure [18:15:58] Krenair: when I disable it, my clients wants a password every time I ssh to an instance [18:16:51] Luke081515: https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_instances_with_ProxyCommand_ssh_option_.28recommended.29 [18:17:26] hrm, the console output no longer shows the boot sequence? odd. [18:18:24] not sure what's going on there [18:19:04] hm, I'm using putty for sshing, how can I tell putty to use that linked method? [18:20:44] I've seen lots of console logs missing that for some reason [18:20:55] * Krenair will be back later [18:21:00] Luke081515: not sure... [18:23:36] i think, there's no possibility for that link method [18:23:57] the disadvantage of windows.... [18:23:59] but what really is the problem to use agent forwarding [18:27:21] doctaxon: the intermediate party (i.e. the bastion) can use all your private keys [18:27:45] so you're not just trusting us with your labs private keys, but with all others as well [18:35:30] * Luke081515 is back later, now first at dinner [19:16:26] valhallasw`cloud: is having agent forwarding enabled a problem if you only have one key loaded (the labs one)? [19:17:16] tom29739: anyone who is root on bastion can probably already impersonate you on all of labs, so as long as you don't use that key anywhere else it's probably OK. [19:26:39] I'm a bastion projectadmin and AFAIK my access can't be used to impersonate people using ProxyCommand-based ssh inside labs instances where I cannot sudo [19:29:29] (admittedly at this stage that's probably not a huge percentage of instances) [19:31:12] I think all of the bastion projectadmins could also just add themselves to a project [19:32:21] using different rights, yes, true. [19:34:58] among other various other potentially-abusable actions [20:03:38] Luke081515: I found this blog post on proxycommand and putty -- https://monkeyswithbuttons.wordpress.com/2010/10/01/ssh-proxycommand-and-putty/ -- if you can confirm it works that would be a nice thing to add to wikitech [20:03:57] ok :) [20:04:03] bd808: btw, do you got salt access? [20:04:40] I don't. Only full roots have that as far as I know. [20:05:00] I *think* puppet runs on reboot though so you could try that [20:05:04] * Luke081515 don't know´who are "full roots" [20:05:14] bd808: IIRC, I already rebootet it about 4 times [20:05:20] I tried normal and hard reboot [20:05:41] The "full roots" are -- https://wikimediafoundation.org/wiki/Staff_and_contractors#Labs [20:05:57] wikimedia ops too? [20:06:15] Oh and Madhu now. Yeah anyone in techops has root everywhere [20:06:57] hm, ok, then I have to wait for one of them [20:08:30] luckily I replaced most of that instance already, but I need one repo from that. I mean I have that repo localy too, but copying that repo at that instance is ways more easier then setup a new repo, so I can avoid mass of duplicate-messages [20:16:32] ops who are not on the labs team can also do the same as them [20:28:41] Luke081515: I can try and help, what's up? [20:29:52] madhuvishy: my problem is, that after a reboot I can't access rcm-2.rcm.eqiad.wmflabs, I get Permission denied (publickey,keyboard-interactive). all the time [20:30:00] valhallasw`cloud tried already to login via root [20:30:12] he's assuming, that puppet is not running there [20:31:34] madhuvishy, I was thinking maybe either salt or interactive console access may work [20:32:09] I don't trust salt, and I'm not sure if interactive consoles were fully set up yet [20:32:30] o.O [20:32:36] Krenair: yeah I don't know much about this either [20:32:38] looking anyway [20:45:20] labtesthorizon seems broken again :S [20:45:32] all it logs is my login failure [20:59:23] Krenair: I can get an interactive console but not sure what to do with it [21:01:35] not having such access I have no idea what you can do at that level [21:01:50] are you logged in as root or something? [21:09:34] yeah [21:11:22] madhuvishy: tail /var/log/auth.log is probably the best source of information [21:11:43] and possibly a forced puppet run can help? i.e. puppet agent -tv [21:11:44] I wonder why I can't find an interactive console UI in labtesthorizon? [21:12:02] unbroke the damn thing just to log in... [21:12:31] got a labtestspice url from novaclient though that too seems not to work [21:14:51] valhallasw`cloud: puppet run from where? [21:15:40] i'm looking at auth logs on labvirt1006 - which is where this instance seems to be - I am not even sure that's the right place - but the logs don't tell me much [21:15:40] madhuvishy: on that host (rcm-2?) [21:15:49] ah i cant get in [21:15:54] madhuvishy: but maybe I'm misunderstanding, I thought you had an interactive console on that host [21:15:55] I thought you got an interactive console? [21:16:06] yes from labvirt1006 [21:16:25] so can you use it to run puppet inside the instance? [21:16:27] but nothing i type even shows up [21:16:30] i tried [21:16:36] okay [21:16:39] did salt not work either? [21:16:40] but ^ [21:17:11] didn't try [21:17:28] madhuvishy: ok, just to make sure I'm following. You logged in to labvirt1006, and ran virsh console --devname serial1 , and that hangs? [21:17:55] Pretty sure that didn't work last time I tried it in labtest [21:17:58] one thing to try might be to reboot the box and see if that interactive console shows anything [21:18:14] yeah, I only know what wikitech says :/ [21:18:19] valhallasw`cloud: yeah so [21:18:24] https://www.irccloud.com/pastebin/HJ4Zaoqu/ [21:18:30] this is what it says [21:18:44] and then there's a cursor [21:18:45] but if i type stuff it doesn't show up [21:18:55] :/ [21:18:58] not sure if I'm being dumb somehow [21:19:06] but nothing happens [21:19:26] madhuvishy: I can try rebooting the host from wikitech, maybe it'll show something then? [21:19:40] may be [21:19:51] i'll keep it open while you try [21:19:54] yeah this wasn't the interactive console I was talking about... a new system was (is being?) set up recently [21:20:03] the wikitech console was also super silent.. that might be related [21:20:12] madhuvishy: rebooted [21:20:22] the console quit [21:20:27] bah. [21:20:29] didn't show anything [21:20:30] he he [21:20:35] I've now (finally) edited that page to mark the virsh console instructions as not working [21:21:05] valhallasw`cloud: can you try rebooting again? [21:21:08] I'm not sure it ever worked [21:21:15] madhuvishy: sure! [21:21:22] madhuvishy: rebooting [21:24:32] valhallasw`cloud: hmmm [21:27:32] madhuvishy: maybe https://wikitech.wikimedia.org/wiki/OpenStack#Get_a_web-based_console_and_root_password ? [21:28:38] valhallasw`cloud: I did virsh destroy to force reboot, the docs say rebooting from nova should bring it back up [21:28:43] but i'm not sure that happened [21:29:18] ah, it's now in shutdown state [21:29:22] yeah [21:29:36] I get error: domain not running [21:30:24] horizon says 'Unable to start instance: rcm-2' [21:30:34] but it does seem to be running! [21:30:35] huh [21:31:19] yep, host is back online, but auth still fails [21:31:30] madhuvishy: wait. can't you just ssh in with the root password? :D [21:32:47] I thought that host didn't accept passwords when you tried? [21:33:05] I don't have access to the root password [21:34:01] anyway I was trying the web-based console thing in labtest earlier, didn't seem to work properly [21:34:22] one of the labtest instances I tried showed someone had logged in already, but it couldn't keep the connection for some reason [21:35:54] valhallasw`cloud: the root password change was reverted [21:40:07] Luke081515: can you check now? [21:40:14] I rebooted instance via nova [21:40:21] i can now get in as room [21:40:24] root [21:40:28] valhallasw`cloud: ^ [21:40:35] yeah, root seems to work for me as well now [21:40:50] cool [21:40:56] says puppet run already in progress [21:41:28] Aug 13 21:40:23 rcm-2 sshd[1867]: error: AuthorizedKeysCommand /usr/sbin/ssh-key-ldap-lookup returned status 1 [21:41:29] Aug 13 21:40:23 rcm-2 sshd[1867]: error: key_read: uudecode AAAAB....== 42:..:2b luke081515 rcm root key\n failed [21:41:35] >_< [21:42:01] huh [21:42:27] and there's an oddity with sshd trying to start twice ('error: Bind to port 22 on :: failed: Address already in use.') [21:42:40] hmmm [21:42:41] might explain why it was failing earlier [21:42:43] i got to go now [21:42:53] meeting a friend in 20 minutes [21:42:54] thanks for your help madhuvishy [21:42:58] madhuvishy: thanks! [21:43:09] np! thanks for helping me learn a bit about this stuff today :) [21:43:12] byeee [21:43:27] madhuvishy: thank you very much :) [21:43:28] no, wait, I think that's just a red herring. The key_read is actually reading from /etc/ssh/userkeys [21:47:12] ok, I have the feeling two different sshds are trying to start on that host, and one understands my key and the other doesn't [21:47:28] does it run gerrit or something? [21:48:39] seems to be two sshds [21:48:42] https://www.irccloud.com/pastebin/JOPQpgAC/ [21:49:15] ! there's /etc/ssh/sshd_config.phabricator [21:49:15] hello :) [21:49:22] which also specified port 22 [21:49:51] and that only accepts user vcs etc [21:49:55] oh it runs phab [21:49:59] that'll be for git-ssh.wikimedia.org [21:50:03] or the equivalent thereof [21:50:07] *nod* [21:50:15] not sure how prod sets this up, most likely two IPs? [21:50:28] yeah, or maybe sshd on a different port? [21:50:33] like gerrit. [21:50:45] now's a bad time for me to investigate multiple-IPs-from-nova :) [21:51:07] Luke081515: ^ summary: don't reboot that box ;-) [21:51:16] ok :) [21:51:36] I will put the data from it, (I already replaced the most), and then in 2-3 I will delete it [21:51:59] *2-3 weeks [21:53:33] or you could just fix your phab ssh config to not interfere [21:54:23] * valhallasw`cloud tests if puppet reverts that [21:55:11] iff it does, stick role::puppet::self on there and make the fix a little more permanent? [21:55:40] puppet doesn't revert it (the phab role isn't applied anymore, I guess?) [21:55:50] Luke081515: so phabricator ssh will now start on port 2222 instead [21:56:03] Krenair: do you know, whoch service I have to stop that phab stops ssh things? most of phab service is already down [21:56:12] I'm just make sure actually that everything is at the new one [21:56:14] not off the top of my head [21:56:33] anyway, I'm off to bed [21:56:35] good night! [21:56:58] valhallasw`cloud: gn8, and thanks for your help [21:58:07] Krenair: and for your help too :) [21:58:12] np [22:05:55] Back in Feb, I submitted a task (https://phabricator.wikimedia.org/T127633) to install a package on exec servers. The package has its .deb repo so it should be very straightforward. It's a morphological analyser of Polish words, potentially useful for Wiktionarians (but also others who do NLP). Is there any way of bumping the task up? I'd help if only I could [22:06:46] exec servers? this is for the tools project? [22:07:02] yeah, okay, I added that to the task [22:07:38] looking through tools' workboard they have quite a few package install requests [22:08:08] unfortunately valhallasw just went to bed [22:08:22] might be best to try asking again in the morning or, preferably, monday [22:08:27] hm, wikibugs is quiet? [22:08:36] though I should warn you tasks do not accumulate priority over time [22:08:56] Luke081515, I'll give it a kick [22:09:19] Krenair, I imagine so, but it seems it hasn't even been triaged... [22:09:40] what do you mean alkamid? [22:09:51] Luke081515, ^ [22:10:00] Krenair: thank you :) [22:10:17] ah, automatic nickchange :) [22:10:19] Krenair, oh, it was triagged, sorry [22:10:56] so I should try and catch valhallasw and bugger him [22:11:04] um [22:12:35] 'to bug' someone, and 'to bugger' them, have extremely different meanings in the english language :) [22:13:23] At least it does here anyway: https://en.wiktionary.org/wiki/bugger#Verb [22:15:35] Krenair, OMG, I meant badger, it's time to go to bed [22:16:17] I certainly don't want to bugger anyone (; [22:25:21] Alkamid: short answer is: we install if in official ubuntu repo, not from 3rd party [22:25:30] Will respond in mote detail tomorrow [22:28:05] valhallasw`cloud: 3 minutes to late :-/ [22:28:10] 00:22 -!- alkamid [~adam@wiktionary/alkamid] has quit [Quit: Leaving] [23:27:01] hi! I'm trying to run a website under mono and got this: http://stackoverflow.com/questions/24872394/ Granting special rights shouldn't really be required. Could someone just try creating /etc/mono/registry/LocalMachine && /etc/mono/registry/Users for me please? That should fix it