[03:30:06] <wikibugs>	 6Labs: Delete tool labs project "commonstools" - https://phabricator.wikimedia.org/T99868#1300678 (10Fastily) 3NEW
[03:54:02] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1407 is CRITICAL 60.00% of data above the critical threshold [0.0]
[04:19:01] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1407 is OK Less than 1.00% above the threshold [0.0]
[06:18:16] <wikibugs>	 6Labs, 3ToolLabs-Goals-Q4: Investigate kernel issues on labvirt** hosts - https://phabricator.wikimedia.org/T99738#1300775 (10yuvipanda) Moritz pointed out that labvirt1005 doesn't have any instances running so we could do this there.
[06:18:35] <wikibugs>	 6Labs, 6operations, 3ToolLabs-Goals-Q4: Investigate kernel issues on labvirt** hosts - https://phabricator.wikimedia.org/T99738#1300776 (10yuvipanda)
[06:23:09] <wikibugs>	 6Labs, 6operations, 3ToolLabs-Goals-Q4: Investigate kernel issues on labvirt** hosts - https://phabricator.wikimedia.org/T99738#1300793 (10yuvipanda) (labvirt1005 is empty because of T97521 - it hasn't been repooled yet)
[06:44:09] <shinken-wm>	 PROBLEM - Puppet failure on tools-submit is CRITICAL 33.33% of data above the critical threshold [0.0]
[07:09:13] <shinken-wm>	 RECOVERY - Puppet failure on tools-submit is OK Less than 1.00% above the threshold [0.0]
[08:27:07] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1212 is CRITICAL 33.33% of data above the critical threshold [0.0]
[08:27:21] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1216 is CRITICAL 44.44% of data above the critical threshold [0.0]
[08:27:23] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1201 is CRITICAL 40.00% of data above the critical threshold [0.0]
[08:27:49] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1203 is CRITICAL 50.00% of data above the critical threshold [0.0]
[08:28:31] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1209 is CRITICAL 30.00% of data above the critical threshold [0.0]
[08:28:33] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1207 is CRITICAL 40.00% of data above the critical threshold [0.0]
[08:28:40] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1213 is CRITICAL 40.00% of data above the critical threshold [0.0]
[08:30:10] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1208 is CRITICAL 44.44% of data above the critical threshold [0.0]
[08:30:17] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1218 is CRITICAL 55.56% of data above the critical threshold [0.0]
[08:30:31] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-wmt is CRITICAL 20.00% of data above the critical threshold [0.0]
[08:31:21] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1211 is CRITICAL 60.00% of data above the critical threshold [0.0]
[08:31:21] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1202 is CRITICAL 44.44% of data above the critical threshold [0.0]
[08:31:27] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1206 is CRITICAL 40.00% of data above the critical threshold [0.0]
[08:31:36] <shinken-wm>	 PROBLEM - Puppet failure on tools-master is CRITICAL 30.00% of data above the critical threshold [0.0]
[08:31:50] <shinken-wm>	 PROBLEM - Puppet failure on tools-mail is CRITICAL 20.00% of data above the critical threshold [0.0]
[08:31:56] <shinken-wm>	 PROBLEM - Puppet failure on tools-mailrelay-01 is CRITICAL 30.00% of data above the critical threshold [0.0]
[08:32:42] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1205 is CRITICAL 30.00% of data above the critical threshold [0.0]
[08:33:08] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1210 is CRITICAL 44.44% of data above the critical threshold [0.0]
[08:33:24] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1201 is CRITICAL 55.56% of data above the critical threshold [0.0]
[08:33:42] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1202 is CRITICAL 40.00% of data above the critical threshold [0.0]
[08:34:46] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1208 is CRITICAL 40.00% of data above the critical threshold [0.0]
[08:34:50] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1203 is CRITICAL 20.00% of data above the critical threshold [0.0]
[08:38:32] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1215 is CRITICAL 60.00% of data above the critical threshold [0.0]
[08:39:18] <shinken-wm>	 PROBLEM - Puppet failure on tools-shadow is CRITICAL 20.00% of data above the critical threshold [0.0]
[08:39:58] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1209 is CRITICAL 50.00% of data above the critical threshold [0.0]
[08:40:08] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1204 is CRITICAL 33.33% of data above the critical threshold [0.0]
[08:40:45] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1207 is CRITICAL 50.00% of data above the critical threshold [0.0]
[08:41:17] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1205 is CRITICAL 22.22% of data above the critical threshold [0.0]
[08:41:33] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1206 is CRITICAL 20.00% of data above the critical threshold [0.0]
[08:43:01] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1219 is CRITICAL 20.00% of data above the critical threshold [0.0]
[08:43:21] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1210 is CRITICAL 55.56% of data above the critical threshold [0.0]
[08:44:21] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL 22.22% of data above the critical threshold [0.0]
[08:44:39] <shinken-wm>	 PROBLEM - Puppet failure on tools-precise-dev is CRITICAL 40.00% of data above the critical threshold [0.0]
[08:45:04] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1217 is CRITICAL 55.56% of data above the critical threshold [0.0]
[08:45:10] <shinken-wm>	 PROBLEM - Puppet failure on tools-submit is CRITICAL 55.56% of data above the critical threshold [0.0]
[08:46:00] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1214 is CRITICAL 50.00% of data above the critical threshold [0.0]
[08:46:01] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1204 is CRITICAL 60.00% of data above the critical threshold [0.0]
[08:55:40] <wikibugs>	 6Labs: Fix monitor_labs_salt_keys.py to handle the new labs naming scheme - https://phabricator.wikimedia.org/T95481#1300894 (10Andrew) Yes and no. I'm trying to eliminate use of the ec2 ID.  So, for example, my instance util-abogott used to have a salt key of 'i-000000ee.eqiad.wmflabs'; now it uses 'i-000000ee....
[08:57:28] <wikibugs>	 6Labs, 6operations, 3ToolLabs-Goals-Q4: Investigate kernel issues on labvirt** hosts - https://phabricator.wikimedia.org/T99738#1300898 (10Andrew) yep, we can do this on labvirt1005 at any time.  We can cold-migrate some test instances there to test the suspend/resume issue.  Well, ok, actually, let's make s...
[08:58:22] <wikibugs>	 6Labs, 6operations, 3ToolLabs-Goals-Q4: Investigate kernel issues on labvirt** hosts - https://phabricator.wikimedia.org/T99738#1300899 (10Andrew) And, btw, there's no short-term plan to repool labvirt1005 -- we've always planned to keep a server empty as a backup, and since labvirt1005 was down during the m...
[09:37:24] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1201 is OK Less than 1.00% above the threshold [0.0]
[10:11:21] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1205 is OK Less than 1.00% above the threshold [0.0]
[10:12:49] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1203 is OK Less than 1.00% above the threshold [0.0]
[10:21:25] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1206 is OK Less than 1.00% above the threshold [0.0]
[10:30:11] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1204 is OK Less than 1.00% above the threshold [0.0]
[12:04:45] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1208 is OK Less than 1.00% above the threshold [0.0]
[12:05:45] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1207 is OK Less than 1.00% above the threshold [0.0]
[12:11:01] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1214 is OK Less than 1.00% above the threshold [0.0]
[12:12:22] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1216 is OK Less than 1.00% above the threshold [0.0]
[12:13:42] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1213 is OK Less than 1.00% above the threshold [0.0]
[12:16:16] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1211 is OK Less than 1.00% above the threshold [0.0]
[12:17:06] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1212 is OK Less than 1.00% above the threshold [0.0]
[12:18:06] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1210 is OK Less than 1.00% above the threshold [0.0]
[12:23:33] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1215 is OK Less than 1.00% above the threshold [0.0]
[12:25:03] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1209 is OK Less than 1.00% above the threshold [0.0]
[12:30:03] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1217 is OK Less than 1.00% above the threshold [0.0]
[12:32:58] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1219 is OK Less than 1.00% above the threshold [0.0]
[12:40:14] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1218 is OK Less than 1.00% above the threshold [0.0]
[14:30:38] <ankita-ks>	 hi
[14:30:42] <ankita-ks>	 i need some help
[14:31:00] <ankita-ks>	 I have been trying to setup language tool server on tools labs.
[14:31:17] <ankita-ks>	 unsuccessfully
[14:31:54] <ankita-ks>	 Here are some doubts I have :
[14:32:08] <ankita-ks>	 1. what is portgrabber and what does it do?
[14:32:53] <ankita-ks>	 2. This one is pretty basic, let's say i set the server up, where do I access it?
[14:33:12] <ankita-ks>	 3. If it is running as a service, does it have an ip associated with it?
[14:33:28] <ankita-ks>	 Basically to test if it is running or not, what ip and port do I test?
[14:34:03] <ankita-ks>	 also, I haven't figured out the tools labs yet. I mean I can set it up and follow the instructions on the wiki, but I don't get how it works.
[14:34:15] <ankita-ks>	 For instance when say I become <sometool>
[14:34:20] <ankita-ks>	 what am I doing exactly
[14:35:08] <ankita-ks>	 or if i start webservice, what happens? (apart from the toolset being up)
[14:35:17] <ankita-ks>	 what changes is it making to the system and where?
[17:31:11] <Betacommand>	 ankita-ks: what are you trying to do?
[17:49:03] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1410 is CRITICAL 25.00% of data above the critical threshold [0.0]
[18:02:21] <russblau>	 When qstat shows a line with status "Eqw", I assume the "E" stands for "Error", but how to find out what caused the problem?
[18:05:42] <mutante>	 russblau: try  qstat -j jobid
[18:05:54] <mutante>	 if you have the ID
[18:06:07] <mutante>	 supposedly that shows an "error reason"
[18:06:55] <mutante>	 "Eqw means is that the queue is in error state, thats the E, and is in queue wait, thats the qw."
[18:07:20] <russblau>	 thanks, but output of that isn't very helpful; is the whole grid having problems?
[18:08:11] <russblau>	 error reason    1:          05/21/2015 17:59:16 [51290:13568]: execvlp(/var/spool/gridengine/execd/tools-webgrid-lighttpd-1201/j
[18:11:28] <mutante>	 russblau: sorry, i dont know, but it seems like it then  
[18:17:42] * yuvipanda waves
[18:19:03] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1410 is OK Less than 1.00% above the threshold [0.0]
[18:20:56] <Superyetkin>	 cannot connect to labsdb via mysql workbench
[18:21:16] <Superyetkin>	 this is the error message: incompatible ssh peer no acceptable hex algorithm
[18:21:28] <Superyetkin>	 this has worked fine before
[18:22:13] <mutante>	 russblau: yuvipanda: < shinken-wm> RECOVERY - Puppet failure on tools-webgrid-lighttpd-1410 is OK
[18:22:21] <mutante>	 ^ that sounds like it might be better now
[18:22:54] <mutante>	 but not sure about tools-webgrid-lighttpd-1201  vs. tools-webgrid-lighttpd-1410
[18:23:12] <Superyetkin>	 hello?
[18:23:48] <mutante>	 Superyetkin: is it "kex" instead of hex?
[18:24:07] <Superyetkin>	 yes, sorry
[18:24:12] <mutante>	 that is indeed a recent change to ssh config for security
[18:24:31] <mutante>	 would there maybe be a newer version of the software you use?
[18:24:40] <mutante>	 it sounds a bit outdated 
[18:25:01] <Superyetkin>	 yes, it is an old version because I am on Windows XP SP3
[18:25:13] <Superyetkin>	 cannot upgrade my OS for now
[18:25:16] <mutante>	 Superyetkin: any chance you can upgrade? the old ciphers are kind of insecure
[18:25:25] <mutante>	 hmm
[18:26:01] <mutante>	 well, it's hard to find a balance between supporting all clients and a secure config
[18:26:18] <mutante>	 XP is really end of life 
[18:26:31] <Superyetkin>	 according to Microsoft, yes
[18:26:35] <Superyetkin>	 for me, a huge NO
[18:26:41] <Superyetkin>	 this is my work computer
[18:26:48] <mutante>	 just saying there are going to be more of these problems
[18:27:14] <Superyetkin>	 I am using MySQL Workbench 5.2.46
[18:27:15] <mutante>	 do you have any settings in the app that can influence the ciphers used ?
[18:27:35] <Superyetkin>	 and XP SP3 does not support newer versions, as far as I know
[18:28:05] <Superyetkin>	 I do not know the innerworking of the software
[18:29:16] <ankita-ks>	 Betacommand : I am trying to run an http server on tools labs. : http://wiki.languagetool.org/http-server
[18:29:54] <yuvipanda>	 Superyetkin: if  you are looking to just run sql queries - try quarry.wmflabs.org
[18:30:20] <Superyetkin>	 no, I have a local database on labs
[18:30:38] <mutante>	 Superyetkin: it's similar to this bug http://bugs.mysql.com/bug.php?id=51405  just newer.. looking
[18:31:04] <Superyetkin>	 I was using the labs for the needs of trwiki
[18:31:09] <Superyetkin>	 I am a tool developer
[18:31:16] <sitic>	 Superyetkin: I'm getting the same error with debian and mysql workbench 6.3.3.0: https://dpaste.de/vWv0
[18:31:19] <Superyetkin>	 http://tools.wmflabs.org/superyetkin/index.html
[18:31:47] <sitic>	 https://bugs.mysql.com/bug.php?id=74658 seems to be recent enough
[18:32:21] <Merlissimo>	 i have the same problem "ssh_dispatch_run_fatal: no matching MAC found" can sb. post me the list of supported MACs
[18:32:30] <mutante>	 sitic: thank you, that's it
[18:33:11] <Superyetkin>	 no solution for me?
[18:33:23] <mutante>	 Merlissimo: on jessie or trusty:
[18:33:26] <mutante>	 MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-
[18:33:28] <Superyetkin>	 this is frustrating...
[18:33:29] <mutante>	 512,hmac-sha2-256,umac-128@openssh.com
[18:33:39] <mutante>	 on precise: MACs hmac-sha2-512,hmac-sha2-256
[18:35:11] <mutante>	 here are the relevant changes:
[18:35:21] <mutante>	 https://gerrit.wikimedia.org/r/#/c/185329
[18:35:39] <mutante>	 https://gerrit.wikimedia.org/r/#/c/185321/
[18:36:20] <mutante>	 Superyetkin: not an easy one unless mysql workbench fixes it or we have to revert it for everything
[18:37:09] <Superyetkin>	 why on earth can you apply such disastrous changes without informing anyone?
[18:37:34] <Merlissimo>	 thx mutante
[18:37:34] <Superyetkin>	 I am watcing the wikimedia lists and cannot se any notification at all
[18:37:38] <Superyetkin>	 this is really bad
[18:38:45] <Superyetkin>	 WHO needed this change?
[18:38:53] <Superyetkin>	 WHO applied it?
[18:38:57] <Superyetkin>	 WHO discussed it?
[18:39:03] <mutante>	 Superyetkin: http://stribika.github.io/2015/01/04/secure-secure-shell.html
[18:39:04] <Superyetkin>	 WHERE is the community?
[18:39:17] <Superyetkin>	 really frsutrating...
[18:39:25] <Superyetkin>	 lost hours
[18:39:50] <mutante>	 it was in code review for a long time and reviewed by several peers
[18:39:50] <Superyetkin>	 this is not the right way to do it
[18:40:07] <mutante>	 Superyetkin: it's really hard to achieve both goals at the same time, security and supporting all clients that exist 
[18:40:33] <mutante>	 it's an improvement for most , unfortunately it affects a few
[18:40:42] <mutante>	 we are definitely discussing this and can open a bug
[18:41:16] <Superyetkin>	 bad, man
[18:41:37] <Superyetkin>	 this is not what Wikimedia is tring to achieve...
[18:41:48] <Superyetkin>	 I am an active sysop and tool developer on trwiki
[18:42:05] <Superyetkin>	 I am devoting a reasonable time for the project,
[18:42:46] <Superyetkin>	 and one day, I come back and see what? CANNOT connect to the database I had been using for months...
[18:42:50] <Superyetkin>	 really frustrating
[18:43:03] <mutante>	 i understand, but there must be some kind of balance, we cant just support any client until the end of time, it would make it very insecure
[18:43:27] <mutante>	 we are not just ignoring it
[18:43:34] <Superyetkin>	 I am not expecting for you to support ALL clients
[18:44:41] <Superyetkin>	 ok, thanks for the explanation
[18:44:51] <Superyetkin>	 I will try to live with this hard fact
[18:45:04] <Superyetkin>	 do not ever rely one anybody...
[18:45:10] <Superyetkin>	 THANKS
[18:45:25] <mutante>	 Superyetkin: maybe you could just give us a little time? we are talking about it
[18:45:36] <mutante>	 also going to open a bug 
[18:45:55] <Superyetkin>	 opening a bug will not help
[18:46:05] <Superyetkin>	 this needs to be reverted
[18:46:19] <Superyetkin>	 I am only one of those developers affected
[18:46:28] <Superyetkin>	 you cannot know the real impact of it
[18:46:30] <Superyetkin>	 never
[18:46:50] <mutante>	 not without a bug, right
[18:46:55] <bblack>	 it's a security enhancement that's meant to help protect you/us.  there was unpredicted fallout with ancient/broken clients that are apparently common
[18:47:07] <bblack>	 why don't you relax, have a drink, and let us discuss the fallout for a few minutes :P
[18:47:28] <Superyetkin>	 bblack, you are not right
[18:47:50] <Superyetkin>	 man, this is a serious issue
[18:48:02] <bblack>	 so is everything we do, all day, every day
[18:48:02] <Superyetkin>	 and I can see therei is no community consensus behind that
[18:48:17] <bblack>	 if we got community consensus on every patch, we'd never get anything done.
[18:48:45] <Superyetkin>	 you MUST seek community consensus for this type of things
[18:48:46] <bblack>	 relax, we'll sort things out.  filling a screen with ranting doesn't help.  we're Aware 
[18:49:09] <Superyetkin>	 relaxed than ever now
[18:49:19] <Superyetkin>	 but this is not good news...
[18:49:52] <Superyetkin>	 at least, there should have been a notification somewhere
[18:50:06] <Superyetkin>	 this was working last week
[18:50:09] <Superyetkin>	 oh my god...
[18:59:13] <yuvipanda>	 Superyetkin: can you check if putty works?
[19:02:17] <Superyetkin>	 yes, putty works fine
[19:03:20] <yuvipanda>	 alright, so it's basically just old versions of mysqlbench that are problematic
[19:03:53] <Superyetkin>	 that is a huge problem
[19:04:09] <Superyetkin>	 you cannot force people to upgrade!
[19:04:16] <yuvipanda>	 alright, so calm down
[19:04:28] <yuvipanda>	 freaking out isn't going to have you help people
[19:04:41] <Superyetkin>	 man, watch your words
[19:04:46] <yuvipanda>	 what?
[19:05:03] <Superyetkin>	 I just mentioned a serious issue here
[19:05:15] <Superyetkin>	 so, do not take it for granted
[19:05:24] <Superyetkin>	 that is important
[19:05:27] <Superyetkin>	 understand?
[19:06:01] <bblack>	 We're looking at a broader scope here.  We have very few users reporting about clients broken by this change, which upgrades security for many many other clients.  This was a broad change that runs through all of our production access, etc.
[19:06:02] <Superyetkin>	 I am just telling you a system working last week is BROKEN now
[19:06:05] <Superyetkin>	 understand that?
[19:06:32] <yuvipanda>	 Superyetkin: you need to be calmer and more respectful if you want to be helped. We are working on a way to restore access
[19:06:37] <bblack>	 the broken case is the vast minority.  the reason it's broken is primarily that the client in question is ancient, and there is a workaround using putty for a more up-to-date tunnel implementation.
[19:06:39] <yuvipanda>	 also, this probably wasn't working last week
[19:06:47] <yuvipanda>	 Superyetkin: are you sure this has been broken for only a week?
[19:06:57] <yuvipanda>	 bblack: mutante that change was merged a longer time ago, not last week
[19:07:14] <Superyetkin>	 panda, this WAS working last week
[19:07:19] <bblack>	 the mac/kex changes were just today
[19:07:31] <yuvipanda>	 aaah, I see 
[19:07:35] <yuvipanda>	 so this isn't the ciphersuite changes
[19:07:39] <bblack>	 right
[19:07:42] <yuvipanda>	 fair enough
[19:08:09] <Superyetkin>	 no easy workaround for this
[19:08:15] <Superyetkin>	 putty does not help me at all
[19:08:17] <yuvipanda>	 now we just need to find a windows machine to document the putty workaround
[19:08:29] <Superyetkin>	 I prefer Workbench for its GUI
[19:08:48] <Superyetkin>	 without that, no way to actually visualize and design smart queries
[19:09:23] <Superyetkin>	 putty works, so what?
[19:09:27] <bblack>	 well, I think someone's putting together an email on the putty workaround
[19:09:31] <Superyetkin>	 it is irrelevant
[19:09:32] <yuvipanda>	 no, you can use putty to setup an ssh tunnel
[19:09:36] <bblack>	 the idea is you can use putty as the tunnel access for the workbench GUI
[19:09:37] <yuvipanda>	 and then point mysql workbench to it
[19:09:51] <Superyetkin>	 without pagent?
[19:09:52] <bblack>	 http://realprogrammers.com/how_to/set_up_an_ssh_tunnel_with_putty.html
[19:09:56] <Superyetkin>	 be more specific
[19:10:08] <bblack>	 ^ that, plus some specific instructions on modifying your workbench config to use that tunnel endpoint, basically.
[19:10:17] <bblack>	 but none of us have XP to test/write the specific instructions out
[19:10:32] <Superyetkin>	 what needs to be done exactly?
[19:10:38] <yuvipanda>	 and a lot of us are in flights or getting on flights, and most of our windows documentation is historically from volunteers
[19:11:30] <Superyetkin>	 hmm
[19:11:39] <yuvipanda>	 anyway I'm filing a bug
[19:12:07] <Superyetkin>	 do you know how the putty can be configured to allow that?
[19:13:50] <wikibugs>	 6Labs: Document PuTTY based tunneling workaround for accessing labsdb via ancient MySQLBench versions - https://phabricator.wikimedia.org/T99942#1302301 (10yuvipanda) 3NEW
[19:13:56] <yuvipanda>	 ^ there we go
[19:14:06] <yuvipanda>	 Superyetkin: there's a link in that bug, same link bblack pasted a while ago
[19:14:44] <bblack>	 basically you want to set up the tunnel as documented there, then not use the SSH Tunnel stuff in MySQLBench, instead have it connect directly to localhost:3306, which is the local end of the PuTTY tunnel.
[19:15:01] <bblack>	 (not use the built-in tunnel stuff in mysqlbench, I mean)
[19:16:35] <Superyetkin>	 use pagent anyway?
[19:16:58] <wikibugs>	 6Labs: Document PuTTY based tunneling workaround for accessing labsdb via older MySQLBench versions - https://phabricator.wikimedia.org/T99942#1302319 (10yuvipanda)
[19:19:06] <Superyetkin>	 I am also using WinSCP
[19:19:20] <Superyetkin>	 any chance we can use the SSH config for that?
[19:19:28] <valhallasw>	 Coren: on the ubuntu versioning, the ~ sorts before everything else -- I think so one can do 1.4~rc1 etc
[19:20:00] <yuvipanda>	 valhallasw: are you on Windows? :)
[19:20:04] <valhallasw>	 yuvipanda: yes
[19:20:30] <yuvipanda>	 Superyetkin: is having ssh issues on windows, do you think you can help? (https://phabricator.wikimedia.org/T99942)
[19:20:57] <yuvipanda>	 valhallasw: ^
[19:21:23] <yuvipanda>	 gotta run, packing and then have to trek all the way to the airport
[19:21:43] <valhallasw>	 yuvipanda: I think your suggestions there are sane?
[19:21:45] <Superyetkin>	 what is your destination?
[19:21:59] <valhallasw>	 upgrade mysql workbench, if that doesn't work, hit mysql workbench with a hammer and tunnel manually with putty
[19:22:39] <bblack>	 I think you have to go with the putty tunnel, I think even the latest mysql binary releases for windows still use an outdated paramiko (which is the bundled ssh tunnel implementation)
[19:22:50] <bblack>	 at least, that's how the bug reports read for mysql users complaining.
[19:22:55] <valhallasw>	 if it's paramiko, it might be possible to upgrade it manually
[19:23:38] <bblack>	 this was the relevant upstream mysql bug: https://bugs.mysql.com/bug.php?id=74658
[19:23:42] <valhallasw>	 'create an oracle account'
[19:23:46] <valhallasw>	 *look of disapproval*
[19:24:01] <valhallasw>	 oh wait, there's a skip registration button
[19:27:28] <wikibugs>	 6Labs: Document PuTTY based tunneling workaround for accessing labsdb via older MySQLBench versions - https://phabricator.wikimedia.org/T99942#1302377 (10valhallasw) Apparently an issue with older Paramiko versions. Possible workaround:   - download a more recent paramiko, e.g. https://github.com/paramiko/parami...
[19:33:13] <mutante>	 just mailed labs-l with details re: sshd
[19:37:53] <valhallasw>	 mutante: err, shouldn't https://gerrit.wikimedia.org/r/#/c/199936/ come with a change to make sure bastions can still do agent forwarding?
[19:38:02] <wikibugs>	 10Tool-Labs: (Some) Precise grid jobs seem to be failing - https://phabricator.wikimedia.org/T99946#1302404 (10scfc) 3NEW
[19:39:25] <mutante>	 valhallasw: ideally no, everybody would just use ProxyCommand for everything
[19:40:06] <valhallasw>	 mutante: that's not realistic for multi-hop situations
[19:40:22] <valhallasw>	 but those could be solved by HBA instead
[19:40:28] <mutante>	 valhallasw: there is the keyholder thing
[19:40:42] <mutante>	 ehm.. wait
[19:41:14] <mutante>	 08:57 < ori> for scap, there is an ssh-agent on tin armed with a deploy key that is owned by root, and an ssh-agent-proxy instance that is g+rw for deployers and which  allows deployers to use the deployment key
[19:41:18] <mutante>	 08:57 < ori> it has worked very well
[19:41:31] <valhallasw>	 typical situation: someone is logged in to tools-login and needs to connect to tools-exec-something to read a log file. Blocking forwarding means they need to open a new putty, configure the right proxy command, log in, etc
[19:41:40] <valhallasw>	 instead of typing 'ssh tools-exec-1201'
[19:42:53] <bd808>	 forwarding to tools-login is scary
[19:43:12] <valhallasw>	 it's also the only thing that works
[19:43:22] <bd808>	 so many opportunities for a local priv exploit to harvest agents
[19:44:47] <mutante>	 valhallasw: so i just type "ssh instancename" for that and jump via the bastion with ProxyCommand. i believe the equivalent is plink.exe ?
[19:45:02] <mutante>	 http://fixunix.com/ssh/74073-putty-proxycommand.html
[19:45:06] <valhallasw>	 there is no equivalent on windows
[19:45:08] <valhallasw>	 yes, you can do it
[19:45:09] <valhallasw>	 it's a PITA
[19:45:24] <valhallasw>	 and you can't do a simple "do this for all hosts that end with .eqiad"
[19:46:14] <mutante>	 *nod*, point taken. at least would need good docs for windows users what to do
[19:46:16] <valhallasw>	 and if it requires plink, that categorizes as 'impossible' rather than 'a pita'
[19:46:56] <mutante>	 cygwin .. but i get it
[19:47:11] <valhallasw>	 cygwin also borders on impossible :P
[19:47:17] <valhallasw>	 it's not a realistic solution
[19:47:27] <wikibugs>	 10Tool-Labs: (Some) Precise grid jobs seem to be failing - https://phabricator.wikimedia.org/T99946#1302428 (10yuvipanda) Related to the gridengine upgrade @coren did on precise hosts earlier?
[19:47:30] <mutante>	 https://www.ibm.com/developerworks/community/blogs/Dougclectica/entry/ssh_key_authentication_with_putty_part_2_tunneling_with_plink12?lang=en
[19:48:16] <valhallasw>	 yes. and you have to do that again for *every host you want to connect to*
[19:49:33] <mutante>	 at least only once since you can save a profile
[19:50:32] <mutante>	 but yea, not planning to merge it soon
[20:10:03] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1410 is CRITICAL 30.00% of data above the critical threshold [0.0]
[20:40:02] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1410 is OK Less than 1.00% above the threshold [0.0]
[22:16:47] <Cyberpower678>	 I can't get into my account on Labs
[22:17:14] <Cyberpower678>	 My client says Key exchanged failed.
[22:18:38] <Cyberpower678>	 Hello?
[22:20:02] <Cyberpower678>	 Coren, ^
[22:21:11] <Cyberpower678>	 legoktm, ^
[22:21:18] <valhallasw>	 Cyberpower678: could be you're using an ancient SSH client
[22:21:32] <Cyberpower678>	 Umm no.  I got in this morning.
[22:22:13] <Cyberpower678>	 valhallasw, ^
[22:23:13] <valhallasw>	 Cyberpower678: dunno, could be related to https://gerrit.wikimedia.org/r/#/c/185321/
[22:23:42] <valhallasw>	 what client are you using?
[22:23:47] <Cyberpower678>	 Smart FTP
[22:23:52] <Cyberpower678>	 My solution to everything.
[22:25:29] <Cyberpower678>	 valhallasw, ^
[22:25:40] <valhallasw>	 Cyberpower678: putty is working for me, so I'm inclined to say it's smartftp. I don't have time to look at it now, though, as it's half past midnight
[22:25:56] <mutante>	 Cyberpower678: https://lists.wikimedia.org/pipermail/labs-l/2015-May/003730.html
[22:26:12] <mutante>	 sorry about that, yea, that is a recent change
[22:26:27] <valhallasw>	 Cyberpower678: Try to see if smartftp can generate verbose logs, and check those for more detailed info
[22:27:29] <Cyberpower678>	  /facedesk I'm really starting to hate labs.
[22:28:02] <Cyberpower678>	 I think smartFTP uses NIST
[22:30:54] <Cyberpower678>	 I'm about 3 clicks away from placing a retired banner on my userpage.
[22:40:38] <comets>	 needs help with the first 2 clicks? :D
[22:41:51] <Cyberpower678>	 Labs is supposed to make a volunteer's life easier when providing tools and bots, not have them jumpt through hoops and loops to do what they could've done 10 times as fast on toolserver.
[22:44:33] <Cyberpower678>	 valhallasw, SmartFTP does the following key exchange methods
[22:44:35] <Cyberpower678>	 Key Exchange Methods
[22:44:35] <Cyberpower678>	 �diffie-hellman-group-exchange-sha256
[22:44:35] <Cyberpower678>	 �diffie-hellman-group-exchange-sha1
[22:44:35] <Cyberpower678>	 �diffie-hellman-group14-sha1
[22:44:35] <Cyberpower678>	 �diffie-hellman-group1-sha1
[22:44:36] <Cyberpower678>	 �ecdh-sha2-nistp256
[22:44:38] <Cyberpower678>	 �ecdh-sha2-nistp384
[22:44:40] <Cyberpower678>	 �ecdh-sha2-nistp521
[22:44:42] <Cyberpower678>	 �curve25519-sha256@libssh.org