[00:00:24] <andrewbogott>	 Yeah, I can't make anything but 'small' start up.
[00:00:55] <andrewbogott>	 PiRCarre: is UTRS your project?
[00:00:59] <PiRCarre>	 Nope.
[00:01:07] <bd808>	 I guess that means I'm done for the night and can start drinking beer and eating tacos :)
[00:01:24] <PiRCarre>	 andrewbogott: I just saw it on [[w:WP:VPT]], a page I watch
[00:01:36] <andrewbogott>	 PiRCarre: I switched it off yesterday as part of the migration.  No one has claimed it, I presumed it to be abandoned.
[00:01:44] <andrewbogott>	 I guess I'll respond on that page...
[00:01:44] <PiRCarre>	 Oh.
[00:01:50] <PiRCarre>	 Thanks :-)
[00:06:33] <PiRCarre>	 andrewbogott: I think enwiki uses it a lot - https://en.wikipedia.org/wiki/Wikipedia:UTRS
[00:06:45] * andrewbogott  scowls

[00:08:03] <andrewbogott>	 PiRCarre: Could it be that the project is no longer maintained, but still has users?  There has been quite a lot series of announcements on various lists requesting people to take ownership of projects.  No one has mentioned this one until just this second.
[00:08:22] <PiRCarre>	 Hm, did you try poking the admins?
[00:08:43] <PiRCarre>	 I suppose many people do not follow labs-l (but they should)
[00:09:06] <andrewbogott>	 PiRCarre: Yes, I've send emails to labs-l, wikitech-l, and every user that has ever created a labs instance.
[00:09:15] <andrewbogott>	 *sent
[00:09:21] <PiRCarre>	 I have no idea, then.
[00:10:20] <PiRCarre>	 Are all instances with no owners "switched off"?
[00:11:01] <bawolff>	 So it seems my crontab disappeared (?)
[00:11:50] <andrewbogott>	 PiRCarre: http://lists.wikimedia.org/pipermail/labs-l/2014-February/002152.html
[00:16:49] <andrewbogott>	 PiRCarre: I hope that email makes sense, and that my post on village pump isn't too rude...
[00:41:34] <bawolff>	 I've noticed it seems like the xmllint program is not available on job runners
[01:36:53] <Krinkle>	 Coren: andrewbogott: I've rebooted my new eqiad (to migrate from pmtpa) instance 3 times now (several hours apart), still unable to login.
[01:37:20] <andrewbogott>	 what instance and project?
[01:38:10] <andrewbogott>	 Krinkle: ?
[01:39:49] <Krinkle>	 cvn-app3
[01:40:01] <Krinkle>	 but this has been reported for all new eqiad instances afaik for project after the switch
[01:40:13] <Krinkle>	 https://bugzilla.wikimedia.org/show_bug.cgi?id=62771
[01:40:23] <Krinkle>	 channel 0: open failed: connect failed: Connection timed out
[01:40:24] <Krinkle>	 ssh_exchange_identification: Connection closed by remote host
[01:43:12] <andrewbogott>	 Krinkle: which bastion are you using?
[01:43:21] <Krinkle>	 the one documented in ProxyCommand on wiki
[01:43:44] <Krinkle>	 https://github.com/Krinkle/dotfiles/blob/master/hosts/KrinkleMac/templates/sshconfig
[01:43:44] <andrewbogott>	 Are you a project admin for this project?  (If you are I can tell you want to fix, if not I can just fix it :)  )
[01:43:55] <andrewbogott>	 *what
[01:44:10] <Krinkle>	 I'm an admin for this project
[01:44:50] <Krinkle>	 Has the source problem been fixed (e.g. does this apply only to instances created last week, or still for new instances? e.g. is new instance creation broken for eqiad, or was it and haven't fixed up those yet?
[01:45:01] <andrewbogott>	 Hm, wait, actually, maybe this is something else...
[01:45:15] <Krinkle>	 If it's the latter, I'm happy to be told how to fix it, otherwise, I'd rather wait for the overal fix as I plan to create more instances during the migration.
[01:45:27] <andrewbogott>	 nope, that was it.  Try to ssh now?
[01:45:47] <Krinkle>	 works
[01:45:55] <Krinkle>	 If you are having access problems, please see:https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances
[01:45:55] <Krinkle>	 Creating directory '/home/krinkle'.
[01:45:56] <Krinkle>	 Last login: Tue Mar 18 05:49:46 2014 from 10.4.1.85
[01:45:57] <Krinkle>	 krinkle@cvn-app3:~$ 
[01:45:59] <andrewbogott>	 OK, so if you check out the security groups...
[01:46:07] <andrewbogott>	 I just added a rule for 10.0.0.0/8 on port 22.
[01:46:11] <Krinkle>	 Looks like it has the motd of the old pmtpa template, it was changed to wikitech a while back.
[01:46:11] <andrewbogott>	 That's what I changed.
[01:46:34] <Coren>	 andrewbogott: We're going to run into this over and over.  How complicated do you think changing them would be wholesale?
[01:46:48] <andrewbogott>	 Coren: I emailed you about that over the weekend :)
[01:46:54] <andrewbogott>	 Or, tried to, maybe it got spam-trapped :(
[01:47:05] <Coren>	 You asked my opinion on whether we should; I need moar data to opine.  :-)
[01:47:35] <Krinkle>	 I don't remember from back when I created the pmtpa instances, but iirc ssh worked by default there (via bastion)
[01:47:37] <Coren>	 I guess my principal worry is on what the risk of error is.
[01:47:57] <Krinkle>	 is there a concern for this in eqiad? like higher risk or something?
[01:48:02] <Coren>	 Krinkle: It did, because the default firewall rules allow it -- same rules being all wrong in eqiad.  :-)
[01:48:28] <Krinkle>	 I did apply the default security group.
[01:48:39] <Krinkle>	 I guess that one is a different 'default' in eqiad at the moment 
[01:48:40] <Coren>	 !
[01:48:40] <wm-bot>	 hello :)
[01:48:54] <andrewbogott>	 Coren:  What do you mean by 'risk of error'?
[01:48:58] <Coren>	 Wait, those are /new/ instances?
[01:49:14] <Coren>	 andrewbogott: Opening up something we shouldn't -- or did you intend to do just port 22?
[01:49:43] <Coren>	 Krinkle: The issue is that the default still allows access.  From pmtpa.  :-)
[01:50:13] <Krinkle>	 Wiat, so that's why it broke when I updated my ProxyCommand config
[01:50:22] <andrewbogott>	 Coren:  I don't really know enough sql to do anything, hence my email.
[01:50:26] <Krinkle>	 it didn't add the new rule for eqiad
[01:50:35] <Krinkle>	 there was a rule for it, but it's changed from pmtpa bastion to eqiad
[01:51:02] <andrewbogott>	 But, yeah, there are multiple options -- we could just do port 22 or we could just find every place that has a pmtpa-specific rule and change it to 10.0.0.0/8
[01:51:32] <Krinkle>	 what does that place currently have if not 0./8
[01:51:35] <Coren>	 I'm a little concerned the latter might do something that ends up being troublesome.
[01:52:12] <Coren>	 But just port 22 should be safe enough.
[01:52:20] <Coren>	 Especially since we don't do password auth.
[01:52:40] <andrewbogott>	 The default group is 22 and 5666 -- shall we do both?
[01:53:01] <Coren>	 5666 is monitoring, right?  I say yes.  That's also harmless.
[01:53:09] <andrewbogott>	 Krinkle: the old rule was limited to just pmtpa labs.  The new rule I've been using opens a pretty big hole, in order to encompass both eqiad and pmtpa labs.
[01:53:23] <andrewbogott>	 Coren: is your sql-fu strong enough to just do that?  Or feed me a query to do it?
[01:53:39] <Coren>	 andrewbogott: If you point me at the right DB, I can do.
[01:54:35] <andrewbogott>	 Coren:  virt1000, db name 'nova', table 'security_group_rules'
[01:55:09] <andrewbogott>	 I /think/ that the auth is already set in the root environment on virt1000.  If not, I'll dig up a password.
[01:55:19] <Coren>	 It is.
[01:55:47] <andrewbogott>	 The rules are local to the host.  So changes on virt1000 are limited to eqiad instances.
[01:55:57] <andrewbogott>	 which is what we want
[01:57:06] <Coren>	 andrewbogott: *nod*  All done.
[01:57:12] <Coren>	 For ports 22 and 5666
[01:57:25] * andrewbogott  reloads

[01:57:27] <andrewbogott>	 looks good!
[01:57:32] <andrewbogott>	 What do you think about doing 80 as well?
[01:57:45] <andrewbogott>	 If someone has 80 open for only pmtpa labs, that means they're using the dynamic proxy
[01:57:56] <Coren>	 Ah, there were a few 10.4.0.0/14; change those too?
[01:58:13] <andrewbogott>	 huh, I don't know why that would be...
[01:58:37] <andrewbogott>	 but, sure.  Maybe that was an older default
[01:59:14] <Coren>	 Yeah, doing 80 too; limiting it to pmtpa is pointless now.
[01:59:25] <andrewbogott>	 Btw, can I see the query line you're using?  For the sake of learnin'
[01:59:56] <Krinkle>	 andrewbogott: btw, that fix for my instance, was that changed on your end in direct relation to that instance or more generic?
[02:00:14] <Krinkle>	 I know y'all are working on it as I speak, but if I were to create another instance now, would that one be fine?
[02:00:19] <andrewbogott>	 Krinkle: security groups are project-wide.  You can see them if you click on 'Manage Security Groups'
[02:00:23] <Krinkle>	 (I'm not doing that, will start on Monday)
[02:00:37] <Coren>	 andrewbogott: {{done}}
[02:00:37] <Krinkle>	 I thought those were baked into the instance upon creation thouguh
[02:00:42] <andrewbogott>	 And, anyway, Coren just fixed it EVERYWHERE
[02:00:54] <andrewbogott>	 Krinkle: It's weird.  The assignment of groups to instances is fixed.
[02:01:00] <andrewbogott>	 But the assigning of rules to groups, not fixed.
[02:01:04] <Krinkle>	 or maybe the wiki interface just doesn't expose the ability to change them after the fact
[02:01:05] <andrewbogott>	 Does that make sense, sort of?
[02:01:19] <andrewbogott>	 So, your instance is still in the default group.  But the default group has changed.
[02:01:27] <andrewbogott>	 Coren:  thank you!
[02:01:32] <Krinkle>	 I've found myself deleting an instance and re-creating it because I forgot to give it the right security
[02:01:40] <Krinkle>	 andrewbogott: OK :)
[02:03:16] <Krinkle>	 k, preparing for my flight now, off for a day or two
[02:03:28] <andrewbogott>	 bd808|BUFFER: It looks to me like only the default ('small') image type is working right now.  I don't know why but probably won't sort it tonight.
[02:03:30] <andrewbogott>	 Sorry for the hold-up.
[02:20:50] <Magog_the_Ogre>	 coren, sorry I was away
[02:21:05] <Magog_the_Ogre>	 the log file is upload.log for profile tools-magog
[02:21:10] <Magog_the_Ogre>	 the crontab is in magog
[02:21:14] <Magog_the_Ogre>	 (not tools-magog)
[02:23:29] <Magog_the_Ogre>	 does anyone know if Coren will see this message, and if not, how to contact him off IRC?
[02:23:41] <PiRCarre>	 Magog_the_Ogre: someone could call him if it's urgent
[02:23:52] <Magog_the_Ogre>	 it's not THAT urgent
[02:24:01] <Magog_the_Ogre>	 but I'd like it resolved in the next day or two if possible
[02:24:05] <PiRCarre>	 why not just MemoServ, email, talk page, etc. ?
[02:24:52] <Magog_the_Ogre>	 a) I'm a newb, don't know how to  MemoServ b) unless IRC just provides emails somehow, can't do it, c) talk page on which project is reliable?
[02:25:47] <PiRCarre>	 enwiki probably. or wikitech. idk
[02:26:23] <PiRCarre>	 Magog_the_Ogre: memoserv is /msg MemoServ send Username write your message here
[02:26:49] <PiRCarre>	 email would just be using the email address he uses (or EmailUser)
[04:05:02] <TParis>	 andrewbogott_afk: Ping
[06:54:23] <grrrit-wm>	 (03PS2) 10Adamw: add Extension:FundraisingChart; notify wikimedia-dev about FR commits [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/119643 
[06:55:03] <grrrit-wm>	 (03PS3) 10Adamw: add Extension:FundraisingChart; notify wikimedia-dev about FR commits [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/119643 
[07:00:30] <grrrit-wm>	 (03PS2) 10Adamw: Make the "blacklist" variable an array cos that's slightly nicer than a space-separated string [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/119924 
[13:49:52] <grrrit-wm>	 (03PS1) 10Gerrit Patch Uploader: Various changes to wmt-ko [labs/tools/WMT] - 10https://gerrit.wikimedia.org/r/120206 
[13:49:55] <grrrit-wm>	 (03CR) 10Gerrit Patch Uploader: "This commit was uploaded using the Gerrit Patch Uploader [1]." [labs/tools/WMT] - 10https://gerrit.wikimedia.org/r/120206 (owner: 10Gerrit Patch Uploader)
[13:52:43] <grrrit-wm>	 (03PS1) 10Gerrit Patch Uploader: Add status checker to bot-operator [labs/tools/WMT] - 10https://gerrit.wikimedia.org/r/120207 
[13:52:46] <grrrit-wm>	 (03CR) 10Gerrit Patch Uploader: "This commit was uploaded using the Gerrit Patch Uploader [1]." [labs/tools/WMT] - 10https://gerrit.wikimedia.org/r/120207 (owner: 10Gerrit Patch Uploader)
[13:53:09] <Coren>	 Magog_the_Ogre: I've copied both in ~tool.magog/ (the crontab in magog.crontab).  Please remember to not restore that crontab to your user account.
[14:04:54] <grrrit-wm>	 (03CR) 10PiRSquared17: [C: 032 V: 032] Various changes to wmt-ko [labs/tools/WMT] - 10https://gerrit.wikimedia.org/r/120206 (owner: 10Gerrit Patch Uploader)
[14:05:16] <grrrit-wm>	 (03CR) 10PiRSquared17: [C: 032 V: 032] Add status checker to bot-operator [labs/tools/WMT] - 10https://gerrit.wikimedia.org/r/120207 (owner: 10Gerrit Patch Uploader)
[14:32:58] <valhallasw>	 Coren: I'm trying the jmail pipe, but it doesn't seem to function.
[14:33:41] <Coren>	 Yeah, I'm still struggling with exim.  That said, it /should/ work if you send the mail from within labs.
[14:33:49] <valhallasw>	 Ahhh. I see.
[14:34:01] <valhallasw>	 I'll wait some longer then :-)
[14:34:04] <Coren>	 At least, it worked for me.  :-)  So you can test while I'm beating exim up.  :-)
[14:40:29] <jimmyxu>	 Hi, it seems that tools.wmflabs.org's web service can't be accessed on -login. Is this the wrong way to do things?..
[14:44:13] <Betacommand>	 jimmyxu: what are you trying to do?
[14:44:33] <jimmyxu>	 Betacommand: to grab /wikiviewstats/ in a script
[14:45:05] <Betacommand>	 jimmyxu: your script shouldnt be executing on -login
[14:46:17] <jimmyxu>	 Betacommand: well that's still work in progress. Besides, -dev fails also
[14:46:26] <Coren>	 jimmyxu: Also, however, from /inside/ tools you should use the hostname 'tools-webproxy'
[14:46:53] <Betacommand>	 jimmyxu: use jsub and have to run on one of the exc nodes
[14:47:04] <jimmyxu>	 Coren: thanks/
[14:47:13] <valhallasw>	 Coren: except tools-webproxy is an implementation detail, so one should not use that either.
[14:47:45] <valhallasw>	 there really is no reason why tools.wmflabs.org/* cannot just work
[14:47:47] <jimmyxu>	 Coren: afaics this is not documented on wikitech?
[14:47:59] <valhallasw>	 be it via a hosts file or via iptables rules
[14:48:01] <Betacommand>	 Coren: those links should work
[14:48:13] <Betacommand>	 I had a ticket about that
[14:48:30] <Betacommand>	 otherwise some scripts become a nightmare to maintain
[14:48:57] <Betacommand>	 if is on labs do_foo() else do_bar()
[14:49:09] <Betacommand>	 makes for some really ugly code
[14:50:14] <Coren>	 Well, it can't be the same IP though I suppose I could do some uglies with /etc/hosts
[15:21:17] <Coren>	 FYI, I'm rejiggering the way cron is handled along the lines of a suggestion from Kernair that should greatly improve reliability.  Yeay.
[15:31:35] <Betacommand>	 Coren: ??
[15:32:04] <Coren>	 A seperate host to run cron jobs, with validation of the jobs proper.
[15:33:25] <Betacommand>	 validation of jobs?
[15:33:55] <valhallasw>	 probably blocking everything that's not jsub/qsub
[15:34:24] <Betacommand>	 valhallasw: that will really piss me off until cron is fixed
[15:34:52] <Betacommand>	 Ive got 1 script that I run that doesnt use jsub because I need the output emailed to me
[15:34:58] <valhallasw>	 have you noticed the part that said 'a seperate host'?
[15:35:28] <valhallasw>	 also, you can just ask SGE to mail you stdout
[15:35:55] <Betacommand>	 valhallasw: and SGE will probably say fuck you, hell no
[15:36:10] <Betacommand>	 Ive had too many issues
[15:36:18] <valhallasw>	 Yes, and you're being overly agressive again. Anything else you'd like to point out?
[15:36:19] <Betacommand>	 just ask coren 
[15:37:03] <Betacommand>	 valhallasw: I end up getting most of the bugs so that the rest of you dont
[15:37:35] <Betacommand>	 It gets old fast
[15:38:31] <valhallasw>	 ok, so qsub does not support mailing stdout, unfortunately, but you can still just let the script pipe to 'mail betacommand'
[15:39:26] <valhallasw>	 sounds like a useful option for jsub, though.
[15:40:01] <Betacommand>	 valhallasw: thats one of several requests that Ive made and have been sitting on a shelf for 6 months
[15:40:24] <valhallasw>	 Betacommand: probably because the | mail betacommand   option is fairly trivial to implement yourself
[15:40:48] <Betacommand>	 valhallasw: odds are it wont work most of the time
[15:41:11] <Betacommand>	 or some other unexpected issue will crop up
[15:41:12] <valhallasw>	 *shrug* then don't use the new server
[17:12:25] <grrrit-wm>	 (03CR) 10Siebrand: "Why merge a .orig file? That's what we have version control for." [labs/tools/WMT] - 10https://gerrit.wikimedia.org/r/120206 (owner: 10Gerrit Patch Uploader)
[17:21:17] <grrrit-wm>	 (03PS1) 10John F. Lewis: Removed useless .orig file [labs/tools/WMT] - 10https://gerrit.wikimedia.org/r/120222 
[17:21:40] <grrrit-wm>	 (03CR) 10John F. Lewis: [C: 032 V: 032] "Self-merge this" [labs/tools/WMT] - 10https://gerrit.wikimedia.org/r/120222 (owner: 10John F. Lewis)
[18:53:16] <PiRCarre>	 pirsquared@tools-login:~$ sql zhwikivoyage
[18:53:17] <PiRCarre>	 This is unknown db to me, if you don't like that, blame petan on freenode
[18:53:19] <PiRCarre>	 Make sure to ask for a db in format of <wiki>_p
[18:53:28] <PiRCarre>	 zhwikivoyage_p, then ?
[18:53:50] <petan>	 PiRCarre: I /think/ that someone changed that script
[18:53:53] <petan>	 let me checks
[18:56:01] <petan>	 PiRCarre: no there is indeed no such db, but that command it wrong, it should be blame Coren now
[18:56:31] <petan>	 nowadays 98% of stuff can be fixed by Core n only because everything else is restricted :/
[19:03:37] <scfc_de>	 PiRCarre: It looks like the alias zhwikivoyage.labsdb on which sql depends isn't there yet.  Looking at meta_p.wiki, zhwikivoyage seems to sit in s3.labsdb, so "mysql -h s3.labsdb zhwiktionary_p" gets you to its database.
[19:04:13] <scfc_de>	 PiRCarre: I'll add the alias in a bit.
[19:04:17] <PiRCarre>	 scfc_de: ok, thanks
[19:36:36] <scfc_de>	 PiRCarre: The alias should now work from all user-accessible hosts.
[19:36:52] <PiRCarre>	 great, thank you
[19:56:01] <YMS>	 Hi
[19:57:24] <PiRCarre>	 Hi.
[19:57:53] <YMS>	 I'd like to restore my bulk-copied tool (actually just a dumb library file), but I'm too stupid to connect to tools-login
[19:58:02] <YMS>	 Who can help me?
[19:58:45] <scfc_de>	 YMS: What host are you connecting to and which error message do you get?
[19:59:54] <YMS>	 tools-login.wmflabs.org
[20:00:10] <YMS>	 "Permission denied (publickey,hostbased)"
[20:01:48] <scfc_de>	 YMS: Your shell username in Labs is "yms"?  Is that your username on your own machine as well?  Otherwise, you need to "ssh yms@tools-login.wmflabs.org".
[20:02:22] <YMS>	 My shell username is yms, yes
[20:02:55] <YMS>	 ssh yms@tools-login.wmflabs.org is what I did
[20:04:37] <Krenair>	 YMS, is your key on labsconsole?
[20:04:39] <Krenair>	 er, wikitech*
[20:04:41] <scfc_de>	 Does ~/.ssh/id_rsa.pub on your own machine start with "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIBeZNR2ou73DuEaYKzIAe0vKHMhHcQBsgprP8iBIUbAy0w3xGhN5DIK/q5Md0W7MIah0AHoPtT8K1MnR"?
[20:04:45] <Magog_the_Ogre>	 Coren?
[20:05:59] <Coren>	 Magog_the_Ogre: hmm?
[20:06:53] <YMS>	 D'oh, that just might be the problem
[20:06:59] <Magog_the_Ogre>	 did you get my memoserv and/or the messages I left last night, Coren 
[20:07:00] <Magog_the_Ogre>	 ?
[20:07:18] <YMS>	 I don't have a public key file stored in ~/.ssh
[20:08:03] <Coren>	 [09:53:09] <Coren> Magog_the_Ogre: I've copied both in ~tool.magog/ (the crontab in magog.crontab).  Please remember to not restore that crontab to your user account.
[20:10:10] <scfc_de>	 YMS: So you don't have any ssh key at the moment?  Hmmm.
[20:10:55] <YMS>	 Just my private key
[20:10:55] <Magog_the_Ogre>	 OK
[20:10:55] <Magog_the_Ogre>	 could you restore upload.log as well?
[20:11:08] <Coren>	 That's what "both" meant\
[20:11:19] <YMS>	 Next stupid question: Looks like I can't just copy my public key from the wikitech preferences, do I?
[20:11:39] <Magog_the_Ogre>	 thanks, coren
[20:15:04] <scfc_de>	 YMS: *ughhh*  I'm not sure if ~/.ssh/id_rsa.pub is actually needed by ssh as it's really needed on the server.  But my bigger assumption is that your private and public key no longer match.  So if you only used this key with Labs, I would generate a new one and upload the public part to wikitech.
[20:16:25] <Magog_the_Ogre>	 coren, I'm still getting:
[20:16:25] <Magog_the_Ogre>	  can't open or create /var/run/crond.pid: Permission denied
[20:16:27] <YMS>	 I do use them only here ... however, why should they no longer match? Both were untouched.
[20:17:15] <Coren>	 Magog_the_Ogre: ... that's perfectly normal.  Why would you even try to write in /var/run?
[20:17:38] <Magog_the_Ogre>	 oops I was typing cron instead of crontab
[20:17:41] <Magog_the_Ogre>	 sorry
[20:17:44] <Magog_the_Ogre>	 thanks again
[20:17:50] <Coren>	 Heh.  No worries.  :-)
[20:18:33] <scfc_de>	 YMS: General distrust in all things brittle :-).
[20:18:59] <valhallasw>	 YMS: you can check it with   ssh-keygen -y
[20:19:37] <valhallasw>	 that should give you the same data as the contents in id_rsa.pub
[20:35:44] <YMS>	 Thanks guys!
[20:35:57] <YMS>	 I generated a new pair of public and private key and I'm in
[20:45:07] <YMS>	 Wikidata Label Collector up and running again :)
[20:45:56] <YMS>	 Maybe I'll move the whole tool to Labs some time, so I get more used to how things work here ;)
[21:02:57] <YMS>	 Thanks again & bye