[00:18:34] <Krinkle>	 Funny potential bug I could file "Labs is slower than Toolserver". Running a speedtest such as https://github.com/sivel/speedtest-cli from login.toolserver.org and tools-login.eqiad.wmflabs respectively shows a pretty dramatic difference
[00:18:57] <Krinkle>	 tried about a dozen different times to account for different servers and fluctuations
[00:19:38] <Krinkle>	 from eqiad: Download 200-300 Mbit/s, Upload 25-30 Mbit/s
[00:20:00] <Coren>	 Krinkle: You mean except for the fact that it's entirely specious by definition?  They are, literally, on the same network.  :-)
[00:20:03] <scfc_de>	 What's your target?
[00:20:07] <Krinkle>	 from esams (toolserver): Download 800-900 Mbit/s, Upload: 350-400 Mbit/s
[00:20:31] <Coren>	 Oh, wait, you're doing test from one point inside to one point /outside/?
[00:20:42] * Coren  chuckles.

[00:20:56] <Coren>	 Then that's even worse; it's geographically dependent.  :-)
[00:21:02] <Krinkle>	 for eqiad it picked Internode (San Jose, CA), SoftLayer Technologies, Inc. (San Jose, CA) and Monkey Brains (San Francisco, CA) 
[00:21:10] <Krinkle>	 [3km]
[00:21:28] <Krinkle>	 from esams it picked Solcon Internetdiensten N.V. (Apeldoorn) [34.77 km] and  Solcon (Dronten) [4.33 km]
[00:21:30] <Betacommand>	 Krinkle: thats about 100 miles different i think
[00:21:38] <Betacommand>	 *1000
[00:21:43] <Betacommand>	 for eqiad
[00:21:57] <Krinkle>	 No, It's testing towards the closest server offering speedtest
[00:22:06] <Krinkle>	 it's not connecting to the same point from esams and eqiad.
[00:22:07] <Betacommand>	 eqiad is in VA
[00:22:16] <Betacommand>	 Not CA
[00:22:18] <Coren>	 Krinkle: Then it sucks; it picked west coast servers for a DC that's on the east coast!
[00:22:23] <Krinkle>	 I know it's a pointless test because in all likelihood the target is the bottleneck
[00:22:54] <Krinkle>	 or at least in grey area where limits are not relevant
[00:22:59] <Betacommand>	 Krinkle: like I said the "closest" server was 1000 miles
[00:23:34] <scfc_de>	 I'm not sure if we should optimize for bulk downloads from random servers :-).
[00:23:38] <Betacommand>	 Coren: what city is the DC in?
[00:23:44] <Coren>	 Betacommand: Ashburn, VA
[00:24:18] <Krinkle>	 Strangely though, it said those are 4-70km away from eqiad
[00:24:27] <Krinkle>	 from tools/eqiad: Hosted by SoftLayer Technologies, Inc. (San Jose, CA) [70.93 km]: 12.716 ms
[00:24:30] <Betacommand>	 Krinkle: more like 2,392.93 miles
[00:24:35] <scfc_de>	 Betacommand: Look at the Google Maps for Washington Dulles Airport and follow the approach path for runway 18C (IIRC) a mile out :-).
[00:24:38] <Krinkle>	 and Monkey Brains (San Francisco, CA) [3km]
[00:24:47] <Krinkle>	 Something must be throwing it off
[00:25:00] <Betacommand>	 Krinkle: depends on how  its geolocating
[00:25:13] <Coren>	 Monkey Brains is, indeed, about a few blocks away from the WMF office.  Which is about 4000 km away from the DC.  :-)
[00:25:30] <Betacommand>	 Coren: 2.3K
[00:25:34] <Betacommand>	 wiat
[00:25:42] <Betacommand>	 *wait wrong unit
[00:25:48] <Coren>	 Yeah, I was about to say.  :-)
[00:26:12] <Betacommand>	 Coren: so sue me, its been a long week
[00:26:18] <Coren>	 (In fact, IIRC, Monkey brains is the ISP for the office)  :-)
[00:26:40] * Krinkle  runs the speedtest-cli with a forced ---server to see if it makes a diff

[00:26:40] <Krinkle>	 http://www.speedtest.net/speedtest-servers.php
[00:26:41] <Betacommand>	 Im in the middle of working out a logistics nightmare
[00:26:46] <Krinkle>	 Lets pick one in ashburn
[00:27:15] <Krinkle>	 lol
[00:27:17] <Krinkle>	 Hosted by RPM WEB SERVICES (Ashburn, VA) [3874.60 km]: 16.292 ms
[00:27:17] <Krinkle>	 Testing download speed........................................
[00:27:18] <Krinkle>	 Download: 844.61 Mbit/s
[00:27:19] <Krinkle>	 Testing upload speed..................................................
[00:27:21] <Krinkle>	 Upload: 314.69 Mbit/s
[00:27:34] <Krinkle>	 4000 km away, ok, it's clearly just the distance comp that's broken
[00:29:09] <Matthew_>	 Hey, so... anyone intimately familiar with privacy policy... and I allowed to collect IPs in replacement of usernames?  Like, if I tell users that they can provide a username but if they don't then I will use the IP?
[00:29:49] <Coren>	 Matthew_: Provided to give the required disclaimer before you do so, yes.
[00:30:02] <Coren>	 Matthew_: (And also that you aren't on tools)
[00:30:08] <Matthew_>	 Coren: Is there a specific wording?  
[00:30:12] <Matthew_>	 NVM, I'm on tools...
[00:31:09] <Coren>	 Ah.  No, on tools you have to not take IPs at all as we allow hotlinking and transclusion from the projects; the tools must respect the intersection of both policies.
[00:31:15] <Betacommand>	 Matthew_: what are you trying to do?
[00:32:00] <Matthew_>	 Coren: Ah... That kinda makes sense..
[00:32:20] <Matthew_>	 Betacommand: Build a system that allows users to request articles for [[WP:RA]].  
[00:32:33] <Matthew_>	 http://tools.wmflabs.org/articlerequest-dev <-- old working copy.
[00:32:49] <Matthew_>	 Except I just got migrated, nvm >.>
[00:32:53] <Coren>	 Matthew_: That sounds like something suited to OAuth
[00:33:03] <Matthew_>	 Coren: OAuth?
[00:34:15] <Coren>	 Trying to find a link.  Basically, a way to get a token to edit on an editor's behalf without having to hold their credentials.
[00:34:25] <Coren>	 That's what the croptool uses, for instance.
[00:34:35] <Matthew_>	 Coren: Huh, that's actually cool.  I was just gonna have a bot...
[00:35:10] <Krinkle>	 <client ip="208.80.155.130" lat="37.7898" lon="-122.3942" isp="Wikimedia Foundation" isprating="5.0" rating="0" ispdlavg="39308" ispulavg="5431" loggedin="0" /> 
[00:35:14] <Krinkle>	 Yeah, that explains.
[00:35:26] <Krinkle>	 It's mapping client IP to ISP, and then ISP to address and address to lat/long
[00:35:52] <Krinkle>	 Like everyone lives near their ISPs office yeah? What's this 1985?
[00:36:13] <Krinkle>	 we're an isp it says
[00:36:15] <Coren>	 Matthew_: https://www.mediawiki.org/wiki/Extension:OAuth
[00:37:07] <Coren>	 Krinkle: Heh.  I'm guessing they get confused because we're not a leaf node but actually peering with ISPs.
[00:37:16] <scfc_de>	 I think anomie has developed an example app for OAuth, and Magnus has gone through the school of hard knocks for a few of his tools.
[00:37:35] <Matthew_>	 Coren: That's actually really cool o.O Can it handle IPs?
[00:37:47] <Coren>	 Matthew_: There's a flask example there: https://wikitech.wikimedia.org/wiki/Setting_up_Flask_cgi_app_as_a_tool/OAuth
[00:38:36] <Coren>	 Matthew_: I'm pretty sure it can't; IPs don't have credentials to edit on behalf of.  But in a case like this, your tool could edit as itself rather than present a token.
[00:39:01] <Coren>	 Matthew_: I don't expect that'd be an issue if you rate-limit anonymous requests.
[00:40:01] <Matthew_>	 Coren: OK, makes sense.  The reason I ask is I wanted to have every request tagged with some sort of username...
[00:40:11] <Matthew_>	 Hmmmm... that's really cool actually o.O
[00:41:19] <Krinkle>	 Coren: About tool labs mail, is that supposed to be working?
[00:41:24] <Krinkle>	 I tried 4 different addresses, but all bounced
[00:41:56] <Krinkle>	   tools.delinker@tools.wmflabs.org,    delinker@tools.wmflabs.org,   delinker.*@tools.wmflabs.org, delinker.foo@tools.wmflabs.org
[00:42:14] <Krinkle>	 [(10) mail.tools.wmflabs.org. [208.80.155.162]:25: Connection timed out] 
[00:42:14] <Krinkle>	 [(10) mail.tools.wmflabs.org. [208.80.155.162]:25: Connection timed out] 
[00:42:23] <Coren>	 Krinkle: Actually, I should be okay to turn that on now.
[00:42:46] <Krinkle>	 https://support.google.com/mail/answer/7720 (linked in the delivery notification)
[00:42:48] <Coren>	 Krinkle: The correct emails would be 'tools.delinker' or 'delinker.anything'.
[00:44:43] * Coren  turns it on.

[00:44:45] <Matthew_>	 Coren: I'll look into this more, thank you very much.  I may not be able to do it in v.1.0 but I will be migrating to it probably in the future.  
[00:44:47] <Matthew_>	 Noice :)
[01:17:37] <legoktm>	 Coren: around? I'm having trouble finding my tool's database on tools-db...
[01:29:37] <legoktm>	 I'll just shoot you an email
[01:56:40] <tgr>	 what are the troubleshooting steps if i get permission denied when trying to ssh to a labs instance where i am a projectadmin?
[01:57:07] <tgr>	 i verified that the SSH key i use for bastion gets forwarded to the instance
[02:03:58] <scfc_de>	 tgr: When was the instance created?  Where is it located (eqiad or pmtpa)?  Have you already looked at Help:Access?
[02:05:38] <tgr>	 uhh, guilty :)
[02:05:46] <tgr>	 but i did now and it doesn't help
[02:06:27] <tgr>	 scfc_de: equiad, created about a week ago i think
[02:09:53] <scfc_de>	 tgr: Hmmm.  Well, in that case /my/ troubleshooting would be to call for andrewbogott or Coren :-).
[02:11:38] <scfc_de>	 BTW, you get "permission denied(public key)"?
[02:12:52] <scfc_de>	 Are we talking about uploadwizard-osm-equiad.eqiad.wmflabs?  That times out on ssh; have you looked at the security groups for your project to allow ssh from 10.0.0.0/8?
[02:14:37] <tgr>	 scfc_de: no, multimedia-alpha.equiad.wmflabs
[02:14:59] <tgr>	 apparently my ssh key is not present there
[02:15:25] <tgr>	 i am not very familiar with labs, but if i am a project member, that should be automatic, right?
[02:17:38] <scfc_de>	 *eqiad* (no "u").  multimedia-alpha.eqiad.wmflabs:ssh times out for me from tools-login as well.  Which error message do you get?
[02:18:26] <andrewbogott>	 tgr, the firewall for that project is blocking ssh.  Hang on, I'll fix it.
[02:19:19] <andrewbogott>	 tgr, better now?
[02:21:18] <tgr>	 andrewbogott: thanks, works now
[02:21:24] <andrewbogott>	 great
[02:21:45] <tgr>	 i still dont understand how i ended up with a permission denied error though
[02:21:56] <tgr>	 shouldnt it just have timed out?
[02:22:02] <andrewbogott>	 that's what it did for me.
[02:22:28] <tgr>	 no, i definitely got a permission error
[02:22:53] <tgr>	 i could evem see with ssh -v that the various keys got offered and refused
[02:23:00] <tgr>	 weird
[02:23:06] <tgr>	 anyway, works now, thanks
[02:23:20] <scfc_de>	 There's an instance with that name in pmtpa as well, so maybe the wrong one?
[02:24:41] <tgr>	 scfc_de: probably that's it
[02:28:49] <grrrit-wm>	 (03PS7) 10Tim Landscheidt: Simple tool to simplify using the backup snapshots [labs/toollabs] - 10https://gerrit.wikimedia.org/r/76313 (owner: 10Platonides)
[03:14:51] <Krinkle>	 Coren: Still failing
[03:14:54] <Krinkle>	 Google tried to deliver your message, but it was rejected by the server for the recipient domain tools.wmflabs.org by mail.tools.wmflabs.org. [208.80.155.162].
[03:14:54] <Krinkle>	 The error that the other server returned was:
[03:14:54] <Krinkle>	 550 Unrouteable address
[03:14:57] <Krinkle>	 different error this time ^
[03:15:00] <Krinkle>	 From 20 minutes ago
[03:16:19] <Coren>	 Yeah, wasn't quite tested yet; moar config needed.  I'll probably get the time to work on that tomorrow.
[03:51:31] <grrrit-wm>	 (03PS4) 10Adamw: Reformat using a slightly more yaml-y style [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/112311 
[03:51:34] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Reformat using a slightly more yaml-y style [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/112311 (owner: 10Adamw)
[03:57:49] <grrrit-wm>	 (03PS5) 10Adamw: Reformat using a slightly more yaml-y style [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/112311 
[04:03:57] <grrrit-wm>	 (03CR) 10Adamw: "Which style u like is just a matter of preference--no worries if you want to leave the syntax as it was!" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/112311 (owner: 10Adamw)
[04:09:34] <grrrit-wm>	 (03PS1) 10Adamw: Make the "blacklist" variable an array [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/119924 
[04:09:57] <torin8>	 Please help! ! When I run a MySQL query on tools-login.wmflabs.org, I get ERROR 1045 (28000): Access denied for user 'torin8'@'10.68.16.7' (using password: NO)
[04:10:23] <torin8>	 what do I need to do to solve this?
[05:52:38] <hedonil>	 torin8: did you fix your problem?
[06:03:14] <torin8>	 nope
[06:03:54] <torin8>	 hedonil: still have the same issues
[06:06:09] <hedonil>	 torin8: your denied for user 'torin8'@'10.68.16.7 says you are not using your MySQL credentials, but your console credentails 
[06:06:21] <hedonil>	 torin8: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Shared_Resources/MySQL_queries#Accessing_the_databases
[06:09:28] <torin8>	 hedonil: which password do I need when connecting to MySQL?
[06:09:43] <torin8>	 when I try my password, i get ERROR 1045 (28000): Access denied for user 'torin8'@'10.68.16.7' (using password: YES)
[06:10:23] <torin8>	 hedonil: do I need to register separately for sql access?
[06:10:43] <hedonil>	 torin8: the MySQL username and password are stored in a file called  replica.my.cnf.
[06:11:15] <hedonil>	 torin8: it's registered automatically. this file should be there
[06:15:06] <hedonil>	 torin8: your MySQL username should be like: uXXXXX.   it's different from your console username torin8.
[06:15:48] <torin8>	 hedonil: replica.my.cnf is stored in my tool folder, right? when I try to open that file, it says permission denied
[06:16:30] <hedonil>	 torin8: try this command first:   become <toolname>
[06:16:49] <hedonil>	 torin8: what's your tool?
[06:17:00] <torin8>	 hedonil: icelab
[06:17:50] <hedonil>	 torin8: ok. first type:   become icelab
[06:18:38] <hedonil>	 torin8: then type:    sql enwiki_p
[07:13:50] <torin8>	 hedonil: hi again, sorry, I lost internet connection
[07:14:17] <hedonil>	 torin8: I sent you an email to labs-l
[07:21:23] <torin8>	 hedonil: i don't think I ran chmod on my directory. Only on my script file because it wouldn't run otherwise
[07:22:31] <torin8>	 hedonil: when I became my tool, i was able to retreive my password and login for sql. They are assignd once and don't change when I reconnect?
[07:23:15] <hedonil>	 torin8: yep. they will stay untouched.
[07:24:20] <torin8>	 hedonil: great! how do i use them?
[07:24:59] <hedonil>	 torin8: most simple from console (example):   sql enwiki_p 
[07:25:52] <hedonil>	 torin8: with this command, the MySQL creds are invoked automatiaclly from replica.my.cnf
[07:26:30] <hedonil>	 torin8: or more in detail https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Shared_Resources/MySQL_queries
[07:27:28] <torin8>	 hedonil: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading authorization packet', system error: 0 happened when i entered my sql password
[07:28:04] <hedonil>	 torin8: this can happen sometimes (it shouldn't). try again
[07:43:40] <torin8>	 hedonil: all servers time out when i try to connect. I can't access anything at the moment, so I will try later. So, when I enter password, i should be able to run Bash scripts, or do I query directly?
[07:44:55] <hedonil>	 torin8: Hmm. right now there seems to be trouble at tools-login. (slow as heck)
[07:45:54] <hedonil>	 torin8: you may want to log out and login to the alternative host:  tools-dev.wmflabs.org
[07:46:35] <hedonil>	 torin8: it's almost the same. maybe it works from there.
[07:48:34] <torin8>	 hedonil: woop, thanks! tools-dev loads fine! I tried running take /home/torin8/ and it said 'you must own the directory'
[07:49:26] <torin8>	 hedonil: chmod also didn't work. It says operation not permitted
[07:49:48] <hedonil>	 torin8: ah, ok. then you'll need more power to fix this ;) try to contact scfc_de or Coren later.
[07:50:39] <torin8>	 hedonil: ahh i see. I'll wait till the guys are online =)
[07:53:41] <torin8>	 hedonil: when i become my tool and run sql enwiki_p, it opens MarinaDB. So MySQL is no more? Is the database schema still the same??
[07:55:09] <hedonil>	 torin8: So you are in now! congrats. what database are you trying to access?
[07:56:30] <hedonil>	 torin8: to simply browse the database servers and take a look at current schemas you can use this https://tools.wmflabs.org/tools-info/?dblist=s1.labsdb
[07:56:44] <torin8>	 hedonil: thank you! I need enwiki, precisely, the languaglinks table
[07:57:56] <hedonil>	 torin8: Yeah. then everything should be fine (but the owner issue has to be fixed though)  ;)
[07:59:00] <torin8>	 hedonil: I don't really know how to use MariaDB. Does it understands scripts?
[07:59:58] <hedonil>	 torin8: whats your script? can you paste it http://tools.wmflabs.org/paste/
[08:05:12] <torin8>	 hedonil: https://tools.wmflabs.org/paste/view/2c13b7c3
[08:07:10] <hedonil>	 torin8: looks perfectly fine ;
[08:07:47] <torin8>	 hedonil: i think the server should be s1.labsdb instead of sql-s1?
[08:08:07] <hedonil>	 torin8: yep . you're right.
[08:09:10] <hedonil>	 torin8: so almost perfectly fine ;)
[08:09:10] <torin8>	 hedonil: it took me forever to figure it out =) How do I run it now? from under Marina DB? or as my tool?
[08:10:56] <hedonil>	 torin8: it's a bash script. so run it from console. first  chmod +x <script> .  then ./<script>
[08:15:53] <torin8>	 hedonil: when i run it like this, i get ERROR 1045 (28000): Access denied for user 'torin8'@'10.68.16.8' (using password: NO)
[08:16:57] <hashar>	 hello
[08:17:08] <hedonil>	 torin8: yep try https://tools.wmflabs.org/paste/view/716c21fa
[08:18:51] <hedonil>	 torin8: either:  mysql --defaults-file=~/replica.my.cnf -h s1.labsdb  -e "foo"    or   sql enwiki_p -e "foo"
[08:25:56] <torin8>	 hedonil: this is funny. it says access denied
[08:26:16] <hedonil>	 YuviPanda: 
[08:26:30] <torin8>	 torin8@tools-dev:/data/project/icelab$ sql enwiki_p -e "foo" hedonil: Enter password: ERROR 1045 (28000): Access denied for user 'torin8'@'10.68.16.8' (using password: YES)
[08:26:45] <torin8>	 hedonil: torin8@tools-dev:/data/project/icelab$ sql enwiki_p -e "foo" Enter password: ERROR 1045 (28000): Access denied for user 'torin8'@'10.68.16.8' (using password: YES)
[08:28:15] <hedonil>	 torin8: Hmm. it's torin8 again. try  sql ruwiki_p -e "show databases;"  what's with that simple thing?
[08:29:37] <torin8>	 hedonil: do i run it as my tool or as torin8?
[08:30:44] <hedonil>	 torin8: try this simply from your console, should work without entering any password
[08:31:14] <torin8>	 hedonil: torin8@tools-dev:/data/project/icelab$ sql ruwiki_p -e "show databases;" Enter password: ERROR 1045 (28000): Access denied for user 'torin8'@'10.68.16.8' (using password: YES)
[08:31:47] <torin8>	 hedonil: what do you mean simply from console? from my Windows local terminal?
[08:32:30] <hedonil>	 torin8: if you are logged in to labs with PuTTY (I assume)
[08:32:46] <torin8>	 hedonil: yes
[08:32:55] <hedonil>	 torin8: 'k. just a sec
[08:33:46] <hedonil>	 torin8: ok. try this one:  mysql --defaults-file=~/replica.my.cnf -h s1.labsdb enwiki_p -e "show databases;"
[08:34:58] <torin8>	 hedonil: mysql: unknown option '--defaults--'
[08:35:28] <hedonil>	 YuviPanda: Are you around? Can you take a look at tools-login. It's completely clogged. fubar.
[08:35:39] <hedonil>	 torin8: you are scaring me ;)
[08:36:10] <torin8>	 hedonil: was i cursed? =)
[08:36:39] <hedonil>	 torin8: what happens if you do:  cat replica.my.cnf 
[08:37:42] <torin8>	 hedonil: permission denied
[08:37:56] <torin8>	 hedonil: and we are back to where we started =(
[08:37:59] <hedonil>	 torin8: aha!
[08:38:30] <hedonil>	 torin8: can you   chmod 700 replica.my.cnf   ?
[08:39:53] <hedonil>	 torin8: or paste the result of:  ls -lisa   from just where you are right now (console)
[08:40:19] * hedonil  tries something different

[09:08:12] <a930913>	 Can't get into tools-login? :/ ssh_exchange_identification: Connection closed by remote host
[09:08:42] <hedonil>	 a930913: yeah it's f*cked up
[09:08:49] <a930913>	 hedonil: D:
[09:08:57] <a930913>	 Is dev ok?
[09:09:03] <hedonil>	 yep
[09:23:51] <hashar>	 !log deployment-prep fixing l10ncache on deplkoyment-bastion : <tt>chown -R l10nupdate:l10nupdate /data/project/apache/common-local/php-master/cache/l10n    </tt>  The l10nupdate UID/GID has been changed and are now in LDAP
[09:23:53] <labs-morebots>	 Logged the message, Master
[09:29:01] <hashar>	 !log deployment-prep l10ncache is now rebuild properly : https://integration.wikimedia.org/ci/job/beta-code-update/53508/console
[09:29:04] <labs-morebots>	 Logged the message, Master
[09:37:11] <a930913>	 ClueBot NG is down.
[09:38:32] <a930913>	 Wonder what broke it. It still looks like it's running on the status page.
[09:47:12] <hedonil>	 torin8: hey cursed you ;), I got a new approach
[09:47:28] <torin8>	 Firefox kills IRC, anybody has similar issues?
[09:47:43] <torin8>	 hedonil: hey =)) I am back!
[09:48:17] <hedonil>	 torin8: first copletely logout from tools-dev
[09:48:23] <torin8>	 hedonil: I am all ears!
[09:48:23] <Amir1>	 hi, Is it tools okay?
[09:48:30] <a930913>	 Amir1: Nope.
[09:48:39] <Amir1>	 ssh ladsgroup@tools-login-eqiad.wmflabs.org
[09:48:40] <a930913>	 We're doomed. Doomed I say.
[09:48:40] <Amir1>	 ssh_exchange_identification: read: Connection reset by peer
[09:49:01] <Amir1>	 a930913: yeah, I'm doomed
[09:49:22] <Amir1>	 :)
[09:49:27] <a930913>	 Amir1: ssh ladsgroup@tools-dev-eqiad.wmflabs.org
[09:49:30] <hedonil>	 Amir1: yeah doomed & cursed
[09:49:50] <a930913>	 Does anybody know what's wrong?
[09:50:05] <hedonil>	 torin8: you still have 2 connections to tools-dev, they must be terminated first
[09:50:12] <a930913>	 It doesn't seem limited to tools-login.
[09:50:13] <Amir1>	 yeah it's working
[09:50:17] <Amir1>	 but still doomed
[09:53:24] <torin8>	 hedonil: how do I do i? i closed both putty and WinSCP
[09:53:34] <hedonil>	 torin8: yep
[09:54:50] <torin8>	 hedonil: and you still see 2 active connections?
[09:55:01] <hedonil>	 torin8: yes
[09:55:24] <torin8>	 hedonil: how do i close them?
[09:56:10] <hedonil>	 torin8: hmm. try a new putty connection
[09:57:32] <torin8>	 hedonil: i logged into tools-dev
[09:57:51] <hedonil>	 torin8: ok. now type:  id
[09:58:09] <hedonil>	 torin8: and maybe paste the result
[09:58:09] <gifti>	 this ...
[09:58:27] <torin8>	 hedonil: uid=4502(torin8) gid=500(wikidev) groups=500(wikidev),50062(project-bastion),50380(project-tools),51481(tools.newwebtest),51716(tools.icelab),51813(tools.torintool)
[09:58:59] <hedonil>	 torin8: 'k. I've added you to a play tool newwebtest
[09:59:14] <hedonil>	 torin8: try:   become newwebtest
[09:59:34] <torin8>	 hedonil: ok, what do i do?
[09:59:35] <a930913>	 hedonil: Won't he need to relogin?
[10:00:13] <hedonil>	 a930913: is a new connection and the group membership seems ok
[10:00:53] <hedonil>	 torin8: if you are newwebtest now, try   sql enwiki_p
[10:01:24] <torin8>	 hedonil: it worked, i'm in MariaDB
[10:02:00] <a930913>	 I said "he", was I wrong? :o
[10:03:18] <hedonil>	 torin8: to be sure:  sql enwiki_p -e "Show databases;"  works from console?
[10:03:56] <torin8>	 my console looks like this now: MariaDB [enwiki_p]>
[10:04:20] <hedonil>	 torin8: looks good! 
[10:04:41] <hedonil>	 torin8: exit this prompt with:   exit
[10:05:10] <torin8>	 hedonil: the latter command gave me a list of databases
[10:05:26] <hedonil>	 torin8: perfect
[10:07:00] <torin8>	 hedonil: what's next? =)
[10:07:22] <hedonil>	 torin8: you can worj with this tool account now (if you want). or exit from neweebtest and try   become icelab
[10:08:52] <torin8>	 hedonil: so i just move my script there and run it as ./script?
[10:09:13] <hedonil>	 torin8: so yeah, try
[10:11:59] <torin8>	 hedonil: so, i became Icelab and ran the same commands. It showed me the database list. But when i ran the script, it said ERROR 1045 (28000): Access denied for user 'tools.icelab'@'10.68.16.8' (using password: NO) 
[10:12:57] <hedonil>	 torin8: always a new variant
[10:13:10] <torin8>	 hedonil: strange, when i did sql enwiki_p, it didn't ask for a password
[10:13:11] * hedonil  laughs

[10:13:44] <hedonil>	 torin8: that's absolutely correct
[10:14:26] <hedonil>	 torin8: cat  replica.my.cnf   shows you some data
[10:14:55] <torin8>	 hedonil: lol, we need to sprinkle some holy water or fairy dust on it =)
[10:14:55] <hedonil>	 torin8: don't paste that ;)
[10:15:35] <torin8>	 hedonil: yep, it shows my credentials
[10:17:18] <hedonil>	 torin8: please goto this page  https://wikitech.wikimedia.org/wiki/Special:NovaServiceGroup   (login required)
[10:18:04] <hedonil>	 torin8: and then to your tool  local-icelab
[10:19:28] <hedonil>	 torin8: then  -> Manage members
[10:19:38] <torin8>	 hedonil: i found my tool, do i click manage members?
[10:20:44] <hedonil>	 torin8: yep. then put cursor in Member field and add someone
[10:21:49] <torin8>	 hedonil: done
[10:23:00] <perohanych>	 WWho knows what has happened with the Wikitrends tool?
[10:23:16] <torin8>	 hedonil: now I delete that user, right?
[10:23:28] <hedonil>	 torin8: no. wait a few minutes. then logout. then login again. 
[10:24:03] <hedonil>	 torin8: then try:   become icelabs
[10:24:12] <torin8>	 hedonil: just to clarify, what's the right way to log out? I just close Putty window, but there should be a command?
[10:24:53] <hedonil>	 torin8: exit 
[10:25:39] <hedonil>	 torin8: but all that matters is a new session (connection)
[10:27:12] <torin8>	 hedonil: oops. Server unexpectedly closed network connection
[10:27:41] <russblau>	 Is tools-login down?
[10:27:55] <torin8>	 russblau: looks like it to me
[10:27:55] <hedonil>	 russblau: yes, try tools-dev
[10:29:18] <torin8>	 hedonil: i still get the permission thing
[10:29:45] <hedonil>	 torin8: can you become icelabs?
[10:29:52] <torin8>	 yes i did
[10:30:20] <hedonil>	 torin8: pls paste   ls -lisa   
[10:31:02] <torin8>	 hedonil: tools.icelab@tools-dev:~$ ls -lisa
[10:31:17] <torin8>	 hedonil: gave me 64 files
[10:31:51] <hedonil>	 torin8: ok. pls paste the result  http://tools.wmflabs.org/paste/
[10:32:35] <torin8>	 hedonil: https://tools.wmflabs.org/paste/view/35058f7d
[10:34:11] <hedonil>	 torin8: looks ok. and as you said  cat  replica.my.cnf  shows you some creds
[10:34:11] <a930913>	 perohanych: "wikidata-analysis and wikitrends, ...are still being copied over and may take up to another day to complete given the large amount of data to copy." [Yesterday]
[10:35:52] <torin8>	 hedonil: then what's the issue?
[10:36:19] <hedonil>	 torin8: so  sql enwikip -e "show databases;"    *should*  work. Does it?
[10:36:38] <hedonil>	 torin8: sql enwiki_p -e "show databases;"
[10:37:39] <torin8>	 hedonil: yes it works
[10:38:03] <hedonil>	 torin8: \o/ \o/
[10:38:23] <torin8>	 hedonil: but it doesn't hel with running the script =(
[10:39:08] <hedonil>	 torin8: how do you call your script?
[10:39:12] <torin8>	 hedonil: or am i not running it right?
[10:39:39] <torin8>	 hedonil: ./lang.scr 500px.txt
[10:39:54] <Arnaugir>	 hi, Do you know why I might be getting a UnicodeEncodeError here, using mwparserfromhell:  template.get(u"edició").value . Pyhton script is in UTF-8...
[10:40:22] <torin8>	 hedonil: returns error access denied
[10:40:54] <hedonil>	 torin8: you have still  the mysql command  not the sql command 
[10:41:31] <hedonil>	 torin8: replace:   mysql -h s1.labsdb enwiki_p    with    sql enwiki_p
[10:41:32] <torin8>	 hedonil: wait,  I missed that. What does it mean?
[10:42:31] <hedonil>	 torin8: the 'sql' command is a wrapper script that invokes your replica.my.cnf automatically.
[10:44:34] <hedonil>	 torin8: the 'mysql' is a binary which needs additional information for your MySQL user, otherwise your tools account is sent to MySQL server which doesn't work
[10:45:14] <torin8>	 hedonil: it worked!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[10:45:31] <hedonil>	 torin8: \o/
[10:46:11] <torin8>	 hedonil: thank you so much!!!!!! you have no idea how many days I have already spent on making it work
[10:46:20] * hedonil  mumbles that torin8 is not cursed

[10:46:25] <torin8>	 hedonil: many-many thanks
[10:46:28] <hedonil>	 torin8: yw
[10:46:46] <torin8>	 hedonil: hehehe =) not cursed, good to know =)
[10:52:47] <Arnaugir>	 hi, Do you know why I might be getting a UnicodeEncodeError here, using mwparserfromhell:  template.get(u"edició").value . Pyhton script is in UTF-8... any idea? sorry to insist
[10:56:01] <hedonil>	 tools-login seems really to be FUBAR. even the cronjobs don't work any longer
[10:57:26] <hedonil>	 Coren: petan: Reedy. Any admin around to take a look at tools-login? really bad state.
[11:02:57] <hedonil>	 perohanych: if your looking for TOP top viewstats, you can check out  https://tools.wmflabs.org/wikiviewstats/ 
[11:42:50] <a930913>	 hedonil: Are all admins on US time?
[11:48:07] <valhallasw>	 a930913: that depends on your 'admin' definition. Petrb is on CET afaik
[11:48:13] <valhallasw>	 as is hashar, for instance
[11:48:52] <valhallasw>	 Arnaugir: the same issue as before: if you parse a str, use str to query the parsed page
[11:51:57] <Arnaugir>	 valhallasw: I don't understand at all. What do you mean exactly?
[11:51:59] <Arnaugir>	 this is not a str object
[11:54:10] <uduwage>	 I am unable to ssh into tools-login-eqiad.wmflabs.org
[11:54:26] <uduwage>	 is anyone having same experience?
[12:02:39] <a930913>	 uduwage: Yup it's b0rked.
[12:02:54] <a930913>	 uduwage: Use tools-dev for now.
[12:04:29] <uduwage>	 @a930913 thanks
[12:07:57] <`fox`>	 Coren, did you have time to install stuff on labs? I would need this: https://bugzilla.wikimedia.org/show_bug.cgi?id=62649
[12:12:01] <kart_>	 Q: Do I need anything else in config for ssh to eqiad?
[12:41:06] <Alchimista>	 Coren: login isn't working, on linux with openssh, ssh alchimista@tools-login.wmflabs.org just keeps tryng, without giving any error
[12:41:32] <Coren>	 "keeps trying"?
[12:41:50] <HenriqueCrang>	 same for me here
[12:42:14] <Alchimista>	 it does nothing, just keeps the signal flashing on the cmd
[12:42:25] <HenriqueCrang>	 using -v it stops on: debug1: identity file /home/handrade/.ssh/id_ecdsa-cert type -1
[12:42:30] <Coren>	 Ah; times out.
[12:42:35] <Coren>	 Looks like -login is OOM
[12:43:59] <Coren>	 Ah, yes it is.
[12:44:25] * Coren  grumbles.

[12:44:48] <Coren>	 And I pretty much doubled its size, too.  Someone is being rude with their crontab it looks like.
[12:45:35] <Coren>	 `fox`: Today I have to work on mail, then I'm catching up bugzillas; so I'll probably get to yours.
[12:49:14] <HenriqueCrang>	 now login is working. thanks Coren :)
[12:58:25] <HenriqueCrang>	 Coren, I know we are kind of late with it, but, where we can find our old crontab on equiad?
[12:59:17] <Coren>	 HenriqueCrang: They should have been copied over; but unless you ran 'finish-migration <tool>', they're still in temporary files (...DATA.crontab in your tool's home)
[12:59:57] * Coren  points at labs-l

[13:00:01] <HenriqueCrang>	 just found it there! thanks again!
[13:02:52] <russblau>	 crontabs on tools-login seem to be gone (not a migration issue; I migrated tools several days ago and uncommented the crontabs then, they just disappeared this morning)
[13:03:19] <russblau>	 disregard... my mistake
[13:25:56] <hashar>	 Coren: good morning, would you mind merging for me a bunch of packages installation for contint  please ?  https://gerrit.wikimedia.org/r/#/c/119983/ 
[13:26:06] <hashar>	 will let me phase out yet another instance from pmtpa \O/
[13:31:14] <hashar>	 !log integration deleting integration-jenkins2 which had gerrit/zuul/jenkins.  Will rebuild it in integration-dev.eqiad.wmflabs
[13:31:17] <labs-morebots>	 Logged the message, Master
[13:32:24] <hashar>	 !log integration created web proxy entry  ci.wmflabs.org pointing to  integration-dev.eqiad.wmflabs:80
[13:32:26] <labs-morebots>	 Logged the message, Master
[13:35:58] <Coren>	 hashar: Sure.  Gimme a sec.
[13:36:02] <hashar>	 \O/
[13:36:32] <hashar>	 and if you get time I could use the migration of two unpuppetized instances from pmtpa to eqiad
[13:36:51] <hashar>	 and then I would have only one instance left in the labs project "integration" !
[13:37:21] <Coren>	 hashar: For instance migration, you probably want to talk to andrewbogott_afk.
[13:38:31] <hashar>	 ahh
[13:38:40] <hashar>	 will drop him an email with the instance I need to transfer. Thx!
[13:40:14] <hashar>	 Coren: sorry did a typo :  package {}  is using singular, not plural 'packages'   https://gerrit.wikimedia.org/r/119991
[13:40:15] <hashar>	 :(-
[13:40:47] <Coren>	 o_O  How did Jenkins not catch that?
[13:41:35] <hashar>	 packages {} is valid puppet syntax hehe
[13:41:52] <hashar>	 Alexandros is working on a script to compile the catalog on a bunch of representatives nodes
[13:42:07] <hashar>	 such typo would be caught by the catalog compilation and fail it
[13:42:20] <Coren>	 That would be extra cool.
[13:48:41] <rluts>	 hi. what happened with ImageMagick on tool labs? my script return "sh: 1: convert: not found"
[14:00:30] <scfc_de>	 Coren: I suppose you rebooted tools-login due to the OOM?  It looks like the iptables didn't come back up.
[14:09:55] <Coren>	 scfc_de: I have, and it clearly didn't.  Bleh.  Ima puppetize this now because it has gotten annoying.
[14:11:02] <Coren>	 Also because we'll have to move the replicas soon to a different row and that will make it easier to be less disruptive.
[14:15:20] <scfc_de>	 Coren: Good luck! :-)  I still think if sweat needs to be invested, the biggest return would give https://bugzilla.wikimedia.org/61897 ("Move LabsDB aliases and NAT to DNS and LabsDB servers").
[14:16:19] <Coren>	 scfc_de: Yeah, but can't be done before the DB move.
[14:16:31] <Coren>	 Well can, but not productively.
[14:17:25] <Coren>	 ... but you're correct; I think we can live another week or two with that annoyance.
[14:21:28] <hashar>	 and I just found out labs instances have a 2GB /var/  :-(
[14:24:49] <anomie>	 Coren: Since it has concerns related to tool labs, bug 62615 might be something you'd want to opine at. Low priority, don't interrupt yourself if you're busy with other stuff.
[14:25:41] <Coren>	 hashar: That's a good thing; it prevents runaway logs from breaking things (as often happens)
[14:25:47] <hashar>	 yeah
[14:25:52] <hashar>	 that is nice
[14:26:17] <hashar>	 the contint slaves are using pbuilder which has cowimages for lucid and precise in /var/cache/pbuilder :]
[14:26:37] <hashar>	 gotta tweak the misc::package-builder infamous class \O/
[14:27:11] <Coren>	 Heh.  "Fun".  That said, we do have a bad habit of putting random junk in /var/ that really doesn't belong here at WMF.
[14:29:40] <hashar>	 at previous job we had /  /var /tmp in different partitions
[14:29:58] <hashar>	 and application received a company wide ID and were required to use /<theircode>
[14:30:12] <hashar>	 with log to /<their code>/v1.2.3/log/ 
[14:30:13] <hashar>	 etc
[14:30:52] <Coren>	 hashar: That seems overkill.  Traditionally, you'd put that in /opt/something although the modern fad is /srv for "applicative data"
[14:31:09] <hashar>	 yeah that was definitely overkill
[14:31:14] * hasteur  waves at their gerrit mistarget buddy.

[14:31:15] <hashar>	 though all apps were using the same layout
[14:31:21] <hashar>	 that is very handy for production
[14:32:02] <Coren>	 hashar: I can see how it would be.  As long as you don't trample over well-defined LFS semantics, any convention will work.
[14:33:11] <Coren>	 But I've an aversion to polluting /.  There are enough things in there already.  :-)
[14:36:12] <hashar>	 well each app would have its own partition :]
[14:36:30] <hashar>	 but yeah that is overkill for us
[14:36:32] <hashar>	 will get pbuilder to be parameterizable so the cow images can be written to /mnt
[14:37:17] <Coren>	 Well, also /mnt for a permanent thing is Evil™ but in our case hysterical raisins.  That really should have been /srv
[14:45:35] <Cyberpower678>	 hedonil, ping
[14:48:19] <hashar>	 Coren: I am willing to convert to /srv :-]
[14:48:43] <hedonil>	 Cyberpower678: Hi
[14:48:59] <Coren>	 hashar: Consider it for the future.  You got enough moving parts already.  :-P
[14:49:12] <Cyberpower678>	 hedonil, https://tools.wmflabs.org/supercount/index.php?user=ClueBot%20NG&project=en.wikipedia .  Not even X!'s edit counter could handle that without crashing.
[14:50:32] <hedonil>	 Cyberpower678: hehehe
[14:50:48] <Cyberpower678>	 hedonil, look on the bottom of the page. :p
[14:53:08] <hedonil>	 Cyberpower678: the new design and the pie graphic is awesome
[14:53:19] <Cyberpower678>	 Thank you.
[14:53:26] <hedonil>	 Cyberpower678: and we are all looking forward to the non JS version, too !!
[14:54:06] <Cyberpower678>	 :)
[14:54:18] <Cyberpower678>	 I'm busy until the weekend.
[14:54:36] <Cyberpower678>	 So no mods today.
[14:54:37] <Cyberpower678>	 \
[15:06:02] <hashar>	 I am puppet fluent \O/
[15:09:16] <hedonil>	 scfc_de: can you do torin8 and annabel a favour and fix/chown  /home/torin8/   and  /home/annabel ?
[15:12:08] <scfc_de>	 hedonil: Yeah (still catching up).  So that's that bug again?  My brain's wired to parse "permission denied" = "permission denied (public key)".
[15:12:51] <hedonil>	 scfc_de: we managed to get it work for her tools account icelab
[15:13:08] <scfc_de>	 Odd, /home/torin8 has been filled by the user, the ownership to root only changed later?!
[15:13:08] <hedonil>	 scfc_de: but her home ist still owned by root
[15:14:08] <hedonil>	 maybe a magic back-to-the-root task ;)
[15:14:09] <scfc_de>	 !log tools sudo chown -R torin8.wikidev /home/torin8
[15:14:12] <labs-morebots>	 Logged the message, Master
[15:15:16] <scfc_de>	 !log sudo cp -R /etc/skel /home/annabel && sudo chown -R annabel.wikidev /home/annabel
[15:15:16] <labs-morebots>	 sudo is not a valid project.
[15:15:45] <scfc_de>	 !log tools sudo cp -R /etc/skel /home/annabel && sudo chown -R annabel.wikidev /home/annabel
[15:15:46] <labs-morebots>	 Logged the message, Master
[15:29:21] <a930913>	 Damianz: Does CBNG rely on tools-login?
[15:32:47] <andrewbogott>	 Coren, see what I mean?
[15:40:20] <hashar>	 andrewbogott: going to mail you some instances I would need to be migrated from pmtpa to eqiad. It is not urgent, can wait monday :]
[15:40:23] <hashar>	 have a good week-end!
[15:40:41] <hashar>	 unless there is another procedure to request instances moves
[15:40:44] <andrewbogott>	 hashar:  Can you make a bug?  Will be easier for me to keep track...
[15:40:47] <hashar>	 sure
[15:40:51] <hashar>	 one bug per instance?
[15:41:16] <andrewbogott>	 https://bugzilla.wikimedia.org/enter_bug.cgi?product=Wikimedia%20Labs&component=Infrastructure&blocked=62042 
[15:41:19] <andrewbogott>	 One big bug is fine.
[15:41:33] <andrewbogott>	 Just, make sure 62042 depends on it.
[15:41:41] <hashar>	 okkk
[15:41:46] <Coren>	 andrewbogott: Wait, what?  <andrewbogott> Coren, see what I mean?
[15:41:56] <andrewbogott>	 Coren: I mean, re: labstore4
[15:42:17] <Coren>	 Ah.  Yes, yes I do.  I'm not sure how that happened, I'm trying to see if there is a pattern.
[15:42:31] <andrewbogott>	 ok.
[15:42:36] <andrewbogott>	 It's not urgent, just giving me the creeps
[15:43:06] <andrewbogott>	 It looks like the kind of thing that would result from an rsync with the wrong number of terminal slashes...
[15:43:15] <andrewbogott>	 but I'm pretty sure I haven't rsyncd /to/ labstore4 ever
[15:47:16] <hashar>	 andrewbogott: bugs filled. Thank you!
[15:49:43] <scfc_de>	 !log tools sudo cp -R /etc/skel /home/csroychan && sudo chown -R csroychan.wikidev /home/csroychan; that should close [[bugzilla:62132]]
[15:49:45] <labs-morebots>	 Logged the message, Master
[15:53:02] <scfc_de>	 Nemo_bis: On Tools, you own an empty /home/icinga ("drwxr-xr-x 2 nemobis 50383 4096 Mär 17 16:42").  Do you remember how you managed to create it on Monday?
[15:58:04] <Nemo_bis>	 scfc_de: absolutely not
[15:58:13] <Nemo_bis>	 what does it contain?
[15:58:33] <Nemo_bis>	 actually, did I even login on monday
[15:59:03] <Nemo_bis>	 you said empty, sorry
[16:00:07] <Nemo_bis>	 I don't remember when Andrew migrated dumps data, but I don't think it was this week and I don't think we had any icinga directory
[16:07:34] <scfc_de>	 Nemo_bis: It's totally empty.  I'm pretty sure the dumps project (?) has separate home directories; perhaps it's an artifact by the mass migration of Tools on that day?  Anyhow, I'm just gonna delete then.  Grazie!
[16:08:48] <Nemo_bis>	 Grazie mille
[16:13:20] <scfc_de>	 !log tools rmdir /home/icinga (totally empty, "drwxr-xr-x 2 nemobis 50383 4096 Mär 17 16:42", perhaps artifact of mass migration?)
[16:13:23] <labs-morebots>	 Logged the message, Master
[16:23:07] <legoktm>	 Coren, scfc_de: ping? I'm having trouble finding one of my databases after the migration...
[16:24:47] <scfc_de>	 legoktm: I'm gonna yield that to Coren as I don't have knowledge in that area.
[16:25:02] <legoktm>	 ok
[16:28:54] <perohanych>	 exit
[16:49:23] <scfc_de>	 tools-login OOM again?  *argl*
[16:50:20] <scfc_de>	 !log tools tools-login: pkill -u tools.bene (OOM)
[16:50:23] <labs-morebots>	 Logged the message, Master
[16:51:16] <scfc_de>	 lbenedix: Are you Labs user "bene"?
[16:52:01] <legoktm>	 scfc_de: no, that's benestar
[16:52:14] <legoktm>	 [09:52:05] -NickServ- Last seen  : Mar 16 22:57:26 2014 (4 days, 17:54:38 ago)
[16:53:38] <scfc_de>	 legoktm: k, thanks.
[17:06:31] <Coren>	 legoktm: It wasn't named "normally" so it'll need a manual copy.  I'll do it for you right after lunch.
[17:06:42] <legoktm>	 ok, thanks :)
[17:36:20] <juul>	 hi, I just requested shell access and the server told me to come here and ask someone to add me 
[17:59:37] <chrismcmahon>	 Coren:  do you have a minute to help me reason about a possible issue managing cookies on the beta cluster?  not urgent, just mysterious. 
[18:06:07] <Arnaugir>	 valhallasw: if you are around let me know
[18:06:10] <Arnaugir>	 thx
[18:07:05] <chrismcmahon>	 (or if anyone else would like to help me puzzle about a session storage anomaly on beta labs, speak up)  :-)
[18:09:12] <kaldari>	 Question: Both my pre-migration and post-migration database login credentials seem to work fine on tool labs. Will the old one stop working at some point?
[18:09:37] <kaldari>	 Coren, petan: ^
[18:09:44] <petan>	 hi
[18:10:23] <petan>	 kaldari: login credentials to clone or tools-db?
[18:10:34] <kaldari>	 to clone
[18:10:59] <petan>	 kaldari: I have no idea :( this database is maintained ultimately by Coren, so he is likely the only person who can asnwer this
[18:12:00] <kaldari>	 Thanks. I'm inclined to leave it alone per "If it ain't broke, don't fix it." but I don't want it to unexpectedly break in the future.
[18:12:18] <petan>	 I recommend you to update everything to new version
[18:12:25] <kaldari>	 Hopefully Coren will pop up and let me know
[18:12:46] <kaldari>	 OK, if you think that's safer, I'll go ahead and update.
[18:13:22] <scfc_de>	 kaldari: I don't think there's a technical reason to remove the old credentials; the purpose IIRC was to avoid possible future collisions, but I'd agree with petan, if you can change them, you should.
[18:15:19] <kaldari>	 updated the credentials and everything seems to still be working :)
[18:18:28] <Coren>	 legoktm: What's your db name?
[18:18:38] <legoktm>	 locallegobot I think
[18:19:11] <Coren>	 kaldari: I don't think I'm going to ever take the time to actually /disable/ the old credential, but they wouldn't be restored in case we have to wipe and reload a shard.
[18:19:49] <Coren>	 kaldari: So considering them "legacy", and there only for the transition is safest.
[18:20:08] <Coren>	 chrismcmahon: Sure.  Do tell?
[18:20:19] <kaldari>	 Coren: Thanks!
[18:20:48] <valhallasw>	 Arnaugir: I'm here.
[18:21:36] <Arnaugir>	 good. Sorry to bother you valhallasw . I told you before that template.get("edició").value raises a UnicodeDecode error. You told me to parse it as str? How? I don't understand. (sorry about my noobiness)
[18:21:52] <chrismcmahon>	 hi Coren, so I have what looks at first glance to be a race condtion in MobileFrontend code... 
[18:22:08] <Coren>	 legoktm: 'locallegobot'?
[18:22:14] <legoktm>	 Coren: yeah
[18:22:21] * Coren  dumps.

[18:22:31] <legoktm>	 :)
[18:22:51] <chrismcmahon>	 Coren: here's the code: 
[18:22:53] <chrismcmahon>	 http://pastebin.com/CMPk6xhv
[18:23:16] <valhallasw>	 Arnaugir: it's the same issue as before. You have parsed a page without converting it to an unicode object. Thus mwpfh expects str parameters to functions and str for assignments
[18:23:24] <valhallasw>	 Arnaugir: You are passing an unicode object now
[18:23:34] <Coren>	 legoktm: What's your tool current sXXXXX username?
[18:24:05] <legoktm>	 Coren: s51043
[18:24:16] <Arnaugir>	 valhallasw: the problem is, mwparserfromhell obtains the text this way: mwparserfromhell.parse(text) and this is not a unicode object
[18:25:11] <chrismcmahon>	 Coren: so what's happening is that the browser is clicking the toggle on the page at http://en.m.wikipedia.beta.wmflabs.org/w/index.php?title=Special:MobileOptions.  then intermittently the line "if ( $request->getVal( 'token' ) != $context->getMobileToken()" evaluates to "true" and I get an error in the browser. 
[18:26:22] <Arnaugir>	 valhallasw: if I convert mwparserfromhell.parse(text) into unicode, I cannot use functions over it
[18:26:24] <chrismcmahon>	 Coren:  the Mobile team swears that a race condition in that code snippet is impossible, so I'm wondering if there is some way that beta labs could be handling cookies in a way that would return differently for the $context and $request calls. 
[18:26:26] <valhallasw>	 Arnaugir: well, it's not a problem as long as you only interact with it using str objects
[18:26:31] <valhallasw>	 Arnaugir: huh.
[18:26:44] <valhallasw>	 Arnaugir: you'd do .parse(text.decode(...))
[18:26:58] <Coren>	 chrismcmahon: I'm not seeing anything funky in there; but I don't know the mobile code at all.
[18:27:19] <Arnaugir>	 valhallasw: I'm doing that. text = wikipage.getWikiText().decode('utf-8', 'replace') // code = mwparserfromhell.parse(text)
[18:27:33] <Coren>	 legoktm: You have a lego.sql.tgz in your tools' home.  Just feed that to mysql and you're all set.
[18:28:30] <Withoutaname>	 Does anyone have a mass unblock script?
[18:28:55] <valhallasw>	 Arnaugir: OK, I'm confused now. You originally complained template.get(u'sométhing') did not work, but now you're using template.get('sométhing')?
[18:29:18] <chrismcmahon>	 Coren:  I know jack about PHP.  bug is https://bugzilla.wikimedia.org/show_bug.cgi?id=62614
[18:30:49] <Arnaugir>	 valhallasw: yes, I'm confused too. This is the situation. The python script is UTF8. The wikitext I obtain is decoded with .decode('utf-8', 'replace') , then I parse it with code = mwparserfromhell.parse(text). Once I have worked with it with mwpfh functions, I do unicode(code) and it always works.
[18:31:15] <Arnaugir>	 template.get("llengua") works fine. template.get("lléngua") does not
[18:31:32] <valhallasw>	 Arnaugir: right, because you should use template.get(u'lléngua')
[18:31:35] <legoktm>	 Coren: thanks!
[18:31:38] <Arnaugir>	 valhallasw: not working either
[18:32:04] <Arnaugir>	 valhallasw: it works in template.has(u"edició")
[18:32:13] <Arnaugir>	 valhallasw: but doesn't work in template.get(u"edició").value
[18:32:16] <chrismcmahon>	 Coren: what's odd about that code is that it does not depend on being logged in, it is strictly setting a cookie for the local browser session.  so the Mobile theory is that something is hinky with how beta labs is processing session/cookie storage, and that's sort of a black hole for us all. 
[18:33:10] <Arnaugir>	 valhallasw: Okay, I think I found the problem
[18:33:31] <Arnaugir>	 the problem is not with "edició" parameter, but with the text I'm trying to print, which has special characters too (is a parameter value from the wikipedia page).
[18:33:44] <Arnaugir>	 before I had print template.get(u"edició").value to debug
[18:33:48] <Arnaugir>	 it worked with print unicode(template.get(u"edició").value)
[18:33:56] <Arnaugir>	 this encoding things will drive me mad...
[18:34:08] <valhallasw>	 Arnaugir: read Joel Spolsky's article.
[18:34:17] <valhallasw>	 there is no way you
[18:34:21] <Arnaugir>	 valhallasw: it's pending
[18:34:27] <valhallasw>	 there is no way you're going to get it right by just trying around
[18:34:40] <Arnaugir>	 valhallasw: until now I did ;) just kidding. I really should.
[18:34:45] <Arnaugir>	 sorry to bother you!
[18:39:45] <Coren>	 chrismcmahon: I see absolutely no reason why that would be the case; but I'm really not the betalabs expert either.  Hashar is probably your best bet there; he's the one who maintains its configuration.
[18:39:47] <juul>	 how do i get my ssh account approved?
[18:40:55] <Coren>	 juul: That'll happen as soon as one of us gets a minute to walk through the list.  It normally takes 24-48 hours.
[18:40:59] <juul>	 Coren: ok thanks!
[18:43:01] <chrismcmahon>	 Coren: OK, thanks. I still think it smells like a race condition where either that $context or $request does not DTRT, but whatever, not my area of expertise. 
[18:45:46] <Damianz>	 So who broke tools this morning for a while?
[18:49:50] <Coren>	 legoktm: Did you manage to get your DB okay?
[18:50:10] <Coren>	 Damianz: AFAIK, someone managed to run tools-login into the ground (OOM) but nothing else broke.
[18:50:32] <legoktm>	 Coren: yes, it's working perfectly. thanks!
[18:52:13] <Damianz>	 Hmm.. bot stopped at 10:20 and started again at 13:00, might not be tools - I've only just sat down so not looked yet.
[18:52:33] * Damianz  notes it's normally tools or someone breaking the api is subtle ways

[19:12:17] <Coren>	 Damianz: From what I've seen to date, the new setup is rock-solid.  There were minor issues here and there migrating, but once migrated everything seems to be going well.
[19:12:26] * Coren  reconsiders actually /saying/ that.

[19:12:35] <Coren>	 I've just jinxed it, didn't I?  :-)
[19:14:31] <valhallasw>	 Coren: I've seen NFS/jsub issues already.
[19:14:35] <Damianz>	 Just need to wait for nfs to explode again
[19:15:09] <Coren>	 No NFS explosion allowed.
[19:15:17] <Coren>	 valhallasw: ... what?  Moar details!
[19:15:24] <valhallasw>	 Use of uninitialized value $prog in scalar chomp at /usr/bin/jsub line 112. Use of uninitialized value $prog in -l at /usr/bin/jsub line 113. Use of uninitialized value $prog in pattern match (m//) at /usr/bin/jsub line 127. Use of uninitialized value $cwd in scalar chomp at /usr/bin/jsub line 129. Use of uninitialized value $cwd in concatenation (.) or string at /usr/bin/jsub line 130. /gerrit_reviewer_bot.sh: No such file or directo
[19:15:43] <valhallasw>	 ^ that stuff. My mailbox is full of ptmpa ones, but this one is from eqiad
[19:16:14] <valhallasw>	 this was today at 17:45 CET
[19:17:08] <valhallasw>	 in ptmpa, those typically came in batches, around once a month, but more frequently in february
[19:17:22] <scfc_de>	 valhallasw: I believe that was at the time of the OOM.
[19:17:30] <scfc_de>	 (That today.)
[19:18:24] <Coren>	 valhallasw: Yeah, that's the login server going OOM; the invocation of 'which $exe' on line 112 fails in a bad way and the rest of the script gets confused.
[19:18:40] <Coren>	 I need to start enforcing the rules a bit more severely.
[19:18:58] <Coren>	 I've already had to remove shell from someone again today.
[19:19:44] <Damianz>	 hmmm
[19:19:48] <valhallasw>	 I would really like an SGE submit host
[19:20:11] <Coren>	 ... you mean where, like, only cron runs?
[19:20:14] <Damianz>	 Any replication delays or such? Seem to be missing some user data for a few hours
[19:20:17] <valhallasw>	 Coren: Right.
[19:20:19] <Damianz>	 I'd like a better system than cron ;)
[19:20:32] * Damianz  is guilty of using login for his crons... but they're only little

[19:20:34] <Coren>	 Damianz: So would I.
[19:20:43] <valhallasw>	 Coren: maybe even without internet access except for tools-login and SGE? >:)
[19:20:51] <Coren>	 Damianz: No, that's cool -- it's the people who abuse -login to run bots that are the issue.
[19:21:11] <scfc_de>	 And ulimit to VMEM(jsub) + margin.
[19:21:19] <Coren>	 valhallasw: That's not going to prevent the current issue, which is people ignoring or choosing to ignore the rule about not running stuff on the bastions.
[19:21:39] <valhallasw>	 Coren: except they won't do it on that host, so it won't affect my tasks.
[19:21:59] <valhallasw>	 the only reason I care about people doing that is that they break my cron jobs
[19:22:00] <Coren>	 valhallasw: Your optimism is noted.  Also funny.  :-)
[19:22:31] <valhallasw>	 These are the people that are too lazy to read the SGE manual. Hence, they also won't know the SGE submit hosts exists in the first place!
[19:22:56] <Coren>	 Hmm.  There's some amount of sense in that.
[19:23:27] <scfc_de>	 tools-submit in line with submit.toolserver.org?
[19:23:31] <Coren>	 Enough sense that I shall do so.
[19:23:36] * valhallasw  cheers

[19:23:37] * Coren  ponders.

[19:23:55] <Coren>	 I'm also going to disable cron for non-tools.
[19:25:18] <Damianz>	 I like having a non tool account running tools :P
[19:25:28] <scfc_de>	 Don't make it too complicated :-).  I think valhallasw's right that being hush-hush about it might keep trouble far enough.
[19:25:46] <Coren>	 Damianz: Don't let me catch you doing that.
[19:27:22] <Damianz>	 Coren: Useful for debugging things and monitoring scripts... though mostly I trust my own servers cron to monitor tools more than labs to monitor labs :D
[19:27:57] <Coren>	 Damianz: None of which you cannot do from a tool account.  Don't do it.
[19:28:11] <Dragonfly6-7>	 Can someone create for me a dynamic list of all pages -- but not subpages -- in the Wikipedia namespace? I've found an annoying trend of moving drafts into Wikipedia namespace instead of mainspace
[19:29:42] <Damianz>	 Can you delete tools now? That's the main reason not to use tool accounts because I'd end up with loads being used as disposable... otherwise switching user is silly and makes switching betwean users longer
[19:30:14] <Coren>	 Damianz: We could always delete tools; but it requires an admin (it's not complicated though)
[19:30:27] * Dragonfly6-7  waits politely

[19:31:04] <valhallasw>	 Dragonfly6-7: https://bugzilla.wikimedia.org/enter_bug.cgi?alias=&assigned_to=wikibugs-l%40lists.wikimedia.org&attach_text=&blocked=&bug_file_loc=http%3A%2F%2F&bug_severity=normal&bug_status=NEW&cf_browser=---&cf_platform=---&comment=&component=Database%20Queries&contenttypeentry=&contenttypemethod=autodetect&contenttypeselection=text%2Fplain&data=&dependson=&description=&form_name=enter_bug&keywords=&maketemplate=Remember%20values%20
[19:31:28] <valhallasw>	 hoping that entire URL got passed through.
[19:31:43] <scfc_de>	 valhallasw: Nope
[19:31:46] <valhallasw>	 http://tinyurl.com/4td856b
[19:31:51] <valhallasw>	 ^ shorter ;-)
[19:32:27] <Dragonfly6-7>	 thank you
[19:32:42] <scfc_de>	 valhallasw: Doesn't work either :-).
[19:32:45] <Dragonfly6-7>	 .... no, sorry, TinyUrl truncated that
[19:32:54] <Damianz>	 lol
[19:33:01] <scfc_de>	 Dragonfly6-7: Please file a bug under "Tool Labs tools" => "Database queries".
[19:33:33] <valhallasw>	 tinyurl did *what*? That's hilarious.
[19:34:20] <Damianz>	 Why does bugzilla need so much crap in the url
[19:35:58] <valhallasw>	 BZ doesn't, but I was too lazy to filter the crap out
[19:42:32] <Damianz>	 Oh yay, found the problem... an actual bug!
[19:48:03] <Dragonfly6-7>	 scfc_de - you're assuming a skillset not in evidence.
[19:52:22] <Damianz>	 Apparently the level of vandalism from ipv6 addresses is minimal :(
[19:55:32] <Damianz>	 https://github.com/DamianZaremba/cluebotng/commit/1601b8ccd564cb27e7dc87c3f08ca60156f05505 < boo php
[20:00:51] <valhallasw>	 Coren, are tool .forward files supposed to work yet?
[20:03:15] <valhallasw>	 Coren: I've tried various forms (pipe, then file, then a simple forward), but mail to tools.tsreports-dev@tools.wmflabs.org does not seem to get delivered anywhere
[20:03:21] <valhallasw>	 also not in /var/mail
[20:07:40] <Coren>	 valhallasw: From outside you mean?  No, that's what I'm working to debug now.
[20:09:09] <valhallasw>	 Okay, great. Then I'll fiddle around with it next week or so :-)
[20:11:44] <Coren>	 valhallasw: I should have it working well enough within the hour actually if you want to play with it.
[20:12:10] <Coren>	 valhallasw: But also, if you want to process mail, there is exactly one allowed command with pipe forwards:  "|jmail some_executable"
[20:12:49] <valhallasw>	 I see.
[20:13:02] <valhallasw>	 Is local mail delivery also possible?
[20:13:54] <valhallasw>	 I'm not sure yet which will be the easiest (a long-running job that polls a file or short running jobs via SGE)
[20:14:10] <Coren>	 valhallasw: No, forbidden by legal (we do not want to be storing mail for users).  Sorta-kinda indirectly because you could cheat and have jmail just run something that stores the mail, but it's not kosher -- it's mean for meal that will end up as input to a tool.
[20:14:27] <Coren>	 mail*
[20:14:41] <valhallasw>	 Right. I'm a bit afraid jmail would swamp the SGE queues, but I guess it will be OK
[20:14:48] <Coren>	 It has its own queue.
[20:15:15] <Coren>	 The aptly-named 'mailq'  :-)
[20:16:02] <Damianz>	 legal knows how to spoil all the fun
[20:17:31] <Coren>	 Legal is working had to make sure the fun can be had long-term.  :-)
[20:26:49] <benestar>	 Coren: hey
[20:40:36] <chrismcmahon>	 hi hashar are you around?  I have a question about session/cookie  storage on beta labs in recent times
[20:40:49] <hashar>	 chrismcmahon: yes I am 
[20:41:07] <hashar>	 I think the session are in memcached
[20:42:30] <chrismcmahon>	 hashar:  OK.  I have this strange issue for MobileFrontend where sometimes comparing tokens from the server fails.  background is:  https://bugzilla.wikimedia.org/show_bug.cgi?id=62614
[20:42:46] <andrewbogott>	 Coren: sorry to nag, but I'd like to get rsyncs going on labstore4 before the weekend… any idea which of the project dirs are the canonical/latest?  I could copy both, but that seems wrong...
[20:43:06] <andrewbogott>	 hashar, do you have everything you need for your many migrations?
[20:43:44] <hashar>	 andrewbogott: beta is pending the puppet mariadb module by Sean.  Might use some review there.  Then I guess he will be able set up the databases.
[20:44:05] <Coren>	 andrewbogott: Gimme a sec, I'm checking something.
[20:44:09] <andrewbogott>	 hashar:  ok… but you have all the instances/datadirs that you need copied already?
[20:44:17] <hashar>	 andrewbogott: bd808 created a local salt/puppet masters and taught me how to migrate our  beta instances to it. So we can cherry pick puppet patches :]
[20:44:39] <hashar>	 there is still a bunch of instances in integration project I dont know what to do with. I sent mail earlier today and waiting reply.
[20:44:56] <hashar>	 whenever I get repoies (probably on monday) I will follow up and either ask for moves or destroy them
[20:45:00] <hashar>	 so for tonight I am fine :]
[20:45:07] <andrewbogott>	 hashar: are you going to have a cron do a periodic fetch/rebase?  Or will that be managed by hand?
[20:45:17] <andrewbogott>	 hashar:  OK, just make me more bugs if you need more instances rescued.
[20:45:37] <hashar>	 before switching beta cluster I will have to rsync the beta /data/project . That is quite fast.
[20:45:39] <bd808>	 andrewbogott: At the moment we are doing it by had, but I'd like to automate the rebase
[20:45:46] <bd808>	 *by hand
[20:46:00] <hashar>	 the databases we will mysqldump  / mysql import
[20:46:18] <andrewbogott>	 bd808: it should be just a 10-line cron script, right?  fetch, rebase, on error rebase --abort and send an email
[20:46:33] <hashar>	 chrismcmahon: sorry that is way above my knowledge :-/   The mobile team should be able to help there sorry :/
[20:46:37] <Coren>	 andrewbogott: The only thing I can think of is that there was a rsync that went crazy at some point recently.
[20:47:08] <andrewbogott>	 Coren, did you do a test of touching a file from an instance and seeing what changes?  I'll do that now, if not.
[20:47:17] <bd808>	 andrewbogott: Yeah. You happen to have one lying around? That would be a great addition to the puppet::self role in general
[20:47:29] <jorm>	 w00t.  may is up on unicorn now.  thanks for all the help yesterday, guys.
[20:47:54] <andrewbogott>	 bd808: I don't, although I think there's functionality in the git::clone code to sync on every puppet run.  No notifications though.
[20:48:04] <hashar>	 git::clone  ensure => latest  probably does git pul
[20:48:05] <hashar>	 l
[20:48:10] <Coren>	 andrewbogott: AFAICT, the "subdirectory" isn't mounted (or mountable) from teh instances.
[20:48:19] <andrewbogott>	 Coren, awesome, I will ignore it.
[20:48:22] <andrewbogott>	 thanks
[20:48:38] <hashar>	 I would love to find a solution with a dedicated branch in puppet for beta with some job ensuring the branch is mergeable on master
[20:48:57] <hashar>	 this way we can put our patches in Gerrit and merge them while waiting for them to be merged back in prod
[20:51:24] <Coren>	 andrewbogott: Hang on.  Something odd is going on with some of those.
[20:51:31] * andrewbogott  hangs

[20:53:31] <andrewbogott>	 Coren, I guess those dirs aren't managed by manage-nfs-volumes-daemon?
[20:53:37] <andrewbogott>	 Since it doesn't seem to be running on labstore4
[20:53:43] <andrewbogott>	 labstore3 is off, right?
[20:53:55] <Coren>	 Yeah, labstore3 is off (that's the one with the buster controller)
[20:54:21] <Coren>	 andrewbogott: Something really odd is going on, depending on the project.  Give me time to do a couple more tests.
[20:54:29] <andrewbogott>	 sure, take your time.
[20:58:54] <hashar>	 andrewbogott: we have some patches for labslvm mounting that might be interesting to you
[20:59:14] <hashar>	 andrewbogott: bd808 adjusted your role::labs::lvm::mnt to accept a different mount point :-] https://gerrit.wikimedia.org/r/#/c/119524/
[20:59:19] * Damianz  un-hangs andrewbogott

[20:59:26] <hashar>	 Damianz: thx :]
[20:59:42] <Coren>	 andrewbogott: Yeah, okay, AFAICT what I said earlier seems to hold.  /srv/project/project seems to be a rsync of /srv/project done some time between Oct 10 and Dec 15
[21:00:02] <Coren>	 I mean, /srv/$project/$project
[21:00:04] <Coren>	 and so on.
[21:01:19] <Coren>	 I got confused because for a few projects ./home and ./project and empty, while the subdir isn't; but that's projects that have migrated and blew away their data.
[21:02:34] <hashar>	 !log integration pmtpa /data/project is now emptied up!!! \O/
[21:02:37] <labs-morebots>	 Logged the message, Master
[21:07:09] <hashar>	 heading to bed. Have a good week-end !
[21:08:14] <andrewbogott>	 Coren, ok -- that's not so scary.
[21:08:18] <andrewbogott>	 Thanks for investigating.
[21:12:31] <liangent>	 is it possible (and easy) to make use of http://tools.wmflabs.org/?status 's code to view my own SGE cluster?
[21:12:49] <liangent>	 or is there a good qstat viewer to recommend?
[21:26:01] <Coren>	 liangent: It should be fairly simple to adapt; IIRC scfc_de has ported it to the toolserver for use with its grid.
[21:26:53] <Withoutaname>	 Does anyone have a mass unblock script?
[21:27:40] <andrewbogott>	 Coren, if one of those nfs vols contains only '.snapshot/' then I can regard it as empty, right?
[21:27:55] <Coren>	 Withoutaname: I don't remember having ever seen one; that doesn't seem to be a frequent operation.
[21:27:58] <Coren>	 andrewbogott: Yep.
[21:28:32] <andrewbogott>	 Coren: well… I should probably ask a different question as well.  Is copying directly out of e.g. /srv/$project/home going to get me a proper image of the volume?  Or do I have to mount it elsewhere?
[21:29:00] <andrewbogott>	 (With gluster I definitely had to mount it, because there were duplication/cow/etc. shenanigans with the raw file storage)
[21:29:27] <Coren>	 On labstore4 /srv/$project/{home,project} are the canonical sources.
[21:29:55] <andrewbogott>	 great.
[21:30:28] <Withoutaname>	 ok guess I'll try to make one myself :( with whatever scant knowledge I have of python
[21:30:58] <Coren>	 Withoutaname: Beware that most projects (enwiki in particular) require an approval process for admin bots.
[21:31:35] <Withoutaname>	 thanks for the heads up
[21:31:50] <Withoutaname>	 Can I run it on toollabs though?
[21:32:34] <Damianz>	 I always think the BAG is such a funny name
[21:33:02] <Withoutaname>	 what else can you think of for it
[21:33:17] <Withoutaname>	 "bot owners' group"? "bot approvals committee"?
[21:34:47] <Damianz>	 'The dude', because everyone should get approval from the dude
[21:37:02] <liangent>	 Coren: and where's its source..?
[21:39:48] <Nettrom>	 Withoutaname: you might want to look into the Pywikibot framework, it takes away a lot of the hassles
[21:40:48] <Withoutaname>	 I'll read the documentation for it and try to see if I can make heads or tails
[21:42:16] <liangent>	 https://git.wikimedia.org/blob/labs%2Ftoollabs/49472c7b15ed0e7f81359ad88186a4fb14ef3695/www%2Fcontent%2Fstatus.php right?
[21:42:43] <Mpaa>	 Hi. Need some help with ssh connection. I uploaded ssh key to bot gerrit and wikitech. Gerrit is working and bastion no, any clue?
[21:44:00] <Mpaa>	 with gerrit I get:
[21:44:08] <Mpaa>	 debug1: Offering RSA public key: /home/squiddy/.ssh/id_rsa
[21:44:19] <Mpaa>	 debug1: Offering RSA public key: /home/squiddy/.ssh/id_rsa
[21:44:30] <Mpaa>	 debug1: Server accepts key: pkalg ssh-rsa blen 279
[21:45:08] <Mpaa>	 with mpaa@bastion.wmflabs.org, I do not get anything and move on until it fails
[22:00:34] <Magog_the_Ogre>	 have we had any luck with restoring the logs and/or restoring the cron tabs?
[22:03:35] <Magog_the_Ogre>	 I cannot restore my bot until I get the log back
[22:18:49] <awjr>	 on tool labs, it does not appear that my .bashrc is being sourced when i log in - is that expected?
[22:19:24] <awjr>	 Coren: ^
[22:39:20] <Coren>	 awjr: Yes; when logging in interactively only your .profile is sourced -- most people source their .bashrc from /it/ to have consistent environments.
[22:39:30] <Coren>	 (The default .profile doesn't)
[22:40:29] <Coren>	 Magog_the_Ogre: I can restore the crontab easily enough; though I'm a little surprised you don't have it.  What's your tool name again?
[22:40:43] <Coren>	 Magog_the_Ogre: Logfiles I have to do by hand though.  What's the filename?
[22:45:44] <awjr>	 roger that Coren, thanks
[22:46:43] <awjr>	 i didnt have a .profile so my .profile was certainly not sourcing .bashrc :p
[22:46:44] <awjr>	 problem solved.
[22:46:49] <Coren>	 liangent: It's in git.  labs/toollabs (in the www directory)
[22:47:02] <Coren>	 liangent: Alternately, you can just look in ~tools.admin/public_html
[22:47:42] <Coren>	 Withoutaname: Yeah, you can run it here.
[22:49:28] <bd808>	 Coren: I'm trying to create some new instances for the deploymnet-prep project in eqiad and they are blowing up on initial boot with ldap connection timeout errors to the instance console.
[22:49:48] <Coren>	 o_O
[22:49:49] <bd808>	 Instances are deploymnet-es[012]
[22:50:36] <Coren>	 bd808: Checking.
[22:50:55] <bd808>	 The last time I saw this Andrew eventually found out the server they we placed on was missing a network cable
[22:51:23] <Coren>	 bd808: That's not the case here; the box is actually alive and reachable.
[22:51:31] <bd808>	 i-0000026[cde] if that helps
[22:52:00] <Coren>	 It also seems to have only a partial puppet run done.
[22:53:00] * bd808  is jinxed in eqiad labs ;)

[23:00:06] <Coren>	 bd808: I'm honestly not seeing anything wrong with your instances, except for the fact that it does't look like puppet ran correctly.
[23:00:38] <Coren>	 bd808: LDAP is up, at least, so I don't know where the connection errors might come from except, perhaps, that the config files weren't puppeted in?
[23:00:57] <Coren>	 Stupid question, have you tried rebooting them to force a new puppet run?
[23:01:46] <bd808>	 Coren: So … reboot and hope?
[23:01:53] <bd808>	 jinx
[23:02:02] <bd808>	 I can totally do that
[23:02:15] <bd808>	 and I can nuke them and try again if that doesn't work
[23:04:23] <bd808>	 "deployment-es0 puppet-agent[1200]: Could not request certificate: Connection timed out"
[23:04:51] * bd808  will blow them up and start over

[23:04:56] <Coren>	 Wait, that has nothing to do with LDAP; that's the puppet master being out of its gourd (which would explain why you don't have a complete puppet run)
[23:06:38] <bd808>	 They look jacked up. "Could not set 'directory on ensure: File exists - /var/run/puppet"
[23:07:14] <bd808>	 puppet agent failed to start on reboot
[23:07:36] <Coren>	 Well yeah, if it doesn't have a cert then it can't work.
[23:07:55] <Coren>	 bd808: Try just one at first.  I want to see why the first run failed.
[23:08:25] <bd808>	 Ok. I'll start with es0
[23:11:13] <bd808>	 Coren: Could not parse configuration file: Certificate names must be lower case; see #1168
[23:11:20] <bd808>	 Coren: Starting puppet agent       [80G [74G[[31mfail[39;49m]
[23:12:05] <bd808>	 That's initial boot on the "new" es0 (i-0000026f)
[23:14:05] <valhallasw>	 Coren: did you need any more info on the jobs running on the tomcat server? It's still happening.
[23:15:34] <Coren>	 Not really, I don't have the bandwidth to look at it just yet.
[23:16:23] * Damianz  lowers the pps going to Coren

[23:16:25] <valhallasw>	 OK. Should I just make a bugzilla bug with all the data I have?
[23:16:41] <Coren>	 valhallasw: That sounds sane.
[23:19:56] <bd808>	 Coren: Same final result "deployment-es0 puppet-agent[1194]: Could not request certificate: Connection timed out - connect(2)"
[23:22:32] <andrewbogott>	 bd808: what project is this?
[23:22:44] <bd808>	 andrewbogott: deployment-prep
[23:24:05] <bd808>	 4 m1.large image creations in a row have died on first boot with logs full of ldap timeouts from nslcd followed by failure to get the cert from the puppet master
[23:24:11] <valhallasw>	 Coren: ok, done: https://bugzilla.wikimedia.org/show_bug.cgi?id=62942
[23:24:55] <andrewbogott>	 bd808: is it just large instances that fail?
[23:25:16] <bd808>	 andrewbogott: I haven't tried other sizes today
[23:26:49] <bd808>	 I'm setting up the cirrus cluster. Created 3 m1.large in rapid succession, got an "instance not created" error when trying to create the 4th. Went to console of es0 (first one made) and saw these errors.
[23:27:27] <bd808>	 The next two instances showed the same error logs. Nuked es0 and created it again
[23:27:30] <bd808>	 same outcome
[23:28:35] <^demon|vcu>	 Since we're nuking...
[23:28:50] <^demon|vcu>	 Call them elastic[\d+] like production.
[23:28:53] <andrewbogott>	 bd808: your project was pushed right up against the quota for cores.  I don't know if that was the problem, but… I just raised it quite a bit.
[23:29:02] <^demon|vcu>	 es[\d+] confuses people with external store.
[23:29:26] <bd808>	 ^demon|vcu: can do!
[23:51:30] <andrewbogott>	 bd808: have you ever had a large size instance work?
[23:51:50] <andrewbogott>	 I just tried, small is working but large it not… trying medium now
[23:52:06] <andrewbogott>	 Why would that affect network connectivity?  I cannot guess.
[23:52:20] <bd808>	 andrewbogott: That's a good question. I don't know that I've tried to build one before. small and xl have worked in the past
[23:53:52] <bd808>	 I'm sure Nik wouldn't mind having xlarge instances if that's the case
[23:59:42] <PiRCarre>	 andrewbogott: hi http://icinga.wmflabs.org/cgi-bin/icinga/status.cgi?hostgroup=utrs&style=detail
[23:59:46] <PiRCarre>	 https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#UTRS_has_fallen_over
[23:59:48] <PiRCarre>	 What's up?
[23:59:59] <bd808>	 andrewbogott: Not totally confirmed yet, but it looks like xlarge may be having the same issues