[00:05:46] @seen addshore [00:05:46] rschen7754: addshore is in here, right now [00:14:20] @notify addshore [00:14:20] This user is now online in #huggle so I will let you know when they show some activity (talk etc) [00:21:16] * Ryan_Lane yawns [00:22:19] Ryan_Lane: coffee? :) [00:22:30] nah. already had a beer [00:22:38] and had a giant coffee and tea earlier [00:22:46] huh, we should do that. but we're at starbucks... [00:22:57] 4 wikipedians here atm :) [00:24:13] legoktm: what about dive into python and pythonchallenge? [00:24:30] what about them? [00:24:45] legoktm: was just responding to 14 20:18:13 < legoktm> I learned Python from Wikibooks: https://en.wikibooks.org/wiki/Non-Programmer%27s_Tutorial_for_Python_2.6 [00:24:53] Oh [00:24:54] Idk [00:24:58] Just didnt use them [00:25:10] It was Wikibooks + stackoverflow [02:01:18] the java compiler and java runtime are the same version, riiiight? [02:02:50] java version "1.6.0_27" [02:02:51] OpenJDK Runtime Environment (IcedTea6 1.12.3) (6b27-1.12.3-0ubuntu1~12.04.1) [02:02:53] OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode) [02:02:54] javac 1.7.0_15 [02:02:56] :( [02:03:13] * rschen7754 facepalms [02:05:55] javac: target release 1.6 conflicts with default source release 1.7 [07:04:53] petan: Coren there's a ghost in labs' shell. [07:04:53] err, tools labs shell, at lesat [07:05:02] I killed *everything*, set it up all over again, same issue. [07:05:14] I made sure that everything that was ever open('d) [07:06:50] was killed [07:06:54] (from strace) [07:06:57] and the problem persists [07:07:02] so I'm setting it up on my VPS now [07:25:47] I'm going to give it one last shot and build git from source [07:28:18] * YuviPanda builds git [07:30:18] this is insane. I'm not going to hunt for curl and install that, etc. [07:30:21] * YuviPanda goes to set it up on his vps for now [07:38:19] * YuviPanda builds curl [07:41:53] hey petan. around? [07:57:58] !tunnel [07:57:58] ssh -f user@bastion.wmflabs.org -L :server: -N Example for sftp "ssh chewbacca@bastion.wmflabs.org -L 6000:bots-1:22 -N" will open bots-1:22 as localhost:6000 [07:58:13] * YuviPanda pings petan [07:58:18] pig [07:58:21] pong [07:58:33] petan: I'm manually locally compiling git [07:58:43] where [07:58:45] petan: can you install the -dev packages? [07:58:46] for me? [07:58:52] sure [07:58:53] apt-get install libcurl4-gnutls-dev libexpat1-dev gettext libz-dev libssl-dev [07:58:55] but do this on -dev pls [07:59:00] not on -login [07:59:06] hmm? [07:59:11] i did 'become suchaserver' [07:59:13] and started building [07:59:17] it's only a local copy [07:59:19] mhm... [07:59:22] ok [07:59:23] sec [07:59:25] in a private directory. [07:59:29] not going to affect anyone else [08:00:16] !log tools petrb: installing dev packages needed for YuviPanda on login box [08:00:18] Logged the message, Master [08:01:25] petan: Hello :) Have you got my message? --Could you run "sudo a2enmod suexec" at bots-apache and restart apache2 please. [08:01:40] oh really suexec [08:01:52] petan: thank you :) [08:01:58] Nullzero no and yes [08:02:13] didn't get any message but sure [08:02:20] petan: were they installed? [08:02:28] YuviPanda working on it [08:02:32] ah, okay :) [08:04:24] YuviPanda it's on -login [08:04:28] whee [08:04:44] petan: <3 thanks [08:04:51] yw [08:05:44] Nullzero done [08:05:46] !log bots petrb: restarted http to get new module [08:05:48] Logged the message, Master [08:05:53] petan: thx ;) [08:06:20] petan: don't you use puppet to maintain your config? [08:08:16] * YuviPanda watches git build [08:27:30] paten: can you send me the output when running "sudo /usr/lib/apache2/suexec -V"? I think that docroot is incorrect. In /var/log/apache2/suexec.log, it shows something like "cannot get docroot information", "command not in docroot". I don't know how to fix it, though. [08:27:41] petan: [08:31:25] petan: morebots is dead; Is that running on the bots labs project? ( bug is https://bugzilla.wikimedia.org/show_bug.cgi?id=47228 ) [08:32:13] hmm apparently not [09:07:08] back [09:07:17] hashar at some point [09:07:55] !log bots petrb: restarted morebots [09:08:01] !log bots petrb: restarted morebots [09:08:03] Logged the message, Master [09:08:32] hashar but that is production morebots :D [09:08:35] what you mean [09:08:46] that doesn't run on labs... that run on some box I have no access to [09:09:23] petan: ah in prod. Thx for confirming :-] [09:12:24] !tunnel [09:12:24] ssh -f user@bastion.wmflabs.org -L :server: -N Example for sftp "ssh chewbacca@bastion.wmflabs.org -L 6000:bots-1:22 -N" will open bots-1:22 as localhost:6000 [09:12:27] :o [10:07:58] addshore your bot stopped running or what? [10:07:58] addshore bsql is totally unloaded :D [10:08:02] yes :p [10:08:05] fixing a bug :P [10:08:08] ah [10:08:25] addshore: hi! [10:12:37] HI rschen7754 ! [10:12:45] did you get my talk page notice? [10:12:50] *nite [10:12:51] *note [10:18:58] which talk page? :D [10:19:16] enwiki [10:23:35] checking :) [10:48:19] !log bots addshore: bots-gs installed python-twisted, sqlite, sqlit3. Updated jre [10:48:21] Logged the message, Master [11:01:11] good evening [12:00:08] evening hippietrail [12:30:36] i was thinking of reimplementing some toolserver stuff of mine on labs [12:30:45] such as? :) [12:31:20] my #wiktionary bot, the en.wiktionary random page per language tool [12:34:36] [bz] (NEW - created by: Željko Filipin, priority: High - normal) [Bug 47203] Math extension broken at beta.wmflabs.org: "No backend defined with the name `global-multiwrite`." or "Missing texvc executable" - https://bugzilla.wikimedia.org/show_bug.cgi?id=47203 [12:49:29] !log deployment-prep -cache-upload03 refreshing local puppet repo [12:49:32] Logged the message, Master [12:58:58] Morning labs. [12:59:05] Hi Coren [13:00:54] Oh, hey, a Sumanah. Rare to see you here. [13:01:15] Oh, I'm here a bit more often than you'd think, I just don't speak much [13:03:23] * Coren struggles against python. [13:04:35] hey ottomata [13:05:18] hihi [13:06:00] hey i'm confused by a step on [[Help:Getting Started]] [13:07:10] i'm supposed to upload my SSH key via the openstack prefs page, but when i go there i seem to already have one there [13:07:42] is the page out of date or just not clarifying that i should expect to have various keys [13:13:13] hippietrail: You, normally, you'd have just the ones you put there yourself. Is it possible you simply forgot you already put one up? [13:13:51] no i never put one there. i was just following the steps for the first time and saw there was already one there [13:14:08] ... as far as i know ... is there a way to see if i did it so long ago that i forgot i'd done it? [13:16:12] Not that I know of. That said, it's possible that they ported over keys from some other system (svn?) when they initially set things up -- but that was before my time. :-) [13:16:28] If you don't know the key, feel free to remove it and put a new one in place. [13:17:24] hmm i didn't try to compare it to my key - i used to use with the wikimedia repo way back when ... [13:18:09] That may well be it, then. [13:19:31] well yes it is the same! [13:20:26] i assumed it must've been some new generated one (-: [13:27:15] cool logged in with putty (-: [13:30:20] hey Coren [13:30:30] so mark had the exact problem I was having / am having [13:30:34] it turned out to be a glusterfs issue [13:30:42] can you check if that is the case again? [13:30:48] apparently a 'gluster brick got stuck' [13:30:53] Heh. Thankfully, glusterfs is going away. [13:31:09] ... on tools? [13:31:09] the exact same setup works on my VPS and on other servers. [13:31:11] yes [13:31:12] on tools [13:31:13] well [13:31:16] his problem was on Labs [13:31:17] not tools [13:31:18] but mine is on tools [13:32:59] hey Ryan_Lane. [13:33:11] mark told me he had the same git issue I was having on labs a while back [13:33:21] and that you unstuck a glusterfs 'brick' to fix it for him? [13:33:46] do you remember any details? [13:33:57] Coren: tools labs is on Glusterfs, right? [13:34:12] YuviPanda: For now, yeah. [13:34:43] Coren: any estimate for db replication yet? ;p [13:34:51] Ryan_Lane: Speaking of, do you have a pointer to a python-primer-for-perl-hackers that isn't a condescending screed? [13:35:08] Coren: hmm, I'm thinking of just running this in my VPS for now [13:35:14] addshore: No further news than my last status email. :-) [13:37:31] :< [13:39:15] addshore: Three word summary: Still on schedule. :-) [13:40:24] Coren: is there a way for me to access a non-glusterfs filesystem from tools labs? [13:41:04] YuviPanda: If you arm yourself with 24h of patience or so, yes. I'm almost done with the new fileserver. [13:41:23] hmm, okay :) [13:49:22] hmm [13:49:22] I got a bug [13:49:42] I have asked to create an instance a few minutes ago and it is already available. Has something changed recently in the way we create instances? [13:49:48] cause it is blazing fast now. [13:50:06] yeah, it has changed [13:50:17] i think Ryan made it so the images were much more complete [13:50:19] and I don't have my /home :-((( [13:50:24] so you can log in before the initial puppet run [13:50:52] run puppet once and maybe things will fix themselves...actually [13:50:55] run it and tell me how it goes [13:51:00] i created 2 new instances on friday [13:51:09] no puppet log on the instance :-D [13:51:10] and they're puppet.conf files were not right when they started up [13:51:19] so they couldnt' actually run puppet [13:51:24] i thought it was a temporary thing [13:51:31] but i just tweaked them this morning and now they work [13:51:35] what happens when you run puppet? [13:51:50] it is being run by some process [13:51:56] but it surely log stuff in syslog :-] [13:52:01] just not in /var/log/puppet.log [13:52:04] not a big issue though [13:52:26] at least it is fast [13:52:28] \O/ [13:59:23] [bz] (NEW - created by: Antoine "hashar" Musso, priority: Unprioritized - normal) [Bug 47243] puppet does not log to /var/log/puppet.log or /var/log/puppet/puppet.log - https://bugzilla.wikimedia.org/show_bug.cgi?id=47243 [14:02:14] bah [14:02:18] even ganglia monitor does not work :-d [14:02:22] [bz] (NEW - created by: Antoine "hashar" Musso, priority: Unprioritized - normal) [Bug 47244] ganglia-monitor does not start (override_hostname not understood) - https://bugzilla.wikimedia.org/show_bug.cgi?id=47244 [14:03:52] !log deployment-prep Rebooting the fresh deployment-cache-upload-test6 instance [14:03:55] Logged the message, Master [14:07:45] hmm [14:08:27] now I have got to find out how to have /home mounted again [14:10:45] hashar, ottomata: can you please file bugs with issues you've found with the new image? [14:12:34] looks like the manage-volumes daemon hasn't added this instance yet [14:12:48] which likely means its broken [14:13:22] hm. it says it updated it a minute ago [14:13:51] Ryan_Lane: I did, against the Infrastructure component [14:14:04] I suspect nagios-nrpe starts with the wrong user as well (aka nagios instead of icinga) [14:14:18] maybe so, yeah [14:14:33] that's done by puppet [14:14:40] does the first puppet run fail? [14:14:40] we should have a icinga-nrpe package :-] [14:14:44] yes, we should [14:14:49] well [14:15:02] otherwise I have to do this stuff in the bootstrapping, which is silly [14:15:04] puppet will attempt to restart nrpe over and over because the init script can't find it [14:15:17] so we have to manually kill nrpe and start the service again [14:15:18] that fix puppet [14:15:22] ugh [14:15:34] that definitely needs to be fixed [14:15:43] (well that does not fix, that just let it run the init script with 'status' and get a correct result) [14:15:44] we should have just kept the user as nagios [14:15:50] or we should have made a custom package [14:15:56] or migrated the package to use icinga :-] [14:16:07] hm. gluster daemon may be dead on labstore1 or labstore2 [14:16:15] or just make a generic nrpe-server package that will take the username out of /etc/default/nrpe [14:25:26] Ryan_Lane, I think puppet looks fine on a new instance now [14:25:30] i just created another one [14:25:35] and puppet.conf looks good [14:25:43] the ones I created on friday didn't have server set [14:25:51] but the new one I just created is fine [14:26:19] o.O [14:26:25] that doesn't make any sense :D [14:26:54] I wonder if this was due to the search limit length issue I fixed [14:27:41] $ ls /home [14:27:41] gmetric hashar icinga ubuntu [14:27:41] $ [14:27:52] not sure they those home dirs should be created there :-] [14:29:01] bah [14:29:03] I found the issue [14:29:18] [bz] (NEW - created by: Antoine "hashar" Musso, priority: Unprioritized - normal) [Bug 47245] gmetric , icinga and ubuntu user dirs created in /home - https://bugzilla.wikimedia.org/show_bug.cgi?id=47245 [14:29:20] I deleted a volume the other day and gluster said it failed on a node [14:29:23] biking to friend's office, back in 20ish [14:29:29] it left a directory [14:29:43] so, when gluster went to rotate its logs on that host, it didn't restart [14:29:44] it just died [14:31:47] Coren: I dont know what the schedule is though ;p [14:34:23] addshore: Gunning to have it up by the Hackaton (end of may) [14:34:32] oh yes, I remember :P [14:34:46] you gonna be at said hackathon? :P [14:34:54] addshore: Ayup. [14:35:11] god dam, im still not sure if I can make it >.< [14:35:11] addshore: You get three guesses as to what I'll be doing there. :-) [14:36:36] drinking? [14:39:11] time to look into db replication.. [14:40:40] Coren: please make sure labstore3 and labstore4 are not running gluster [14:40:43] server [14:40:51] and make sure it's purged [14:40:54] it's causing issues [14:41:28] Oy! Puppet fun. Forgot to check that. [14:41:35] Ryan_Lane: Fixing now. [14:43:31] Ah. Node config still had labstore[1-4] even though you had removed 3-4 elsewhere. [14:43:34] hashar: /home and /data/project are working now [14:44:00] I only removed it from a script [14:44:13] I didn't remove it from anything else. I had assumed you would ;) [14:44:37] I don't have access to any instance right now though :D [14:44:38] Yeah, and I had assumed you did. Doing so now. :-) [14:46:27] hashar: why not? [14:46:44] ugh. seems /public/keys is missing [14:47:35] what the fuck gluster? [14:48:30] Ryan_Lane: https://gerrit.wikimedia.org/r/#/c/59143/ [14:49:04] Ryan_Lane: Just split them for now. [14:49:05] Coren: you need aggregators [14:49:11] $ganglia_aggregator = "true" [14:49:18] also [14:49:23] don't use cluster nfs [14:49:24] Oh, I noted you only had them on 1-2, I expected that was on purpose. :-) [14:49:29] use cluster labsnfs [14:49:40] you'll need to add a new cluster in ganglia.pp for it [14:49:49] I had patterned it on not being 'labsgluster' [14:49:54] :-) [14:50:07] I did. Oh. Forgot to git add it. [14:50:09] hashar: should be fixed now [14:50:35] well, we've had nfs servers in production historially [14:50:38] and still do [14:50:43] *historically [14:50:49] ok. I have to run [14:50:54] back in a while [14:59:57] Coren how complicated would it be to get some hardware / even some cheapest possible for a dedicated sql server? [15:00:20] addshore raised some doubts regarding performance of current sql server :D [15:02:11] petan: Not especially hard, but also a bit moot. The new replication servers (where user DB can live) is seriously kickass. :-) [15:02:44] well, but it isn't available [15:02:57] yeah "Exiting; no certificate found and waitforcert is disabled" ;D [15:03:16] * hashar dishes out the instance [15:03:31] Coren if there is some existing box we could install db on... :P [15:03:40] petan: The timeline to set up a physical box to use as a mysql server would be pretty much the same as that of getting the new DB available; I'm not sure what we'd gain. [15:03:44] idc but I guess it's again pretty restricted [15:04:10] what do you mean [15:04:23] what that "user DB" actually is? [15:04:36] petan: I mean that by the time we set aside and configure the hardware for a dedicated mysql box, we'd already have the big DB up most likely. [15:04:58] petan: You can still ask Ryan; he's the gatekeeper to using hardware. :-) [15:05:09] "big DB" is what [15:05:15] you mean replicated production? [15:05:18] * Coren nods. [15:05:21] ah [15:05:39] ok but this is completely different db instance would that be hosted on same hw? [15:06:02] I would ask Ryan but he's not here [15:06:02] :D [15:06:02] Why would it be completely different? Labs users can create DBs on that cluster too. [15:06:08] aha [15:06:16] I thought that it's going to be like toolserver [15:06:27] Coren: just a thought from where i am :P [15:06:30] like a read only db server [15:06:36] then another mysql server for users [15:06:39] the dbs will be on tools first? [15:07:06] so to migrate my db from bots to the new db I will need to have it on tools? :P [15:07:10] addshore: Yeah; it's going to be the trial run. bots- shortly after and, once tested, the other projects. [15:07:33] how long a time perios is shortly after? ;p [15:07:39] addshore I think you can actually access both db's simultaneously [15:08:07] addshore: That's unknown. It depends entirely on how well it goes on tools, clearly. [15:08:15] addshore tbh it would be good to move your bot to tools and keep using bsql for now until new server is ready [15:08:32] addshore: The more testing we get to do on tools, the faster it will be declared ready for bots I suppose. [15:08:32] I see little point in moving huge db from one virtual box to another virtual box and then to another box [15:15:31] xD [15:15:59] right [15:16:01] well [15:16:06] give me 5 or 10 mins ;p [15:25:44] Coren: what is the easiest way to update files in /project/addbot owned by local-addbot? [15:26:04] define "update files"? [15:26:32] replace with one I have on my pc ;p [15:27:34] What I'd do is copy a tarball/zip to your account first with scp/sftp, then unpack them in the tool directory from the tool user. [15:27:41] I.e.: [15:27:44] $ become addbot [15:27:46] [bz] (NEW - created by: Antoine "hashar" Musso, priority: Unprioritized - normal) [Bug 47249] puppetmaster::self unusuable - https://bugzilla.wikimedia.org/show_bug.cgi?id=47249 [15:27:58] $ unzip ~addshore/the.zip [15:28:59] If you have the files in a git or svn repo somewhere, you can also just check them out from the tool account. [15:32:58] Hi [15:33:25] Coren: Is it possible to install 7zip on Tools? [15:34:15] Darkdadaah yes [15:34:21] Darkdadaah: Do you need the rar support? [15:34:54] Coren: nope. [15:35:10] Darkdadaah: Then yes, like petan said. :-) [15:36:51] Thanks :) Who can do that? [15:37:04] I can do that :P [15:37:18] where do you need it to be, just on -login or on exec nodes too for some reason [15:38:03] Both: I may use it directly by hand or through jobs. [15:38:12] ok [15:41:29] !log tools petrb: installing p7zip everywhere [15:41:31] Logged the message, Master [15:44:02] Darkdadaah done [15:44:21] \o/ Thanks ! [15:46:33] Coren where is teh sw page [15:46:35] on wiki [15:46:51] I think it was somewhere in ur userspace [15:47:14] got that [15:47:40] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/Notepad was modified, changed by Petrb link https://www.mediawiki.org/w/index.php?diff=673712 edit summary: [+6] + [15:48:02] petan: http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Notepad [15:48:09] Yeah, that. :-) [15:51:13] !toolspad is http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Notepad [15:51:14] Key was added [15:55:51] Coren: with jsub is there a way to supress output files? [15:56:19] addshore: Entirely? Not that I know of, but you can set them to /dev/null [15:56:45] -o /dev/null -e /dev/null [15:57:27] Completely getting rid of the error output might not be a good idea, however, because if things crash you'll have to trace of it. [15:58:00] no* trace [16:05:11] ill send all errors to a single file [16:05:17] [= [16:05:28] That works. [16:06:47] jstart is the same as jsub but it starts a continuous job? [16:07:41] addshore: It's essentially jsub -once -continuous [16:18:26] Coren: if you see Yuvi, ask him if git is working on gluster now [16:18:41] I'm betting labstore2 was broken the entire time he was trying [16:19:37] hmm Coren one thing I would say is it would be great if queued jobs could be executed faster [16:34:43] addshore: "faster"? IIRC, the master checks the queue every 30s [16:34:52] Ryan_Lane: hah! [16:34:56] Ryan_Lane: let me try! [16:35:11] Coren: how many can it deply after each check? [16:35:19] *deploy [16:35:27] "Could not intern from pson" <- my favorite error message ever [16:36:16] addshore: I think its starts everything there is room for; but I've admitedly been a bit frugal in the number of open slots to keep performance acceptable. [16:36:25] hmm [16:36:30] Coren: I don't know any perl -> python guides [16:36:32] i dont think i can run on tools then [16:36:46] Ryan_Lane: I found some. They're all condesending screeds. :-) [16:36:48] Ryan_Lane: no change, really. still the same [16:37:03] Coren: I take up about 200 slots for this code [16:37:04] do they all start of with "So, you've decided to finally use a good language…"? [16:37:08] *off [16:37:23] addshore: That seems very much overkill. [16:37:34] nope, one per wiki :P [16:38:57] Ryan_Lane: Most of the time is "compare how this pretty python code is superior to this badly written perl code; clearly if you write perl you are a poor schmuck". I've yet to find one that actually shows how perl idoms map to python, every tutorial I found seem to presume you want to do web stuff and know little but visual basic. [16:39:39] idoms? [16:39:48] Ryan_Lane: yup, tried again. No change [16:39:56] GrumpyPanda: ok [16:39:57] I'm setting it up on my VPS for now. [16:40:14] I'll move it back later. [16:40:30] I don't know what tutorials you're looking at. most shouldn't be aimed at web [16:40:36] Perl has more than one way of doing everything, but most people tend to use the common idioms for the normal stuff. while(<>), foreach my $foo in (sort keys %hash), and so on. [16:40:45] oh. idioms :D [16:41:00] * Coren belatedly notes the typo. [16:41:09] Coren: http://learnpythonthehardway.org/ [16:41:36] ^^ that's a really good python tutorial [16:42:25] ... "A good first program" -> a bunch of prints hello world? Seriously? [16:42:41] * Coren despairs. [16:42:44] what language tutorial doesn't start with hello world? [16:42:52] even C tutorials do [16:43:34] I've yet to see one that spends a chapter on it and has exercises saying "make it print something different". [16:46:25] Different people learn different ways. "learn Python the hard way" might be meant more for people for whom Python is their first programming language - I think that's what I heard from others who've done it [16:46:40] it's an exercise based tutorial [16:46:48] it's meant to make you actually do things [16:46:54]