[00:40:37] hi kraigparkinson robla [00:40:58] kraigparkinson: i haven't heard from drdee for a bit. [00:41:01] fwiw [00:41:44] no worries. I figure he's done for the day. [01:04:19] word. [01:17:17] me too, I'm out. [01:19:28] hi [01:34:50] drdee: I'm getting a 500 internal server error when i try to access mingle [01:35:15] just sent an email in response to the mingle request with a list of issues that relate to hosting umapi on stat1001 [01:36:47] if it's possible that I can crunch through those issues with andrew are some ops person early next [01:36:49] week [01:38:06] working via IRC with someone who has access should do it. The work may entail some commits to operations-puppet but I'm confident that it won't be extensive. [06:53:41] erosen: ping? [06:53:48] sup GroggyPanda [06:53:52] erosen: heya [06:54:05] ori-l: is trying to puppetize dependencies for limnpy [06:54:18] erosen: any idea how hard is it going to be to use pandas 0.7 instead of 0.9? [06:54:35] hmm [06:54:43] i can't say for sure [06:55:09] i suspect it will cause a few minor issues [06:55:21] but they should be minor [06:55:29] GroggyPanda: whats the use case? [06:56:03] erosen: use case is puppetizing stat1001 [06:56:11] since apparently we... won't get virtualenv? [06:56:23] oh yeah, i've slowly been hearing about this [06:56:42] ori-l: if we do get virtualenv, the only packages needed are python-dev and python-virtualenv, whatever versions are there in the repo [06:57:08] GroggyPanda: just out of curiosity, is there any reason not make the files on stat1 and then rsync them or something? [06:57:25] leslie might actually kill you [06:57:29] hehe [06:57:34] in a gruesome way, too [06:57:38] i see [06:57:40] good to know [06:58:16] GroggyPanda: well in that case, I'm sure we can make it work with pandas 0.7, it just might require a litlte tweaking [06:58:17] :D [06:58:38] ok, so i'll go ahead and submit a patch for 0.7 & co [06:59:03] no reason we couldn't change it later. but this way at least you can run things and see whether or not they work [06:59:20] ori-l: I can do that right now, actually. [06:59:23] * GroggyPanda tries [07:06:47] ori-l: can you tell me the version of python-mysql from the ubuntu repo? [07:06:54] I am reproducing a virtualenv with exact versions [07:07:30] 1.2.3-1build1 [07:07:32] GroggyPanda: which version of pandas did you find a deb for, btw. I'm doing the same thing [07:07:55] erosen: ori-l found 0.7.0 i think [07:08:27] apt-cache show python-pandas | grep ^Version [07:08:36] 0.7.0-1 [07:09:36] ori-l: python-jinja2 version? [07:10:16] ori-l: wait, is this the same as on stat1? [07:10:40] so, 2.6 [07:12:27] this is going to power some kind of web service, right? [07:12:31] ori-l: no? [07:12:42] no? [07:12:44] well [07:12:45] eventually [07:12:50] good enough [07:12:51] but the web service is running from labs [07:12:56] hmm [07:13:08] i'm looking for an excuse to to include this with role::statistics::www [07:13:09] there will be a webdrop, so files being served out of it on stat1001 [07:13:12] so yeah, good enough :P [07:13:12] which would make things easy [07:13:15] how? [07:13:23] because stat1001 already has that role [07:14:00] ok? [07:15:35] erosen: for some reason I can't do 'import limnpy' even after installing it [07:15:36] weird [07:15:41] i see the egg file [07:15:51] weird [07:16:03] created a virtualenv and it imported fine [07:16:20] hmm [07:16:22] on stat1? [07:16:31] no, on my local machine [07:16:37] hmm [07:16:44] erosen: can you share output of your pip freeze? [07:17:26] GroggyPanda: git-remote-helpers==0.1.0 [07:17:26] -e git://github.com/wikimedia/limnpy.git@47103e699658d6890e42abf9107c0f50c7a031e6#egg=limnpy-dev [07:17:28] numpy==1.7.0 [07:17:29] pandas==0.7.0 [07:17:30] python-dateutil==1.5 [07:17:31] wsgiref==0.1.2 [07:17:58] erosen: interesting. I'm installing from a local git clone [07:17:59] also [07:18:00] errors :( [07:18:21] erosen: http://pastebin.com/VYyZze5w [07:18:59] GroggyPanda: that looks like an exception after it was installed [07:19:08] I'm running the script [07:19:12] installation installs fine. [07:19:19] GroggyPanda: oh good [07:19:23] ok that seems reasonable [07:19:38] I think the inplace=True arg is actually something that changed between versions [07:19:39] erosen: as in, limnpy doesn't seem to install, but I could simlink it to same folder so that it could do a relative import instead [07:20:41] GroggyPanda: i'm not sure what error you were hitting with the install but it fails unless you have numpy insatlled [07:21:01] it is sort of weird about numpy in that it requires it but won't automatically install it [07:21:21] (probably for good reason because it didn't used to work through pip, half the time) [07:21:51] yeah, I installed it [07:22:06]

New patchset: Ori.livneh; "Ensure Python deps for limnpy are on stat1001" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54116 [07:22:19] erosen: http://pastebin.com/YK506x7C [07:24:00] wsgiref is in stdlib [07:24:06] no? [07:24:06] GroggyPanda: i'm working on getting a no-site-packages env up--i forgot to do that the first time [07:24:08] hmm, why does that turn up there? [07:24:18] erosen: yeah, this one is no-site-packages [07:24:19] GroggyPanda: I think it must be a pandas dep [07:24:25] probably because it was separate while wsgi was a pep [07:25:04] should i add jinja2 to that patch? [07:25:14] ori-l: could you? [07:25:25] sure [07:25:29] thanks :) [07:29:42] done [07:30:10] poke otto to merge it and the packages will automagically appear within 30 minutes or so [07:30:13] so now we just need to port limnpy to work on the older pandas version [07:30:48] * erosen compling ... [07:31:18] well, i haven't done much data analysis, but pandas seems to get mentioned an awful lot [07:31:38] so if it's the bee's knees, you should update the package [07:31:47] ori-l: it is the bees knees, I guess. [07:32:29] even an out-of-date deb makes your life a lot easier since there are scripts for importing the latest source from upstream and annotating the changes file accordingly [07:32:49] ori-l: http://neuro.debian.net/pkgs/python-pandas.html?highlight=pandas [07:33:00] ori-l: updated ones for a lot of stats packages [07:33:05] while we probably can't use their repo [07:33:11] we should be able to pick up something? [07:33:17] GroggyPanda, ori-l: the funny thing is that it is really just a convenient tabular data representation, so everything could be done without pandas [07:33:30] well, convenience is pretty nice :) [07:33:44] yeah, I'm not saying I think it is bad [07:33:50] yeah, IIRC it's just a DataFrame thing on top of numpy [07:34:14] but lugging around the numpy dep might be too high a price [07:34:29] yeah, numpy is a bitch to build [07:34:32] GroggyPanda: yup [07:35:02] we should hire a full-time debianizer to fulfill these requests IMO [07:35:31] debianizing things is the hardest thing in all of software engineering; i hate it with vim and verve [07:35:56] do we have our own repos? [07:36:06] are they our sole providers, or do we pull from the official ones too? [07:36:10] debian people love to taunt you by pointing out that if you had just used deb-foo-foomatic-4 it would have worked with a single invocation with no arguments [07:36:43] we maintain our own mirror of the official repos from ubuntu for the versions we run, with additional packages [07:36:58] A few of my friends used to be debianizers. but then they all switched to arch [07:37:31] http://ubuntu.wikimedia.org/ [07:37:54] ah [07:37:55] right [07:37:55] ok [07:38:13] ori-l: what does debianization entail? Just producing the .deb packages? [07:38:20] or the build script that produces them? [07:38:50] we have to have source packages for everything [07:39:26] and there are a set of tools for structuring and working with git repositories such that you can create a deb out of the state of the repository [07:39:41] there's a 'debian' branch and an 'upstream' branch etc [07:39:46] as much as there seems to be a desire to avoid gruesome death [07:39:56] i think rsync was actually the most sensible choice here [07:40:13] hehe [07:40:13] as stat1 is intended as a computation env, and stat1001 is intended as a webserver without access to data [07:40:27] which makes it questionable that limnpy should be installed there in the first place [07:41:24] well, [07:41:32] it wouldn't be the first time in the history of analytics that intentions conflict with reality [07:41:47] but up to you [07:41:59] happy to abandon the patch if you want. [07:44:54] i kno rite? [07:45:02] i have no horse in this pagent [07:45:09] merely mentioning what i'm told [07:45:22] well, let's figure it out [07:45:26] GroggyPanda, why stat1001? [07:45:41] as to how the baroque and poorly communicated reality shall be navigated, i leave that to better men [07:45:58] ori-l: because I'm told that stat1 shouldn't be serving web pages. [07:46:20] is limnpy serving web pages? [07:46:31] nope. *nothing* here is serving web pages [07:46:34] see? [07:46:44] if we could rsync without getting killed that does solve everything [07:46:47] compute on stat1 [07:46:50] rsync to stat1001 [07:46:55] cron on stat1, rsync to stat1011 [07:46:56] yeah [07:46:58] be spattered with gore as god intended [07:47:00] *1001 [07:47:03] oh, that's what you meant [07:47:08] well, i am sassy idiot [07:47:17] i thought you meant rsync the python packages [07:47:29] worry not. you are in good company. [07:47:30] rsyncing data would be fine [07:48:24] so... an unpuppetized stat1 is fine? [07:48:47] i'll just update the patch to create a dir on stat1 that gets rsync'd [07:48:57] /var/poop or somesuch [07:49:24] i mean, stat1 is a shared, semi-federated env [07:49:27] by design [07:49:38] at the very least, everyone has to share some databases [07:49:58] GroggyPanda: what does the data have in common? generated by limnpy? [07:50:02] ops has always begrudgingly admitted that it doesn't fit with their beautiful crystalline castle [07:50:04] fishing for a good directory name [07:50:05] yes. [07:50:16] I think 'csv' would be a good name [07:50:26] it doesn't *have* to be generated by limnpy, it just usually is [07:50:51] cruft separated values! [07:52:09] what i want to know: [07:52:16] how is it that acne sells exactly ONE tie [07:52:19] and it's actually GOOD [07:52:22] (this baffles me) [07:52:23] http://shop.acnestudios.com/shop/men/accessories/webber-ch-grey-check.html [07:52:35] I mis-read 'sells' as 'smells' [07:52:40] (these are the things that keep me up at night) [07:52:42] the rest of the sentence was... [07:52:53] ...about ties [07:52:54] yes yes. [07:52:57] i know. [07:53:12] :D [07:54:05] are the files actually comma-separated? [07:54:10] ori-l: they are. [07:54:11] weren't we standardizing on tabs or something? [07:54:15] hmm [07:54:18] erosen: ^ [07:54:19] * ori-l considers 'sv' [07:54:24] how about [07:54:25] i'm pretty into tabs. [07:54:25] 'data'? [07:54:27] dsv! [07:54:28] but that's suspicious [07:54:36] i'm confused, i can't see any comments fro ori-l [07:54:38] "delimiter separated values" [07:54:45] 'thingamaggingies' [07:54:48] maybe unblock him, erosen? [07:54:51] dsv sounds politically correct [07:54:52] hehe [07:54:53] erosen put me on /ignore, yeah [07:55:03] ... why? [07:55:03] i don't know how [07:55:06] hahah [07:55:14] actually [07:55:25] i mean i didn't think i did [07:55:25] csv could totally be "character separated values" [07:55:33] and then BAM [07:55:34] TABS [07:55:36] but now that you mention it, I did accidentally right click and select something [07:55:39] TOTALLY NOT COMMAS [07:55:45] and i was surprised that nothing happened [07:55:50] man. the perils of a GUI. [07:55:51] 'NOBODY EXPECTS THE TABS' [07:56:42] how about just 'limn'? presumably all this data is being processed for visualization by limn? [07:56:53] yes [07:56:57] limn-data? [07:57:00] this isn't actually limn [07:57:04] i like that [07:57:09] though, you know, i am kinda partial to "dsv" due to 0x20AB [07:57:14] http://www.fileformat.info/info/unicode/char/20ab/index.htm [07:57:32] limn-data, then [07:57:34] yup i had accidentally ignore ori-l, sorry ori-l [07:57:55] honestly, erosen, it would have been way more productive to ignore dschoon instead. [07:58:00] dschoon: :D [07:58:04] hehe, i know [07:58:06] ori-l is totally doing useful things atm. [07:58:06] at what point in this conversation did you begin to suspect that you weren't getting everything? [07:58:28] GroggyPanda, if I were you I'd be miffed that he seemed so nonplussed that you were acting schizophrenic [07:58:28] ori-l asks a fascinating information theory question! [07:58:46] is IRC generally lossless enough that "shit don't make sense" is cause to believe things are amiss? [07:58:51] if so, when? [07:58:53] for some reason I am laughing like a maniac right now [07:59:12] this was the firs nonsequitur: ops has always begrudgingly admitted that it doesn't fit with their beautiful crystalline castle [07:59:13] [12:50am] [07:59:14] yes. [07:59:39] wait [07:59:41] that makes no sense [07:59:47] amusingly, that may well have been a real nonsequitur in context. [07:59:57] :D [08:00:29] actually, with a bit more context it makes sense [08:00:34] the next line was: ] [08:00:35] I think 'csv' would be a good name [08:00:47] which made no sense w.r.t the castles [08:00:55] in context: i like castles [08:00:59] but i dislike crystals. [08:01:00] castle separated values [08:01:05] castle seperated villages [08:01:07] hence my general distaste for marin [08:01:36] pretty okay with berkeley, though, as they are more anti-crystal than they are anti-castle. [08:01:58] (not that it's likely there isn't a city ordinance against castles. but whatever.) [08:02:02] * GroggyPanda does crystal healig on dschoon [08:02:43] grr. [08:02:48] i'll crystal heal YOU. [08:02:49] it's working :P [08:03:00] hehe [08:03:18] I'm surprised that not one person has made a pandas joke [08:03:40] i think we all mentally reviewed our options and came up empty handed. [08:03:57] sigh [08:03:59] * GroggyPanda will wait [08:04:02] on the other hand, i have ALL SORTS of bay-area civic politics jokes. [08:05:11] the 21 year old drinking thing sucks [08:06:10] erosen: so if we have puppetized rsync, we only have to get a webdrop working on stat1001 [08:06:19] generally speaking, we're not so good at legislating culture across the board. [08:06:36] ask women how we're doing on the equality front for an object lesson. [08:06:37] seems right to me [08:07:01] GroggyPanda: i mean you have complete control on stat1 [08:07:13] yeah, virtualenv is good enough [08:07:57] does stat1001 currently have an /a/ ? [08:08:10] yes [08:08:25] dschoon: yeah, my favorite pas-times are: 1. reading archives of wikimedia-l looking for drama, 2. reading memeorandum looking for stupid things US people in positions of high power say [08:08:38] much to my annoyance, however, it does not reliably bear the same structure or permissions as stat1 [08:08:54] so parental discretion is advised. [08:09:24] but my parents don't know anything about permissions [08:10:30] must have had a good set of teenage years then, eh? [08:11:09] they don't know anything about linux permissions, but I had to be home by 9:30 and my house was a good 1.5 hour commute away from the city [08:11:22] would be it fair to only sync .gz files? [08:11:27] but that didn't matter, since I spent most of it on IRC anyway. [08:11:38] ori-l: well, these have to be web-accessible at some point as .csv files [08:11:40] (this is the pattern established by otto for rsync jobs; i want to be consistent) [08:11:42] (which limn will hit) [08:11:51] oh. [08:11:56] so if you want to do .gz and then unzip once it hits here. [08:12:17] does the limn executable read them or are they requested by the browser via http? [08:12:32] basically do they need to be readable by apache [08:12:33] they're read by limn via http [08:12:41] well, if consistency is our goal, let's take care to exclude something essential in the rsync, so all things analytics can remain high-touch and frustrating [08:12:50] http, yes. so apache, yes. [08:13:31] (limn is approximately 14k lines of javascript; as far as that server-side executable is concerned, it's grep duct-taped onto webdav.) [08:13:55] (which is to say, it does little on the server other than "read this file from disk") [08:14:09] yeah, but the limn instance is on labs... [08:14:19] hmm, wait - is the limn instance going to move to stat1001 at some point? [08:14:25] No. [08:14:32] limn has a proxy built-in. [08:14:52] so long as the target is whitelisted, it will get the remote files without CORS support on the target [08:14:56] ok, so i'm going to set this up to rsync to /var/www/limn-data [08:15:43] on the assumption that all the data is intended to be world-readable, right? [08:15:57] rsync from where? [08:16:09] from stat1:/a/limn-data [08:16:17] to: stat1001:/var/www/limn-data [08:16:25] ah, okay [08:16:28] is this a cron? [08:16:41] behind eight levels of misdirection, yes [08:17:05] about what you'd expect from an abstraction on top of shell scripts [08:17:37] heh [08:17:52] that harkens amusingly to our conversation about IoC, erosen [08:17:58] indeed [08:18:29] world-readable, yes? [08:18:34] someone say 'yes', please [08:18:38] or 'no', if the answer is no. [08:18:42] "sure!" [08:19:09] our movement is all about the free and open dissemination of information! [08:19:17] o+r 4 all! [08:19:21] ori-l: yes [08:19:25] thanks [08:25:18] god spelunky is hard. [08:26:55] i'm going to call it limn-public-data [08:27:02] more explicit [08:27:04] nice [08:37:18] the default rsync-n-cron config is once daily [08:37:22] too infrequent? [08:37:27] or ok? [08:37:32] ori-l: good enough for me. [08:37:39] ori-l: also, what time daily? [08:37:55] 08:00 UTC [08:38:01] 30 minutes ago [08:39:10] good enough i guss [08:43:05] it wouldn't be very hard to have this be hourly [08:43:11] if that is better [08:43:16] ori-l: it is better, yes. [09:07:39] boo, zz_GroggyPanda [09:12:05] you're welcome, bastards [09:13:06] ori-l: i appreciate it [09:13:25] :P [09:13:42] np. didn't expect it to take three hours, but: puppet. [09:14:01] well, thanks for pushing on [09:14:12] this will be useful for a lot of things [09:14:30] assuming the wrath doesn't come too soon [09:15:04] naw, that was predicated on a completely false notion of what you were proposing [09:15:09] it should be totally fine [09:15:20] great [09:15:29] just don't put private data there by accident because it'll get whisked right to a world-readable web dir [09:15:34] yup [09:16:41] anyways, good night [09:16:47] good night [09:25:35] ori-l, know you are appreciated always [09:25:54] (wrath and gore notwithstanding) [09:26:06] (certainly notwithstanding al gore) [09:26:12] (oy.)