[00:20:21] erosen_: i guess the real question is who should the zero people talk to when they add a new partner? [00:20:39] yeah... [00:20:40] that would be probably preilly, [00:20:43] k [00:20:50] there is no need to request a filter anymore [00:21:01] but someone will need to maintain that varnish file [00:21:05] as our logging infrastructure matches the varnish config [00:21:18] that's an ops responsibility so far [00:21:22] k [00:21:24] i'll tell them that [00:21:28] we could assist them if they want [00:27:38] l [00:27:38] k [00:28:10] hey erosen_ deployed that label thing: http://gp-dev.wmflabs.org/graphs/global_north_south_active_editors [00:30:01] nitey nite everyone, see you 'morrow [00:30:08] laterz milimetric [00:34:45] milimetric: thankye! [01:07:02] who likes raphaeljs ? [01:07:03] http://jsfiddle.net/XtZhh/ [01:07:35] this will be present in the new mobile reports [01:09:44] it's a sketch of what I want to do so the person who reads the report understands what the numbers mean [01:10:23] but more of a legend than for each cell [01:11:14] *rather_than [01:25:14] two suggestions: [01:25:47] 1) make the output of the current mobile report compatible with wikistats so that wikistats can take your output in convert it in the wikistats html files [01:25:57] 2) talk to erik about raphaeljs and work with him to implement it [01:26:03] erikz that is [01:51:04] why raphael.js? [01:51:09] ^^ drdee [01:51:23] ^^ average_drifter [01:51:28] we use d3.js in limn [01:51:37] generally, we've really found it to be awesome [01:51:53] http://d3js.org/ [14:09:57] yoyo [14:11:18] morning drdee [14:11:22] i gotta restart but i'll brb [14:14:54] back [14:17:13] oh no!!! another ruby upgrade f-ed up my system [14:17:24] drdee, I wanted to work on funnel stuff with you [14:17:28] doh! [14:17:31] not funnel [14:17:32] mobilel [14:17:42] the pig script you're writing [14:20:12] hold on [14:42:23] mr dan! [14:42:25] ready! [14:42:37] do you have shell access to an01? [14:44:36] milimetric ^^ [14:44:51] i can't do any pig no [14:44:54] but i have access [14:45:08] but i can help look stuff up and debug [14:45:52] https://plus.google.com/hangouts/_/2da993a9acec7936399e9d78d13bf7ec0c0afdbc [15:11:58] mooorning [15:12:25] :-) [16:42:13] hi sumanah! [16:42:13] so did you start the search for Limn mentees? :) [16:42:13] hi milimetric! I did! [16:42:13] cool, how do you do that sort of thing? Is there a page I should be watching for people asking questions or anything? [16:42:13] I originally started in private email [16:42:13] I will finish up today and ping you here [16:42:14] cool beans. I'd like to revise my answer to "who would be a good mentee?" [16:42:14] because I heard Sue on NPR and I thought - you know, if it increases the diversity of our community, I don't really care what kind of coder they are :) [16:42:14] YoYO OTTOMATA!!!!! [16:42:14] as long as they're interested in the project, I'm happy to have them on board sumanah [16:42:14] Cool :) [16:42:15] Some of them may not be interested in the project at the start [16:42:15] but they could be persuaded I think [16:42:15] yoyoyooy [16:42:17] drdee, now that we are using tabs [16:42:17] we need to change our pig scripts! [16:42:18] i am on it baby! [16:42:18] sorry for the delay milimetric right now I am in #wikimedia-tech having my blog post about Ops improved by Guillaume [16:42:18] no worries :) i'm deep in thought sumanah [16:42:18] as far as getting people interested, the catch phrases Limn will soon be famous for: "reactive programming, d3 js, grammar of graphics" [16:42:25] so maxmind does not seem to support 3 char countries natively :( [16:42:25] so i have to add a lookuptable [16:42:26] milimetric remember were david put those country files? [16:42:26] hm, lemme search [16:42:27] https://gerrit.wikimedia.org/r/gitweb?p=analytics/reportcard/data.git;a=blob;f=geo/country-codes.json;h=5f926143aad32242f593fd3e9b1f295f845a3392;hb=refs/heads/develop [16:42:27] drdee^ [16:42:27] https://gerrit.wikimedia.org/r/gitweb?p=analytics/reportcard/data.git;a=tree;f=geo;h=2996adc4076e4c821f7ef0f21e3b3e99f0b5bb84;hb=refs/heads/develop [16:42:27] those are the links dschoon posted before [16:42:28] if you search "country codes" in this room's logs you'll find the discussion around it [16:42:29] actually, i am not gonna change it in pig udf [16:42:29] it needs to be changed as part of limnify [16:42:29] the python code is already there [16:50:56] lunch time! back before standup [17:03:02] ping dschoon [17:03:22] howdy. not working yet, tho [17:21:03] hey dschoon - I tried to leave you alone but can't :) [17:21:21] your push from last night makes it so js files don't get re-calculated on request [17:21:22] so: [17:21:28] rm -rf var [17:21:42] coke -d ../../../../reportcard-data -t rc link_data [17:21:48] npm start [17:21:59] and then notice how the js files don't get generated and limn doesn't work [17:24:05] i tried to figure out what broke things but can't - something about moving the modules.yaml [17:38:45] apparently the board's in town [17:38:58] does anyone know if we're required to wear pants to work? [17:39:08] milimetric: aiight. i'll take a look. i'm sure it's simple. [17:39:33] I think pants are safe [17:39:39] like - safer than no pants [17:39:59] I'm cautious like that. And thanks dschoon, I'm sure it's some tiny line somewhere which makes it all the more harderer to find via git bisect [17:40:01] boo! [17:41:10] hey everyone, I have my 1 on 1 during our standup today so I'll give a quick update here [17:41:26] I've been making slow progress on the "build/debianize/puppetize" set of tasks [17:42:09] I'm stuck currently so I'm helping Diederik by figuring out a generic way to import his PIG script output into a Limn "geojson" datafile [17:42:32] milimetric: you know the cache is for 10s, right? [17:43:05] did you try my steps above? You'll see that it's not related to the cache [17:43:16] k [17:43:19] it simply does not respond to any /js/limn/* requests [17:43:21] i didn't yet [17:43:29] i have to deal with that "pants" thing first [17:43:30] rather, it responds to them with "unknown" [17:43:54] exciting! [17:44:17] er, not unknown, dammit what's wrong with me today - 404 [17:44:39] ugh - index.jade - i'm gonna stop talking now [17:45:12] fixed it [17:45:19] you're gonna love this patchset [17:47:36] milimetric: https://github.com/dsc/limn/commit/09a332b356183565cdfdf148ec0b551af3194508 [17:48:00] lol, nice [17:48:03] i knew it! [17:48:10] which is why I was smart enough to not LOOK for it :) [17:48:17] * milimetric pats self on back [17:48:44] that would have literally taken me months to find [17:48:55] i would have torn connect-compiler apart limb from limb searching for the answer [17:49:10] and probably would've shot myself once I found it [17:49:17] so I think I owe you my life [17:49:20] heh [17:49:41] always look where things changed! [18:15:18] heading into the office [18:15:20] back in few [18:25:04] milimetric: can you access kripke? [18:28:01] i don't think packet loss is dead yet :D [18:30:40] hey erosen [18:30:42] no - it seems down [18:30:52] i can't even access bastion1 [18:31:03] yeah, was just gonna say the same [18:31:04] the labs list seems to think it is working... [18:31:11] hm... [18:31:24] http://reportcard.wmflabs.org/ down [18:31:26] ottomata - any ideas how this could be or should we go in labs and ask ryan [18:31:28] just forwarded you the link [18:31:41] uh oh, thanks Nemo_bis [18:31:47] wait, that works [18:31:47] that's not good [18:31:59] milimetric: I hear the WMF board has a meeting and needs it right now [18:32:00] yeah, i just tried it [18:32:07] or something like that [18:32:08] but it works for me [18:32:09] seems to be working [18:32:24] doesn't resolve here? [18:32:28] i even tested in incognito [18:32:33] interesting [18:32:41] *** Can't find reportcard.wmflabs.org: No answer [18:32:52] Server: 62.101.93.101 [18:32:55] Address: 62.101.93.101#53 [18:33:31] oh man, we seriously have some of the worst luck ever [18:33:33] just to see if it is dns, try http://208.80.153.208/ [18:33:42] ottomata - any idea what's going on with labs? [18:33:58] naw [18:34:10] saw some emails from ryan recently about some ldap issues [18:34:36] erosen: yes, slowly but that works [18:34:53] interesting, glad to hear it at least sort of works [18:35:38] so it's some DNS problem over there [18:35:39] what channel is Ryan in again? [18:37:50] well for labs, #wikimedia-labs is a channel [18:37:55] but he's in #wikimedia-operations too [18:37:59] usually #wikimedia-operations too, he doesn't seem to be online [18:38:01] ^ [18:38:04] aye [18:38:39] yeah, he's of [18:38:41] *off [18:39:30] you know what I am currenlty using instead of dsh to do something [18:39:34] iterm2 multiple input! [18:39:36] so cool! [18:39:43] i have 4 terminals opened to 4 analytics nodes [18:39:49] I needed to change the kraken upstream url [18:39:54] haha [18:45:18] ok, IP worked for Sj too [18:45:32] ottomata - nobody seems to care in labs either :) [18:45:37] haha aww [18:45:47] I'm feeling kind of helpless [18:47:12] do any labs instances work? [18:48:28] reportcard seems to work fine, but Nemo_bis is having trouble hitting it [18:48:35] it seems to work via IP [18:49:03] kripke just seems inaccessible because of bastion [18:49:24] I'll bug CT and find out what's up [18:49:52] I may drop off of IRC momentarily though as I move around [18:49:53] yep, I can't hit reportcard anymore - so the DNS was probably cached for a while [18:50:06] robla - it's looking like DNS more and more [18:50:16] and thank you for the help :) [18:52:16] CT doesn't seem to be at his desk (nor Ryan right now) [19:03:00] drdee, you back? [19:11:27] yaaaa [19:12:27] robla, erosen, Nemo_bis: still no idea what went wrong, but things seem to be back up [19:12:30] mind testing? [19:12:38] I didn't touch anything [19:12:54] yup [19:12:57] i gave it a magic kick [19:12:59] ssh works [19:14:36] hey guys [19:14:39] i'm going to reinstall an01 [19:14:46] anything in particular need saved? [19:14:48] i'll save our homedirs [19:14:54] anythign in /a need saved? [19:15:01] still doesn't work here, maybe DNS need spreading [19:15:03] mmm, i have some stuff in ~/tmp [19:15:28] actually nvm ottomata reinstall away [19:21:34] back [19:23:30] hey dschoon - you missed the magical fun of "our limn labs instances were not accessible due to DNS issues AT THE EXACT MOMENT the board was trying to access them" [19:23:39] ... [19:23:44] that's lovely. [19:24:10] i think this week had it all [19:24:33] i'll go sit upstairs just in case i'm needed. [19:37:48] FYI, RobH and I are reinstalling an01 with a private IP, and putting a new public IP on analytics1026 to use it as the new analytyics bastion [19:38:39] aight [19:39:04] this will mean the haproxy entry point will be down [19:39:11] soooo no web access to kraken [19:39:42] milimetric: man, grammar of graphis is so good [19:41:25] no hue [19:41:26] ? [19:41:40] ottomata: so what's the end-goal of this plan? [19:41:47] don't we have dashboard services on an26? [19:41:50] free up an01 for beefier stuff [19:41:55] no, on an27 [19:41:56] we really ought not run them on the bastion box [19:41:56] ok [19:41:58] good good [19:42:05] an26 is unpurposed at this time [19:42:10] we were going to get a new crappy machine [19:42:32] awesome. [19:42:40] i'm glad to hear we'll be getting an01 back for work. [19:42:50] but we realized that networking was going to get a wee complicated [19:42:50] so, once we have another crappy box [19:42:58] oh. [19:42:58] especially with the ACL request in [19:43:04] okay, so basically what i'm wondering is: [19:43:08] and an26 was free for this use [19:43:10] so i think its cool [19:43:46] can we get another public IP so we can run the bastion/web gateway on a different machine than the one that's going to have to relay incoming data from esams? [19:44:11] because esams isn't on the same LAN -- we don't have a fiber link that crosses the atlantic :P [19:44:40] milimetric: http://reportcard-dev.wmflabs.org/ == 404 [19:44:47] http://reportcard-test.wmflabs.org/ == 404 [19:44:54] wait [19:44:58] i have it wrong, don't i [19:45:04] derf [19:45:05] http://dev-reportcard.wmflabs.org/ [19:45:08] lol [19:45:10] it's good now [19:45:14] no worries [19:45:28] aiight. [19:45:40] HMMMMmmmMMm [19:45:42] good q dschoon [19:46:22] grammar of graphics is sitting at page 27 sadly unprogressed since this week has been nuts [19:46:27] but I can't wait to get back to it [19:46:42] i am on page 3! [19:46:54] ls [19:46:57] hehe [19:47:11] "Objects are relatively stupid. They do a few things well, as do lobsters." [19:47:42] erosen I tried to come up with a fake output for ls but nothing funny came to mind [19:47:48] hehe [19:47:57] yeah, wrong shell--glad you got it [19:48:56] love that dschoon, and most everything else he says - he's one very reasonable dude [20:15:19] ottomata, drdee: quick log format q: what are the last four fields? [20:15:40] i have _,_,html-lang,x-cs [20:15:43] http://wikitech.wikimedia.org/view/Squid_log_format [20:16:04] aah thanks [20:16:14] sorry if this was already updated [20:16:19] when you sent it last [20:17:49] what's up ? [20:28:41] friday is sort of like monday [20:29:15] anyway, I'm gonna pump some more on the mobile reports then I'll take some rest [20:30:18] milimetric: do you know something for d3.js that draws arrows and stuff ? [20:30:26] milimetric: I got this made with raphaeljs http://jsfiddle.net/XtZhh/ [20:30:41] milimetric: but dschoon mentioned that we're using d3 mainly [20:30:49] so I wanted to port it to d3 [20:30:53] howdy [20:31:02] taking a look [20:31:04] milimetric: hi :) [20:31:26] it's not really worth the effort to port it [20:31:30] hm, no, because the arrows wouldn't be "data driven" as it were [20:31:43] so d3 only really handles relationships between data and SVG [20:31:43] afaic, it's fine to use raphaeljs in wikistats [20:32:11] drdee: hi :) [20:32:14] raphael is great though - even falls back to VML on IE [20:32:30] so I think it's a good choice [20:32:38] cool [21:08:54] OH DEOM! [21:08:55] ? [21:09:27] drdee: https://plus.google.com/hangouts/_/2da993a9acec7936399e9d78d13bf7ec0c0afdbc [21:09:28] [4:08pm] drdee: yo guys [21:09:29] [4:08pm] drdee: erosen, dschoon, ottomata, DarTar [22:08:01] okay updated https://gerrit.wikimedia.org/r/gitweb?p=analytics/reportcard/data.git;a=blob;f=geo/country-codes.json;h=cc9618c21ae367759d994822a77ecc8234e9d8a1;hb=HEAD [22:12:27] cool. [22:12:37] you added continents! neat. [22:13:25] you should probably update the csv version also [22:13:26] https://gerrit.wikimedia.org/r/gitweb?p=analytics/reportcard/data.git;a=blob;f=geo/country-codes.csv;h=b3e86aa732583992737f2c27f207664f8fd0ec54;hb=HEAD [22:13:28] ^^ drdee [22:13:41] k [22:28:01] drdee [22:28:05] have you seen this with oozie before? [22:28:05] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2] [22:28:12] i think it must be a share lib error [22:28:13] many times :) [22:28:22] could be [22:28:26] look in hue [22:28:26] how do I feeeeex? [22:28:34] let me help [22:29:12] can you give me access to Cannot access: /var/log/hadoop-yarn/apps/otto. [22:29:15] (on hdfs) [22:30:45] i think the analytics team folks should be able to read everybody's log files by default [22:31:05] yeah i agree [22:31:10] not sure how to best fix that though [22:31:31] try now [22:31:33] (on an01) [22:31:48] nah [22:31:56] it's not on an01, it's on hdfs [22:32:01] i know [22:32:07] ohhh right namenode [22:32:07] uhhh [22:32:24] ok try now [22:32:37] nah [22:33:22] bwah [22:33:27] are you running on an10? [22:33:30] hadoop fs? [22:33:31] hmmm [22:33:32] naw that shouldn't matter [22:33:33] hm [22:33:45] i put you in the mapred grou [22:33:45] hm [22:33:54] yup [22:33:55] works now [22:33:58] cool [22:34:51] http://hue.analytics.wikimedia.org/filebrowser/view/var/log/hadoop-yarn/apps/otto/logs/application_1355947335321_12152/analytics1020_8041?offset=0&length=4096&mode=text&compression=none [22:34:56] piggybank.jar does not xist [22:35:20] piggybank! [22:35:25] PIGGYBANK!!!! [22:35:47] where should it be? [22:35:49] my homedir? [22:35:53] yup [22:35:54] why does this work for the continent script [22:35:54] hmm [22:36:12] hoookay [22:36:13] is this for packet loss? [22:36:36] yes [22:36:54] not sure if you still need piggybank [22:37:08] that was only needed to extract the hostname from the kafka field [22:37:16] we really ought to have a shared /lib dir, organized in the maven style [22:37:32] it's more complex than that [22:37:42] it's not directly related to maven [22:37:48] i know it's not. [22:37:56] pig expects jar in your local home folder [22:37:56] that's a good pattern for organizing jars [22:38:03] oh? you can't give it a path? [22:38:05] that seems strange [22:38:27] i'm sure there is a good way to do that [22:38:28] maybe /lib/m2/repository/org/hadoop/pig/piggybank/1.0.1/piggybank-1.0.1.jar [22:38:29] but yeah [22:40:13] not exactly what y'all a talking about, but i know that this line works for importing piggybank: REGISTER 'hdfs:///libs/piggybank.jar' [22:40:29] see? bingo :) [22:40:33] not sure about local paths [22:40:45] so we can keep them out of home directories [22:40:51] (i meant in hdfs) [22:40:55] streaming stuff is another matter [22:40:59] (not local fs) [22:41:01] *nod* [22:42:02] hmsame prob [22:42:04] hm [22:42:37] grrrrrr [22:44:58] but oozie expect stuff somewhere else again [22:46:03] i don't even need piggybank for this! [22:46:10] git outta here piggs! [22:49:25] grrr so close [22:49:28] so quitting time! [22:49:36] it ran! [22:49:39] but it didn't have any output [22:49:41] and the next one isn't running [22:49:49] oozie is very smart but very cumbersome [22:50:03] aight have an awesome weekend! [22:50:18] oh there it goes [22:50:20] theres one more running [22:50:28] i'll wait til it finishes hopefully output [22:53:58] ottomata: is the new canonical kraken repo on wikimedia? [22:54:06] yes [22:54:10] https://github.com/wikimedia/kraken [22:54:11] k [22:54:21] PSSH [22:54:23] still zero bytes [22:54:28] no output [22:54:32] ok i will figure it out monday [22:54:34] lataers all [22:54:43] drdee: I can't clone kraken for some reason [22:54:51] nvm [22:54:54] key forwarding [22:54:58] k [22:54:59] i turned it off after the hubub [22:55:01] b [22:55:01] laterz ottomta [22:59:45] nite ottomata [22:59:54] have a cool weekend [23:19:57] * andrewbogott is adjusting the logbot slightly [23:30:11]