[04:45:24] New review: Ori.livneh; "(2 comments)" [analytics/log2udp2] (master) C: -1; - https://gerrit.wikimedia.org/r/58449 [13:17:32] mooooorning [13:19:15] New review: coren; "(2 comments)" [analytics/log2udp2] (master) - https://gerrit.wikimedia.org/r/58449 [13:28:41] morning [13:36:12] ping average_drifter [13:37:07] drdee, did we already book mutante for today? [13:37:14] yes [13:37:18] do we have a time? [13:37:47] New patchset: coren; "New implementation of log2udp" [analytics/log2udp2] (master) - https://gerrit.wikimedia.org/r/58449 [13:38:54] it's in your calendar :) [13:39:12] Upgrade Precise [13:39:12] Thu, April 11, 11:30am – 1:30pm PST [13:39:20] oh looky there [13:39:21] great thank you [13:39:37] but i guess you can you start now [13:39:44] shall we start an ether pad doc [13:39:50] to keep track of things? [13:40:11] i guess, start with mingle card content? i just updated it [13:40:34] http://etherpad.wmflabs.org/pad/p/EmeryUpgrade [13:40:41] k [13:43:01] New review: coren; "Verified to build and run." [analytics/log2udp2] (master); V: 2 - https://gerrit.wikimedia.org/r/58449 [13:58:21] drdee, up to you if you want to allocate my time to what mark is suggesting :) [14:01:17] yes i will talk about this with kraig and rob [14:14:28] analytics-logbot, still feeling ok? [14:14:28] I am a logbot running on tools-exec-02. [14:14:28] Messages are logged to www.mediawiki.org/wiki/Analytics/Server_Admin_Log. [14:14:29] To log a message, type !log . [14:26:10] ping average_drifter [15:46:09] ping ottomata, need help regarding emery? [15:48:20] yooy [15:48:24] um, i think its pretty much ready [15:48:29] i backed up the filters dir [15:48:35] we really just need to do it, tis all [15:48:52] ergh, the upgrade is scheduled at the same time as my robla 1on1 :p [15:52:11] why not start now ? [15:58:38] mutante was going to do the actual reinstall, right? [16:01:44] no, i think you were and he would keep an eye on it but you can ask him to do it as well, whatever works out the best [16:02:08] brb [16:02:13] (relocating) [16:02:18] average_driftter......around??? [16:04:39] hm, ok i can do it…i'm not 100% sure if I know everything to do [16:04:45] i'll start after the standup [16:04:49] in the middle of kakfa deb stuff now [16:21:51] pong drdee [16:21:55] yo [16:22:02] hi [16:24:17] mornin [16:24:23] hi dschoon [16:24:27] aye [16:24:29] i live, sorta [16:24:30] hi erosen [16:24:38] yo [16:26:37] drdee, is our Problem Scrum now in https://plus.google.com/hangouts/_/2da993a9acec7936399e9d78d13bf7ec0c0afdbc ? [16:26:58] scrum in 30m right ? [16:27:10] dschoon: yes [16:27:14] average_drifter: yes [16:27:18] ok [16:27:19] coolio [16:28:03] dschoon: quick python question [16:28:09] sup [16:28:16] dschoon: do you know of any ways of figuring out why a python process is sleeping? [16:28:27] i guess it's not python specific [16:28:43] one sec [16:28:43] meeting [16:28:43] erosen: gdb attach to it [16:28:43] then i will answer [16:28:45] but i've got these jobs which are all sleeping for no apparent reason [16:28:49] k [16:28:55] average_drifter: can you do that? [16:29:01] erosen: which process is that ? [16:29:06] PID please [16:29:10] erosen: don't do that :P [16:29:13] pdb is better [16:29:25] well it is a long running job that I'd rather not kill if possible [16:29:33] so I might be in the system level world [16:30:53] average_drifter: the pid is 6205, but please don't accidentally maim it [16:33:46] sorry, haven't tried this with python before [16:33:57] need to try it on a process locally first [16:34:12] if it works then I'll make a small recipe and we can try that [16:34:20] there's a program [16:34:20] k, don't put too much effort into this [16:34:21] pdb [16:34:27] which is gdb, but python [16:34:30] it is useful. [16:34:39] it lets you step through the process, dump stack,e tc [16:35:09] fair, the problem is that this issue seems to have only arisen after the job has been running for some time [16:49:38] drdee: missing scrum today; "GLEE" meeting [16:49:45] :( [16:49:46] k [16:56:18] erosen: you might also try running dstat on the box [16:56:22] see what's busy [16:56:44] what does that do? [16:57:08] rtfming [16:57:34] dschoon, I actually think it is a sql issue [16:57:42] *nod* [16:57:44] for some reason queries that were once fast are now slow [16:57:51] https://ishmael.wikimedia.org/more.php?host=db1047&hours=24&checksum=17499807313791228470 [16:57:54] dstat is an amalgam of vmstat iostat, etc [17:01:13] hangoutbeing wonky [17:01:19] https://plus.google.com/hangouts/_/2da993a9acec7936399e9d78d13bf7ec0c0afdbc [17:01:22] ^^ ottomata [17:43:35] I'm away for 20m brb [17:44:03] drdee: if you want to talk about 60, I'll be on the scrum hangout link [17:44:08] in 20m [17:56:43] ottomata, do we need to send email about scheduled maintenance for emery?/ [17:57:31] i'm not sure who would care? not many people have access to it [17:58:01] might as well just send a notice to ops@ and analytics@ [17:59:01] i will logbot in ops [18:03:10] ok [18:31:54] New patchset: coren; "New implementation of log2udp" [analytics/log2udp2] (master) - https://gerrit.wikimedia.org/r/58449 [18:33:00] New review: coren; "New option is of little use unless it's actually /checked/ for." [analytics/log2udp2] (master); V: 2 - https://gerrit.wikimedia.org/r/58449 [20:03:08] k [20:06:08] dschoon: can we host average_drifter test runs of his wikistats html reports on kripke? (just need an apache server and a box with public ip address) [20:06:46] sure [20:07:09] dschoon, mind if I import sampled logs only once a day? [20:08:10] several of the cards say hourly... [20:08:29] gonna be small files [20:08:57] isn't it 700MB/hour? [20:09:07] which of the cards say hourly? [20:09:27] it is 700mb/day IIRC [20:09:38] i thought it was 92 [20:09:41] but i could be wrong [20:09:50] if it's 700mb/day, then yeah. daily is fine [20:10:05] okay [20:10:08] 92 says daily [20:10:11] so that's fine with me [20:10:20] no sampled [20:10:25] 100 [20:10:26] 1000 [20:10:30] 700MB / day [20:10:33] coolio [20:10:35] k [20:10:48] well, uncompressed its 3.2G /day [20:11:06] shrug [20:11:13] speaking of [20:11:18] HDFS compression is on, right? [20:12:00] the snappy stuff? can't remmeber [20:13:20] i thought compression happens at the job level, you can store compressed output of a MR job if you want but you have to set that explicitly [20:13:40] i also thought there was an option to transparently compress everyting [20:16:39] there is, don't remember, i think we tried it, but don't remember [20:16:41] dschoon [20:17:44] how's this look? [20:17:44] http://localhost:8888/filebrowser/view//wmf/raw/webrequest/archive/webrequest-sampled-1000.sample/2013-04-11/part-1364239892421_10750-m-00000 [20:18:20] why's the part file so weird? [20:19:27] ? [20:19:53] part-1364239892421_10750-m-00000 [20:20:06] oh, huh [20:20:11] i guess the mobile ones are weird too [20:20:13] my bad! [20:20:15] that's what it normally looks like [20:20:16] ja [20:20:20] ohh [20:20:23] i think thats the hadoop consumer job idea [20:20:23] because they're mapper-parts [20:20:28] yeah [20:20:30] no reduce phase, right? [20:20:35] right, think so [20:20:43] okay. [20:20:47] gotcha, makes sense [20:21:13] ok but! [20:21:16] that shoudl import daily now [20:21:25] coolio [20:21:26] i have it starting the job at 0:002 [20:21:28] sounds good to me [20:21:30] 0:02 UTC [20:21:38] but, you should probably still run jobs with overlaps [20:21:41] yep. [20:21:44] even though most of the data won't be in the overlaps [20:21:52] no harm in it [20:21:54] the datasize is tiny [20:21:57] you'll have to parse x3 the data [20:22:01] but yeah [20:22:08] oh no! 2G! [20:22:20] fwiw, my mobile job for all of march took 2h [20:22:24] yea…. [20:22:30] and that was like several hundred TB [20:22:30] nice! [20:22:35] ok cool, then we won't worry abou tit [20:23:03] hm, i could import every 4 hours or something [20:23:46] should be fine [20:28:33] 4 hours or daily? [20:28:35] which do you prefer? [20:29:52] dschoon^ [20:30:08] probably 4h, but if you're already done with daily, that's fine [20:30:17] naw its easy to change [20:33:37] cool, every 4 hours: [20:33:37] http://localhost:8888/filebrowser/view/wmf/raw/webrequest/archive/webrequest-sampled-1000.sample#/wmf/raw/webrequest/archive/webrequest-sampled-1000.sample/webrequest-sampled-1000 [20:33:48] hotness [20:33:50] http://localhost:8888/filebrowser/view/wmf/raw/webrequest/archive/webrequest-sampled-1000.sample/webrequest-sampled-1000 [20:34:24] i find the filename format kind of compical [20:34:27] *comical [20:34:34] wait wait. is it sampled? [20:34:42] i'm confused. [20:34:50] * ori-l hugs ottomata [20:34:55] thanks again for the jsonschema stuff [20:35:03] very much appreciated [20:40:07] hah, yup! that was fun [20:40:17] been learning quite a bit about git buildpackage, (and git, and debian/ etc) [20:40:24] getting easier, its good experience [20:40:28] it is sampled [20:40:30] dschoon [20:40:34] and [20:40:39] i was joking :) [20:40:42] haha [20:40:42] .../webrequest-sampled-1000.sample/webrequest-sampled-1000 [20:40:45] haha [20:40:47] well [20:40:51] that's just a quick import for testing [20:40:52] it will be at [20:41:00] /wmf/raw/webrequest/webrequest-sampled-1000 [20:41:06] okay :) [20:41:14] qq [20:41:17] could perhaps it be at: [20:41:29] /wmf/raw/webrequest/all-sampled-1000 [20:42:18] hmm, it could [20:42:20] sure [20:42:29] that makes sense [20:42:31] k [20:43:19] i also tend to favor many nested directories [20:43:38] because that's not really a cost, and it's a lot harder to undo names with - instead of / in them later [20:43:46] like, for the mobile job, it's: [20:43:57] well, that would be harder to configure right now [20:44:06] /wmf/public/webrequest/mobile/device/props/2013 [20:44:07] ahh [20:44:08] okay. [20:44:15] the consumer takes a directory (/wmf/raw/webrequest) and then names it after the kafka topic [20:44:27] (to complete the example, instead of /wmf/public/webrequest/mobile_device_props/2013 [20:44:27] unless i can put slashes in the kafka topic name... [20:44:43] lemme try that real quick [20:46:13] i suspect it won't work [20:46:22] because zk uses / to mean directory [20:46:28] and kafka keeps its group info in sk [20:46:29] zk [20:48:19] heyayyy [20:48:21] it works! [20:48:22] http://localhost:8888/filebrowser/#/user/otto/tmp/kafka/slashes/test/slashes/in/topic [20:48:29] lol [20:48:31] nice1 [20:48:37] ok [20:48:38] cool [20:48:44] so [20:48:47] whatchuwant? [20:48:50] that's really awesome to know [20:49:00] webrequest/sampled/1000? [20:49:01] /wmf/raw/webrequest/all-sampled-1000 [20:49:02] i think? [20:49:09] i thought you wanted slashes! [20:49:13] all/sampled-1000 [20:49:16] i think [20:49:18] all/sampled-1000 [20:49:19] hmm [20:49:19] no [20:49:19] ok [20:49:22] you're totally right [20:49:24] use slashes! [20:49:26] go nuts! [20:49:32] i don't really like 1000 [20:49:34] 1/1000 [20:49:35] hahah [20:49:36] no [20:49:39] lol [20:49:40] no [20:49:44] that's a little TOO nuts [20:49:46] .001 [20:49:59] all/sampled/.001? [20:50:00] naw [20:50:02] 0.001 [20:50:03] 1000 is good [20:50:06] all/sampled/1000 [20:50:08] yeahhhhhh [20:50:09] its fine [20:50:10] it's the convention used on stat1 [20:50:13] yeah [20:51:01] all/sampled/0.001? [20:51:02] no? [20:51:04] 1000 better? [20:51:11] ok ok ok 1000 [20:52:13] erggggh [20:52:13] well [20:52:15] hmm [20:52:16] hmmmmm [20:52:18] hmm [20:52:20] hmm? [20:52:51] i want the kafka topic to have 'webrequest' in it [20:52:53] to start with it [20:52:54] hm [20:52:54] oh [20:52:56] but I can still do that [20:53:03] and then just consume to /wmf/raw [20:53:08] webrequest dir will come from the topic [20:53:09] oooo [20:53:10] i like [20:53:33] haha [20:53:33] nice [20:54:52] so good! [20:54:52] Consuming -1 messages from topic 'webrequest/all/sampled/1000' in consumer group ottotest0 into /user/otto/tmp/kafka... [20:57:01] ohh wait [20:57:06] it didin't actually consume anything...>>.... [20:57:12] it created dirs properly [20:57:13] h [20:57:13] hm [21:00:03] byte offset? [21:00:09] maybe it had nothing to consume? [21:01:38] poop nope [21:01:42] the topic doesn't exist in zookeeper [21:01:46] it produced just fine [21:01:50] dunno what happened to it [21:02:22] ha, yup [21:02:24] currently [21:02:28] no idea where those logs go [21:02:33] yeeeah [21:02:34] without a slashed topic [21:02:34] so, uh [21:02:37] back to dashes. [21:02:45] at least i called it :) [21:02:58] haha, yeah [21:03:00] so up [21:03:01] um [21:03:13] webrequest-all-sampled-1000? [21:03:20] yep! [21:03:20] or just sampled? [21:03:24] wait [21:03:25] webrequest-sampled-1000 [21:03:26] ? [21:03:28] you can set the prefix, right? [21:03:38] i want the topic to start with webrequest [21:03:42] /wmf/raw/webrequest [21:03:52] /wmf/raw/webrequest/webrequest-all-sampled-1000 [21:03:54] seems fine [21:04:01] eyah that's what i think we got [21:04:06] all-sampled or just sampled? [21:04:10] webrequest-sampled-1000 [21:04:12] i guess all [21:04:21] webrequest-all-1000? [21:04:24] webrequest-all.1000? [21:04:39] naw - [21:04:41] that's how we're doing it [21:04:42] ok [21:04:47] webrequest-all-sampled-1000 [21:05:15] ja [21:09:09] ok that works [21:09:22] first import should happen in 3 hours I think [21:09:53] alright. good to know. [21:12:05] if i'm still about, i'll check on it then [21:12:44] oh, 5. yeah, i'll def check on it then [21:12:47] (thought it was later.) [21:17:21] yeah, shoudl go at 0:02, 4:02, 8:02, 12:02, 16:02, 20:02 UTC [21:18:10] brb lunch! [21:37:10] http://www.youtube.com/watch?v=KmzkgSpeKfQ [21:37:31] taking a break with tea [21:47:50] thaaaank you ottomata for your work on emery! [21:58:54] d [21:59:01] (wrong window) [22:00:21] ZZ [22:00:23] (wrong window) [22:00:28] hehe [22:38:43] hey dschoon, any news regarding the device class pig job? [22:38:52] fiddling [22:40:08] i'll ping you when there's progress, yo [23:07:45] dschoon: is there anything in particular you are fiddling with :) ? [23:08:50] oozie, as always [23:09:11] what's the problem? [23:09:21] some totally inscrutable XML error [23:09:32] i'm sure it's a dumb typo [23:09:42] but the problem with subworkflows is that it could be in any of the related files [23:10:30] i'll have it done tonight no matter what [23:10:39] gonna take a break though [23:13:32] if you solve it, can you maybe document the solution? [23:14:35] of course! [23:14:44] i've been collecting a doc of pig and oozie notes [23:14:49] it is ... half finished, as always [23:15:13] not that much time for extra-curriculars like that [23:16:45] awesome [23:16:58] alrighty guys! see ya tomorrow [23:17:16] word [23:17:18] thanks man [23:18:51] i like DeployedPanda's [23:18:56] :D [23:19:02] they are better than UndeployedPandas [23:19:04] me too! [23:19:16] same as UndeployedPanda [23:19:17] bad panda! [23:19:19] haha [23:19:29] ModeratelyResponsiblePanda [23:19:56] dammit [23:20:03] haha [23:20:18] later guys