[09:09:49] <grrrit-wm>	 (PS51) Nuria: [WIP] Run recurring reports using the scheduler [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/112165 (owner: Milimetric)
[09:26:13] <grrrit-wm>	 (PS7) Nuria: Adding coding guidelines to README.md file [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/120542
[09:29:15] <grrrit-wm>	 (CR) Nuria: Adding coding guidelines to README.md file (5 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/120542 (owner: Nuria)
[10:13:49] <grrrit-wm>	 (PS52) Nuria: Add ability to create recurrent reports via UI. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/112165 (owner: Milimetric)
[10:14:30] <grrrit-wm>	 (PS2) Nuria: Added coalesced recurrent reports [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/122638 (owner: Csalvia)
[13:46:37] <ottomata>	 mornin qchris!
[13:46:42] <qchris>	 Morning, sir!
[13:46:49] <ottomata>	 did you mean to have two jobs called camus-webrequest-import-* running at once?
[13:46:50] <ottomata>	 :)
[13:46:55] <qchris>	 Yes.
[13:46:58] <ottomata>	 ok!
[13:47:01] <qchris>	 Is that too much?
[13:47:06] <ottomata>	 naw its fine
[13:47:17] <ottomata>	 just thought maybe it was an accident, since the second one is called -try4
[13:47:25] <ottomata>	 i was about to restart some stuff on an12
[13:47:28] <qchris>	 The first one is for a directory, where there is already some data in.
[13:47:33] <ottomata>	 ok cool
[13:47:35] <qchris>	 The second one is for a fresh directory.
[13:49:54] <grrrit-wm>	 (CR) QChris: "Code looks good. Just nit-picking in commit message." (2 comments) [analytics/wikistats] - https://gerrit.wikimedia.org/r/123603 (owner: Stefan.petrea)
[14:38:03] <grrrit-wm>	 (CR) QChris: [C: -1] "Considering this a WIP commit, as per your email," [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/120542 (owner: Nuria)
[14:47:07] <ottomata>	 oo hey qchris
[14:47:11] <ottomata>	 so, i've been testing camus stuff
[14:47:13] <ottomata>	 using the test topic
[14:47:17] <ottomata>	 instead of webrequest mobile
[14:47:26] <ottomata>	 which means you can test using much less data :)
[14:47:28] <ottomata>	 want to know how?
[14:48:10] <ottomata>	 also, i just restarted some things on an12, so the task attempts that were running there just got killed
[14:48:20] <qchris>	 Sure tell me :-D
[14:48:22] <qchris>	 Whaaaaat?
[14:48:33] <qchris>	 That means I have to restart my tests. Hahahaha.
[14:49:15] <qchris>	 Ok. I'll restart them afterwards.
[14:49:17] <ottomata>	 ha, nope
[14:49:17] <ottomata>	 you don't
[14:49:24] <ottomata>	 those attempts shoudl get reassigned, right?
[14:49:32] <qchris>	 Yes. They should.
[14:49:38] <ottomata>	 i hadn't restarted the hadoop services since I fixed that IPs
[14:49:43] <ottomata>	 that's why jmx ganglia stuff wasn't working
[14:49:48] <qchris>	 But the whole point of testing was that they go through on analytics1012 right away :-)
[14:49:56] <ottomata>	 aye ok, so
[14:50:12] <qchris>	 No worries. I'll have to leave for a few hours anyways.
[14:50:20] <ottomata>	 so, i'm using Snaps' kafkacat to pipe from webrequest_mobile | test
[14:50:23] <qchris>	 I'll ping you afterwards to see if it's safe to restart testing.
[14:50:25] <ottomata>	 on an26
[14:50:44] <ottomata>	 https://github.com/edenhill/kafkacat
[14:50:47] <qchris>	 kafkacat ... Cool.
[14:51:22] <ottomata>	 if you wann
[14:51:24] <qchris>	 Gonna have to look at that.
[14:51:46] <ottomata>	  /home/otto/bin/kafkacat
[14:51:50] <ottomata>	 i've got it compiled and just put there
[14:51:56] <ottomata>	 i'll make a package for it one day...
[14:51:56] <ottomata>	 :)
[14:52:04] <ottomata>	 its there on an26
[14:52:51] <ottomata>	 kafkacat -b analytics1021.eqiad.wmnet -C -t webrequest_mobile -p 1 | head kafkacat -b analytics1021.eqiad.wmnet -P -t test -z snappy
[14:52:54] <ottomata>	 :)
[14:52:54] <ottomata>	 \
[14:53:00] <qchris>	 Missing broker ..
[14:53:01] <qchris>	 Ah!
[14:53:02] <qchris>	 Right.
[14:53:02] <ottomata>	 kafkacat -b analytics1021.eqiad.wmnet -C -t webrequest_mobile -p 1 | head | kafkacat -b analytics1021.eqiad.wmnet -P -t test -z snappy
[14:53:42] <qchris>	 Cool!
[15:01:28] <grrrit-wm>	 (PS8) Nuria: Adding coding guidelines to README.md file [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/120542
[15:01:46] <Snaps>	 whoop, early adopter achievement unlocked!
[15:05:04] <milimetric>	 ooh, chsalvia?!
[15:05:09] <milimetric>	 I wonder what the h stands for
[15:05:11] <milimetric>	 herbert?
[15:05:14] <milimetric>	 hugo?
[15:05:30] <milimetric>	 hector!
[15:05:42] <milimetric>	 hermione?!
[15:06:50] <ottomata>	 hellcat
[15:07:15] <ottomata>	 hey Snaps, what version would you say kafkacat is?
[15:07:16] <ottomata>	 :p
[15:07:18] <ottomata>	 0.0.1?
[15:07:25] <ottomata>	 0.1.0?
[15:07:41] * Snaps  waits for another shift

[15:08:35] <Snaps>	 ottomata: 1.0.0? :)
[15:08:48] <ottomata>	 sure!
[15:08:54] <Snaps>	 Be bald, and all that
[15:08:57] <Snaps>	 ;)
[15:08:59] <ottomata>	 bald, right
[15:09:05] <ottomata>	 was thinking of buildling a package
[15:09:08] <ottomata>	 wanna tag it?
[15:09:11] <Snaps>	 cool!
[15:09:11] <Snaps>	 yeah
[15:09:18] <ottomata>	 danke
[15:13:17] <Snaps>	 ottomata: done
[15:13:30] <grrrit-wm>	 (PS1) QChris: Remove outdated LOCs in README.md [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/124333
[15:14:24] <grrrit-wm>	 (CR) QChris: "Not sure which part one should measure, but 1.5k seems" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/124333 (owner: QChris)
[15:14:34] <Snaps>	 ottomata: it requires rdk 0.8.4 though, which has not been released yet (current master)
[15:15:24] <grrrit-wm>	 (CR) Milimetric: [C: 2] "That makes sense. Though it's nice to remember that the codebase was once so small. We should aim to increase it only if absolutely nece" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/124333 (owner: QChris)
[15:16:18] <qchris>	 Hey ... milimetric you're too fast. I wanted to use that change in an email to you ... now it's merged and gone :-P
[15:16:44] <milimetric>	 :) sorry qchris
[15:17:01] <milimetric>	 if I don't see [WIP] or actual Work In Progress, I merge ruthlessly
[15:17:10] <qchris>	 Hahaha :-D
[15:17:13] <milimetric>	 did you want to talk about it?
[15:17:30] <qchris>	 Nono. It was about the dependency injection.
[15:17:43] <qchris>	 You're email sounded like wikimetrics is still a small code base.
[15:17:57] <qchris>	 It no longer is. And I wanted to point you to that commit ...
[15:18:00] <qchris>	 But you were faster.
[15:18:15] <milimetric>	 hm, I agree it's not still 1.5k
[15:18:21] <qchris>	 So no worries. Just merge away if I do not WIP a change.
[15:18:31] <milimetric>	 but I consider 0-10k small, 10k-100k medium and over that large
[15:18:42] <qchris>	 Ja. It's ok.
[15:19:17] <qchris>	 Have to leave for some hours. I'll rejoin afterwards.
[15:19:25] <milimetric>	 have a nice night
[15:19:28] <ottomata>	 ah ok Snaps
[15:19:31] <ottomata>	 when do you think 0.8.4 will be out?
[15:20:47] <Snaps>	 ottomata: I'm pretty much done with it, but since there are some big changes (C++ interface and configure script) I'd like some second eyes on stuff, the debian packaging in particular (faidon)
[15:20:53] <ottomata>	 ah ok
[15:20:55] <ottomata>	 hm, ok cool
[15:21:05] <Snaps>	 ottomata: did ./configure in kafkacat work without problems for you?
[15:22:13] <ottomata>	 yeah, worked like a charm
[15:22:33] <ottomata>	 I think kafaktee built statically is a good idea
[15:22:50] <ottomata>	 could we make it do that by default?  or is that a bad idae?
[15:22:58] <ottomata>	 (or does it already do that by default?)
[15:24:59] <Snaps>	 ottomata: it doesnt, and I dont think it should because it gets all its features automagically from rdkafka through the config properties interface. so I think it should be an explicit static when desired. Okay?
[15:25:54] <ottomata>	 yeah that's fine, you are probably right
[15:26:21] <ottomata>	 sometimes its nice to be able to just copy utils like that around without needing build tools or root privs, ya know?
[15:26:31] <ottomata>	 but, yeah, I can always just build my own static version of it and use it
[15:26:42] <ottomata>	 probably won't really matter much
[15:27:03] <Snaps>	 ./bootstrap.sh builds a static version
[15:27:12] <Snaps>	 (or ./configure --enable-static)
[15:28:24] <ottomata>	 ayeyyy,e, yeah I used bootstrap, not ./configure,:)
[15:29:48] <Snaps>	 and I do agree, a static version is most handy. (albeit wrong from a debian perspective)
[15:42:13] <ottomata>	 yup
[17:49:00] <grrrit-wm>	 (PS53) Csalvia: [WIP] Run recurring reports using the scheduler [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/112165 (owner: Milimetric)
[17:49:01] <grrrit-wm>	 (PS3) Csalvia: Added tests for coalesced recurrent reports [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/122638
[17:49:03] <grrrit-wm>	 (PS1) Csalvia: Added tests for coalesced public reports [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/124361
[18:15:58] <Ironholds>	 so, the sampled requestlogs
[18:16:03] <Ironholds>	 are those produced internally or provided to us by ops?
[18:40:33] <grrrit-wm>	 (PS54) Csalvia: Add ability to create recurrent reports via UI. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/112165 (owner: Milimetric)
[18:41:10] <grrrit-wm>	 (PS55) Milimetric: Add ability to create recurrent reports via UI. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/112165
[18:45:54] <ottomata>	 Ironholds: they come from udp2log
[18:45:59] <ottomata>	 via the varnishes themselves
[18:46:01] <ottomata>	 so umm, by ops
[18:51:32] <grrrit-wm>	 (PS4) Csalvia: Added tests for coalesced recurrent reports [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/122638
[18:54:07] <grrrit-wm>	 (PS2) Csalvia: Added tests for coalesced public reports [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/124361
[18:57:01] <grrrit-wm>	 (PS5) Csalvia: Added tests for coalesced recurrent reports [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/122638
[18:58:13] <grrrit-wm>	 (Abandoned) Csalvia: Added tests for coalesced public reports [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/124361 (owner: Csalvia)
[19:34:31] <Ironholds>	 ottomata, aha, ta
[19:35:19] <ottomata>	 Ironholds: : https://wikitech.wikimedia.org/wiki/Analytics/Kraken/Overview#Current_Architecture
[19:35:27] <ottomata>	 uhhh
[19:35:31] <ottomata>	 that deals with kraken
[19:35:46] <ottomata>	 in that diagram, replace analytics1006/udp2log kafka producer
[19:35:51] <ottomata>	 with udp2log log box instances and filters :)
[19:35:58] <ottomata>	 + an rsync job over to stat1002
[19:36:17] <ottomata>	 haha
[19:36:19] <ottomata>	 i love this:
[19:36:20] <ottomata>	 "Analytics' spring 2013 deliverables requires running analysis on unsampled mobile webrequest log"
[19:36:24] <ottomata>	 har har har
[20:07:45] <hashar>	 explain it to me like i am five years old:  why do we need unsampled web logs ?:D
[20:10:30] <ottomata>	 hashar, I am not analyst, so many things I do not konw
[20:10:32] <ottomata>	 but here's one:
[20:10:35] <ottomata>	 funnel analysis
[20:11:05] <ottomata>	 and behavior analysis
[20:11:06] <ottomata>	 hm
[20:11:07] <ottomata>	 right?
[20:12:28] <hashar>	 ottomata: yeah that sounds buzz word approved
[20:12:46] <hashar>	 i.e. I have no clue but there must be a reason for us to do such analysis hehe
[20:16:38] <ottomata>	 well, those allow people to figure out how, for example, changes in UI make people behave
[20:16:43] <ottomata>	 really good for editor experiments stuff
[20:17:13] <ottomata>	 with sampled you can't do that, because then you can't track individual sessions at all
[20:19:32] <qchris>	 No tracking of individual sessions. It's evil!
[20:23:27] <ottomata>	 haha
[20:23:30] <ottomata>	 just sayin!
[20:23:36] <ottomata>	 in theory that is why?
[20:28:33] <tnegrin>	 I'm in a meeting but I have some comments in a little bit
[20:31:31] <grrrit-wm>	 (CR) Nuria: "Besides the comments the UI side is still missing." (4 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/122638 (owner: Csalvia)
[20:41:38] <ottomata>	 anybody here aware of this?
[20:41:38] <ottomata>	 http://wikistats.wmflabs.org/
[20:46:58] <qchris>	 Seeing the page, I recall that I have seen it before some time ago. But I know nothing about it.
[20:48:36] <ottomata>	 name collision!
[21:16:10] <tnegrin>	 re: unsampled data -- here's something that I wrote a few months ago for the privacy policy.
[21:16:11] <tnegrin>	 https://meta.wikimedia.org/wiki/Talk:Privacy_policy#WMF_Response_to_concerns_about_unsampled_data
[21:17:13] <ottomata>	 hashar ^ ahh, he's gone
[21:17:22] <tnegrin>	 regarding qchris's points, I do not want to track users, but I think there are mechanisms we can use to get data that we do need without tracking
[21:17:37] <tnegrin>	 ok -- I tried -- I can send him an email
[21:18:58] <ottomata>	 tnegrin:  Antoine Musso <amusso@wikimedia.org>
[21:19:15] <tnegrin>	 thanks -- it was not obvious :)
[21:19:43] <tnegrin>	 but everyone calls him hasher right?
[21:28:40] <qchris>	 ottomata: I do need permissions to create a kafka topic. Right?
[21:28:55] <qchris>	 (kafkacat allows me to produce to the test topic, but not test_qchris)
[21:29:07] <qchris>	 % ERROR: Failed to produce message (734 bytes): Local: Unknown topic
[21:29:37] <ottomata>	 his irc nick is hashar
[21:29:50] <ottomata>	 yeah, you'd need perms to do that qchris
[21:30:09] <ottomata>	 qchris: can you just produce to test?
[21:30:11] <qchris>	 Ok. Thanks.
[21:30:17] <ottomata>	 i'd prefer to keep the number of topics low
[21:30:19] <qchris>	 Yes, I am doing that right now.
[21:30:32] <qchris>	 But I did not want to get in the way of others, so I wanted to append my nick.
[21:30:52] <qchris>	 "test" is fine for now.
[21:35:01] <ottomata>	 naw, it won't hurt anything, test topic is meant to be for whatever you want
[21:35:10] <ottomata>	 there's currently no way to delete topics in kafka
[21:35:14] <ottomata>	 so I do'nt want to just create them
[21:35:24] <ottomata>	 well, you can delete them
[21:35:31] <ottomata>	 but you have to shutdown kafka and do a buncha manual stuff
[21:36:00] <grrrit-wm>	 (PS56) Milimetric: Add ability to create recurrent reports via UI [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/112165
[21:56:11] <grrrit-wm>	 (CR) QChris: [C: -1] "Patch Set 8 has conflicts with master." (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/120542 (owner: Nuria)
[22:52:33] <average>	 0'9;i4~['po-{);o'p
[22:53:04] <average>	 wtf, was tryin to clean my keyboard, dropped some coffee on it