[00:41:05] drdee 1234 [00:41:17] 1234 [00:41:17] drdee hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/['1234'] [00:41:17] drdee hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/1234 [00:43:38] 1234 [00:43:38] drdee hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/1234 [00:43:44] 1234 drdee [00:45:23] 1234 drdee [00:45:23] drdee hopes that drdee will have a look at https://mingle.corp.wikimedia.org/projects/analytics/1234 [00:45:39] 1234 [00:45:39] drdee hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/1234 [13:04:59] milimetric [13:05:02] check this out [13:05:29] yo milimetric have a look at 1112 [13:05:30] drdee hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/1112 [13:07:40] does it still freak out if I talk about having 12 beers? [13:07:40] milimetric hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/12 [13:07:42] yep [13:07:48] we should just do #12 [13:07:53] that's fine [13:08:02] but if you do this: [13:08:10] https://mingle.corp.wikimedia.org/projects/analytics/12 [13:08:14] then nothinghappens [13:08:39] I still think we should use the # prefix [13:08:45] but it's cool [13:08:46] nice job [13:08:50] no problem [13:08:58] i will add the hash [13:23:34] 12 beers [13:30:10] (PS3) Milimetric: work in progress on timeseries [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/82636 [13:32:44] milimetric check 1112 [13:32:58] milimetric check #1112 [13:32:58] drdee hopes that milimetric will have a look at https://mingle.corp.wikimedia.org/projects/analytics/#1112 [13:33:02] nice [13:33:15] oh [13:33:19] the link is wrong ;) [13:33:22] 1 sc [13:33:27] maybe it should be SirData, the name gets truncated [13:33:34] oh no! that's too male [13:34:02] I got it [13:34:08] it should be Marvin the Paranoid Android [13:36:13] #1112 [13:36:13] drdee hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/1112 [13:36:40] k [13:36:48] milimetric, i think this works [13:37:02] so let's have 20 beers and check out card #1112 [13:37:03] milimetric hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/1112 [13:37:06] nice :) [13:37:15] and then we should look at #1 and #2 [13:37:15] milimetric hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/1 [13:37:15] milimetric hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/2 [13:37:33] well played CommanderData [13:57:48] morning qchris_away, average, ottomata [13:57:52] yoyo [13:58:06] have you met CommanderData? [14:05:03] i just read the introductino emamil [14:05:07] CommanderData: hello! [14:05:17] what do you think about #1133? [14:05:28] #1133 [14:05:28] ottomata hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/1133 [14:28:09] hey jwidl! [14:28:13] jwild [14:28:16] i mean [14:28:34] hey ottomata [14:28:35] hi!! :) I decided this morning I need to be on IRC more often :D [14:28:42] and we notice! [14:28:53] I feel very welcomed, drdee! [14:28:55] ottomata can you have a look at the gdoc i sent you [14:29:23] ja k [14:29:37] jwild: do you know more about today's demo of wikimetrics by jaime? [14:29:50] hiya jwild! [14:29:57] drdee I didn't know there was one -at the metrics meeting? [14:30:04] hi ottomata :) [14:30:10] no separately [14:30:21] oh. no i have no idea. [14:30:24] is it a google doc? [14:30:26] i mean [14:30:29] google hangout? [14:30:45] jwild have you met CommanderData? you can ask it questions [14:30:52] i think google hangout, not sure though [14:31:27] CommanderData: what kind of questions can we ask you? [14:31:43] I dunno, drdee, CommanderData doesn't answer my questions [14:31:47] why is wikimetrics cool? [14:31:48] I'm not familiar with it so I didn't fix it in case I made it worse [14:32:11] hm [14:32:21] it's a bit of an abrasive bot [14:32:23] what do you think about logging infrastructure? [14:32:28] why are yo so harsh? [14:32:29] It's never shown unexpected behaviour like this before [14:32:34] haha [14:33:01] why do ants not contemplate their existence? [14:33:01] I can't make that a priority right now [14:33:09] hmm, why questions I see :) [14:33:10] Nobody has ever complained about it [14:33:15] why why why? [14:33:15] We should have updated our software years ago [14:33:25] true dat [14:35:16] haha. When do you guys come to SF? [14:35:18] ottomata, gdoc ok? [14:35:20] monday [14:36:52] tuesday [14:37:42] drdee, can I reword things? [14:37:48] 1) and 2) say almost the same thing [14:38:01] are you gonna shakespear me? [14:38:06] haha [14:38:12] yeah go for it [14:39:21] this is nto true, btw: [14:39:21] Once data has been consumed from the Broker, the Broker deletes it’s copy. [14:39:30] k [14:39:39] data is only deleted once it has expired [14:39:42] ok [14:39:42] like, after 1 week [14:39:43] or whatever [14:40:14] this sentence incomplete? [14:40:14] Where we install the Kafka Brokers is a matter of design, we can install them in each datacenter or in some. [14:40:16] ? [14:40:49] drdee? [14:41:17] some datacenters? [14:41:45] oh ok [14:44:41] how's that now? [14:46:29] looks good to me [14:46:30] send it? [14:46:35] k [14:46:36] do it [14:46:40] aight [14:48:01] done [14:49:04] average around? [14:53:03] yooo Snaps_, you around? [14:53:19] what are the possible value for format.key.type? [14:53:21] string and? [14:53:22] json? [14:53:29] oh sorry [14:53:33] it is the same as format.type [14:53:34] got it sorry [14:55:34] oh hm, another q Snaps_ [14:55:48] does a single varnishkafka instance only send to a single toppar at a time? [14:55:55] it doesn't balance between multiple partitions? [14:57:43] it will balance it by a random partitioner [15:00:53] and it will saturate all partitions it knows about (from the broker metadata) [15:01:16] (unless you specify partition = ) [15:03:06] hm [15:03:26] k, i just noticed on the prod kafka tests [15:03:29] that only one partition is filling up [15:08:07] CommanderData: Do you take some form of the usual "!" notation as well? Like in !card 112 ? [15:08:14] !card 113 [15:08:29] I take that as a no :-) [15:11:58] whooo [15:12:02] feature request? [15:12:05] ottomata: huhm, and there are leaders for all partitions? [15:13:42] yes, oh but on all the same broker [15:13:43] topic: varnish partition: 0 leader: 4 replicas: 4,3 isr: 4,3 [15:13:43] topic: varnish partition: 1 leader: 4 replicas: 3,4 isr: 4 [15:13:43] topic: varnish partition: 2 leader: 4 replicas: 4,3 isr: 4,3 [15:13:43] topic: varnish partition: 3 leader: 4 replicas: 3,4 isr: 4,3 [15:13:43] topic: varnish partition: 4 leader: 4 replicas: 4,3 isr: 4,3 [15:13:43] topic: varnish partition: 5 leader: 4 replicas: 3,4 isr: 4,3 [15:13:48] one of the partitions, 1 [15:13:55] does not have a full isr [15:14:02] partition 1 is the one being produced to [15:14:22] lemme see what happens if I do a preferred replicata election [15:15:10] try "kafka.debug = metadata,topic" and see what it says [15:15:19] in varnishkafka? [15:15:38] eee, this is marks' setup on some prod varnish instance somewhere [15:15:41] we might have to rope him in [15:17:56] yeah, in varnishkafka.conf [15:17:58] okay [15:18:02] but it is up to date? [15:18:12] (i,e. not more than a week old or so) [15:19:53] no for sure not [15:22:21] milimetric: [15:22:24] naming things time! [15:22:29] i've got two json fields I want to name [15:22:31] each a timestamp [15:22:31] :) [15:22:37] one is our usual string format [15:22:41] the other will be epoch long [15:22:44] timestamp1 timestamp2 : [15:22:45] :D [15:22:56] and [15:22:59] timestamp is a reserved word [15:23:00] string format looks like what? [15:23:01] can't use it [15:23:02] uh [15:23:04] 2013-01-01... [15:23:38] does the length of this matter since it'll be replicated billions of times? or does snappy handle that well [15:23:38] 2013-08-06T06:27:25 [15:23:48] i think it doesn't matter [15:23:51] dunno [15:23:56] and epoch long is like mediawiki, 20130101000000? [15:24:01] no [15:24:06] epoch is just an integer number [15:24:10] unix timestamp [15:24:12] oh the millis [15:24:13] k [15:24:29] um... [15:24:32] timestamp_epoch [15:24:32] timestamp_string [15:24:46] timestamp_unix [15:24:47] timestamp_iso8601 [15:25:05] timeYMDHms [15:25:09] no [15:25:17] (thinking out loud) [15:25:45] time_millis [15:25:47] time_string [15:25:52] it is seconds [15:25:53] pretty sure [15:26:02] current is 1378394756 [15:26:19] yeah that's 43.708610984 years [15:26:21] yep, seconds [15:26:25] so since 1970 [15:26:27] time_seconds [15:26:34] time_string [15:26:36] ts_utc and ts_unix [15:26:46] oo hm [15:27:00] time_utc and time_unix [15:27:00] ts_utc, ts_unix eh? [15:27:00] hmm [15:27:02] I like that [15:27:09] ts_epoch or ts_unix? [15:27:14] i'm more used to unix i think being used, [15:27:18] i'd do time instead of ts, I like things spelled out [15:27:18] like in MySQL and elsewhere [15:27:23] i do too [15:27:30] but timestamp sounds more right than time [15:27:36] time sounds like a duration for somereason [15:27:36] you are such byte wasters [15:27:38] haha [15:27:40] cool [15:27:54] i agree, was trying to shorten it but timestamp works [15:28:00] I know Snaps_, this is probably somewhere I should relent [15:28:16] i'm used to doing more app software engineering stuff, where bytes like this don't matter [15:28:27] but in the webrequest data, we probably should reduce , eh? [15:28:32] Snaps, we are big data :D [15:28:33] slightly shorter: datetime_string, datetime_unix [15:28:41] big data -> big variable names [15:29:06] Snaps_: with compression, how much will the longer var names matter? [15:29:08] Snaps_ ^^ [15:29:23] what does it matter [15:29:32] :( [15:29:33] dt_string, dt_unix if we want something short, because ts isn't a common abbreviation [15:29:45] dt_utc, dt_unix [15:29:46] ? [15:29:58] I dont know, actually, my guess is that there will be no mentional difference between ts_ and timestamp_ or wahtever [15:29:59] hmm, but i guess they are really both in utc though [15:30:00] hm [15:30:06] utc is more expressive [15:30:12] than string [15:30:22] yeah, but is still ambiguous [15:30:25] yep [15:30:30] since both are utc [15:30:41] the fact that they're in utc is known [15:30:49] none of the mediawiki columns say utc for example [15:30:56] everyone here knows utc is the norm [15:31:09] but the fact that it's a string is important so that it's parsed properly [15:31:25] dt_unix, dt_iso8601, if we want to be really clear [15:31:51] iso8601 is that format exactly? [15:32:02] that works for me then [15:32:41] yeah, just such a lame name though [15:32:42] but yeah [15:32:59] Date and time expressed according to ISO 8601: Combined date and time in http://en.wikipedia.org/wiki/UTC: [15:32:59] 2013-09-03T20:46Z [15:33:31] i don't really like 'dt', but i'll go with it for now [15:34:45] i don't like it either, would rather timestamp if snappy handles it well [15:34:55] but we're talking GAzilliobytes [15:37:59] ottomata what's the status of #736 [15:37:59] drdee hopes that ottomata will have a look at https://mingle.corp.wikimedia.org/projects/analytics/736 [15:38:29] drdee, the link doesn't seem to work correctly [15:38:30] Either the resource you requested does not exist or you do not have access rights to that resource. [15:38:40] mmmmm [15:38:45] whoopsie [15:38:47] you are right [15:38:57] oh status is i can almost do it anytime [15:40:15] #736 [15:40:18] drdee hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/736 [15:40:36] !736 [15:40:36] drdee hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/!736 [15:42:16] that put a ! in the url [15:43:40] alrady fixed :) [15:43:58] ottomata, what about #607 [15:44:06] drdee hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/607 [15:44:18] uhhh [15:44:22] is that for hue? [15:44:28] i think i don't know what that is [15:44:43] for hue yes [15:44:52] for hue, ssl is done, there is no official cert though [15:44:55] just a self signed [15:44:56] hmm, so if someone puts a mingle number on a gerrit message, then grrrit-wm says it and it'll echo? [15:45:00] that must be fun [16:49:06] ottomata: it seems we need an official certificate for hue, don't you think? [16:49:58] yeah, but let's figure that out later [16:50:02] we don't even have a public IP that we can use yet [16:50:23] k [16:54:09] drdee: anything I can do to help with the analyst section of the monthly report? [16:54:26] you own it :) i don't know where you put it [16:54:52] I thought you'd take care of that like last time, I added my updates to the pad as requested [16:55:23] nm, let me see where things are [16:56:11] also, I haven't heard back at all from you guys on the Wikimetrics page on mw.org [17:00:37] guys, scrum [17:03:02] average scrum [17:37:04] (PS4) Milimetric: work in progress on timeseries [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/82636 [17:37:50] I think that fixes the transaction isolation part of #1135 [17:38:01] aw, where's CommanderData? [17:40:00] he was not running :( [17:40:15] ottomata, can you help me with a simple upstart service for CommanderData? [17:41:02] drdee: want me to help? [17:43:48] sure, CommanderData is a simple python-based IRC bot installed on the tools-login instance, it would be useful if some service would make sure it's always running [17:44:05] drdee: don't have it run on tools-login! [17:44:08] does it fork? [17:44:10] it'll probably get killed often [17:44:16] use jsub, put it on the grid! [17:44:27] ori-l: no [17:44:28] i.e., if you just run it from the command line, does it stay in the foreground, or does it go to the background and drop you back to your shell? [17:44:29] kk [17:44:31] put it in the grid as a continuous job, and the grid will make sure it is continuously running [17:44:39] stays in the foreground [17:44:43] (https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help) [17:44:46] any environment variables? [17:44:50] no [17:44:53] also, what's the full path to the script? [17:44:56] & any command line args? [17:46:13] drdee: [17:46:20] 1 sec [17:47:52] ............ [17:48:12] /data/project/bingle/code/CommanderData/bot.py CommanderData '#wikimedia-analytics' (nick, channel) [17:50:53] Hi, didn't make it to scrum, so I'll give a quick update here: "I [17:51:38] I'm sill working on 701, Dario sent an e-mail yesterday explainin the new changes to the metric [17:52:00] drdee: https://dpaste.de/9Ac57/raw/ [17:52:03] but 701 is blocked right now, right? [17:52:19] save as /etc/init/commanderdata.conf [17:52:41] average: ^^ [17:52:46] awesome ori-l [17:52:50] blocked as in? [17:53:16] brb [17:53:46] average, assuming you are blocked can you take on #1109 else stick with 701 [17:53:46] drdee hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/1109 [17:54:51] drdee: I will have to re-read Dario's e-mail, [17:55:01] but I read it yesterday and it seemed clear [17:56:30] Not blocked at the moment. I will read 1109 also [17:57:39] k [18:25:33] sounds like we should demo Wikimetrics next meeting! [18:27:16] tnegrin: we should! [18:27:37] nice work everybody -- sweet call out [18:28:18] tnegrin: +1! [18:28:28] :) [18:39:47] two call outs today [19:17:30] yo ottomata [19:18:38] yoyo [19:18:55] you see my q about the VSL_ARGS? [19:20:08] huhm, no, where? [19:23:58] ottomata: I'm prefixing all kafka-related varnishkafka config properties with "kafka." as to avoid ambigiogionousness [19:24:25] if i had to spell ambigiogionousness i would fail [19:24:37] haha, ok [19:24:38] great [19:24:40] * YuviPanda adds that to quips [19:26:16] paravoid's debianization passes this to varnishkafak: [19:26:16] DAEMON_OPTS="-a -w ${LOGFILE} -D -P ${PIDFILE}" [19:26:16] but those aren't really arguments to varnishkafka [19:26:16] varnishkafka says [19:26:16] VSL_ARGS are standard Varnish VSL arguments: [19:26:16] are those standard arguments that paravoid is trying to pass? [19:26:16] yep [19:26:31] https://gist.github.com/ottomata/6454912 [19:27:06] but I dont really see why [19:27:07] There were too many developers working on that same thing [19:27:08] thank you [19:27:08] hahah [19:27:09] don't see wh-y? [19:27:22] wh-y he is passing them? [19:27:34] not sure either, probably just copy/paste from some other varnish* deb? dunno [19:27:45] i've removed passing anything extra to varnishkafka [19:27:46] yeah, he based it off the varnishncsa thingie [19:27:53] but then start-stop-daemon doesn't create the pidfile properly [19:28:05] so /etc/init.d/varnishkafka status|stop don't work [19:28:16] You will need -D and -P .. [19:28:26] but -a -w (write to offline log file), I dont get [19:29:00] ok [19:30:16] /usr/bin/varnishkafka: invalid option -- 'D' [19:30:20] Snaps_: ^ [19:31:18] huhm [19:31:33] none of those options are supported actually :) [19:31:44] daemonization is controlled through the config file. [19:31:54] milimetric: hi [19:32:04] hey stefan [19:32:05] morning [19:32:08] :) [19:32:11] it doesnt support pidfiles, but I think start-stop-daemon can handle that, no? [19:32:18] i've got a meeting in 20 min. but do you want to catch up? [19:32:20] hangout [19:34:32] right [19:34:35] average ^ [19:34:36] yeah, it hsould [19:34:39] but its not [19:34:40] so dunno [19:34:47] guess i'll ask paravoid [19:39:42] ottomata: I only think the pidfile is the issue. the args should go away [19:41:01] soo, there should be a pidfile arg for varnishkafka? [19:41:06] or shoudl just start-stop-daemon [19:45:08] ottomata: I can implement pidfile support if start-stop-daemon's killall-method isnt enough [19:48:47] hmmm, killall will probably be fine, i'll take out the pidfile arg and try it [19:48:50] but hm [19:48:58]