[00:34:09] <grrrit-wm>	 (PS1) Jdlrobson: Update the ssh script [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/127841
[00:34:11] <grrrit-wm>	 (PS1) Jdlrobson: Add graph to highlight activity on Special:MobileOptions [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/127842
[12:58:16] <grrrit-wm>	 (CR) Nuria: Fix broken test and splitting logic (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/127342 (owner: Milimetric)
[13:59:06] <grrrit-wm>	 (CR) Nuria: "I think we are missing handling the closing of sessions, sessions should be closed at the end of the metric run." (3 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/127157 (owner: Milimetric)
[14:16:52] <yuvipanda>	 milimetric: you guys have been trialling phabricator, right?
[14:16:57] <yuvipanda>	 milimetric: are you using it for CR?
[15:21:13] <grrrit-wm>	 (CR) Hashar: "@qchris I am happy to see the 'recheck' trick works :-]" [analytics/wikistats] - https://gerrit.wikimedia.org/r/126264 (owner: Erik Zachte)
[15:41:46] <nuria>	 yuvipanda: not yet
[15:42:18] <nuria>	 yuvipanda: so far we started to keep the sprint tasks
[15:42:34] <yuvipanda>	 nuria: ah!
[15:42:35] <nuria>	 in phabricator but that's it
[15:43:07] <yuvipanda>	 ok!
[15:43:14] <yuvipanda>	 I was trying to set it up for doing CR
[15:44:18] <nuria>	 i actually think gerrit does pretty fine CRs , they are not linked to tasks but as a CR tool i think is pretty great
[16:19:02] <qchris>	 nuria: About "they are not linked to tasks" ... Actually, Gerrit can link to tasks, cards, or whatever.
[16:19:06] <qchris>	 Look for example at
[16:19:12] <qchris>	 https://gerrit.wikimedia.org/r/#/c/89125/
[16:19:23] <qchris>	 The commit message has a "Card: ..." line.
[16:19:33] <qchris>	 Clicking it takes you to the Mingle card for it.
[16:19:53] <qchris>	 But ... currently ... we have no link targets.
[16:20:05] <qchris>	 As our tasks now are just cells in a spreadsheet.
[16:20:21] <qchris>	 Gerrit has a hard time linking to those.
[16:56:41] <qchris>	 Ironholds: In which hangout do we meet?
[16:59:02] <qchris>	 dartar, halfak:^
[16:59:50] <qchris>	 I am in the batcave ...
[17:00:00] <halfak>	 Good Q.  I'll head to batcave too.
[17:01:52] <halfak>	 Ironholds is out sick
[17:01:55] <halfak>	 qchris, ^
[17:01:58] <halfak>	 meeting cancelled.
[17:02:01] <qchris>	 Oh.
[17:02:02] <qchris>	 Ok.
[17:02:04] <qchris>	 Thanks.
[17:20:08] <ori>	 ottomata: ping re https://gerrit.wikimedia.org/r/#/c/85337/
[17:24:41] <ottomata>	 ah yeahhh, sorry, its on my backburner,
[17:24:44] <ottomata>	 thanks for the ping
[17:24:51] <ottomata>	 think I should be able to get to it this week
[17:25:11] <ori>	 np, i can merge it too but i'd want to do it at a time when you're around to verify
[17:25:22] <ori>	 not urgent at all, if it's on your radar that's good enough for me
[17:25:45] <ottomata>	 yeah, i'd like to do it and make sure its ok, we have to create the tocpi first
[17:26:04] <ottomata>	 actually, if I get a couple of things in for review, I might be able to do that today...
[17:26:07] <ottomata>	 i will ping you if so
[17:26:11] * ori  nods

[17:26:12] <ori>	 no worries if not
[17:28:47] <grrrit-wm>	 (PS1) Ottomata: Calling fflush in kafka_stats_cb [analytics/kafkatee] - https://gerrit.wikimedia.org/r/128950
[17:29:03] <ottomata>	 Snaps: ^
[17:31:03] <grrrit-wm>	 (CR) Edenhill: [C: 2] "deja vu" [analytics/kafkatee] - https://gerrit.wikimedia.org/r/128950 (owner: Ottomata)
[17:31:13] <Snaps>	 ottomata: ^
[17:38:17] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Calling fflush in kafka_stats_cb [analytics/kafkatee] - https://gerrit.wikimedia.org/r/128950 (owner: Ottomata)
[17:51:25] <grrrit-wm>	 (PS1) Ottomata: Updating to version 0.1.2 [analytics/kafkatee] (debian) - https://gerrit.wikimedia.org/r/128952
[18:02:27] <milimetric>	 hey qchris/nuria/csalvia
[18:02:34] <nuria>	 yes!
[18:02:34] <milimetric>	 staging is dead due to the unicode assert
[18:02:37] <nuria>	 right here!
[18:02:39] <qchris>	 Hi milimetric
[18:02:42] <qchris>	 :-D
[18:02:43] <milimetric>	 i'd love to fix it
[18:02:46] <qchris>	 Revert!
[18:02:48] <milimetric>	 but that would require self merging right now
[18:02:52] <milimetric>	 no! :)
[18:02:57] <csalvia>	 yes
[18:02:59] <nuria>	 no i did CR
[18:03:02] <nuria>	 that
[18:03:02] <milimetric>	 so, batcave?
[18:03:15] <csalvia>	 batcave it is
[18:03:15] <qchris>	 coming ...
[18:03:19] <nuria>	 we can merge it, my CR was "this assert can be removed entirely"
[18:03:21] <milimetric>	 we have a meeting in 15 minutes anyway, if you guys want to join early and help me fix / merge that'd be super
[18:03:25] <nuria>	 ok
[18:07:19] <grrrit-wm>	 (PS3) Milimetric: Fix broken test and splitting logic [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/127342
[18:09:57] <grrrit-wm>	 (CR) Nuria: [C: 2] Fix broken test and splitting logic [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/127342 (owner: Milimetric)
[18:21:04] <grrrit-wm>	 (PS1) Milimetric: Fix Flask logging [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/128959
[18:21:32] <grrrit-wm>	 (CR) Csalvia: [C: 2 V: 2] Fix Flask logging [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/128959 (owner: Milimetric)
[18:42:55] <grrrit-wm>	 (Abandoned) Milimetric: Refactor metrics to take a project, not a session [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/127157 (owner: Milimetric)
[18:55:58] <ottomata>	 ori, gonna do it!
[18:56:00] <ottomata>	 just created the topic
[18:57:52] <ori>	 oh woot
[18:57:56] * ori  checks vanadium

[18:58:02] <ottomata>	 running puppet there now
[18:59:15] <ottomata>	 ah hmm, oof, ori, i think we should have included kafka::config, not kafka::client
[18:59:26] <ottomata>	 kafka client installs the kafka package, which requires java, scala, bla, etc.
[19:00:05] <ori>	 uhoh
[19:00:27] <ori>	 okay, going to disable puppet on vanadium so it doesn't pick them up
[19:00:31] <ottomata>	 too late
[19:00:44] <ottomata>	 its ok, it isn't hurting anything
[19:00:46] <ottomata>	 it just installed the package
[19:00:48] <ottomata>	 i think
[19:00:58] <ori>	 i don't think so
[19:00:59] <ottomata>	 yeah, i'm going to fix in puppet, we'll have to manually remove packages
[19:01:05] <ori>	 i don't see it in /var/log/puppet.log
[19:01:14] <ottomata>	 oh it says failed
[19:01:18] <ottomata>	 giterr: /Stage[main]/Kafka/Package[kafka]/ensure: change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install kafka' returned 100: Reading package lists...
[19:01:18] <ottomata>	 hm
[19:01:27] <ottomata>	 it isntalled for usre
[19:01:30] <ottomata>	 root@vanadium:~# which kafka
[19:01:30] <ottomata>	 /usr/sbin/kafka
[19:01:48] <ori>	 maybe puppetd -tv doesn't log to /var/log/puppet.log
[19:02:29] <ori>	 ok, i'll acknowledge the icinga alert (it's only for the kafka consumer)
[19:03:06] <ottomata>	 k
[19:05:04] <tnegrin>	 we messing around with the event logging?
[19:05:34] <ori>	 are we ever
[19:06:34] <ori>	 tnegrin: ottomata and i (well, mostly ottomata) are deploying the eventlogging->kafka thing. the alerts are specific to it; other eventlogging things are unaffected.
[19:06:43] <tnegrin>	 cool -- thanks
[19:07:01] <ottomata>	 aye, so ok, ori, what's up with the eventlogging job then
[19:07:02] <ottomata>	 ?
[19:07:15] * ori  looks

[19:07:16] <tnegrin>	 just got the alerts and wanted to make sure things were ok
[19:08:06] <ottomata>	 ori, reenabling puppet, wanna make sure its ok now,
[19:08:07] <ottomata>	 ok?
[19:08:21] <ori>	 just a sec
[19:08:22] <ottomata>	 k
[19:10:01] <ori>	 hm
[19:10:06] <ori>	 from kafka.producer import KeyedProducer
[19:10:06] <ori>	 ImportError: cannot import name KeyedProducer
[19:10:14] * ori  investigates further

[19:10:17] <ori>	 you can re-enable puppet tho
[19:12:19] <ottomata>	 k
[19:12:41] <ori>	 okay, the version of the package we are using doesn't have KeyedProducer, but the SimpleProducer class can be made to behave the same way
[19:12:47] <ori>	 i'll have a patch in a sec
[19:14:40] <ottomata>	 ok cool
[19:14:43] <ottomata>	 puppet looks better too
[19:14:53] <ori>	 ottomata: it does mean that the events won't be keyed by schema
[19:14:57] <ori>	 but that's ok for now i guess
[19:17:24] <ottomata>	 ooook
[19:21:57] <ottomata>	 hey Snaps still around?
[19:22:02] <ottomata>	 got a vk question for ya
[19:31:57] <Snaps>	 shoot
[19:33:37] <ori>	 ottomata: it should be writing now
[19:35:26] <ottomata>	 ooo I see it!
[19:35:26] <ottomata>	 coool!
[19:35:44] <ottomata>	 oh Snaps
[19:35:44] <ottomata>	 hi!
[19:35:54] <Snaps>	 hello!
[19:36:01] <ottomata>	 i was going to ask if buffered messages in varnishkafka were stored batch compressed or not
[19:36:05] <ottomata>	 in memory
[19:36:10] <ottomata>	 i think they are not
[19:36:16] <ottomata>	 but i do not actually know
[19:36:36] <ottomata>	 i guess I mean 'buffered messages in rdkafka'
[19:36:37] <ottomata>	 :
[19:36:38] <ottomata>	 )
[19:37:18] <ottomata>	 so ori, what do we want to do with this now? :)
[19:39:34] <ori>	 are the web-based interfaces for hadoop accessible?
[19:39:51] <grrrit-wm>	 (PS2) Milimetric: Fix Flask logging [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/128959
[19:39:59] <ori>	 brb
[19:42:06] <ottomata>	 ori yes, but this data isn't going into hadoop yet
[19:42:07] <ottomata>	 do we want it to?
[19:42:18] <ottomata>	 (by yes, i mean not really, but I can show you how!)
[19:47:04] <grrrit-wm>	 (PS1) Ottomata: Preparing camus to import webrequest_text as soon as it exists in kafka [analytics/kraken] - https://gerrit.wikimedia.org/r/129017
[19:47:18] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Preparing camus to import webrequest_text as soon as it exists in kafka [analytics/kraken] - https://gerrit.wikimedia.org/r/129017 (owner: Ottomata)
[19:47:50] <grrrit-wm>	 (PS1) Ottomata: Updating kraken [analytics/kraken/deploy] - https://gerrit.wikimedia.org/r/129018
[19:48:00] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Updating kraken [analytics/kraken/deploy] - https://gerrit.wikimedia.org/r/129018 (owner: Ottomata)
[19:48:49] <Snaps>	 ottomata: the original message is kept uncompressed in memory until delivery or failure (if req.acks != 0).
[19:49:23] <Snaps>	 ottomata: additionally, a new buffer is allocated to hold a bunch of compressed messages, if compression is enabled, but this is done just prior to transmission
[19:54:08] <ottomata>	 aye ok, hm, is the message removed from the first buffer if it is placed in the compressed buffer?
[19:54:11] <ottomata>	 for transmit?
[19:54:14] <ottomata>	 Snaps: ^?
[19:54:41] <Snaps>	 no, the original memory remains until acked by broker (or timed out)
[19:55:16] <ottomata>	 ok, is there something that controls the transmit buffer size?
[19:55:42] <grrrit-wm>	 (PS1) Ottomata: Importing 3 topics with 10 partitions, setting map.tasks to 30 [analytics/kraken] - https://gerrit.wikimedia.org/r/129019
[19:55:51] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Importing 3 topics with 10 partitions, setting map.tasks to 30 [analytics/kraken] - https://gerrit.wikimedia.org/r/129019 (owner: Ottomata)
[19:56:11] <grrrit-wm>	 (PS1) Ottomata: Updating kraken [analytics/kraken/deploy] - https://gerrit.wikimedia.org/r/129020
[19:56:20] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Updating kraken [analytics/kraken/deploy] - https://gerrit.wikimedia.org/r/129020 (owner: Ottomata)
[20:02:24] <Snaps>	 ottomata: it will send as much as possible, only paused by the kernel saying the socket buffer is full, so no, not really, but practically it is limited to the transmit window + socket size
[20:04:04] <grrrit-wm>	 (PS1) Milimetric: Remove unnecessary method and test [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/129024
[20:06:45] <ottomata>	 ok, thanks Snaps
[20:07:01] <Snaps>	 ottomata: you are worried about memory usage I assume?
[20:07:55] <ottomata>	 just trying to calcuate a bit, mark was worried
[20:07:59] <ottomata>	 going to turn on text soon
[20:10:10] <grrrit-wm>	 (PS1) Milimetric: Fix user name display in CSV files [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/129025
[20:12:07] <grrrit-wm>	 (CR) Milimetric: Fix user name display in CSV files (2 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/129025 (owner: Milimetric)
[20:17:55] <grrrit-wm>	 (PS2) Milimetric: [WIP] Fix user name display in CSV files [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/129025
[20:31:37] <ottomata>	 oh boy, i just turned on varnishkafka on text varnishes
[20:31:38] <ottomata>	 here we goooo
[20:33:49] <Snaps>	 :)
[20:39:53] <milimetric>	 awesome ottomata: is the server room burning yet?
[20:40:44] <ottomata>	 http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&tab=v&vn=kafka&hide-hf=false
[20:48:41] <ottomata>	 geeez, almost 9M records in like 15 minutes
[20:48:54] <ottomata>	 welp, there it goes
[20:49:05] <ottomata>	 Ironholds: webrequest_text, webrequest_bits and webrequest_mobile now all exist in hive
[20:49:37] <Ironholds>	 ottomata, AWESOME
[20:49:41] * Ironholds  dances

[20:51:23] <Ironholds>	 (hive has a UNION keyword, right? ;p)
[20:55:47] <milimetric>	 dude that looks kind of amazing
[20:56:02] <milimetric>	 it just swallows up data and doesn't even burp or anything
[20:56:16] <milimetric>	 you guys are magicians or something huh?
[20:56:31] <ottomata>	 Ironholds, yeah, but I want to make a big global table for ya
[20:56:42] <ottomata>	 i have to make some big changes to the hive-partitioner script to do it automated
[20:56:44] <Ironholds>	 milimetric, that's pretty much how I see all of you.
[20:56:51] <ottomata>	 i'd like there to be a webrequest table
[20:56:57] <ottomata>	 that has a partition for each of these cache types
[20:57:00] <Ironholds>	 I'm still fairly certain that the entire system is dependent on a script called chicken_innard_divination.sh
[20:57:01] <ottomata>	 so you could do
[20:57:15] <ottomata>	 select * from webrequest where role='bits' and year='2014' ...;
[20:57:21] <ottomata>	 or
[20:57:32] <Ironholds>	 that'd be awesome
[20:57:34] <ottomata>	 select * from webrequest where role='mobile' or role='text' and year='2014' ...
[20:57:39] <ottomata>	 i can easily make that table by hand now
[20:57:44] <Ironholds>	 (partitioning by type as well as year, that is)
[20:57:51] <ottomata>	 but, i want it to update the partition data automatically
[20:58:01] <milimetric>	 ottomata: woa, cool, we can map multiple ways of partitioning on the same files!
[20:58:07] <ottomata>	 and the hive-partitioner infers table names by the hierarchy scheme
[20:58:20] <ottomata>	 yeah, its gonna need some partition configuration support added to it
[20:58:25] <milimetric>	 but ottomata: why would it update automatically, isn't there just bits, text, and mobile?
[20:58:27] <ottomata>	 we wrote it so that everything was convention
[20:58:39] <ottomata>	 yeah, but it needs to have the time partitoins as well
[20:58:44] <milimetric>	 right
[20:58:51] <milimetric>	 good point, that needs some changes
[20:58:56] <milimetric>	 i'm happy to help/review
[20:59:00] <ottomata>	 cool, thanks
[20:59:18] <ottomata>	 i'll try to work on that within the next week or two, we will see
[20:59:44] <ottomata>	 also good news, last week I was worried about camus taking too long to import since I added bits
[20:59:48] <ottomata>	 but it eventually caught up
[20:59:54] <ottomata>	 camus jobs take about 25 minutes now
[21:00:00] <ottomata>	 text almost doubles the data
[21:00:05] <ottomata>	 so it might take up to 50 minutes
[21:00:07] <ottomata>	 which is fine
[21:00:13] <ottomata>	 as long as it is less than 55 minutes its ok
[21:00:18] <ottomata>	 if/when we add upload
[21:00:34] <ottomata>	 if camus takes longer, we'll have to worry about it
[21:00:45] <ottomata>	 i'm sure we can tweak it though
[21:00:51] <Ironholds>	 ottomata, woah. 55 minutes is damn good given the number of datapoints.
[21:01:04] <ottomata>	 well, 55 minutes is an arbitrary number
[21:01:08] <milimetric>	 that's kind of scary close to the limit
[21:01:08] <Ironholds>	 My big fear with this mobile metrics stuff is "it needs to take less time than readers do". Hitting that target is hard.
[21:01:14] <Ironholds>	 well, presumably <60 ;p
[21:01:18] <ottomata>	 its just aht we schedule this to run every hour
[21:01:21] <Ironholds>	 yeah
[21:01:31] <ottomata>	 we'll figure that out
[21:01:36] <Ironholds>	 yeah, it's doable
[21:01:50] <Ironholds>	 my hope is that by having the data available and working on it we'll get a better idea of what we need/don't need
[21:01:50] <ottomata>	 so, when we get this 3rd kafka broker, i'm going to have to make new topics for this anyway
[21:01:56] <ottomata>	 i'm thinking about making more partitions when I do
[21:02:03] <ottomata>	 which will allow more tasks to run and import at once
[21:02:10] <Ironholds>	 and might be able to go "hey, if you hit the limit, R&D totally doesn't need [class of data] consistently"
[21:02:30] <ottomata>	 ahhh, nawww, I'm sure we can get imports to work
[21:02:32] <ottomata>	 storing it all ...
[21:02:35] <ottomata>	 hehh, we'll see :p
[21:04:25] <qchris>	 ottomata: 9M records per 15 minutes ... is that enough?
[21:04:31] <qchris>	 What else goes into sampled-1000?
[21:04:55] <qchris>	 (unsampling sampled-1000 would give ~86M lines per 15 minutes)
[21:05:14] <ottomata>	 qchris, that is a really rough guess
[21:05:21] <ottomata>	 i just ran camus a few minutes after turning this on
[21:05:27] <ottomata>	 and then selected the number of records from the table
[21:05:34] <qchris>	 Oh. I see :-)
[21:05:36] <ottomata>	 i think we are in low load time right now
[21:05:42] <qchris>	 Coolio.
[21:05:43] <ottomata>	 i'm looking at kafka ganglia stuff
[21:05:53] <qchris>	 Yes. Been there as well.
[21:06:02] <ottomata>	 looks like its around 40K / second right now
[21:06:04] <Ironholds>	 9m/15 sounds about right.
[21:06:05] <ottomata>	 for text
[21:06:14] <ottomata>	 yeah, that's just text, btw
[21:06:17] <ottomata>	 qchris, sampeld has everything
[21:06:24] <ottomata>	 which totals around 205K / sec
[21:06:25] <Ironholds>	 I end up with about 9m a day for the 1:1000 when I filter by MIME type to text/html and the API requests
[21:06:34] <Ironholds>	 so, yeah, back-of-the-envelope says it sounds right.
[21:06:53] <Ironholds>	 (well, and exclude uncompleted requests etc)
[21:07:04] <qchris>	 ottomata: Ok. Thanks.
[21:09:45] <grrrit-wm>	 (PS3) Milimetric: [WIP] Fix user name display in CSV files [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/129025
[21:10:26] <grrrit-wm>	 (PS14) Milimetric: Add ability to remove cohorts from database. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/119343 (owner: Terrrydactyl)
[21:11:08] <grrrit-wm>	 (PS4) Milimetric: [WIP] Fix user name display in CSV files [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/129025
[21:46:49] <grrrit-wm>	 (CR) Milimetric: Add ability to remove cohorts from database. (4 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/119343 (owner: Terrrydactyl)
[21:49:07] <milimetric>	 nite everyone
[21:50:08] <ottomata>	 nite!