[00:38:56] <ebernhardson>	 mediawiki should now be writing 1-2 messages per second (sampled 1:1000 until verified no problems) to mediawiki_CirrusSearchRequestSet in kafka
[01:14:33] <harej>	 Does the pageview API have a release date yet?
[02:25:16] <wikibugs>	 Analytics, Analytics-Kanban, Discovery, EventBus, and 8 others: EventBus MVP - https://phabricator.wikimedia.org/T114443#1776521 (GWicke) @ottomata: In my recollection of the discussion & the log you linked to, the question of which REST producer proxy to use was left open. Our priority is to get b...
[07:50:18] <wikibugs>	 Analytics-Wikistats: wrong total number of articles for Tyv.wikipedia.org - https://phabricator.wikimedia.org/T56681#1776661 (Nemo_bis) Open>Invalid a:Nemo_bis [[https://tyv.wikipedia.org/wiki/%D0%A2%D1%83%D1%81%D0%BA%D0%B0%D0%B9:%D0%A1%D1%82%D0%B0%D1%82%D0%B8%D1%81%D1%82%D0%B8%D0%BA%D0%B0|MediaWiki...
[08:46:01] <grrrit-wm>	 (CR) Addshore: "I may just merge this as it is today!" (2 comments) [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/248033 (owner: Christopher Johnson (WMDE))
[08:46:30] <wikibugs>	 Analytics-Wikistats: Cross-link stats.wikimedia.org and ee-dashboard.wmflabs.org - https://phabricator.wikimedia.org/T67994#1776686 (Nemo_bis) >>! In T67994#1773128, @ezachte wrote: > What's the best way to detect which language codes have new stats? (other than screen scraping https://meta.wikimedia.org/wik...
[09:55:07] <wikibugs>	 Analytics, Analytics-Kanban, Discovery, EventBus, and 8 others: EventBus MVP - https://phabricator.wikimedia.org/T114443#1776773 (mobrovac) FWIW, one does not exclude the other: the EL-based service can be used in production, while the node-based REST proxy may be used for development and/or small...
[09:56:51] <wikibugs>	 Analytics, Discovery, EventBus, MediaWiki-General-or-Unknown, and 6 others: Define edit related events for change propagation - https://phabricator.wikimedia.org/T116247#1776776 (mobrovac) Please take a look at the [proposed event definitions](https://github.com/wikimedia/restevent/pull/5) and voi...
[12:14:59] <wikibugs>	 Analytics-Backlog: Make reportupdater output emtpy values when query returns no results. - https://phabricator.wikimedia.org/T117537#1776969 (mforns) NEW
[12:20:23] <wikibugs>	 Analytics-Backlog: Implement re-run script for reportupdater - https://phabricator.wikimedia.org/T117538#1776983 (mforns) NEW
[13:48:10] <grrrit-wm>	 (PS12) Joal: Add refinery-cassandra module [analytics/refinery/source] - https://gerrit.wikimedia.org/r/232448 (https://phabricator.wikimedia.org/T108174)
[14:12:07] <mobrovac>	 hey there guys
[14:12:15] <joal>	 hi mobrovac
[14:12:38] <mobrovac>	 i'd need some info from varnish logs and people tell me you crunch them
[14:12:44] <mobrovac>	 so perhaps you can help me
[14:13:09] <mobrovac>	 I'd need to get IP/UA pairs for reqs to RB for a specific subset of routes
[14:13:11] <joal>	 I'm in a meeting mobrovac, but can read and hopefully answer if not tool ong :)
[14:13:26] <mobrovac>	 hehe kk
[14:14:12] <mobrovac>	 we want to know if there are any clients using https://en.wikipedia.org/api/rest_v1/page/mobile-*/{title} routes
[14:16:31] <joal>	 mobrovac: feasible, using wmf.webrequest table on hive
[14:17:56] <mobrovac>	 aha cool
[14:18:13] <mobrovac>	 ... and how would I go about doing that? :P
[14:18:44] <joal>	 mobrovac: I'll guide you, but after the meeting :)
[14:19:00] <joal>	 the answer to this one is too long ;)
[14:19:01] <mobrovac>	 kk merci joal
[14:19:08] <joal>	 De rien mobrovac :)
[15:05:54] <halfak>	 joal, line 21 in https://etherpad.wikimedia.org/p/research_cluster_loading
[15:05:57] <halfak>	 "sample xml dump"
[15:06:03] <halfak>	 it has about 7k revisions in it
[15:06:15] <joal>	 awesome halfak
[15:07:13] <halfak>	 Thanks for your help this morning.  I think this is going to be pretty easy for me to handle once you have that script done :D
[15:44:58] <joal>	 Hi milimetric
[15:55:36] <wikibugs>	 Analytics, Analytics-Kanban, Discovery, EventBus, and 8 others: EventBus MVP - https://phabricator.wikimedia.org/T114443#1777558 (Ottomata) > In my recollection of the discussion & the log you linked to, the question of which REST producer proxy to use was left open.  I think you may be referring t...
[16:37:14] <ottomata>	 halllloooo nuria
[16:37:27] <nuria>	 ottomata: holaa, still looking at your stuff
[16:37:36] <nuria>	 ottomata: was just about to make sure is all async
[16:38:19] <nuria>	 ottomata: do you have a sample url I can post ?
[16:38:28] <nuria>	 ottomata: like a curl you were using?
[16:38:34] <ottomata>	 ja nuria, do you have access to services project?
[16:38:44] <ottomata>	 kafka-event-bus.services.eqiad.wmflabs
[16:38:50] <ottomata>	 log in there and you can do that
[16:38:53] <ottomata>	 oh, if you run it, i can run it now
[16:38:55] <nuria>	 ottomata: ya, i just instantiated everything (I have comments about this but we can talk later about dev env)
[16:39:04] <nuria>	  ./bin/eventlogging-service kafka:///localhost:9092  --config ./schemas/config.json  --port 8085
[16:39:10] <nuria>	 i instantiated stuff like this
[16:39:10] <ottomata>	 ah
[16:39:17] <nuria>	 'with a local kafka install
[16:39:21] <ottomata>	 oh ok
[16:39:21] <ottomata>	 cool
[16:39:22] <ottomata>	 yeah
[16:39:24] <ottomata>	 then
[16:39:53] <ottomata>	  curl -H 'Content-Type: application/json' -X POST http://localhost:8085/v1/event/mediawiki_EditEventBus -d @EditEventBusRecord2.json
[16:39:54] <ottomata>	 should work
[16:40:19] <ottomata>	 nuria
[16:40:24] <ottomata>	 you need a topic param on that kafka uri though
[16:40:29] <ottomata>	 add
[16:40:30] <ottomata>	 ?topic=%(meta.topic)s'
[16:40:31] <ottomata>	 to it
[16:40:36] <nuria>	 k
[16:40:50] <ottomata>	 also, i may have pushed a new patch since you pulled
[16:40:55] <ottomata>	 last night i worked on this til way too late
[16:41:02] <ottomata>	 (which is against my rules and not good for my brain!)
[16:41:03] <ottomata>	 :p
[16:42:06] <nuria>	 ok, let me check couple things
[16:42:07] <ottomata>	 milimetric: yt?
[16:42:21] <milimetric>	 yes, but on-on-one-ing with kev
[16:42:23] <ottomata>	 k
[16:42:33] <ottomata>	 had a thought about the dotted interpolation thing i wanted to run past ya
[16:44:07] <ottomata>	 haha, nuria except my new patch is broken....good old late night push
[16:44:08] <ottomata>	 fixing
[17:07:09] <wikibugs>	 Analytics-EventLogging, Editing-Department, Improving access, QuickSurveys, and 5 others: QuickSurveys: Schema changes - https://phabricator.wikimedia.org/T114164#1777856 (leila) @Jdlrobson: are we capturing   unhashed IP userAgent x_forwarded_for browser_language uri_query (bonus)  in the updated...
[17:07:35] <wikibugs>	 Analytics-Tech-community-metrics, DevRel-November-2015: Correct affiliation for code review contributors of the past 30 days - https://phabricator.wikimedia.org/T112527#1777857 (Aklapper) Open>Resolved >>! In T112527#1770608, @Aklapper wrote: > Once that's reflected in korma we can close this task....
[17:07:36] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations, DevRel-November-2015: Who are the top 50 independent contributors and what do they need from the WMF? - https://phabricator.wikimedia.org/T85600#1777859 (Aklapper)
[17:09:24] <ebernhardson>	 for an hourly job that does an `alter table ... add partition ...` in hive, should that be an oozie job or just a cron on stat1002? if oozie, is there any docs for the right way to deploy that?
[17:10:52] <ebernhardson>	 also, relatedly, the CirrusSearchRequestSet is now sending unsampled logs to kafka as of ~ 1 hour ago
[17:11:02] <ebernhardson>	 it was sending 1:1000 samples overnight and everything looked reasonable
[17:12:05] <ottomata>	 ebernhardson: awesome!
[17:12:17] <ottomata>	 ebernhardson:  oozie is better, because then you can do it if the data actually exists
[17:12:20] <ottomata>	 not just on current time
[17:12:27] <joal>	 ottomata: no visible diff in grafana for kafka - normal ?
[17:12:39] <ottomata>	 ebernhardson:  how much data is unsampled?
[17:13:07] <ebernhardson>	 ottomata: extrapolating from 1:1000, about ~500M per hour
[17:13:33] <ebernhardson>	 oddly in the last hour though it hasn't written nearly that much, but maybe camus isn't quite realtime? i don't know
[17:13:48] <wikibugs>	 Analytics-Wikistats, Internet-Archive: Discrepancies in historical total active editor numbers - https://phabricator.wikimedia.org/T87738#1777865 (Nemo_bis) Particularly puzzling for me are [[https://web.archive.org/web/20150919045521/http://stats.wikimedia.org/wikisource/EN/TablesWikipediansEditsGt5.htm...
[17:14:17] <ottomata>	 ebernhardson: , joal its here
[17:14:17] <ottomata>	 https://graphite.wikimedia.org/render/?width=588&height=311&target=kafka.kafka1012_eqiad_wmnet_9999.kafka.server.BrokerTopicMetrics.MessagesInPerSec.mediawiki_CirrusSearchRequestSet.OneMinuteRate
[17:14:26] <travis-ci>	 wikimedia/mediawiki-extensions-EventLogging#504 (wmf/1.27.0-wmf.5 - b418df6 : Mukunda Modell): The build has errored.
[17:14:26] <travis-ci>	 Change view : https://github.com/wikimedia/mediawiki-extensions-EventLogging/commit/b418df6afe07
[17:14:27] <travis-ci>	 Build details : https://travis-ci.org/wikimedia/mediawiki-extensions-EventLogging/builds/89051566
[17:14:40] <joal>	 thx ottomata :)
[17:14:47] <ottomata>	 joal: + 250msgs per sec, so probably won't show in main kafka dashboard
[17:14:57] <joal>	 right
[17:14:58] <ottomata>	 ebernhardson:  it isn't realtime
[17:15:26] <ottomata>	 camus is run by cron every 15 mins
[17:16:21] <ebernhardson>	 ottomata: is that 250msgs/s just one partition (kafka1012), or is that aggregate? our other logs were seeing 1500-3000req/s. These new logs aggregate a few things together, but to go from 250 -> 1500 would mean we run an average 6 requests per mediawiki request, which seems off
[17:16:22] <wikibugs>	 Analytics-Wikistats: Discrepancies in historical total active editor numbers - https://phabricator.wikimedia.org/T87738#1777898 (Nemo_bis)
[17:17:10] <ebernhardson>	 our old log, CirrusSearchRequests is currently doing right around 3k/s
[17:17:53] <ottomata>	 ah yes
[17:17:54] <dcausse>	 isn't it per broker?
[17:17:57] <ottomata>	 sorry that is just one broker
[17:18:02] <ebernhardson>	 ok, that makes more sense then :)
[17:18:16] <ebernhardson>	 might still not be noticable next to the webrequest traffic :)
[17:18:37] <ottomata>	 ja, should be that * 6
[17:18:42] <ottomata>	 so sounds rigiht
[17:19:07] <ottomata>	 ebernhardson:  we just changed how we use oozie to add partitions
[17:19:13] <ottomata>	 you could very well KISS and do it in a cron
[17:19:21] <ottomata>	 once an hour just check if the dir exists, and then add the partition.
[17:19:27] <ottomata>	 won't hurt anything if it doesn't
[17:19:59] <ebernhardson>	 ottomata: ok, i imagine we also need to drop the old partitions (90 days?)
[17:20:11] <ottomata>	 ebernhardson: https://github.com/wikimedia/analytics-refinery/tree/master/oozie/util/hive/partition/add
[17:20:12] <ottomata>	 yes
[17:20:20] <ottomata>	 see also
[17:20:31] <ottomata>	 that I do with cron
[17:20:31] <ottomata>	 https://github.com/wikimedia/analytics-refinery/blob/master/bin/refinery-drop-webrequest-partitions
[17:20:35] <ottomata>	 https://github.com/wikimedia/analytics-refinery/blob/master/bin/refinery-drop-eventlogging-partitions
[17:20:49] <ebernhardson>	 awsome, thanks!
[17:21:05] <ottomata>	 also ebernhardson https://github.com/wikimedia/analytics-refinery/tree/master/oozie/webrequest/load
[17:21:12] <ottomata>	 yours will be much simpler though
[17:21:21] <ottomata>	 https://github.com/wikimedia/analytics-refinery/blob/master/oozie/webrequest/load/workflow.xml#L89
[17:34:15] <wikibugs>	 Analytics-Wikistats: Discrepancies in historical total active editor numbers - https://phabricator.wikimedia.org/T87738#1777999 (ezachte) > As [[https://phabricator.wikimedia.org/T87738#1347875  |suggested in June]], I think that lists of users who were counted in one dump but not another might be useful for...
[17:40:10] <wikibugs>	 Analytics-Backlog, Wikimedia-Mailing-lists, operations: Requests to lists.wikimedia.org should end up in hadoop wmf.webrequest via kafka! - https://phabricator.wikimedia.org/T116429#1778032 (JohnLewis) Open>declined a:JohnLewis Closing as declined for several reasons:  * requires Varnish, whic...
[17:45:59] <grrrit-wm>	 (CR) Milimetric: [C: 2 V: 2] Add .gitreview [analytics/limn-ee-data] - https://gerrit.wikimedia.org/r/249393 (owner: Matthias Mullie)
[18:02:03] <nuria>	 ebernhardson: oozie docs; https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Oozie
[18:04:53] <wikibugs>	 Analytics-EventLogging, Editing-Department, Improving access, QuickSurveys, and 5 others: QuickSurveys: Schema changes - https://phabricator.wikimedia.org/T114164#1778130 (jhobs) >>! In T114164#1777856, @leila wrote: > @Jdlrobson: are we capturing >   > unhashed IP > userAgent > x_forwarded_for >...
[18:17:32] <ottomata>	 milimetric:  hiya?
[18:17:38] <milimetric>	 hey
[18:17:43] <ottomata>	 so thought
[18:17:51] <milimetric>	 shoot
[18:17:56] <ottomata>	 i made a function that abstracts where the meta data lives in an event
[18:18:09] <ottomata>	 this makes the whole of eventlogging work with old capsule style, and also new meta subobject style
[18:18:22] <ottomata>	 instead of using dotted to support interpolation of any key in the event
[18:18:33] <ottomata>	 i could just say that interpolation is always done against the meta data
[18:18:43] <milimetric>	 right, that works
[18:18:53] <ottomata>	 it works for all current use cases I can think of
[18:18:56] <ottomata>	 it is less flexible
[18:19:33] <milimetric>	 why less flexible?
[18:19:37] <ottomata>	 but, probably it is better, since the keys in the meta data are more tightly controlled, so you can be more sure that the keys are interpoloate exist
[18:19:46] <milimetric>	 you mean you can't just configure in puppet what the key is
[18:19:46] <ottomata>	 well, in case someone wanted to configure some special kafka writer
[18:19:53] <ottomata>	 that interpolates agains an event content key
[18:19:55] <ottomata>	 no
[18:20:07] <ottomata>	 i mean, say someone wanted to produce to topics based on i dunno, page_title for some weird reason
[18:20:12] <ottomata>	 that's not event metadata
[18:20:16] <ottomata>	 so they can't do
[18:20:28] <ottomata>	 topic=page_%(page_title)s
[18:21:45] <milimetric>	 that's what I mean, you can't configure in puppet what the key to use for the topic is
[18:22:07] <ottomata>	 ?  puppet isn't really relevant here?
[18:22:25] <ottomata>	 oh, but i still interpolate against meta data
[18:22:27] <milimetric>	 isn't that where you'd configure the url that the kafka_writer ends up using?
[18:22:31] <ottomata>	 and that is where topic, and schema live
[18:22:34] <ottomata>	 yes
[18:22:39] <ottomata>	 but either way its the same
[18:22:49] <ottomata>	 so, if i only interpolate against the meta object
[18:22:50] <ottomata>	 i can do
[18:23:02] <ottomata>	 topic=%(topic)s
[18:23:02] <ottomata>	 or
[18:23:11] <ottomata>	 topic=eventlogging_%(schema)s
[18:23:18] <ottomata>	 that still works
[18:23:24] <ottomata>	 because those are explicitly event meta data fields
[18:23:32] <milimetric>	 if there's a meta.schema or meta.topic
[18:23:37] <ottomata>	 sure
[18:23:44] <milimetric>	 what I'm saying, in trying to understand why you say "less flexible"
[18:23:46] <ottomata>	 or just top level schema or topic (i've abstracted finding the meta)
[18:23:53] <ottomata>	 ok, say obj is
[18:24:15] <milimetric>	 is that in this way, you can't use %(some.random.path) from puppet, you can only use %(some.path.that.assumes.meta.prefix)
[18:24:16] <ottomata>	 { 'page_title': 'Boogers', meta: { 'schema': 'Page' ... } }
[18:24:32] <ottomata>	 if I always interpolate against the meta object
[18:24:37] <milimetric>	 and by meta prefix I mean, it works with both meta object or capsule
[18:24:40] <ottomata>	 page_title is not an available interpolation key
[18:24:42] <ottomata>	 yes
[18:24:44] <ottomata>	 exactly
[18:24:56] <ottomata>	 and
[18:24:56] <ottomata>	 so
[18:24:57] <milimetric>	 ok, good, so I understand.
[18:25:01] <ottomata>	 maybe for topic this isn't a big deal
[18:25:06] <milimetric>	 I don't think that loss of flexibility is a big deal then
[18:25:10] <ottomata>	 but we also support interpolation for message keys
[18:25:14] <milimetric>	 as long as it's well documented
[18:25:32] <ottomata>	 say you wanted your parallelized event consumers to each consume messages by key
[18:25:39] <ottomata>	 that is, maybe you are modeling user behavior
[18:25:49] <ottomata>	 so you need any given process to get all of the events from that user
[18:25:56] <ottomata>	 in kafka, this is done by keys
[18:26:00] <ottomata>	 on the message
[18:26:06] <ottomata>	 we don't have any consumers doing this right now
[18:26:11] <ottomata>	 but, if we only interpolate against meta
[18:26:16] <ottomata>	 this wouldn't be possible
[18:26:45] <ottomata>	 that is, one coudln't do
[18:27:00] <ottomata>	 key=%(user_id)s
[18:27:06] <milimetric>	 oh wait... you mean all of interpolation would assume you're looking at meta fields?
[18:27:06] <milimetric>	 not just topic interpolation... right
[18:27:11] <ottomata>	 yes, exactly
[18:30:45] <milimetric>	 hm
[18:31:04] <milimetric>	 what if we made the key meta.something.someone
[18:31:14] <milimetric>	 and then translated "meta" to work with either meta or capsule
[18:31:35] <milimetric>	 so then we can have any arbitrary key and also be backwards compatible with capsule for meta fields
[18:31:50] <ottomata>	 hm, you mean just special case meta.?
[18:31:51] <ottomata>	 like
[18:31:52] <milimetric>	 then the only "trick" would be if your event is encapsulated you'd need to know "meta" means something special
[18:31:53] <milimetric>	 yes
[18:31:59] <ottomata>	 if strcontains meta., use meta object?
[18:32:04] <milimetric>	 right
[18:32:07] <milimetric>	 and strip meta. out
[18:32:15] <ottomata>	 hm
[18:32:23] <ottomata>	 that would work.
[18:32:37] <ottomata>	 kinda hacky
[18:32:45] <ottomata>	 i'd have to regex the key out of the format string
[18:33:08] <ottomata>	 eventlogging_%(meta.topic)s
[18:33:54] <wikibugs>	 Analytics, Discovery, EventBus, MediaWiki-General-or-Unknown, and 6 others: Define edit related events for change propagation - https://phabricator.wikimedia.org/T116247#1778254 (Ottomata) Cool, added some comments.
[18:34:40] <wikibugs>	 Analytics-EventLogging, Editing-Department, Improving access, QuickSurveys, and 5 others: QuickSurveys: Schema changes - https://phabricator.wikimedia.org/T114164#1778261 (leila) Thanks @jhobs. Those were also captured as part of this task, I don't think we should close this task before addressing...
[18:34:40] <ottomata>	 hm, milimetric we could make it work, but i don't really like it
[18:34:44] <milimetric>	 :)
[18:34:46] <ottomata>	 i mena, dotted works...>.>.:/
[18:34:53] <ottomata>	 as is, but you say its lame?
[18:35:09] <milimetric>	 oh, the dotted key idea is fine, i think that's nice for readability
[18:35:16] <milimetric>	 just the code sucks, we could easily write our own
[18:35:38] <milimetric>	 I'll do that if you like
[18:36:49] <milimetric>	 but dotted works and it's not backwards compatible with encapsulated events
[18:37:11] <ottomata>	 oh it is though
[18:37:14] <milimetric>	 could we, instead of trying to deal with both kinds of events, just morph the encapsulated event into a { meta: {capsule} ... }?
[18:37:15] <ottomata>	 oh i see
[18:37:16] <ottomata>	 hm
[18:37:22] <ottomata>	 yeah, the same writer wouldn't be able to do encapsulated
[18:37:37] <wikibugs>	 Analytics-EventLogging, Editing-Department, Improving access, QuickSurveys, and 5 others: QuickSurveys: Schema changes - https://phabricator.wikimedia.org/T114164#1778289 (jhobs) They were captured as "future requests" though, so it might be worth closing this one and splitting those out to anothe...
[18:37:47] <ottomata>	 milimetric:  i sorta did that, sorta
[18:37:48] <ottomata>	 but just for meta
[18:37:50] <ottomata>	 um
[18:37:52] <wikibugs>	 Analytics-EventLogging, Editing-Department, Improving access, QuickSurveys, and 5 others: QuickSurveys: Schema changes - https://phabricator.wikimedia.org/T114164#1778290 (leila)
[18:38:01] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/235671/13/server/eventlogging/schema.py
[18:38:09] <milimetric>	 if we did that, then we can just leave dotted to do its thing on the key and use meta.one.two
[18:38:10] <ottomata>	 line 199
[18:38:11] <ottomata>	 meta_from_event
[18:38:39] <ottomata>	 but you are saying kinda do the opposite
[18:38:46] <ottomata>	 modify the event to have meta: with capsule fields
[18:39:12] <milimetric>	 right, so that function gets the meta object, i mean just change the event itself
[18:39:18] <ottomata>	 i don't like the idea of modifying the event either.  i mean, i could.  i'd still have to actually produce the original event
[18:39:20] <wikibugs>	 Analytics-EventLogging, Editing-Department, Improving access, QuickSurveys, and 5 others: QuickSurveys: Schema changes - https://phabricator.wikimedia.org/T114164#1686646 (leila) "furthe requests" which was really "further". ;-) I'm happy to open another ticket for this @jhobs. Whatever works bett...
[18:39:24] <ottomata>	 but we could modify a temp one for interpolation
[18:39:41] <milimetric>	 so if you pass in: { schema, revision, event: { ... } } you get back out { ... , meta: { schema, revision } }
[18:39:50] <milimetric>	 right, that's ok too
[18:39:51] <ottomata>	 aye, hm
[18:40:01] <ottomata>	 that will work and solve all problems
[18:40:06] <milimetric>	 yeah, temp one would be fine, though this is something we should figure out and test
[18:40:10] <ottomata>	 extra work for this thang to do, but whatev i guess :/
[18:40:13] <milimetric>	 performance wise
[18:40:15] <ottomata>	 yeah
[18:40:16] <milimetric>	 right
[18:40:38] <ottomata>	 ok, milimetric i'm not going to do that now as part of this patch, but will make a note of the idea in comments
[18:40:50] <ottomata>	 the service is meant to only work with meta style events atm anyway
[18:41:08] <milimetric>	 sorry i'm trying to watch this tech talk now, but I should really look at the whole pipeline from where the event gets encapsulated to where it gets used in either shape
[18:41:19] <milimetric>	 intuitively this is pretty insane :)
[18:41:32] <milimetric>	 cool, note for now makes the most sense
[18:43:30] <wikibugs>	 Analytics-EventLogging, Editing-Department, Improving access, QuickSurveys, and 5 others: QuickSurveys: Schema changes - https://phabricator.wikimedia.org/T114164#1778341 (jhobs) Oh I misread that line.  Regardless, I think T117577 should suffice as a "new task" for that work and can probably be p...
[18:43:44] <wikibugs>	 Analytics-EventLogging, Editing-Department, Improving access, QuickSurveys, and 5 others: QuickSurveys: Schema changes - https://phabricator.wikimedia.org/T114164#1778343 (jhobs)
[18:46:08] <ottomata>	 nuria:  hi?
[18:47:41] <nuria>	 ottomata: holaa
[18:48:29] <ottomata>	 batcave?
[18:53:02] <wikibugs>	 Analytics-Backlog, Wikipedia-iOS-App-Product-Backlog, hardware-requests, operations, and 2 others: Request one server to suport piwik analytics - https://phabricator.wikimedia.org/T116312#1778402 (RobH) I'm going to add #vm-requests to this and remove #hardware-requests.  It appears this can live i...
[18:53:10] <wikibugs>	 Analytics-Backlog, Wikipedia-iOS-App-Product-Backlog, hardware-requests, operations, and 2 others: Request one server to suport piwik analytics - https://phabricator.wikimedia.org/T116312#1778405 (RobH) a:JMinor
[18:53:19] <wikibugs>	 Analytics-Backlog, Wikipedia-iOS-App-Product-Backlog, operations, vm-requests, iOS-5-app-production: Request one server to suport piwik analytics - https://phabricator.wikimedia.org/T116312#1746196 (RobH)
[18:57:02] <nuria>	 ottomata: sure
[19:09:44] <grrrit-wm>	 (PS1) Mforns: Optimize queries [analytics/limn-ee-data] - https://gerrit.wikimedia.org/r/250727 (https://phabricator.wikimedia.org/T117220)
[19:26:21] <joal>	 halfak: I have a jar :)
[19:26:39] <halfak>	 Does it have grape jelly in it?
[19:26:40] <halfak>	 ;)
[19:26:51] <joal>	 Only wine :D
[19:26:55] <halfak>	 ooooh!
[19:27:02] <joal>	 French jar you know
[19:27:32] <joal>	 So, still testing a new function, but the default works
[19:27:52] <joal>	 halfak: do you have a full enwiki for me to test generating sorted flat json ?
[19:28:15] <halfak>	 Yes.  Will get path
[19:28:29] <halfak>	  /user/halfak/streaming/enwiki-20150901/dump
[19:29:56] <grrrit-wm>	 (CR) Milimetric: [C: 2 V: 2] "Makes no sense, but I can't argue with performance!" [analytics/limn-ee-data] - https://gerrit.wikimedia.org/r/250727 (https://phabricator.wikimedia.org/T117220) (owner: Mforns)
[19:30:03] <wikibugs>	 Analytics-EventLogging, MediaWiki-extensions-RelatedArticles, MobileFrontend, Patch-For-Review, Reading Web Sprint 59 - Amsterdam and the hamsters: Upstream Schema.js from MobileFrontend to EventLogging - https://phabricator.wikimedia.org/T117140#1778543 (Jdlrobson) p:Triage>Normal
[19:30:29] <wikibugs>	 Analytics-Backlog, Analytics-Wikistats, DevRel-November-2015: Clean the code review queue of analytics/wikistats - https://phabricator.wikimedia.org/T113695#1778552 (Qgil)
[19:31:51] <joal>	 halfak: test launched
[19:32:00] <halfak>	 Woot.
[19:32:06] * halfak looks forward to new data
[19:32:19] <halfak>	 Oh!  Also, it looks like altiscale came through with notes about the memory issues.
[19:32:39] <halfak>	 It looks like the problem is my script.  There's some degenerate input data that happens really really rarely.
[19:32:39] <joal>	 good news ?
[19:32:43] <halfak>	 So I'm looking into that
[19:32:52] <halfak>	 The sad thing is that we just might not be able to do this in hadoop.
[19:32:55] <joal>	 yeah, the unfortunate spike
[19:33:02] <halfak>	 We need flexible memory space.
[19:33:29] <halfak>	 99.5% of the time, we need 300MB of memory.  0.5% of the time, we need 4GB
[19:33:31] <joal>	 hm, I guess we'll discuss that :)
[19:33:56] <halfak>	 Maybe I can figure out what makes this so degenerate and deal with it in the computation.
[19:34:17] <joal>	 Science loves heuristics halfak :)
[19:34:58] <halfak>	 ^ wut
[19:35:46] <joal>	 halfak: other way is to monitor memory space inside the script, but not cool either
[19:35:56] <joal>	 We'll talk again about that I'm sure :)
[19:36:15] <halfak>	 Yeah.  Definitely don't want to do that if I can avoid it.
[19:36:44] <halfak>	 It would be great if I could have an LRU that would trim old words from the revision window if we ended up keeping too many words in memory.
[19:36:56] <halfak>	 The problem is that this is where I rely on garbage collections and references.
[19:37:00] <halfak>	 >:(
[19:37:24] <joal>	 hm ... Another way would to force GC when too many words ?
[19:37:37] <joal>	 Don't know much about python GC though
[19:38:50] * joal is going to have diner !
[19:38:54] <halfak>	 o/
[19:39:00] <halfak>	 Thanks for the jar of wine :D
[19:39:31] <joal>	 no more after my diner halfak : french people ~ wineh
[19:39:40] <joal>	 :)
[19:45:16] <wikibugs>	 Analytics-Engineering, Community-Tech: Add page view statistics to page information pages (action=info) [AOI] - https://phabricator.wikimedia.org/T110147#1569847 (DannyH)
[19:45:31] <wikibugs>	 Analytics-Tech-community-metrics, Phabricator, DevRel-November-2015: Closed tickets in Bugzilla migrated without closing event? - https://phabricator.wikimedia.org/T107254#1778718 (Aklapper) Continuing on my comment in T107254#1748539 >>! In T107254#1748811, @Aklapper wrote: > Starring at https://git.w...
[19:48:04] <joal>	 mobrovac: I am really sorry, time flew today
[19:48:22] <joal>	 I can help either now, or tomorrow if you prefer
[20:43:42] <nuria>	 ottomata: what is teh yaml file you have installed?
[20:43:45] <nuria>	 *the
[20:43:53] <nuria>	 Yaml package that is
[20:44:40] <ottomata>	 nuria?
[20:44:43] <ottomata>	 just built in?
[20:44:44] <ottomata>	 yaml?
[20:44:47] <ottomata>	 import yaml
[20:44:56] <nuria>	 on mine it fails
[20:45:00] <ottomata>	 hm
[20:45:22] <ottomata>	 hm maybe not
[20:45:22] <ottomata>	 PyYAML
[20:45:22] <ottomata>	 ?
[20:45:23] <nuria>	 https://www.irccloud.com/pastebin/fmAaeZzJ/
[20:46:16] <nuria>	 ottomata: ah no, i know...
[20:46:24] <nuria>	 ottomata: it needs to be in requirements , sorry
[20:46:53] <milimetric>	 https://github.com/wikimedia/analytics-wikimetrics/blob/master/requirements.txt#L11
[20:47:04] <milimetric>	 pyaml in requirements ^
[20:47:32] <nuria>	 right, right,
[20:48:33] <milimetric>	 (just pasted 'cause it's really annoying to look up what "import yaml" translates to, which is one of my least favorite things about python.)
[20:54:02] <ottomata>	 aye
[20:57:12] <ottomata>	 nuria: added PyYAML to requirements
[20:57:44] <nuria>	 ottomata: ya, i did too, and tornado. i will make a small patch on top of yours to see how easy is to test, ok?
[20:57:52] <ottomata>	 k
[21:05:52] <wikibugs>	 Analytics, Analytics-Kanban, Discovery, EventBus, and 8 others: EventBus MVP - https://phabricator.wikimedia.org/T114443#1779098 (Ottomata) Ok, still various TODOs around the code, but this is ready for review. https://gerrit.wikimedia.org/r/#/c/235671  There are concepts that it'll be good to do c...
[21:16:06] <wikibugs>	 Analytics-Kanban: Understand the Perl code for Client OS Major Minor Version report {lama} - https://phabricator.wikimedia.org/T117246#1779167 (Milimetric) a:Milimetric
[22:05:09] <addshore>	 ottomata: any chance I could get some comments on or +/-1/2 on https://gerrit.wikimedia.org/r/#/c/247866/ ? :) (trying to push it forward) ;)
[22:07:47] <ottomata>	 hmmm, interesting
[22:08:13] <ottomata>	 addshore:  i guess so, although i'm not crazy about having to prefix everything with daily for this
[22:08:17] <ottomata>	 have you talked with anyone else about this?
[22:08:23] <ottomata>	 any other opsens?
[22:09:08] <addshore>	 I pinged ori yesterday, but dont think he saw it, silly timezones.
[22:09:28] <addshore>	 well, afaik no daily metrics are currently tracked in grpahite
[22:10:47] <addshore>	 Also there was this in May to save space https://github.com/wikimedia/operations-puppet/commit/eae3b530d160419bfc6d53aead968b73de09aa1f#diff-247ef67596593fbfa2bc608e99e39989
[22:11:21] <addshore>	 so I don't think adding longer retention to all metrics would be the way forward
[22:16:48] <addshore>	 but yeh, any comments there or directions to other people to poke would be great ;)
[22:20:30] <ottomata>	 addshore:  yeah indeed, i think you should poke Filippo (godog)
[22:20:37] <ottomata>	 he does a lot of graphite stuff
[22:25:57] <addshore>	 He is the same timezone as me and away currently so I will try to catch him tomorrow :)
[22:27:01] <addshore>	 many thanks!
[22:42:52] <ottomata>	 k, yup! latesr!
[22:54:09] <wikibugs>	 Analytics: Provide weekly app session metrics separately for Android and iOS - https://phabricator.wikimedia.org/T117615#1779777 (Tbayer) NEW
[23:06:31] <ottomata>	 bye all
[23:06:32] <ottomata>	 !
[23:22:12] <wikibugs>	 Analytics-EventLogging, MediaWiki-extensions-RelatedArticles, MobileFrontend, Patch-For-Review, Reading Web Sprint 59 - Amsterdam and the hamsters: Upstream Schema.js from MobileFrontend to EventLogging - https://phabricator.wikimedia.org/T117140#1779926 (Jdlrobson) Waiting for some responses f...