[00:14:59] <neilpquinn>	 halfak: any idea why http://gp.wmflabs.org/graphs/active_editors_total shows the spike but http://reportcard.wmflabs.org/graphs/active_editors doesn't?
[01:34:11] <wikibugs>	 Analytics, Analytics-Backlog, Performance-Team, Patch-For-Review: Collect HTTP statistics about load.php requests - https://phabricator.wikimedia.org/T104277#1439939 (Catrope) On the backend, we should see zero INM/304 traffic for versioned (long cache) requests, and very little 200 traffic and only...
[02:02:57] <halfak>	 neilpquinn, sorry, not sure how those graphs are generated.
[02:04:10] <neilpquinn>	 No worries! It seems like the Wikistats/Report Card numbers are more reliable anyway.
[12:02:32] <milimetric>	 neilpquinn: that graph is based on data that's not as carefully refined as the wikistats ones.  This note talks about that a bit (underneath the graph): "This graph currently over-reports by counting each active editor once for each distinct pair of project and country associated to the IP addresses used by the editor."  In contrast, wikistats carefully
[12:02:32] <milimetric>	 de-duplicates across projects.
[12:49:00] <joal>	 Ironholds: Hi !
[12:59:14] <joal>	 halfak: Hi :)
[12:59:23] <halfak>	 o/ joal
[12:59:46] <joal>	 halfak: I have forgotten to ask you
[12:59:52] <halfak>	 joal, we're in the reduce stage of that diff job :)
[13:00:02] <joal>	 halfak: awesome
[13:00:15] <joal>	 Started long ago (reduce) ?
[13:00:34] <halfak>	 at about 1200 UTC today
[13:00:40] <halfak>	 So, not long
[13:00:43] <joal>	 halfak: Very interesting presentations sent by Aravind
[13:00:57] <halfak>	 Re-snappy splits?
[13:01:02] <joal>	 yup
[13:01:29] <joal>	 I am not sure about why, but snappy is not splittable when using text
[13:01:52] <joal>	 So I'll change the json converter to use bz2
[13:01:56] <halfak>	 bz2 for the future!
[13:01:59] <joal>	 :D
[13:01:59] <halfak>	 :)
[13:02:13] <joal>	 I also have a request for you
[13:02:38] <joal>	 Do you mind having a look at https://phabricator.wikimedia.org/T102161
[13:02:43] <joal>	 halfak: --^
[13:03:07] <joal>	 Vetting new pageview aggregation
[13:33:35] <wikibugs>	 Analytics-Cluster, Analytics-Kanban: Generate test data for Pageview API {slug} [5 pts] - https://phabricator.wikimedia.org/T101785#1441180 (JAllemandou) Distribution analysis for both page_title_hourly and project_cube  {F190564}  {F190565}  {F190581}     # page_title_hourly       - ~60% of page_titles h...
[13:34:10] <joal>	 halfak: yt ?
[13:34:29] <halfak>	 Sorry.  Meeting!
[13:39:44] <joal>	 halfak: np :)
[13:47:22] <joal>	 Ironholds: Hi again :)
[14:18:09] <milimetric>	 joal: so the aggregations you just merged from marcel should / would end up in https://metrics.wmflabs.org/static/public/datafiles/Pageviews/all.csv when deployed, right?
[14:18:32] <joal>	 milimetric: correct !
[14:18:40] <milimetric>	 cool, thx, any ETA on that?
[14:18:42] <joal>	 jgage: Hi Sir
[14:18:47] <milimetric>	 are you waiting for anything to deploy?
[14:19:02] <joal>	 Depends on jgage --> need a puppet merger
[14:20:04] <joal>	 so if it's not today, will be early next week with Andrew
[14:20:08] <joal>	 milimetric: --^
[14:20:14] <joal>	 I'm off tomorrow
[14:20:27] <milimetric>	 cool, makes sense
[14:20:28] <milimetric>	 thx
[14:20:32] <joal>	 np
[14:20:46] <joal>	 milimetric: interesting results about page_title viewcounts
[14:21:00] <joal>	 https://phabricator.wikimedia.org/T101785#1441180
[14:21:16] <milimetric>	 I'll read that in a sec, about to push a patch
[14:21:27] <joal>	 milimetric: np
[14:22:57] <wikibugs>	 Analytics-Backlog: Sanitize aggregated data presented in VitalSign using K-Anonymity {musk} [8 pts] - https://phabricator.wikimedia.org/T104485#1441340 (JAllemandou) Analysis made here : https://phabricator.wikimedia.org/T101785#1441180
[14:33:27] <grrrit-wm>	 (PS1) Milimetric: Handle all.csv whether or not it exists [analytics/dashiki] - https://gerrit.wikimedia.org/r/223789 (https://phabricator.wikimedia.org/T95340)
[14:40:42] <halfak>	 joal, just finished reviewing that phab task.
[14:40:46] <halfak>	 All looks reasonable to me.
[14:41:17] <halfak>	 I know that Ironholds did some work to vet this in the past, so I expect he'll have some suggestions on things to look for.
[14:41:41] <Ironholds>	 what did I do?
[14:41:44] <Ironholds>	 it wasn't me, someone else did it
[14:41:51] <halfak>	 :D
[14:41:59] <halfak>	 hey dude.  See https://phabricator.wikimedia.org/T102161
[14:42:07] <halfak>	 I think all you need to do is read and comment :)
[14:42:15] <halfak>	 "need"
[14:42:29] <halfak>	 NEED URGENT OMG ;)
[14:42:43] <halfak>	 Speaking of urgent
[14:42:49] * halfak gets back to his most recent urgent work
[14:43:25] <Ironholds>	 read it!
[14:43:29] <Ironholds>	 looks good! URL decoding is vital
[14:43:59] <halfak>	 +1
[14:44:50] <joal>	 halfak, Ironholds : Thanks both of you !
[14:45:31] <joal>	 halfak: I promise to tag tasks as urgent next time ;-P
[14:45:32] <Ironholds>	 thank you for doing the work! :D
[14:45:40] <Ironholds>	 I'm totally out of the loop; no access to anything for two weeks. bleh
[14:46:06] <joal>	 Ironholds: no problemo, I just wanted to have that confimred before wikimania :)
[14:47:26] <Ironholds>	 gotcha!
[14:47:35] <Ironholds>	 super excited by how far people got while I was away :D
[14:52:30] <grrrit-wm>	 (PS1) Milimetric: Add overall aggregate to project and language list [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/223793 (https://phabricator.wikimedia.org/T95340)
[15:00:41] <joal>	 Ironholds: while you're at it, can you have a look at k-anonymisation issue: https://phabricator.wikimedia.org/T101785
[15:01:47] <Ironholds>	 joal, will do!
[15:02:01] <joal>	 Ironholds: Thx ! Will you be at Wikimania ?
[15:03:37] <Ironholds>	 nope!
[15:03:47] <Ironholds>	 halfak, etherpad link? I wanna give a report :D
[15:04:08] <halfak>	 huh?
[15:04:25] <halfak>	 Oh yeah.  The haiku lacked a link!
[15:04:52] <halfak>	 http://etherpad.wikimedia.org/p/WMF_Research_Group
[15:19:30] <grrrit-wm>	 (PS2) Milimetric: Add overall aggregate to project and language list [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/223793 (https://phabricator.wikimedia.org/T95340)
[15:37:39] <joal>	 Hey jgage, let me know when you're around :)
[16:43:07] <leila>	 hi madhuvishy. are we using persistent cookies or session cookies for lastAccess?
[16:52:38] <madhuvishy>	 leila: I think it's persistent over a 31 day period
[16:55:03] <jgage>	 joal: good morning!
[16:55:23] <joal>	 Hey jgage :)
[16:55:24] <leila>	 got it. thanks madhuvishy. looking at counting options, while you're pushing on the other fronts.
[16:55:29] <joal>	 Good morning !
[16:56:04] <madhuvishy>	 leila: awesome, thanks
[16:56:18] <joal>	 jgage: Would you have some time for (really easy) review ?
[16:56:22] <jgage>	 joal: ok, i see the change from mforns and i shall merge it
[16:56:33] <joal>	 jgage: You rock :)
[16:57:16] <jgage>	 joal: what host will this be applied to? i'll run puppet there after i merge.
[16:57:41] <joal>	 jgage: stat1002
[16:57:48] <jgage>	 ok, one moment please
[16:57:51] <joal>	 :)
[16:59:56] <jgage>	 joal: ok, merged and applied :)
[17:00:12] <joal>	 Thanks a lot jgage :)
[17:00:16] <jgage>	 my pleasure
[17:00:44] <joal>	 Have a good day :)
[17:00:59] <jgage>	 thanks! you have a nice.. evening ;)
[17:01:23] <joal>	 thanx ;)
[17:03:27] <wikibugs>	 Analytics-EventLogging, Analytics-Kanban: Can Search up sampling to 5%? {oryx} - https://phabricator.wikimedia.org/T103186#1442048 (Ironholds) Okay!  So: in my mind throwing it in HDFS is not the best solution. If we want to throw /all/ eventlogging data in HDFS, great. If we want to throw /no/ eventloggi...
[17:25:59] <joal>	 lzia: Hi leila
[17:26:18] <lzia>	 helloo joal. What's up? :-)
[17:26:52] <joal>	 I just realized I didn't let you know I won't make it tonight to the checkpoint, and that Andrew's in holidays
[17:27:17] <joal>	 Sooooo, I guess it should probably be cancelled
[17:27:26] <joal>	 leila: --^
[17:27:46] <leila>	 np. thanks for the heads up, we can talk more in Mexico.
[17:29:07] <joal>	 leila: for sure, looking forward to that :)
[17:29:12] <joal>	 Have a good day "!
[17:31:42] <grrrit-wm>	 (CR) Joal: "Except for the typo in desccription, looks good to me :)" (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/223793 (https://phabricator.wikimedia.org/T95340) (owner: Milimetric)
[17:33:43] <joal>	 mforns: you there ?
[17:33:45] <milimetric>	 mforns: retro
[17:34:59] <grrrit-wm>	 (PS3) Milimetric: Add overall aggregate to project and language list [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/223793 (https://phabricator.wikimedia.org/T95340)
[18:01:44] <milimetric>	 joal: I fixed that typo btw ^
[18:03:47] <joal>	 milimetric: Actually it was not a typo, but a misunderstanding from me :)
[18:04:02] <joal>	 milimetric: I appove / merge and let you test ?
[18:04:26] <milimetric>	 it was a typo, I had "languages" instead of "Languages"
[18:04:53] <joal>	 You should have all.csv file tomorrow (13:00 UTC)
[18:04:58] <joal>	 milimetric: --^
[18:05:26] <milimetric>	 sweet, ok, so I'll deploy this at some point tomorrow
[18:05:43] <milimetric>	 I just need the dashiki change merged too
[18:06:06] <grrrit-wm>	 (CR) Joal: [C: 2 V: 2] "LGTM :)" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/223793 (https://phabricator.wikimedia.org/T95340) (owner: Milimetric)
[18:06:16] <joal>	 You are merged milimetric :)
[18:06:38] <joal>	 mforns: just realized that we missed some interesting change in aggregator --> Logging
[18:06:44] <joal>	 next change ;)
[18:16:36] <wikibugs>	 Analytics, Analytics-Backlog, Performance-Team, Patch-For-Review: Collect HTTP statistics about load.php requests - https://phabricator.wikimedia.org/T104277#1442344 (Krinkle) Resolved>Open
[18:33:26] <mforns>	 joal, oh so aggregator's logging was improved?
[18:35:27] <wikibugs>	 Analytics-EventLogging, Analytics-Kanban: Can Search up sampling to 5%? {oryx} - https://phabricator.wikimedia.org/T103186#1442438 (Milimetric) @Ironholds, regarding the higher but not as high sampling, how long do you need this data for?  That will affect what sampling you can do because the limiting fac...
[18:38:43] <wikibugs>	 Analytics-Cluster, Analytics-Kanban: Generate test data for Pageview API {slug} [5 pts] - https://phabricator.wikimedia.org/T101785#1442454 (Milimetric) I'm not sure we're making the right choice here to k-anonymize.  I think even if we chose a huge K we would still be vulnerable to the problems that l-di...
[18:45:42] <joal>	 mforns_brb: I was thinking of adding logging for all projects ;)
[18:46:02] <mforns_brb>	 joal, aha
[18:46:10] <joal>	 Anyway, we'll talk about this on monday ;
[18:46:18] <joal>	 I'm off for today :)
[18:46:25] <mforns_brb>	 ok! see you then
[18:46:29] <joal>	 See you guys on Monday, in MEXICO :)
[18:46:31] <mforns_brb>	 :]
[18:46:32] <wikibugs>	 Analytics-EventLogging, Analytics-Kanban: Can Search up sampling to 5%? {oryx} - https://phabricator.wikimedia.org/T103186#1442492 (Ironholds) We're setting a 90 day restriction around other tables right now and that seems totally viable to me (we could even go for less, if Other Dan is okay with being ab...
[18:46:47] <mforns_brb>	 ándale!
[18:46:56] <joal|weekend>	 :D
[19:19:29] <milimetric>	 :) safe travels joseph, see you soon
[19:21:12] <wikibugs>	 Analytics-EventLogging, Analytics-Kanban: Can Search up sampling to 5%? {oryx} - https://phabricator.wikimedia.org/T103186#1442588 (Milimetric) Yeah, 90 days is where we'd like all our EL tables to be.  Less than that would increase the sampling that we could potentially do here.  I'm thinking if the tabl...
[19:22:11] <wikibugs>	 Analytics-EventLogging, Analytics-Kanban: Can Search up sampling to 5%? {oryx} - https://phabricator.wikimedia.org/T103186#1442589 (Ironholds) I'm not :(
[19:44:00] <halfak>	 o/ neilpquinn
[19:55:12] <neilpquinn>	 halfak: o/ What's up?
[19:56:47] <halfak>	 Just getting some preliminary results together for the TAE spike of doom.
[19:57:24] <halfak>	 It looks like there was a burst in new registrations for enwiki that started on 2014/07
[19:58:37] <halfak>	 And that this resulted in an increased count of new actives that bleeds into surviving new actives, old actives and reactivateds
[19:58:59] <halfak>	 I'll have notes on this soon.  Was wondering if you were seeing similar things.
[19:59:42] <neilpquinn>	 No, I haven't done any analysis that would've touched that. Very interesting though Does it look like that's the bulk of the spike? Or just one factor?
[20:00:42] <halfak>	 neilpquinn, for enwiki, the bulk.  I'm splitting the languages into large/medium/small now to take a look at groups.
[20:01:09] <halfak>	 I've got to run right now.  Will make notes and ping you with them later.
[20:01:14] <neilpquinn>	 Cool, thanks!
[20:17:43] <mforns>	 madhuvishy, do you know if kevinator is at the office?
[20:18:02] <madhuvishy>	 mforns: he is, but not at his desk
[20:18:08] <madhuvishy>	 not sure where he went
[20:18:27] <madhuvishy>	 oh wait, i think i do
[20:18:29] <mforns>	 madhuvishy, ok, if you see him, can you please tell him I'm looking for him? :]
[20:18:37] <madhuvishy>	 mforns: sure :)
[20:18:41] <mforns>	 thanks!
[20:31:34] <madhuvishy>	 mforns: around?
[20:34:37] <mforns>	 madhuvishy, hi!
[20:34:53] <madhuvishy>	 Kevin's looking for you now :)
[20:34:54] <kevinator>	 mforns: hi, i heard you were looking from me
[20:43:21] <madhuvishy>	 milimetric: i am looking at your wikimetrics changes - and the only thing that's a bit non-obvious to me is the Include deleted checkbox
[20:43:35] <madhuvishy>	 like when i dont select it but shows up selected as a default
[20:44:11] <madhuvishy>	 i understand that it says if you dont select something it will set wikimetrics' defaults, but its still a little confusing
[20:53:27] <mforns>	 madhuvishy, thanks for pinging kevi-nator :]
[21:04:13] <wikibugs>	 Analytics-Kanban: Troubleshoot EventLogging validation alerts - https://phabricator.wikimedia.org/T105167#1443023 (mforns) By listing all alert timestamps and durations I could not find any matching anomaly in graphite, the logs or the database.  Dan gave the idea that maybe icinga is getting semi-up-to-date...
[21:05:30] <madhuvishy>	 mforns: no problem :)
[23:02:01] <leila>	 tnegrin: I'm in a Hangout that no one is in
[23:02:05] <leila>	 am I in the wrong place?
[23:02:14] <tnegrin>	 we're held up
[23:02:16] <tnegrin>	 be there soon
[23:02:28] <leila>	 ok. thanks.
[23:03:04] <tnegrin>	 10 minutes
[23:03:11] <tnegrin>	 we were just in an epic meeting
[23:51:18] <halfak>	 neilpquinn, check out https://meta.wikimedia.org/wiki/Research_talk:Active_editor_spike_2015/Work_log/2015-07-09
[23:51:33] <halfak>	 I have more work done, but I don't have time to write about it now.
[23:51:38] <halfak>	 I'll get back to it in the morning.