[00:06:31] <mforns>	 nuria, can I help you in something?
[00:09:36] <nuria>	 mforns: I think wikimetrics is doing good
[00:10:52] <nuria>	 mforns: so no need to do anything, i am doing tasks that i had on my queue  here and there.
[00:11:01] <mforns>	 nuria, ok, I'm heading to the airport then
[00:11:21] <nuria>	 mforns: good, see you in the internets in couple days
[00:11:36] <ori>	 ottomata: ping?
[00:11:38] <mforns>	 ok... bye nuria! see you
[00:11:38] <ori>	 i'm around now
[00:14:00] <ottomata>	 ori yoo hooo
[00:14:07] <ottomata>	 you on the busy side?
[00:14:12] <ottomata>	 i am coming searching :)
[00:15:16] <grrrit-wm>	 (PS1) QChris: Set a proper timeout for refining jobs [analytics/refinery] - https://gerrit.wikimedia.org/r/187293
[00:15:42] <qchris>	 ottomata ... before you go searching ... mind a quick review ^
[00:19:03] <qchris>	 mhmmm ... nuria ... you're doing oozie work too :-)
[00:19:14] <qchris>	 Wanna help me with a quick review? ^
[00:19:29] <nuria>	 qchris: suree
[00:19:39] <qchris>	 thanks
[00:20:24] <nuria>	 qchris: gerrit or hangout?
[00:20:44] <qchris>	 gerrit :-)
[00:20:46] <qchris>	  https://gerrit.wikimedia.org/r/187293
[00:25:11] <wikibugs>	 Quarry: Quarry does not respect ORDER BY sort order in result set - https://phabricator.wikimedia.org/T87829#1000128 (MahmoudHashemi) NEW a:yuvipanda
[00:28:03] <nuria>	 qchris: looking
[00:28:34] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Set a proper timeout for refining jobs [analytics/refinery] - https://gerrit.wikimedia.org/r/187293 (owner: QChris)
[00:28:48] <ottomata>	 oh i just merged it :/
[00:28:50] <ottomata>	 :D
[00:28:59] <qchris>	 ottomata, nuria: Thanks :-D
[00:29:28] <nuria>	 ottomata is a merging-ninjaaaaa
[00:30:05] <ottomata>	 nuria, i'll merge this if you fix qchris' comments :)
[00:30:05] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/187011/1/modules/eventlogging/manifests/monitoring.pp
[00:30:55] <nuria>	 ottomata: will do right now
[00:36:31] <nuria>	 ottomata: done, comments corrected.
[01:27:47] <wikibugs>	 Analytics-Engineering, Analytics-Cluster: Researchers have page_id in X-Analytics field of webrequest logs - https://phabricator.wikimedia.org/T77416#828237 (Ottomata) This is going to happen!  I chatted with Ori about this today.  The extension will not work in all cases due to unfortunate output buffering r...
[12:15:28] <grrrit-wm>	 (PS8) Ananthrk: Added Geocode core class that uses Maxmind-V2 Added GeocodedCountryUDF to get country code from IP address Added GeocodedDataUDF to get Geo data from IP address Added ClientIpUDF to determine client IP address given source IP and XFF header Added new depe [analytics/refinery/source] - https://gerrit.wikimedia.org/r/183551
[14:50:59] <qchris>	 no kevinator, no milimetric, no nuria, no mfors ... I guess that means another day without daily scrum.
[15:00:52] <ananthrk>	 oh..i was about to ask whether we are meeting today
[15:01:18] <qchris>	 ananthrk: I thought that today we'd meet again ... but it's only the two of us.
[15:01:39] <qchris>	 I guess we'll just skip it. What do you think?
[15:03:19] <ananthrk>	 okay...i guess we have no other choice today
[15:04:01] <qchris>	 ananthrk: It seeems nuria just joined.
[15:04:11] <qchris>	 And she told me that milimetric is around too.
[15:04:17] <qchris>	 So I guess we meet.
[15:05:22] <qchris>	 nuria: I which hangout? (The batcave is showing as empty for me)
[15:05:29] <ananthrk>	 oh..great
[15:05:36] <nuria>	 what?
[15:05:43] <nuria>	 we are both here...
[15:27:06] <ananthrk>	 nuria: The problem seems to the "Webrequests" class being referred in org.wikimedia.analytics.refinery.hive.IsCrawlerUDF. In refinery-core, the class is called "Webrequest", but the UDF class refers to it as "Webrequests" (with an 's' in the end)
[15:30:01] <ananthrk>	 nuria_: The problem seems to the "Webrequests" class being referred in org.wikimedia.analytics.refinery.hive.IsCrawlerUDF. In refinery-core, the class is called "Webrequest", but the UDF class refers to it as "Webrequests" (with an 's' in the end)
[15:35:44] <milimetric>	 kevinator: hi, we're in the batcave
[15:35:48] <wikibugs>	 operations, ops-core, Analytics: Deprecate HTTPS udp2log stream? - https://phabricator.wikimedia.org/T86656#1000896 (QChris) >>! In T86656#999773, @Ottomata wrote: > The data is still being backfilled in hadoop.  Done.
[17:02:56] <ottomata>	 qchris: mornin!
[17:03:11] <qchris>	 Gooooood morning sir ottomata!
[17:03:33] * qchris feels the smell of decommissioned webstatscollector in the air today!
[17:03:36] <ottomata>	 i came down stairs at the hotel this morning, and an excited brandon told me the backfilling was done!
[17:03:37] <ottomata>	 :)
[17:03:43] <ottomata>	 data looks good?
[17:03:50] <ottomata>	 i can tell him that turning off nginx udp2log is ok?
[17:04:13] <ori>	 only if he turns on SPDY
[17:04:14] <ottomata>	 also, should I remove those *0001 files?
[17:04:16] <ottomata>	 haha
[17:04:29] <ori>	 tit for tat!
[17:05:02] <ottomata>	 bblack says that is the eventual plan!
[17:05:09] <qchris>	 Oh. nginx has priority :-D Let me think things through once again.
[17:06:27] <qchris>	 ottomata: Green light from my side.
[17:08:45] <ottomata>	 awesoooOMe
[17:09:17] <wikibugs>	 operations, ops-core, Analytics: Deprecate HTTPS udp2log stream? - https://phabricator.wikimedia.org/T86656#1000993 (Ottomata) Looking GOOD!  Brandon, you may turn of nginx udp2log! :)
[17:09:26] <ottomata>	 qchris: i should remove the *0001 files from dumps, ja?
[17:09:35] <ottomata>	 i have them backed up
[17:09:44] <qchris>	 Yes, please remove them :-)
[17:10:00] <ottomata>	 done
[17:10:21] <qchris>	 ottomata: btw. the front-page of the pagecounts-raw still shows the old page.
[17:11:19] <qchris>	 Could you run the script by hand?
[17:11:27] <ottomata>	 really?
[17:11:29] <ottomata>	 hm
[17:11:31] <ottomata>	 checking
[17:11:39] <qchris>	 http://dumps.wikimedia.org/other/pagecounts-raw/
[17:11:58] <qchris>	 It's just the front page at that url.
[17:12:04] <qchris>	 The month subpages are ok.
[17:12:04] <ottomata>	 it shouldn't, because puppet runs all of the scripts at once
[17:12:07] <ottomata>	 hm ok
[17:12:47] <qchris>	 I guess it'll be run automatically on 2016-01-01, with the change of the year.
[17:13:59] <ottomata>	 ah, yeah, i think if index.html exists and that, it doesn't run
[17:14:03] <ottomata>	 i'll remove index and rerun
[17:14:14] <qchris>	 Cool. Thanks.
[17:14:27] <ottomata>	 there we go
[17:15:07] <qchris>	 \o/
[17:15:09] <qchris>	 Thanks.
[17:15:18] <ottomata>	 ok, heading to the office, back in a bit
[17:15:30] <qchris>	 k. Thanks again.
[17:28:35] <wikibugs>	 Analytics-Engineering: Calculate per-wiki Edit Completion Rate for Visual Editor edits - https://phabricator.wikimedia.org/T87865#1001019 (Milimetric) NEW a:Milimetric
[17:35:30] <wikibugs>	 Analytics-EventLogging: Multiple user_ids per username in account creation events from ServerSideAccountCreation log - https://phabricator.wikimedia.org/T68101#1001041 (Milimetric) Thanks Chris.  No rush, we were just wondering what the timeline looked like.
[17:43:14] <wikibugs>	 operations, ops-core, Analytics: Deprecate HTTPS udp2log stream? - https://phabricator.wikimedia.org/T86656#1001069 (BBlack) nginx udp2log is off and nginx configs have been reloaded: https://gerrit.wikimedia.org/r/#/c/186257/
[17:44:05] <grrrit-wm>	 (PS1) QChris: Prepare webrequest dump for addition of refined data [analytics/refinery] - https://gerrit.wikimedia.org/r/187416
[17:44:07] <grrrit-wm>	 (PS1) QChris: Add refined tables to webrequest dump script [analytics/refinery] - https://gerrit.wikimedia.org/r/187417
[17:44:09] <grrrit-wm>	 (PS1) QChris: Add pagecounts-all-sites to webrequest dump script [analytics/refinery] - https://gerrit.wikimedia.org/r/187418
[17:44:11] <grrrit-wm>	 (PS1) QChris: Add pagecounts-raw to webrequest dump script [analytics/refinery] - https://gerrit.wikimedia.org/r/187419
[17:44:13] <grrrit-wm>	 (PS1) QChris: Separate different datasets better in webrequest dump script [analytics/refinery] - https://gerrit.wikimedia.org/r/187420
[17:54:20] <grrrit-wm>	 (PS1) Ottomata: Fix bad symbol 'Webrequests' in IsCrawlerUDF.java [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187424
[17:54:41] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Fix bad symbol 'Webrequests' in IsCrawlerUDF.java [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187424 (owner: Ottomata)
[17:59:52] <wikibugs>	 operations, ops-core, Analytics: Deprecate HTTPS udp2log stream? - https://phabricator.wikimedia.org/T86656#1001093 (faidon) Open>Resolved \o/
[18:03:42] <DarTar>	 kevinator: are we blocking 30 mins between 10.30 and 11 and inviting toby?
[18:03:55] <DarTar>	 he’s in other meetings
[18:04:08] <kevinator>	 ok… Toby is at the Collab QR so he’s up there right now
[18:04:37] <DarTar>	 I’ll text him, we need to wrap up his section
[18:19:41] <wikibugs>	 Analytics-Engineering, operations: Decommission webstatscollector - https://phabricator.wikimedia.org/T87868#1001126 (Ottomata) NEW a:Ottomata
[18:42:15] <qchris>	 ottomata: I like those webstatscollector changes that fly by :-)
[18:42:42] <qchris>	 If gerrit had a "Thank you" button ... ;-)
[18:43:23] <ori>	 that's not a bad idea
[18:51:36] <wikibugs>	 Analytics-Engineering, operations: Decommission webstatscollector - https://phabricator.wikimedia.org/T87868#1001260 (Ottomata) Open>Resolved
[18:51:53] * qchris dances
[18:52:21] <qchris>	 Awesome ottomata!
[19:04:54] <ottomata>	 :D
[19:32:21] <wikibugs>	 Analytics-EventLogging: Convert EventLogging to use extension registration - https://phabricator.wikimedia.org/T87912#1001539 (Legoktm)
[19:41:01] <wikibugs>	 Analytics-EventLogging: Convert EventLogging to use extension registration - https://phabricator.wikimedia.org/T87912#1002033 (Legoktm)
[19:55:35] <wikibugs>	 Analytics-Engineering, Analytics-EventLogging: Convert EventLogging to use extension registration - https://phabricator.wikimedia.org/T87912#1002116 (kevinator)
[20:00:54] <tnegrin>	 for realz -- congrats folks
[20:01:04] <tnegrin>	 (on the WSC deco)
[20:01:07] <tnegrin>	 decom
[20:45:05] <ori>	 qchris: the X-Analytics initial commit patch should be ready to go, from my perspective
[20:45:41] <qchris>	 I did not check since back when you said you addressed the MobileFrontend comment.
[20:46:11] <ori>	 yeah, it's resolved
[20:46:21] <qchris>	 Argh ... gerrit is extra slow for me today :-(
[20:47:37] <qchris>	 Finally ... there it is: https://gerrit.wikimedia.org/r/157841
[20:48:17] <qchris>	 No CR-1 left from me. Feel free to merge at will.
[20:48:39] <ori>	 qchris: could you?
[20:49:26] <ori>	 (I tested it with ottomata yesterday, he can vouch :P)
[20:49:30] <qchris>	 I didn't review/test the code.
[20:49:36] <qchris>	 ottomata|meeting: Wanna merge some code?
[20:49:59] <qchris>	 Ah ... I see "meeting"
[20:51:33] <qchris>	 ori: how did the concerns about ob_flush-ing work out? Did those places got fixed in core?
[20:51:38] <qchris>	 (I did not follow core recently)
[20:52:41] <ori>	 qchris: ottomata explained that the thing he's most keenly interested in is page_id for page views. The headers for page views are handled in one place, so ottomata is OK with other requests potentially going "unheadered", like 404s and ResourceLoader module requests.
[20:53:33] <qchris>	 ori: I am sorry. It really seems ottomata should CR+2 then.
[20:53:54] <ori>	 OK, no problem.
[20:53:54] <qchris>	 I basically do not understand all the interplay with core here.
[20:54:08] <qchris>	 Sorry :-(
[20:54:42] <ori>	 it's ok, you will answer for it in the afterlife
[20:55:18] <ori>	 (joking)
[20:56:01] <qchris>	 I AM NOT AFRAID OF SPAGHETTI MONSTERS CHASING ME!
[20:56:14] * qchris arms himself with a fork and a starving belly.
[20:56:38] <qchris>	 (So much for afterlife)
[20:56:39] <ori>	 heh
[20:57:06] <chasemp>	 all hail the noodly appendage
[20:57:48] * qchris wonders who has "spaghetti monster" set as stalk word in his IRC client.
[21:44:49] <milimetric>	 jsahleen: fyi I looked into the query and there's a clear problem with generate.py where it just overwrites all your old data
[21:45:26] <jsahleen>	 milimetric: Your end or ours?
[21:46:07] <jsahleen>	 The numbers are correct now, but the graph is completely flat.
[21:46:34] <milimetric>	 jsahleen: our end
[21:46:39] <milimetric>	 it's just ... broken
[21:46:59] <jsahleen>	 Ah. Well, that makes me feel a bit better. I couldn't see anything wrong with the query. Sorry it's broken.
[21:47:23] <milimetric>	 I have a couple of ideas why, but basically it just doesn't handle this type of query well - where we don't pass in the date but still want daily data
[21:47:30] <milimetric>	 sadly, can't work on it for a bit
[21:47:45] <jsahleen>	 Not a big issue. Quarterly review is over. :)
[21:48:29] <jsahleen>	 So, whenever you get time is fine.
[21:49:04] <milimetric>	 jsahleen: meantime I'm just running the query manually in a cron
[21:49:18] <milimetric>	 you can, btw, feel free to do the same thing and copy the results into the datasets directory
[21:49:31] <milimetric>	 then you can feed the graph from that instead of from the broken generate.py output
[21:49:32] <jsahleen>	 milimetric: which repo?
[21:49:38] <milimetric>	 no repos here, wild west
[21:49:42] <jsahleen>	 crap
[21:50:07] <milimetric>	 jsahleen: I put it in /home/milimetric/daily-content-translation-beta
[21:50:07] <jsahleen>	 Ok. We are still "discussing" our metrics, so that needs to be settled first.
[21:50:12] <milimetric>	 yeah, good
[21:50:32] <milimetric>	 but just fyi, you don't have to chain yourself to this broken data collector
[21:50:42] <milimetric>	 since Limn is pretty easy to make data for
[21:51:16] <jsahleen>	 The team is talking about updating our own stats page that is part of the Content Translation extension. Bypassing the whole limn thing.
[21:51:40] <jsahleen>	 But again, still "discussing."
[21:52:00] <milimetric>	 that's ok with me, and we can ping you when we fix the pipeline
[21:52:05] <jsahleen>	 Sounds good.
[21:52:20] <jsahleen>	 You have other fish to fry
[21:52:49] <jsahleen>	 Big fish
[21:55:39] <milimetric>	 warning: All fish in retrospect are bigger than they appear
[21:55:53] <jsahleen>	 :)
[22:36:38] <grrrit-wm>	 (PS9) Ottomata: Added Geocode core class that uses Maxmind-V2 Added GeocodedCountryUDF to get country code from IP address Added GeocodedDataUDF to get Geo data from IP address Added ClientIpUDF to determine client IP address given source IP and XFF header Added new depe [analytics/refinery/source] - https://gerrit.wikimedia.org/r/183551 (owner: Ananthrk)
[22:40:12] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Prepare webrequest dump for addition of refined data [analytics/refinery] - https://gerrit.wikimedia.org/r/187416 (owner: QChris)
[22:41:06] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Add refined tables to webrequest dump script [analytics/refinery] - https://gerrit.wikimedia.org/r/187417 (owner: QChris)
[22:42:09] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Add pagecounts-all-sites to webrequest dump script [analytics/refinery] - https://gerrit.wikimedia.org/r/187418 (owner: QChris)
[22:42:29] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Add pagecounts-raw to webrequest dump script [analytics/refinery] - https://gerrit.wikimedia.org/r/187419 (owner: QChris)
[22:42:48] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Separate different datasets better in webrequest dump script [analytics/refinery] - https://gerrit.wikimedia.org/r/187420 (owner: QChris)
[22:43:07] <qchris>	 ottomata: Thanks for the merges.
[22:43:16] <ottomata>	 yessuhhh!
[22:43:24] <qchris>	 I am not sure ... should I update the cron that sends out emails to you and jg-age too?
[22:43:32] <ottomata>	 sure!
[22:43:42] <ottomata>	 well, hm.
[22:43:46] <qchris>	 Do you want "--datasets all" ?
[22:43:48] <ottomata>	 for refeined, yes
[22:43:49] <ottomata>	 hm
[22:43:53] <qchris>	 (That's very long lines)
[22:43:53] <ottomata>	 not sure i want pagecounts* ones
[22:44:02] <ottomata>	 i want refined and raw
[22:44:07] <ottomata>	 that'll be good enough for report i think
[22:44:10] <qchris>	 k
[22:45:46] <grrrit-wm>	 (CR) Ottomata: "Hm, still has IpUtil, etc." [analytics/refinery/source] - https://gerrit.wikimedia.org/r/183551 (owner: Ananthrk)
[23:28:06] <wikibugs>	 Analytics, operations: Fix Varnishkafka delivery error icinga warning - https://phabricator.wikimedia.org/T76342#1002702 (Ottomata) Open>Resolved
[23:41:10] <wikibugs>	 Analytics-Cluster: Create another done-flag level for webrequest_*_raw oozie datasets that indicates the partition is mostly good - https://phabricator.wikimedia.org/T87060#1002734 (Ottomata) Open>Resolved
[23:42:06] <wikibugs>	 Analytics-Cluster: Raw webrequest partitions that were not marked successful due to only esams caches causing unknown problems - https://phabricator.wikimedia.org/T74809#1002737 (Ottomata)
[23:42:07] <wikibugs>	 Analytics-Cluster: Raw webrequest partitions for 2014-12-07T20/2H not marked successful - https://phabricator.wikimedia.org/T77024#1002736 (Ottomata) Open>declined
[23:53:06] <wikibugs>	 Analytics-Wikimetrics: Story: WikimetricsUser searches for cohort (filters) using tag name - https://phabricator.wikimedia.org/T75071#1002803 (Capt_Swing) Potential issue: what if I have a tag and a cohort with the same name? What will be returned by my search?
[23:58:55] <nuria>	 holaaaaa