` box on top with just the name. Likely that one would ne... [11:26:36] (PS3) Aklapper: Speed up verification with scripts [analytics/data-warehouse] - https://gerrit.wikimedia.org/r/185239 (owner: Milimetric) [11:27:41] (CR) Aklapper: "Milimetric: This patch has seen no activity for 11 months. How to get this reviewed/merged? If this needs DBA review, should jcrespo be ad" [analytics/data-warehouse] - https://gerrit.wikimedia.org/r/185239 (owner: Milimetric) [13:12:41] Analytics-Wikistats, Internet-Archive: "Top month" and "Trend last 24 months" missing in Wikipedia columns - https://phabricator.wikimedia.org/T72900#1873037 (ezachte) Open>Invalid a:ezachte The empty row is YoY which will return when new pageview def is there for 13 months. (removing it entirel... [14:35:00] Analytics-Backlog: Upgrade Spark to 1.5 - https://phabricator.wikimedia.org/T121159#1873179 (Ottomata) [14:35:01] Analytics-Backlog, Analytics-Cluster: Upgrade to CDH 5.5 - https://phabricator.wikimedia.org/T119646#1873180 (Ottomata) [14:51:23] Analytics-EventLogging, EventBus, Wikimedia-Logstash: eventlogging syslog message not properly recognized by logstash - https://phabricator.wikimedia.org/T120874#1873194 (Ottomata) Hmm, I was working on syslog this week, and it is possible this is a bad message while I was developing. When I look no... [14:52:17] milimetric: Hi :) [14:52:36] milimetric: let me know if you want to spend some time discussing anonymization [14:52:39] (again) [14:53:20] joal: yes, but I wanted to test my patch so we can potentially merge it [14:53:31] joal: but don't merge it after I test, I want Erik's ok on the format as well [14:54:54] milimetric: ok :) [14:57:34] milimetric: Let me know if you want to help, or anything [15:06:17] (CR) Milimetric: [C: 2 V: 2] "might as well merge as this project is sort of de-funct, and just serving for lessons learned purposes." [analytics/data-warehouse] - https://gerrit.wikimedia.org/r/185239 (owner: Milimetric) [15:06:57] (Abandoned) Milimetric: [WIP] Use continue parameter as documented [analytics/mediawiki-storage] - https://gerrit.wikimedia.org/r/187064 (owner: Milimetric) [15:11:09] joal: I ran it again, but I was wondering something: https://hue.wikimedia.org/oozie/list_oozie_coordinator/0130977-150922143436497-oozie-oozi-C/ [15:11:17] how come the coordinator still shows as running? [15:11:23] are all my tests from yesterday still running? [15:11:33] I'm gonna double check [15:11:40] (the actions underneath it all succeeded [15:11:41] Have you provided a stop_date ? [15:11:56] yes, [15:12:07] Woaouch, hue seems broken :) [15:12:10] oh! it stopped :) [15:12:21] * joal is gonna go for CLI [15:12:23] the coordinator says "SUCCESS" now, ok [15:12:33] Then it means it is finishedb [15:13:00] Yup, SUCCEDED :) [15:13:43] joal: and indeed the file name is now correct with the proper date format: geo-20151001-000000.gz [15:13:54] :D [15:13:58] \o/ ! [15:15:14] ok, cool, thx for the review and help joal, we just have to wait for EZ now and then you can merge [15:15:41] milimetric: np at all :) [15:16:08] now... I have 60 notifications in phab. /me needs sugar [15:16:31] milimetric: I have a bunch of stuff to deploy, so probably next week :) [15:16:41] k, cool [15:17:06] milimetric: If you are talking about emails, 95% will be our cleaning garbaging your emails :) [15:24:47] milimetric: Or at least so do I think :) [15:39:55] a-team: will be making up some time and joining standup this morning [15:40:07] cool nuria :) [15:43:14] Analytics-Kanban, Patch-For-Review: Fix EL mysql consumer's deque push/pop and size {oryx} [3 pts] - https://phabricator.wikimedia.org/T120209#1873272 (mforns) [15:43:16] Analytics-Kanban, DBA: 2 hour outage to update mysql on EL slaves - https://phabricator.wikimedia.org/T121120#1873271 (mforns) [15:46:07] Analytics-Kanban, DBA: 2 hour outage to update mysql on EL slaves - https://phabricator.wikimedia.org/T121120#1873279 (mforns) @jcrespo I added the task as 'blocked by', and will close it as resolved in short. Also, I'm ok with //2 hours starting on 2015-12-15 at 10:00 UTC//. I will be logged in IRC and... [15:54:37] hi, what api request do i need to make to get pageview stats for a given page for the past N days/months/years? [15:55:12] mforns was doing a presentation, but it seems his examples don't have that [15:55:49] also, is it possible do get that data for mediawiki.org? [15:58:00] yurik: https://wikimedia.org/api/rest_v1/metrics/pageviews/?doc [15:58:35] joal, thanks, i think i found the doc https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageview_API -- seems to be working for "/mediawiki/". Thanks! [15:59:02] yurik: You're welcome, happy if that works :) [16:00:00] yurik: I have thinking about vega+api, and I think it could be fun :) [16:00:28] ohh yeah - i'm thinking of visualizing this: https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/mediawiki/all-access/user/Extension%3AGraph%2FDemo/daily/2015100100/2015103100 [16:00:42] pageview of the current article for the past N days ) [16:00:59] yurik: Indeed :) [16:01:29] but the potential is much higher of course - like doing word clouds, etc ) [16:01:46] Well, baby stepm hehe :) [16:07:30] ottomata: did you check out the stats that are being sent? [16:08:17] madhuvishy: sorta, ja, i am making sure python packages work, and i'm also messing with your patch to see if i can conditionally inherit in case the mixin isn't available...:) [16:08:23] not sure if possible [16:08:28] (PS8) Bearloga: Functions for identifying search engines as referers. [analytics/refinery/source] - https://gerrit.wikimedia.org/r/247601 (https://phabricator.wikimedia.org/T115919) (owner: OliverKeyes) [16:09:19] (CR) Bearloga: "recheck" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/247601 (https://phabricator.wikimedia.org/T115919) (owner: OliverKeyes) [16:11:44] Hmm, I think it works! [16:12:27] ottomata: nice! [16:12:52] i can append the patch if you want - will also add the default for prefix [16:13:57] cool, am about to submit something to it [16:14:01] go ahead and do your prefix thing firest [16:14:02] i'll rebase [16:15:03] okay [16:15:08] madhuvishy: i was looking at your wikimetrics tests that do not work [16:20:35] ottomata: pushed, but just gonna test on labs [16:23:36] k [16:36:51] madhuvishy: let me know when you have time and i think i can explain why your tests do not work cc milimetric [16:37:11] hey nuria! So as a possible solution to the ENUM problem (because you and I are both very busy rn and so meeting up has fallen through) I asked Mikhail to take a look at the referer patchset [16:37:14] ottomata: gah for some reason it keeps saying ImportError: No module named mixins [16:37:18] according to Jenkins he got it working (with ENUMs!) [16:37:44] madhuvishy: where? [16:37:52] deployment-eventlogging04 [16:38:10] nuria: sure [16:38:14] thanks [16:38:47] Ironholds: ok, will look i see it on my queue, please make sure he tests in the cluster [16:39:03] shall do! Thank you! [16:41:08] madhuvishy: ja i had it uninstall for a sec [16:41:08] try now [16:41:12] was testing something [16:45:39] ottomata: aah [16:45:40] okay [16:48:35] madhuvishy: that has to be done in the prepare method? [16:49:50] madhuvishy: i'm going to do that in another way... :) to make it work with my fancy thing im'a bout to submit [16:50:14] yeah that's what the docs say - Should your Request Handler extend the finish or the prepare methods ensure that your call super otherwise you may run into strange behavior. but it errors for me [16:50:24] https://www.irccloud.com/pastebin/pIoKri4n/ [16:50:31] ottomata: oh howw [16:50:45] using the env var instead [16:50:48] of setting it ourselves [16:50:54] To change this value, set the new prefix with the [16:50:54] ``STATSD_PREFIX`` environment variable. [16:50:59] ottomata: !!! [16:51:02] that's what i did [16:51:06] i sent you notes :D [16:51:09] no, you set self.statsd_prefix [16:51:10] notes? [16:51:13] lol [16:51:21] i sent you one email on how i did this [16:51:26] https://etherpad.wikimedia.org/p/eventbus-statsd [16:51:29] i thoughttt [16:51:36] you wanted to set it in the code! [16:51:39] OH notes! [16:51:46] no i didn't read these ! :o [16:51:50] ha ha [16:52:07] i mean, setting in code is maybe good, but i think it'd be better to just set the env var if it isn't set [16:52:11] and let the mixin code deal with it [16:52:16] sorry for misunderstanding! [16:53:13] ottomata: if we don't set in env var and code, it'll be under sprockets something prefix [16:53:24] put if we set via puppet on env var we are good [16:53:35] no need to set in code if we don't want to [16:53:38] os.environ.setdefault('STATSD_PREFIX', 'eventlogging.service') [16:53:51] okay cool [16:54:00] you can just revert my changes [16:54:11] it throws an error anyway - although i don't know why [16:54:25] heh, ok, just nee to remove those prepare methods, am about to submit a patch [16:54:30] it might be too fancy....:p [16:54:40] :D [16:55:27] nuria: wanna chat now? i'm in batcave [16:55:35] sure, omw [16:57:38] madhuvishy: submitted patch [17:03:15] Sooooory, joining P! [17:08:55] joal, https://www.mediawiki.org/wiki/Extension:Graph/pageviews [17:09:54] yurik: :) [17:09:57] * joal is happy [17:11:58] joal, there is a weird limitation on that API btw - it blocks the "Treat-As-Untrusted" https://www.mediawiki.org/wiki/Extension:Graph/pageviews works because graphoid does not send that header, but client side api request is not working [17:12:44] milimetric: --^ any idea on that ? [17:18:18] Analytics-Backlog: Change e-mail list on oozie communications to e-mail alias - https://phabricator.wikimedia.org/T121241#1873447 (Nuria) NEW [17:23:11] milimetric, https://www.mediawiki.org/wiki/Extension:Graph/Demo#Using_API [17:23:20] ottomata, you might like it too - https://www.mediawiki.org/wiki/Extension:Graph/Demo#Using_API [17:24:34] yurik: that's super awesome, I'm linking to this on the blogpost we're doing [17:25:33] milimetric, hehe, sure, you might want to clean it up a bit visually - e.g. labels, non-manual dates, etc [17:26:06] yurik: tears come to my eyes ... [17:26:14] ~~~~~~=> river of tears [17:26:26] i take it to be a good thing? [17:27:33] :) look what you did, yuri, you made nuria cry [17:28:16] yurik: when you say client side request isn't working, I heard that report from people who had a certain setting in Firefox [17:28:31] but the demo uses that: https://analytics.wmflabs.org/demo/pageview-api/ [17:28:57] and for me that works in FF too, so I never saw the problem for myself [17:29:05] nuria: ok, tested out https://gerrit.wikimedia.org/r/#/c/247601/ on the cluster and it works! :D [17:35:11] milimetric, go to https://www.mediawiki.org/wiki/Extension:Graph/pageviews , edit, and pagepreview -- see debugger output [17:41:38] yurik: COOOL [17:54:53] milimetric: madhuvishy, got a sec to look at this patch with me? [17:54:56] its a little funky [17:55:14] sure [17:55:18] ottomata: my initial feeling is that - we should either add this dependency or not [17:55:26] https://gerrit.wikimedia.org/r/#/c/258029/5 [17:55:28] ummm [17:55:37] lets go to other hangout i guess [17:55:42] ya okay [17:55:48] https://plus.google.com/hangouts/_/wikimedia.org/funky [17:57:24] (CR) Bearloga: [C: 1] "```hive> ADD JAR /home/bearloga/Code/analytics-refinery-source/refinery-hive/target/refinery-hive-0.0.24-SNAPSHOT.jar;" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/247601 (https://phabricator.wikimedia.org/T115919) (owner: OliverKeyes) [18:01:03] Analytics-Kanban, Patch-For-Review: Fix EL mysql consumer's deque push/pop and size {oryx} [3 pts] - https://phabricator.wikimedia.org/T120209#1847820 (mforns) [18:06:41] nuria: wanna continue talking about wikimetrics tests? [18:07:44] OOOok madhuvishy, patched again, thanks :) [18:28:47] madhuvishy: cannot have to met luca (our new devops) [18:29:10] madhuvishy: and after I am signing off [18:29:19] nuria: okay np [18:29:24] lets chat monday [18:29:38] madhuvishy: ok, i might also submit a ptach if i can later later today [18:29:49] okay [18:50:49] Analytics-Backlog, Design-Research: Bot to call global metrics to event page {kudu} - https://phabricator.wikimedia.org/T120330#1873853 (Capt_Swing) [18:52:23] madhuvishy: COol, danke, eventlogging service eventbus now puppetized with your changes [18:52:25] looking good [18:52:33] ottomata: yay :D [18:53:13] Analytics-Backlog, Design-Research: Bot to call global metrics to event page {kudu} - https://phabricator.wikimedia.org/T120330#1873872 (Capt_Swing) Talked with @abit about this. Looks like it's a bigger, hairier development project than I expected; it's outside of my abilities. I'm closing this task as 'd... [18:53:30] Analytics-Backlog, Design-Research: Bot to call global metrics to event page {kudu} - https://phabricator.wikimedia.org/T120330#1873876 (Capt_Swing) Open>declined [18:53:45] Analytics-Backlog, Design-Research: Bot to call global metrics to event page {kudu} - https://phabricator.wikimedia.org/T120330#1851192 (Capt_Swing) declined>Open a:Capt_Swing>None [18:54:00] Analytics-Backlog: Bot to call global metrics to event page {kudu} - https://phabricator.wikimedia.org/T120330#1851192 (Capt_Swing) [18:59:03] mforns_gym: Arf, it's gym time for mforns_gym :) [18:59:08] nuria not gym ? [19:01:43] Aaaah, nuria not working on fridays :) [19:01:56] madhuvishy maybe ? [19:02:10] joal: sup :) [19:02:14] hey ! [19:02:36] Would you have a minute to read that: https://wikitech.wikimedia.org/wiki/Analytics/Data/Pageview_hourly/Sanitization [19:02:42] yess [19:02:47] And tell me if it's reasonnably understandable ? [19:07:58] joal: when you say X Ips [19:08:04] is X the same as K [19:08:06] ? [19:08:15] madhuvishy: Cause we have not yet agreed on X :) [19:08:30] I'd say 5, but prefer to have team agreement [19:08:57] joal: i was a little confused when the title was the good K and the explanation was about X IPs - what is K then? [19:09:12] X is K :) [19:09:20] oh you are just using X as a placeholder for a number [19:09:25] Yessssss [19:09:26] which we will put there [19:09:28] okay cool [19:09:33] Sorry :S [19:11:44] joal: More precisely, it is easy when looking at sets of pageviews from groups of at least X IPs to reasonably correctly guess the pages viewed together by the same IP. [19:11:56] this line is a bit confusing [19:12:06] hm [19:12:23] yes [19:12:39] what is easy to guess [19:12:53] ok [19:13:05] It's the subgroup of pages viewed together [19:13:15] I'll modify the thing [19:13:32] madhuvishy: need to run :S [19:13:39] Thanks for the comment :) [19:13:53] joal: okay i'll read once again and let you know if anything [19:13:58] have a nice weekend :) [19:13:58] Will backlog if you leave more :) [19:14:01] TYHanks you !! [19:14:04] You too :) [19:14:08] Bye a-team :) [19:14:23] bon weekend! [19:15:58] Analytics-Backlog: Display global metrics report results on same page as report inputs - https://phabricator.wikimedia.org/T121262#1873990 (Abit) NEW a:Milimetric [19:19:43] milimetric, did you figure out what it was that was blocking api request? [19:29:55] milimetric, did you figure out what it was that was blocking api request? [20:29:42] yurik: no, I looked at it and was gonna ping you but you weren't on IRC for a bit [20:30:09] milimetric, yeah, was switching to home office instead of cafe office [20:30:12] yurik: I'm not familiar enough with exactly how CORS works to know, but it looks like a client thing [20:30:46] like the client is adding Treat-as-Untrusted and then the server's not allowed to respond to that if it's a different domain? [20:30:49] milimetric, i doubt it - because the "treat as untrusted" is a custom header i introduced into the graph ext to be sent [20:30:52] but maybe we could ask in -services [20:30:56] its not standard [20:31:06] yurik: oh, huh... [20:31:07] and our api checks for that header [20:31:22] and treats the request as anonymous even if it has a session cookie [20:31:24] yurik: but somehow that header is what's messing with the CORS [20:31:35] because if you do a request without it it's fine [20:31:42] yeah, i suspect -services did somehing magic with it too. [20:32:00] well, no, i mean without any magic, plain vanilla jquery can fetch a pageview API url [21:54:21] Analytics-Kanban, Services: Better Response times pageview API.. [8 pts] - https://phabricator.wikimedia.org/T119886#1874425 (Nuria) [21:55:27] Analytics-Kanban, Services: Better Response times pageview API.. [8 pts] - https://phabricator.wikimedia.org/T119886#1838795 (Nuria) [21:56:43] Analytics-Backlog: Create Pageview API dashboard to monitor response times - https://phabricator.wikimedia.org/T121277#1874439 (Nuria) NEW [21:56:54] Analytics-Backlog: Create Pageview API dashboard to monitor response times - https://phabricator.wikimedia.org/T121277#1874446 (Nuria) p:Triage>Normal [23:27:38] Analytics-Backlog: Central repository of global metrics reports - https://phabricator.wikimedia.org/T121286#1874640 (Abit) NEW a:Milimetric