[00:29:20] Hi all [00:29:34] I am Ratnadeep Debnath from India [00:30:12] I am an open source enthusiast and contributor [00:30:39] I mainly contribute to the FedoraProject and sometimes to Mozilla [00:32:00] In the past, I was working with Transifex (initially as a contributor, then an employee) [00:32:48] I am majorly a Python/Django developer. However, lately, I have been working a lot with MeteorJS on https://www.waartaa.com [00:34:03] I want to use my skills to help with the analytics projects at Wikimedia [00:34:54] Currently, I am setting up the environment for https://github.com/wikimedia/analytics-wikimetrics [00:36:46] Please let me know if you have some issues list which I can work on to get my feet wet and start contributing [00:56:11] YuviPanda, Hi [01:04:28] rtnpro, good to see you! We mostly handle issue reports and suchlike on Bugzilla and (after that) Phabricator - http://bugzilla.wikimedia.org/ [01:04:35] GitHub is a mirror rather than where it actually lives. [01:04:58] Ironholds, ok [01:06:46] Ironholds, I see quite some issues here: https://bugzilla.wikimedia.org/buglist.cgi?quicksearch=analytics [01:07:09] Ironholds, I am working on to setup my dev environment [01:07:25] Good luck! I'm a researcher, not an engineer, so I am likely to be of limited utility. [01:07:29] Ironholds, there are some hiccups with the Vagrant setup on Fedora [01:07:55] see above ;p [01:08:04] Ironholds, it was nice to meet you :) [01:10:04] Ironholds, so, what's the timezone of the developer folks here? [01:10:17] largely Europe or the East Coast of North America [01:10:24] with the exception of YuviPanda who flies around a lot [01:10:35] Ironholds, ok :) [01:10:46] Ironholds, yes, I know YuviPanda :D [01:11:59] Ironholds, when you have time, please have a look at https://www.waartaa.com :) [01:12:14] ...why? [01:12:42] Ironholds, it's an open source communication and collaboration tool (around IRC) that I am working on [01:12:48] sure! [01:12:52] again: researcher. [01:13:11] If you have interesting things to say about the state of MaxMind's C++ API, or the systemic bias that affects Wikimedia projects, you lemme know. [01:13:28] Ironholds, sure :) [01:13:30] In the meantime, I have work to do, which is largely centred around, see above. [01:14:13] Ironholds, thanks for responding, I really appreciate it. +1 [02:06:51] (PS1) Bmansurov: Update UI-Daily [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/174055 [06:48:51] do people know where milimetric lives? [06:48:52] NY? [08:08:40] YuviPanda, Hey [08:08:53] heya! [08:09:22] YuviPanda, It's great to see you here :) [08:09:28] you too :) [08:09:58] YuviPanda, I am trying to get started with contributing to Mediawiki analytics [08:10:08] wooo, nice :) [08:10:16] do you want to contribute to analytics or to research? [08:10:39] YuviPanda, to analytics, and then, may be, someday to research [08:10:49] :D [08:10:50] nice [08:10:55] what kind of stuff are you interested in working on? [08:12:15] data processing and make sense out of it, build user friendly web apps to display readable results, improve existing apps [08:12:21] ah [08:12:25] have you seen quarry.wmflabs.org? [08:12:30] no [08:12:45] give it a look! [08:12:49] that could use a lot of love :) [08:12:55] sweet :) [08:13:15] I'm also the lead dev on that. [08:13:20] it's python + jquery [08:13:21] cool [08:13:33] My domain, then :D [08:13:47] :D [08:13:59] you haven't had much experience with gerrit, no? [08:14:06] YuviPanda, I am new to this data world, so, I will need some help to get started [08:14:07] you can get started with github and I can be the bridge for a while [08:14:18] YuviPanda, I haven't used gerrit much [08:14:21] rtnpro: https://github.com/wikimedia/analytics-quarry-web [08:14:23] yeah [08:14:37] YuviPanda, but it's git at the end, right [08:14:46] YuviPanda, so, shouldn't be a problem [08:14:48] yeah :) [08:14:52] well, gerrit is kind of weird git [08:15:20] rtnpro: https://bugzilla.wikimedia.org/show_bug.cgi?id=69544 is a small bug in quarry that you can fix to start. [08:15:27] would help you with the getting started and submitting patches part [08:15:36] YuviPanda, cool :) [08:16:12] YuviPanda, thanks for the pointer, I will work on it after reaching home [08:16:34] :D [08:16:40] YuviPanda, you are contributing to shout? :) [08:16:50] I was :) haven't had time of late [08:16:54] have a few patches in. [08:16:58] implemented their notification support [08:17:12] YuviPanda, awesome and beautiful code, it is [08:17:18] YuviPanda, I am mesmerised [08:18:01] shout? [08:18:05] shout isn't that great, tbh. [08:18:09] uses DOM to store state [08:18:16] YuviPanda, is it? [08:18:26] it is :) [08:18:27] YuviPanda, I was talking about the backend code [08:18:30] well, the software is nice [08:18:31] ah [08:18:34] I haven't touched that [08:18:49] YuviPanda, I was planning to use it as a backend for Waartaa [08:19:00] aaah [08:19:01] nice [08:19:08] YuviPanda, but the API is not that great, it's tightly tied with the client side code [08:19:11] yeah [08:19:54] YuviPanda, did you see our new web page? https://www.waartaa.com [08:20:01] I have! [08:20:02] YuviPanda, waartaa is in 0.2 now :) [08:20:07] nice [08:20:45] YuviPanda, you are in Bangalore, right now? [08:21:15] rtnpro: nope, kerala [08:26:00] ooooooh http://datasets.wikimedia.org/aggregate-datasets/playground/qchris/media-file-accesses/ [08:26:33] YuviPanda, will you be online here in evening? [08:26:47] YuviPanda, then I'd ping you from home then [08:27:29] Sure [08:28:48] YuviPanda, thanks :) [08:29:52] Yw [08:31:36] Analytics / Dashiki: Story: Vital Signs User selects the Daily Pageviews metrics - https://bugzilla.wikimedia.org/72740#c34 (Nemo) PATC>NEW Interesting! Will the CSVs end up on http://datasets.wikimedia.org/aggregate-datasets/ or elsewhere? Wherever they end up, please add a link from the raw fil... [13:28:26] qchris: when is a convenient time to babysit EL during m2-master CNAME attempt #2? [13:28:52] Today is all meetings :-/ [13:29:14] well, no hurry. sometime this week? [13:29:26] Tomorrow morning would by fine by me. [13:29:43] At what UTC hour do you typically stop working? [13:30:21] what is this 'typically' concept? [13:30:25] :) [13:30:25] Hahaha. [13:30:31] * springle calculates [13:30:37] Is 9UTC too late for you? [13:30:55] 9UTC would be fine [13:31:15] Fine by me too. [13:31:23] great, thanks qchris [13:31:36] I thank you springle. [13:31:38] Great. [14:53:11] Analytics: Upgrade Analytics Cluster to Trusty, and then to CDH 5.2 - https://phabricator.wikimedia.org/T1200#23134 (Ottomata) I did analytics1013 and 1014 yesterday. Starting with 1015 today. [15:06:38] YuviPanda, I am back from office :) [15:06:46] hi :) [15:11:35] YuviPanda, pm? [15:29:52] Analytics / Dashiki: Story: Vital Signs User selects the Daily Pageviews metrics - https://bugzilla.wikimedia.org/72740#c35 (nuria) The cvs will be in a repo that we were thinking of checking out under wikimetrics directory for the time coming but with a git clone so anyone can check it anywhere. At th... [16:31:46] ottomata: come back! [16:33:58] YuviPanda, for using mediawiki OAUTH, what credentials shall I use in config.yaml file of quarry? [16:34:08] rtnpro: ah, just request one token. [16:34:09] moment [16:38:12] rtnpro: https://www.mediawiki.org/wiki/Special:OAuthConsumerRegistration [16:38:19] rtnpro: it doesn't need approval to test with your own account [16:39:12] YuviPanda, it says: You do not have permission to propose new OAuth consumers, for the following reason: The action you have requested is limited to users in one of the groups: Autoconfirmed users, Confirmed users. [16:39:30] oh [16:39:33] we can give you that [16:39:43] :) [16:39:56] rtnpro: what's your username? [16:40:06] YuviPanda, rtnpro :D [16:51:54] rtnpro: hmm, need to find someone who can, will be done shortly [16:52:25] YuviPanda, anyways, I am done with the patch, I will send it for review [16:52:33] rtnpro: woot, cool [17:07:49] YuviPanda, here you go: https://github.com/wikimedia/analytics-quarry-web/pull/2 [17:08:39] rtnpro: yay, cool. that should work :) [17:09:56] rtnpro: so I can either just import it into gerrit myself, but you can do that too! [17:10:03] rtnpro: would give you experience with it, and make future contributions easier. [17:10:04] what say? [17:10:22] YuviPanda, let me do it :) [17:10:27] rtnpro: cool :) [17:10:49] YuviPanda, any link to some How to? [17:11:17] rtnpro: https://www.mediawiki.org/wiki/Gerrit [17:17:40] YuviPanda, what's Change-Id, is it the commit hash? [17:17:50] rtnpro: ah, it's this weird thing gerrit uses... [17:17:57] rtnpro: use git review, and install a hook with 'git review -s' [17:18:03] and then it should automatically add a change-id [17:18:12] YuviPanda, ok [17:20:40] (CR) Nuria: [C: 2] Update UI-Daily [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/174055 (owner: Bmansurov) [17:23:47] YuviPanda, I cannot find "quarry" codebase here: https://git.wikimedia.org/ [17:24:00] rtnpro: it'll be inside 'analytics'. it's analytics/quarry/web [17:24:18] YuviPanda, ok [17:28:50] ottomata, got time for a quick Q re. hadoop? [17:29:16] yup [17:29:19] wassususuuuup [17:29:21] I've got some processes that take a long time to crash. I'd like to start looking at the stderr before it fully shuts down. Is there a way I can do that? [17:29:45] * halfak wants to iterate faster. [17:30:10] jgage has been working on that! [17:30:12] i don't know the status of it! [17:30:14] :/ [17:30:24] ideally, yes, you will be able to see your logs in logstash in realtime [17:32:10] halfak: hmm, i thikn you rlogs are in logstash [17:32:15] but i dont' know logstash well [17:32:26] the goal is to be able to tell it to show you all logs from your application_id [17:32:29] your running one is [17:32:29] http://localhost:8088/cluster/app/application_1415917009743_4929 [17:32:38] How do I view that? [17:32:41] wget ? [17:33:25] oop [17:33:26] ssorry [17:33:28] i meant to just paste the id [17:33:32] application_1415917009743_4929 [17:33:33] that is your id [17:33:41] if you got o [17:33:41] to [17:33:48] logstash.wikimedia.org (log in with your labs/ldap creds) [17:33:51] and then search for that [17:33:52] you can see logs [17:33:56] but, i don't know the logstash gui at all [17:34:03] so I don't know how to tell you to filter to see anything useful [17:39:26] (PS1) Rtnpro: Show author info a query in query details page. [analytics/quarry/web] - https://gerrit.wikimedia.org/r/174152 (https://bugzilla.wikimedia.org/69544) [17:41:39] YuviPanda, \o/ [17:44:13] Yay [17:44:42] I'll merge soon [17:44:46] On phone now [17:44:51] YuviPanda, thanks :) [17:57:22] Thanks ottomata. Scoping it out [18:00:35] Hmm.. I don't see logs when I search for application_1415917009743_4929 [18:00:52] hm, me neither. [18:09:13] ottomata: how can I regain access to the research sql user? [18:10:20] whois /tgr [18:10:23] whoops! [18:10:23] hah [18:10:24] awell [18:10:26] hi tgr! [18:10:29] who are ya? [18:10:41] Gergo thank you irc ;) [18:10:44] haha [18:10:45] ummmm [18:10:50] tgr, did you have the research pw before? [18:10:54] yes [18:10:59] who gave it to you? [18:11:09] Gilles, I think [18:18:51] DarTar: The WikiGrok eventlogging tables don't seem to have recorded anything for the past 5 hours or so. Any idea why that might be the case? [18:20:01] DarTar: Or am I expecting the wrong time zone? [18:21:26] kaldari: can you manually generate an event and see if it shows up? [18:22:24] DarTar: Doesn't seem to show up. But both of my tests from yesterday are in there. [18:23:25] DarTar: It looks like new events are showing up, but the timestamps don't make any sense. For example a new error entry showed up a few seconds ago with the timestamp 20141118133333 [18:23:36] kaldari: on a call, brb [18:28:49] tgr, i've added you to the researchers group, running puppet on stat1002 now [18:32:30] sorry, you use stat1003* [18:44:07] YuviPanda: yt? [18:47:45] ottomata: s1-analytics-slave doesn't seem to have gotten any updates for the past 5 hours. Is there anyone I should notify about that? [18:48:29] (CR) Mforns: [C: 2 V: 2] "I've seen you changed the file names, even the ones that already exited. That's perfect." (2 comments) [analytics/dashiki] - https://gerrit.wikimedia.org/r/168488 (owner: Milimetric) [18:49:16] springle: ? [18:55:49] FYI, I found a bug in the syntax highlighting extension that is required by event logging. [18:55:52] https://bugzilla.wikimedia.org/show_bug.cgi?id=73567 [19:00:34] ottomata: thanks! [19:02:25] DarTar, ottomata: also looks like analytics-store.eqiad.wmnet hasn't recorded anything for the past 45 minutes. [19:03:51] nuria__: sup [19:04:33] YuviPanda: there is nothing else to do in your side for eventlogging beta labs to work, correct? [19:05:09] Yup [20:01:59] * halfak can't find any error information in logstash :( [20:02:53] halfak: yeah, sorry, i think we need to talk to gage, i'm not sure what is going in there right now [20:03:24] Hokay. I might need to put this down until I can iterate faster. I'm just eating up time waiting 15 minutes after I start a run to see why it failed. [20:03:35] halfak, what are you running and what against? [20:03:47] can you reduce the size of the data you are running on? [20:03:56] also, hadoop in vagrant works, I often develop there [20:04:06] I'm running some diff generating code. It runs fine on the mini-sample. It's failing on Simple English Wikipedia. [20:04:43] using wikihadoop? [20:05:07] Yup. [20:05:25] I'm actually going back to your mini-sample to make sure that the current version still works there. [20:05:41] do you know if your job actually launches? i tried to run wikihadoop on english wikipedia once, and it never really started [20:05:49] it took the entire time calculating splits [20:05:51] like, days., [20:05:54] before failing [20:06:15] Ugh. [20:06:21] I'm not sure how to know [20:06:28] did you get an application_id? [20:06:49] Oh yes. [20:06:53] ok, then i think it started [20:06:56] i never even got that [20:07:02] simple english wikipedia is much smaller? [20:07:03] anyway? [20:07:31] Yeah. One file as opposed to 270 files too. [20:07:51] Aha! I fail on the sample. [20:07:57] I can iterate on this for a bit. [20:07:58] ? [20:08:04] oh [20:08:06] on the sample [20:08:07] cool [20:08:19] halfak, you know how to cat the logs out of hdfs once the job is complete, yes? [20:08:24] I had a successful run over the sample, then moves to Simple English. looks like one of my changes caused a problem. [20:08:26] complete/failed [20:08:27] Yes [20:08:31] ok cool [20:08:39] It's waiting for it to stop that makes me sad. [20:09:06] ja, push gage! [20:09:12] we've been trying to get this done for months now [20:09:27] i keep telling him it will be really useful for you all, but i don't think he's heard it from you directly :) [20:10:35] * halfak makes a note to say "thank you sir" and "i want moar" [20:44:09] (PS1) Nuria: Adding dist files & config for staging [analytics/dashiki] - https://gerrit.wikimedia.org/r/174268 [20:45:17] (PS2) Nuria: Adding dist files & config for staging [analytics/dashiki] - https://gerrit.wikimedia.org/r/174268 [20:46:05] (CR) Nuria: [C: 2 V: 2] "Self merging dist changes + config per team policy on this regard." [analytics/dashiki] - https://gerrit.wikimedia.org/r/174268 (owner: Nuria) [20:57:28] ottomata: I think this should be ready to get merged, would you be so kind to take a look? [20:57:31] ottomata: https://gerrit.wikimedia.org/r/#/c/172285/ [21:14:14] mforns: yt? [21:14:31] yep [21:14:39] how are you doing? [21:15:27] good :) [21:15:45] so, it looks like we have fresh new data for pentaho /poke Ironholds [21:15:58] ok [21:16:23] can you help do the magic import? there’s a schema change needed (one new boolean field) [21:16:25] I'll import it now [21:16:33] that's ok [21:17:31] mforns, I'd advise against [21:17:34] it's not in the table yet ;p [21:17:45] ha ha [21:17:50] oh, ok [21:18:07] Ironholds, mforns: should we drop the broken cube “Complete monthly” ? [21:18:10] let me know when, and I'll import it [21:18:40] if anyone is going to use it, I can delete it now [21:18:47] sorry [21:18:52] I’d like to leave the current monthly data (just in case the new data introduces a regression/discrepancy) [21:18:55] if anyone is NOT going to use it [21:19:05] but complete monthly can go IMO, Ironholds ? [21:19:06] ok [21:19:52] DarTar, agreed [21:19:56] in order words: [21:20:03] drop complete monthly, keep monthly [21:20:09] “Complete Monthly Pageview Cube”: drop [21:20:12] create monthly 1.1a [21:20:22] “Monthly pageview cube”: keep [21:20:35] ok ok [21:20:35] “Pageviews cube”: keep [21:20:44] ^ this being the legacy daily data [21:21:20] Analytics: Upgrade Analytics Cluster to Trusty, and then to CDH 5.2 - https://phabricator.wikimedia.org/T1200#23258 (Ottomata) Done today: 1015 1016 1017 1019 1020 1029 1030 1031 1032 Something is weird with 1033's console. I will come back to it. [21:21:20] adding a version number doesn’t sound a bad idea [21:21:49] ok, the complete monthly pageviews was deleted [21:22:31] when importing the new data, I'll put a version on it [21:22:49] mforns: the cube is still there when I create a new board [21:23:00] (and I logged out and in again) [21:23:05] you have to click refresh button [21:23:10] oh... [21:23:16] aaaah, magic [21:23:17] try again now, please [21:23:20] it worked :) [21:23:22] ok [21:23:52] I'll add a version number to the current cube [21:24:00] mforns: that’d be great [21:24:04] 0.1? [21:24:08] yes [21:24:11] ok [21:25:02] how about: data fixes, bump subsubversion, schame changes, bump subversion (cc Ironholds) [21:25:14] ? [21:25:35] so the old monthly cube will be 0.1, the new one 0.2 (new field) [21:25:40] gotcha [21:25:45] semantic naming scheme? [21:25:51] 0.1-0 to 0.2-1 [21:26:03] wfm [21:26:11] or subversion, I don’t really care [21:26:49] I added v0.1 to the actual cube [21:31:08] Ironholds: would you be ok learning how to import the data into teh cubes so you guys can drive all teh process in your end? [21:31:11] *the [21:31:23] nuria__, I WOULD LOVE THAT :D [21:31:24] Ironholds: process is actually pretty straight forwrad [21:31:32] seriously, that sounds hella-fun [21:31:34] Ironholds: ok, we'll get some docs [21:31:42] awesome! Thank you so much! [21:31:42] Ironholds: let's touch base later [21:31:49] shall do! Drop me an email :) [22:14:24] mforns, writing now [22:14:32] ok! [22:16:21] mforns, data written! [22:16:26] ok [22:17:43] the table continues to be pentahoviews? [22:17:54] Ironholds, ^ [22:18:45] yup! [23:14:45] DarTar, Ironholds, the data is in the cube. Sorry for the delay, I was documenting in parallel [23:14:55] tis okay [23:17:15] I'm heading off for the evening [23:17:19] emotionally and physically exhausted [23:23:04] thanks mforns! [23:23:17] and get out of here Ironholds :) [23:23:21] np! bye Ironholds [23:29:35] Analytics / EventLogging: Beta setup of event logging not working - https://bugzilla.wikimedia.org/73388#c7 (nuria) Closing, can see event logging events in db and log stream w/o issues. [23:30:06] Analytics / EventLogging: Beta setup of event logging not working - https://bugzilla.wikimedia.org/73388#c8 (nuria) Tables on db: | CentralAuth_5690875 | | Echo_7731316 | | ImageMetricsLoadingTime_10078363 | | MediaVie... [23:30:21] Analytics / EventLogging: Beta setup of event logging not working - https://bugzilla.wikimedia.org/73388 (nuria) NEW>RESO/FIX