[00:00:42] kk [00:17:29] (PS1) Kaldari: Fixing deleted uploads LIMN graph [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/150741 (https://bugzilla.wikimedia.org/68755) [00:18:32] Ironholds: Can you merge https://gerrit.wikimedia.org/r/#/c/150741 ? [00:20:58] kaldari, I do not have +2 [00:21:39] oh, maybe that's owned by mobile web team then [00:22:29] no, I don't have +2 anywhere ;p [00:22:34] I can read R and python and JSON. [00:22:40] everything else is ancient sumerian [00:31:30] (CR) MaxSem: [C: 2] Fixing deleted uploads LIMN graph [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/150741 (https://bugzilla.wikimedia.org/68755) (owner: Kaldari) [00:31:36] (Merged) jenkins-bot: Fixing deleted uploads LIMN graph [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/150741 (https://bugzilla.wikimedia.org/68755) (owner: Kaldari) [00:32:18] (PS4) Terrrydactyl: [WIP] Add ability to delete wiki users [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/142045 [00:32:28] (CR) jenkins-bot: [V: -1] [WIP] Add ability to delete wiki users [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/142045 (owner: Terrrydactyl) [00:33:32] (PS5) Terrrydactyl: [WIP] Add ability to delete wiki users [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/142045 [00:34:16] oof, i still forget to run flake8. eventually i'll learn. [01:18:49] (PS1) Gergő Tisza: [WIP] Query UploadWizard funnel data [analytics/multimedia] - https://gerrit.wikimedia.org/r/150749 [01:26:48] (PS1) Gergő Tisza: [WIP] Query UploadWizard funnel data [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/150750 [07:40:30] (PS2) Gergő Tisza: Query UploadWizard funnel data [analytics/multimedia] - https://gerrit.wikimedia.org/r/150749 [09:51:49] (CR) Gilles: Query UploadWizard funnel data (1 comment) [analytics/multimedia] - https://gerrit.wikimedia.org/r/150749 (owner: Gergő Tisza) [12:03:25] (CR) Gergő Tisza: Query UploadWizard funnel data (1 comment) [analytics/multimedia] - https://gerrit.wikimedia.org/r/150749 (owner: Gergő Tisza) [12:12:11] (PS2) Gergő Tisza: Query UploadWizard funnel data [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/150750 [12:16:52] (CR) Gergő Tisza: "Doesn't work: http://multimedia-metrics.wmflabs.org/dashboards/uw" [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/150750 (owner: Gergő Tisza) [12:49:04] (CR) QChris: [C: -1] Add select_missing_sequence_runs.hql (10 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/150569 (owner: Ottomata) [13:44:48] ohmahgoodness qchris what a review! :) [13:44:59] real quick reply via the substitution depth [13:45:00] https://gerrit.wikimedia.org/r/#/c/150567/ [13:45:07] Ha. [13:45:11] Looking ... [13:45:34] Oh. [13:45:42] Ok :-) [13:45:42] qchris, i know about the month year ranges you are talking about, for the most part all I have done here is adapt the script as it was, and made it more useable [13:45:48] sorry, [13:45:52] month/day/hour ranges [13:45:53] being weird [13:46:02] i'm sure there are many better ways to do all of the things in this script [13:46:15] BUT! ha, I am not going to work on this! that looks like many hours of work, and I am not a sql pro [13:46:29] dan mostly wrote this query, and it works as is, even though i'm sure it could be way better. [13:46:37] i just wanted to get it into the repo, because I kept losing it in gist [13:46:50] I am not sure about the "It works as is" ... [13:47:00] See the comment on line 128. [13:47:02] can we put your review comments into comments in the file as is? and fix later? [13:47:15] looking [13:47:22] Do not tell anybody, and I'll fix the query. [13:47:34] It should be fairly simple with the windowing functions. [13:47:42] (See per file comment) [13:47:48] If it isn't we merge as is. [13:47:52] Sounds like a plan? [13:48:15] yeah, totally, that would be awesome [13:48:24] Ok. [13:48:38] where does the per mapper part come from? the rowSequence udf? [13:48:57] No. It's how the whole map por works. [13:49:02] s/por/part/ [13:49:24] In order for mappers to synchronize on a global state, you'd have to jump through many hoops. [13:49:53] isn't that the point of hive though? evne though you are joining on subquery parts, the results shoudl eventually be joined via several stages of map/reduce jobs? [13:50:13] Yes, but the mappers do not know about hive. [13:50:20] ha, but hive knows about the mappers! [13:50:24] Yes :-D [13:50:29] and how they have to work with data [13:50:39] iunno, i believe you :) [13:50:48] just seems like hive should know better [13:50:54] Just try the sample query I provide. It exhibits the problem. [13:51:08] i believe ya! [13:51:09] (the one from comment on line 128) [13:51:13] Ok :-P [13:53:17] (CR) QChris: Add select_missing_sequence_runs.hql (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/150569 (owner: Ottomata) [14:33:07] milimetric: hey! I made debian packages for all the dependencies of quarry, and have gotten it merged :) Let me know if you want me to make you packages as well :) Won't be useful in vagrant, though. [14:35:44] milimetric: am also considering setting up a debian repository source as a labs project, so other labs projects can depend on it to pull in python / other packages without having to duplicate effort. [14:41:04] hey YuviPanda, milimetric is on vacation, I think he’s not reading on IRC (and if he is, he shouldn’t ;) ) [14:41:13] DarTar: aaah, didn't know :) [14:41:20] DarTar: yay vacations :) [14:42:17] he worked hard to ship all the stuff we needed before Wikimania, I hope he switched off IMAP mail from his phone for good now [14:42:29] DarTar: :) will he be at wikimania? [14:42:34] nope [14:42:54] DarTar: aww, damn. you? [14:43:08] he’ll be in Philly for the hackathon though [14:43:21] yes, I’m flying in this Tuesday [14:44:07] and I intend to steal you from Quarry when you’re done with it to work on Twitter bots [14:44:43] YuviPanda ^ [14:45:02] DarTar: haha :) So far people attempting to steal me would be my current team (apps), future team (ops), the toollabs volunteers, and analytics :D [14:45:23] DarTar: but yeah, bots sound fun :D [14:45:25] celebrity time [14:45:35] DarTar: I like these kind of bite sized projects that can be completed in a few hours and deployed [14:45:44] yesss [14:46:06] see, is Quarry done? [14:46:24] DarTar: *kinda*. still tying up loose ends, and there's no CSV export yet [14:46:30] kk [14:46:50] DarTar: toby was quite enthusiastic about getting this in prod as well, for PMs and others who use EL :) [14:47:32] yeah, although for them a GUI client should do the job as well, I really see this as primarily targeted at external folks [14:50:24] (PS1) Yuvipanda: Separate wiki_id from user id [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150834 [14:51:14] DarTar: true, true. [14:51:26] DarTar: still, this will make it easy to collaborate on / share SQL, instead of using pastebin [14:54:32] yup [14:55:50] (PS2) Yuvipanda: Separate wiki_id from user id [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150834 [15:43:40] (PS3) Yuvipanda: Separate wiki_id from user id [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150834 [15:58:49] (PS1) QChris: Add Hive script to show runs of missing sequence numbers in webrequests [analytics/refinery] - https://gerrit.wikimedia.org/r/150844 [16:07:06] (PS1) Yuvipanda: Simpler way of varying UI for editable/non-editable queries [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150846 [16:28:00] (CR) QChris: "This change is a different approach to" [analytics/refinery] - https://gerrit.wikimedia.org/r/150844 (owner: QChris) [16:28:53] (CR) QChris: "The change" (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/150569 (owner: Ottomata) [16:31:10] (CR) QChris: "> but MapReduce CPU time decreases drastically (" [analytics/refinery] - https://gerrit.wikimedia.org/r/150844 (owner: QChris) [16:36:44] [travis-ci] wikimedia/mediawiki-extensions-EventLogging#231 (wmf/1.24wmf16 - e11e40d : Reedy): The build passed. [16:36:44] [travis-ci] Change view : https://github.com/wikimedia/mediawiki-extensions-EventLogging/commit/e11e40de8c00 [16:36:44] [travis-ci] Build details : http://travis-ci.org/wikimedia/mediawiki-extensions-EventLogging/builds/31348077 [16:37:23] hi qchris -- neat work. [16:37:37] Mhmm? [16:37:48] on the sequence # script [16:37:57] :-D You're reading the gerrit changes? [16:38:00] how much data did you test on [16:38:00] Awesome! [16:38:02] indeed [16:38:11] be careful ;) [16:38:12] A few single partitions. [16:38:22] yikes -- 66 minutes for a partition? [16:38:33] Yup. But that's total cluster time. [16:38:35] (CR) Legoktm: [C: 2] Fix License notice to be consistently CC0 [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150540 (owner: Yuvipanda) [16:38:38] oic [16:38:43] (Merged) jenkins-bot: Fix License notice to be consistently CC0 [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150540 (owner: Yuvipanda) [16:38:43] It's 5 minutes wall-clock time. [16:38:52] And we only run that script, if [16:39:05] the other (faster) script flags missing numbers. [16:39:25] got it -- this script is for debugging then. [16:39:31] Yes. [16:39:56] But the old version we were having does not work reliably, as it [16:40:08] assumes there is a global (cluster-wide) state for UDFs. [16:40:28] in that they are available? [16:40:36] That was from krake... times. [16:40:54] No. In the sense that, when say 37 mappers use the same [16:41:06] UDF, all those 37 instances magically synchronize. [16:41:11] That of course cannot work. [16:41:13] oh shit [16:41:36] thanks for fixing that [16:41:48] Hey. It's our job :-D [16:42:30] heh -- that's true! glad to get some more eyes on it though. [16:44:46] (CR) Legoktm: [C: 2] Separate wiki_id from user id [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150834 (owner: Yuvipanda) [16:44:51] (Merged) jenkins-bot: Separate wiki_id from user id [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150834 (owner: Yuvipanda) [16:45:14] (CR) Legoktm: [C: 2] Simpler way of varying UI for editable/non-editable queries [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150846 (owner: Yuvipanda) [16:45:18] (Merged) jenkins-bot: Simpler way of varying UI for editable/non-editable queries [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150846 (owner: Yuvipanda) [17:04:53] Analytics / EventLogging: Cleaning up of some (?) EventLogging schemata for Growth - https://bugzilla.wikimedia.org/68931 (christian) NEW p:Unprio s:normal a:None Around http://lists.wikimedia.org/pipermail/analytics/2014-July/002351.html it seems some EventLogging schemas need to get p... [17:05:52] Analytics / EventLogging: Cleaning up of some (?) EventLogging schemata for Growth - https://bugzilla.wikimedia.org/68931 (christian) [17:15:51] Analytics / Wikimetrics: Story: Researcher has prototype for wikimania - https://bugzilla.wikimedia.org/68516#c2 (Kevin Leduc) NEW>RESO/FIX Dan's work on the prototype is complete, resolving the issue. [17:19:25] (CR) Ottomata: "Whoa, this looks like some seriously advanced Hive craziness. Awesome. I have no idea how this works!" [analytics/refinery] - https://gerrit.wikimedia.org/r/150844 (owner: QChris) [17:23:21] leila, tnegrin - you guys ok going first in the agenda? [17:23:49] sure! [17:25:00] hold on -- Leila says that Dan G's talk segues into ours [17:25:57] Eloquence, so we wanted to have ours before Dan's, since ours is a more general intro to mobile stats [17:26:29] yep [17:26:39] you'll be first, then dan, then VE [17:26:41] * Eloquence rearranges [17:26:44] aha [17:27:14] sounds good then [17:27:25] ok, these slides are intense - pls make sure to give people enough time to process what they're seeing :) [17:28:35] yup, Eloquence. [17:29:00] there's a takeaways slide at the end [17:29:14] they tell a story :) [17:31:02] The story is solid and should help everyone move forward with it. :-) I also relayed your message to DarTar, to give him a heads up. [17:31:09] it's a great story [17:31:21] we got it from the Brothers Grimm [17:31:29] :) [17:32:13] The site->device classification scheme in particular is fairly complex to understand, esp. wrapping your head around "app" as a "site" [17:32:41] how can there be a non-zero amount of app read access via desktop? [17:32:41] (I can shut up if you guys are busy with prep :) [17:35:03] Eloquence, because desktop as a device class is 'not-mobile' [17:35:13] so it includes 'devices not identified as mobile phones or tablets' ;p [17:35:31] this also potentially includes third-party apps which may or may not provide useful UA info, as well as our app which does not provide useful UA info. [17:35:52] ok, so basically these are misclassified [17:36:58] possibly yes. perfect classification will be possible as and when you instruct everyone at the WMF to stop giving me things to do for about a month...and will only be perfect until people release new devices ;p [17:37:09] :) [17:37:19] I am hoping that some device spec information the Zero team is getting their hands on can be lined up with our UA parsing and provide a shortcut there, though [17:37:28] if at first you don't succeed, make a third party do the scutwork! [17:37:35] and then pay them for binaries of the results. [17:38:27] what this is saying is that mobile edits from tablets > mobile edits from phones as a % of all edits? [17:39:53] also, that there are a lot more mobile edits via the desktop site than via the mobile site? [17:40:08] yes and yes, I think [17:40:14] (but leila can answer better than me) [17:40:27] interesting - if true, that's a pretty big deal [17:40:32] but we'll want to double and triple check that [17:40:35] yeah [17:40:38] and it's edits, not editors [17:40:47] one very enthusiastic german could be throwing things off [17:40:54] shall we caveat these numbers heavily at this point? [17:41:13] * Ironholds had to deal with one editor from dewiki who had fun spoofing his user agent and leaving me wondering why googlebot was editing [17:42:02] Eloquence, we'll try to fit in some last-minute checks but in the absence of those, yeah. as with all research this has limitations around methodology and the time available to apply said methodology [17:42:41] has maryana seen this data yet? [17:43:09] Eloquence, we're checking few things. Give us few min [17:43:11] :-) [17:43:21] Analytics / EventLogging: Story: Product groups have EventLogging validation - https://bugzilla.wikimedia.org/67126#c1 (Kevin Leduc) p:High>Low Adjusting priority to low as this will not get implements in the next coming sprints. [17:43:21] I don't know. I didn't show her; I've been head-down on things for a few days :p [17:43:29] kk! I will shut up now. see you in a bit. [17:43:50] Analytics / EventLogging: Empty objects can pass schemas with required fields - https://bugzilla.wikimedia.org/65607#c1 (Kevin Leduc) p:High>Low Adjusting priority as this will not be fixed in the coming sprints. [17:48:36] Analytics / Refinery: Epic: AnalyticsEng has kafkatee running in lieu of varnishcsa and udp2log - https://bugzilla.wikimedia.org/68139 (Kevin Leduc) [17:50:05] Analytics / Refinery: Epic: AnalyticsEng has kafkatee running in lieu of varnishcsa and udp2log - https://bugzilla.wikimedia.org/68139#c1 (Kevin Leduc) p:Normal>Highes reprioritizing to highest so it gets pulled into next sprint [17:50:21] Analytics / Refinery: Story: AnalyticsEng has kafkatee on analytics1003 - https://bugzilla.wikimedia.org/68246#c1 (Kevin Leduc) p:Unprio>Highes re-prioritizing to highest so it gets pulled into next sprint [17:50:34] Eloquence: btw, desktop can have non-zero amount of app requests [17:50:36] Eloquence: because chinese users hit the desktop URL, since https on mobile web is blocked, but desktop is not [17:50:45] Eloquence: we've code in the app that first tries https mobile, on error then https desktop, and then fails. [17:51:04] Eloquence: so anything with our app's UA that's hitting desktop would be china [17:51:06] Analytics / Refinery: Story: AnalyticsEng generates new datafiles using kafkatee - https://bugzilla.wikimedia.org/68247#c1 (Kevin Leduc) p:Normal>Highes re-prioritizing to highest so it gets pulled into next sprint [17:51:36] Analytics / Refinery: Story: Vet the kafkatee generated files - https://bugzilla.wikimedia.org/68248#c1 (Kevin Leduc) p:Unprio>High re-prioritizing to highest so it gets pulled into an upcoming sprint [17:51:51] Analytics / Refinery: Story: Transparently switch from udp2log datafiles over to kafkatee generated datafiles - https://bugzilla.wikimedia.org/68250#c1 (Kevin Leduc) p:Normal>High re-prioritizing to highest so it gets pulled into an upcoming sprint [17:55:11] halfak: I thought I would skim the discussion page, but it seems easier (and less stressful) to ask you: How's the media viewer research proposal going? [18:01:51] Analytics / General/Unknown: Packetloss issues on oxygen (and analytics1003) - https://bugzilla.wikimedia.org/67694#c9 (Kevin Leduc) NEW>RESO/FIX Marking issue as resolved. I have prioritized focusing on Kafka next so there is no more work needed on UDP2LOG. https://bugzilla.wikimedia.org/show_bu... [18:15:59] marktraceur, badly. :\ [18:16:11] Am. [18:16:13] Aw* [18:16:17] OK [18:16:29] So, it looks like the team decided to postpone the study. [18:16:37] And the trolls are rejoicing. :( [18:16:42] Basically, yes, based on some blockers [18:17:13] However, Dario came in with some good feedback that I'd like to iterate on if "postponed" isn't indefinite. [18:17:43] right. [18:22:38] Analytics / General/Unknown: Turn on PHP on stat servers - https://bugzilla.wikimedia.org/68937 (Ryan Kaldari) NEW p:Unprio s:normal a:None Talked to Toby about this recently... Since we have far more PHP developers than Python developers, we could distribute more analytics work simply by... [19:16:19] Hallo. [19:16:25] Hi aharoni [19:16:39] http://reportcard.wmflabs.org/graphs/active_editors gives me info about active editors in top ten languages or so. [19:16:48] Where can I get info about the rest of them? [19:16:53] i mean languages [19:18:52] There are some grahps on gp.wmflabs.org that come with many caveats. [19:18:59] Let me find an url ... [19:19:51] http://gp.wmflabs.org/graphs/enwiki_editor_counts [19:19:53] http://gp.wmflabs.org/graphs/dewiki_editor_counts [19:20:24] That should be available for all wikipedias, but it overcounts/undercounts a bit. [19:20:46] Let me check if I can find something on stats ... [19:21:40] There you go: http://stats.wikimedia.org/EN/SummaryDE.htm [19:21:44] There you go: http://stats.wikimedia.org/EN/SummaryEN.htm [19:21:53] http://stats.wikimedia.org/EN/SummaryEO.htm [19:22:38] aharoni: I'd go with the http://stats.wikimedia.org/ urls [19:24:26] aharoni: also coming soon, vital signs for all Wikimedia projects https://metrics-staging.wmflabs.org/static/public/dash/ [19:24:46] generated daily [19:24:52] thanks [19:25:05] see also https://meta.wikimedia.org/wiki/Research:Metrics_standardization for some background reading [19:26:11] aharoni: the rolling active editor data you see there is based on a slightly different definition than the canonical one [19:26:26] we’ll be presenting the rationale and background research at Wikimania [19:26:47] Just for fun I am trying to compare the number of VE edits in English and Hebrew. [19:27:26] According to pages cited by qchris , English has 44 times more active editors, [19:27:42] and English has 28 times more articles, [19:28:09] but if I look at Special:Tags, English has 839,000 VE edits, and Hebrew has 105,000 - only 8 times more in English. [19:28:34] :-) [19:28:42] And of course this is a very imprecise measurement, because in Hebrew VE was deployed about half a year later than in English. [19:29:12] I guess that the number of anon users of VE in English is about zero, because it's disabled, [19:29:24] but in Hebrew it's enabled. [19:29:38] where can I see these numbers? Usage of VE by active editors vs anons? [19:31:02] I am not sure we have those numbers. DarTar do you know? [19:31:10] Oh, he's no longer here. [19:31:44] halfak: Do you know whether we have numbers on usage of VE by different editor groups? [19:31:51] (See question of aharoni above) [19:32:04] * halfak is reading scrollback [19:33:30] Hey aharoni. We've got some dashboards. I'm looking right now to see if we have something that will work for you. [19:35:07] http://ee-dashboard.wmflabs.org/dashboards/enwiki-metrics third graph down [19:35:45] aharoni, ^ [19:35:49] See also http://ee-dashboard.wmflabs.org/dashboards/hewiki-metrics [19:35:58] thanks! [19:36:11] hth :D [19:38:50] halfak: it says "Jun 28, 2013" [19:38:58] how do I get something newer? [19:39:07] Ack. [19:40:02] enwiki is recent [19:40:17] hewiki is too. [19:40:34] aharoni, it could be that you're zoomed into an old part of the graph. [19:41:45] halfak: I don't think so. I just went to the URL and didn't click anything more. [19:42:11] The graphs are showing recent data for me too (for enwiki and hewiki). [19:42:38] When I hover with my mouse to the right side of the plots, the legend on the left reports recent dates. [19:43:24] aharoni, you're not alone. limn is not terribly intuitive. We're working on that. [19:53:21] oh :) [20:07:15] halfak, DarTar , StevenW - and is there some trend line that shows the rate of article creation (per day or per week), and of active editors? [20:07:36] Analytics / Wikimetrics: Bring WIKIMETRICS_POOL_SIZE to vagrant's wikimetrics setup - https://bugzilla.wikimedia.org/68825 (christian) PATC>RESO/FIX [20:08:27] aharoni: if you are referring to Vital Signs, not yet (we have the metric definitions, the actual data generation and visualization frontend are under constructions) [20:08:28] aharoni, I don't think that we have any article creation dashboards up right now. [20:09:06] I mean, it's easy to see that Hebrew and French have much more VE edits than English, considering their size, [20:09:39] but the really interesting part is whether this affects the rate of edits, article creation and new editor acquisition [20:09:47] (is "acquisition" an appropriate word here? :) ) [20:12:19] it is :) [20:12:22] Analytics / Wikimetrics: QueuePool pool_size limit should be configurable - https://bugzilla.wikimedia.org/68534 (christian) PATC>RESO/FIX [20:12:23] Analytics / Wikimetrics: Bring WIKIMETRICS_POOL_SIZE to vagrant's wikimetrics setup - https://bugzilla.wikimedia.org/68825 (christian) [20:12:28] Seems like an answerable question. Looking into it could be a good idea for our upcoming research hackathon. [20:12:37] https://meta.wikimedia.org/wiki/Research:Labs2/Hackathons/August_6-7th,_2014 [20:13:15] You could file it as an idea and join us to find some collaborators. https://meta.wikimedia.org/wiki/Research:Labs2/Ideas [20:13:17] aharoni, ^ [20:13:36] which reminds me I still have to add mine [20:14:00] halfak: I’ll do this tomorrow, checking out earlier tonight [20:14:02] * halfak looks forward to Dario's ideas so that he can publicize them. [20:14:07] sounds reasonable [20:14:20] Isn't it like a billion o'clock there? [20:14:30] nah, 10pm [20:14:39] Only a few orders of magnitude [20:14:43] last night was worse :) [20:14:48] 10 vs. 1000000000 [20:20:55] qchris: do you think we could write your fault_hosts files as tsvs? [20:21:19] i'm thinking about turning them into emailed reports [20:21:23] rather than using icinga for this [20:21:35] Sure. [20:21:41] Hive can use custom separators. [20:21:44] ja [20:21:53] just checking that you weren't opposed [20:22:06] Naw. Not opposed. [20:22:17] But Icinga alerts would have been kind of sexy though. [20:23:32] yeah, but its kind of weird (check out convo i'm having with jeff green in #ops) [20:23:47] Ok. Switching over to ops. [20:25:37] (PS1) Ottomata: Use tsv format when outputting webrequest faulty hosts files [analytics/refinery] - https://gerrit.wikimedia.org/r/150962 [20:25:58] (Abandoned) Ottomata: Use tsv format when outputting webrequest faulty hosts files [analytics/refinery] - https://gerrit.wikimedia.org/r/150962 (owner: Ottomata) [20:26:35] (PS1) Ottomata: Use tsv format when outputting webrequest faulty hosts files [analytics/refinery] - https://gerrit.wikimedia.org/r/150963 [20:26:45] DarTar: where should I add the idea? https://meta.wikimedia.org/wiki/Research:Labs2/Ideas ? [20:37:04] https://meta.wikimedia.org/wiki/Research:Ideas/How_does_the_availability_of_VisualEditor_affect_the_rate_of_edits_and_editor_acquisition_in_different_languages%3F [20:53:59] is there an easy way to deploy limn locally? [21:02:53] tgr: I guess, only milimetric can answer that question [21:29:38] (PS6) Terrrydactyl: Add ability to delete wiki users [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/142045 [21:45:08] (PS1) Yuvipanda: Use halfak's MWOAuth library [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150968 [21:45:22] ^ damn right [21:45:33] halfak: :D [21:51:18] (PS2) Yuvipanda: Use halfak's MWOAuth library [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150968 [22:02:37] Open your MWOAuth. [22:36:21] (CR) Legoktm: [C: -1] "CentralAuth makes me cry." (1 comment) [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150968 (owner: Yuvipanda) [22:38:15] (CR) Yuvipanda: Use halfak's MWOAuth library (1 comment) [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150968 (owner: Yuvipanda) [22:38:39] I can't stand it [22:46:13] i'm testing out some code on wikimetrics-dev1 and trying to access metrics-dev.wmflabs.org. tried uploading a cohort and then got a internal server error. is this related to the backfilling or something else? qchris was trying to help me set up the environment, but had to go for the night. [22:46:30] (PS3) Yuvipanda: Use halfak's MWOAuth library [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150968 [22:46:37] legoktm: ^ [23:05:55] legoktm: ^ merge? :) [23:06:11] "CentralAuth makes me cry" [23:06:17] also, [23:06:22] wikitech doesn't even have centralauth [23:06:24] so what [23:06:29] how is that going to work? [23:06:42] legoktm: ah, hmm. so I'll have to switch that over when that changes. I guess they'll send me a userid instead? [23:07:15] I'm still confused why this requires people to have a wikitech account. [23:07:41] legoktm: it doesn't right now, since wikitech oauth is borked :) [23:07:52] legoktm: but other than that, easiest way to make sure they've agreed to labs ToS? [23:08:19] but to make this the easiest to use, you really should have it hooked up to SUL [23:08:25] which requires using mw.o [23:08:41] just have a splash screen that shows up once saying "By using this service you agree to the labs TOS" [23:09:52] YuviPanda: you can copy the wikimetrics boilerplate for this if you want [23:10:04] legoktm: halfak ^ what do you think? [23:12:06] Analytics / General/Unknown: Turn on PHP on stat servers - https://bugzilla.wikimedia.org/68937#c1 (Toby Negrin) Otto -- how difficult to do is this? If it's straightforward, we should probably just do it as it seems like it would make it easier for folks to use the data. thanks, -Toby [23:14:51] Analytics / General/Unknown: Turn on PHP on stat servers - https://bugzilla.wikimedia.org/68937#c2 (Yuvi Panda) Is this just getting php packages installed, or is this about running a php webserver? If the former, it's trivial, but the latter is harder and will also need more careful review / work for... [23:17:07] Analytics / General/Unknown: Turn on PHP on stat servers - https://bugzilla.wikimedia.org/68937#c3 (Toby Negrin) Just the packages. We want to make it easy for Kaldari to access the dbs to generate files for Limn. I have made this point about php myself. Numerous times. [23:22:51] Analytics / General/Unknown: Turn on PHP on stat servers - https://bugzilla.wikimedia.org/68937#c5 (Yuvi Panda) ^ Patch installs the php commandline client only, along with the php5-mysql package. [23:32:29] legoktm: hmm, merge this anyway? We'll make a decision about wikitech vs labs ToS clicking tomorrow [23:34:46] YuviPanda: I can't really support the expansion of CentralAuth's grasp like that. [23:34:56] legoktm: lol [23:35:00] legoktm: there's no other ID for me to use! [23:35:10] local wiki id [23:35:15] local wiki's user id( [23:35:17] legoktm: not part of identify [23:35:17] id* [23:35:36] what is identify? [23:35:56] it's part of our OAuth implementation, returns info like rights, groups, and an identifier to use [23:36:55] so....use whatever the identifier is? [23:38:03] legoktm: that's centrealauth id for WMF wikis :P [23:38:21] legoktm: it's not even called centralauth id in the response, it's called something stupid like 'sud' or something [23:38:42] so just don't call it centralauth in your app [23:39:37] legoktm: hmpf, fine :) [23:43:04] legoktm: ^ [23:43:05] err [23:43:06] (PS4) Yuvipanda: Use halfak's MWOAuth library [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150968 [23:43:07] legoktm: ^ [23:44:52] (CR) Legoktm: [C: 2] ":\" [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150968 (owner: Yuvipanda) [23:44:58] (Merged) jenkins-bot: Use halfak's MWOAuth library [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150968 (owner: Yuvipanda) [23:45:05] okay, can I leave here now? [23:45:08] the colors are getting to me [23:45:13] legoktm: :) I'll get colors in at some point [23:45:15] legoktm: ok [23:45:25] Analytics / Refinery: Epic: Analyst has Page View Report from hadoop prototype - https://bugzilla.wikimedia.org/68961 (Kevin Leduc) NEW p:Unprio s:normal a:None It's a program/script for Hadoop TSV file with columns: Day | Project | Total Page views | Desktop Site Page views | Mobile Site... [23:45:45] QueenOfFrance: if you have op here, can you turn the colors on please? [23:46:07] Analytics / Refinery: Epic: Analyst has Page View Report from hadoop prototype - https://bugzilla.wikimedia.org/68961 (Kevin Leduc) p:Unprio>Low s:normal>enhanc [23:49:11] Analytics / Refinery: Epic: Webstats Collector is replaced in Refinery - https://bugzilla.wikimedia.org/68963 (Kevin Leduc) NEW p:Unprio s:enhanc a:None Replace the functions of Webstats collector with jobs in Hadoop, outputing the same data. [23:49:37] Analytics / Refinery: Epic: Webstats Collector is replaced in Refinery - https://bugzilla.wikimedia.org/68963 (Kevin Leduc) p:Unprio>Low [23:54:52] Analytics / Refinery: Epic: AnalyticsEng has fully dimentionalized Page View counts - https://bugzilla.wikimedia.org/68966#c1 (Kevin Leduc) p:Unprio>Low develop ETL that can scale to handle future dimension requests. Output pageview data to DB? [23:55:06] Analytics / Refinery: Epic: AnalyticsEng has fully dimentionalized Page View counts - https://bugzilla.wikimedia.org/68966 (Kevin Leduc) [23:59:01] Suddenly I can't convince oursql to handle encodings anymore. [23:59:13] heh [23:59:22] * YuviPanda switched from oursql to http://www.pymysql.org/