[00:37:13] Analytics, MediaWiki-API-Team, MediaWiki-Authentication-and-authorization: Create dashboard to track key authentication metrics before, during and after AuthManager rollout - https://phabricator.wikimedia.org/T91701#1147305 (Nuria) @bd808 How are you planning on tracking this data? server side event... [00:40:11] Analytics-Tech-community-metrics, MediaWiki-Developer-Summit-2015, ECT-March-2015: Achievements, lessons learned, and data related with the MediaWiki Developer Summit 2015 - https://phabricator.wikimedia.org/T87514#1147307 (Rfarrand) I added some of the main changes I would make for next year to the t... [00:56:41] Analytics, MediaWiki-API-Team, MediaWiki-Authentication-and-authorization: Create dashboard to track key authentication metrics before, during and after AuthManager rollout - https://phabricator.wikimedia.org/T91701#1147345 (bd808) @nuria I think these will all be server side events, and yes we will w... [01:09:05] Analytics-Kanban, Analytics-Wikimetrics: Utf-8 names on json reports appear as unicode code points: "\u0623\u0645\u064a\u0646" - https://phabricator.wikimedia.org/T93023#1147376 (Fhocutt) It looks like it may be an issue with default Flask settings, specifically JSON_AS_ASCII. From http://flask.pocoo.org/... [01:09:48] Analytics, MediaWiki-API-Team, MediaWiki-Authentication-and-authorization: Create dashboard to track key authentication metrics before, during and after AuthManager rollout - https://phabricator.wikimedia.org/T91701#1147381 (Nuria) @bd808 Excellent, sounds that we are set here. One last comment: let's... [01:34:11] Analytics-Wikimetrics: Can't edit list of users - https://phabricator.wikimedia.org/T67208#1147441 (Fhocutt) From @tflanagan-WMF, there are a few more usecases: * Deleting an invalid username and adding the valid one without having to upload a whole new cohort * Handling renamed users--currently, in order to... [01:35:01] Analytics-Wikimetrics: Feature Request: Cannot edit or delete cohorts - https://phabricator.wikimedia.org/T69664#1147443 (Fhocutt) [01:35:02] Analytics-Wikimetrics: Can't edit list of users - https://phabricator.wikimedia.org/T67208#1147444 (Fhocutt) [09:37:35] Analytics-Tech-community-metrics, MediaWiki-Developer-Summit-2015, ECT-March-2015, developer-notice: Achievements, lessons learned, and data related with the MediaWiki Developer Summit 2015 - https://phabricator.wikimedia.org/T87514#1148071 (Qgil) Very good! Have you seen the [[ https://www.mediawi... [14:41:19] mforns / ottomata: http://realtime.wmflabs.org/ [14:42:22] ??? [14:42:27] druid? [14:43:08] milimetric: ? [14:43:21] yes [14:43:37] it's been ingesting rc stream for the last day [14:43:46] and you can query it in yaml as you please [14:43:55] I can translate, I'm still getting used to the syntax [14:44:02] oh coool rcstream [14:44:02] awesome [14:44:09] sorry [14:44:13] it's just IRC atm [14:44:17] oh [14:44:17] but i'm moving it to rcstream [14:44:19] still, cool [14:45:58] milimetric, awesome! [14:46:42] so, the thinking here is that when we try out different data storage solutions, we should be able to play with them easily [14:46:54] so I made this as a prototype for what that playing might look like [14:47:18] we might want to make it more solid so we can iterate quickly over the other candidates [14:48:35] aha [16:38:24] Quarry: Database dump for analysis - https://phabricator.wikimedia.org/T93907#1149417 (Halfak) NEW [16:40:07] Quarry: Database dump for analysis - https://phabricator.wikimedia.org/T93907#1149427 (yuvipanda) Hmm, so *this* would be somewhat hard to actually produce. The internal DB format is in https://github.com/wikimedia/analytics-quarry-web/blob/master/tables.sql, and I am not sure off the top of my head how to... [16:50:56] Analytics-EventLogging, Analytics-Kanban: Backfill client side data data for 2015-03-20 - https://phabricator.wikimedia.org/T93602#1149447 (Nuria) Open>Resolved [17:14:22] Analytics, Gather, Mobile-Web: Update main menu schema to include collections for limn graphs - https://phabricator.wikimedia.org/T93690#1149518 (bmansurov) [17:22:25] (PS2) Jdlrobson: Update limn graphs (untested) [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/199162 (https://phabricator.wikimedia.org/T93690) [17:22:37] Analytics, Mobile-Web, Patch-For-Review: Update main menu schema to include collections for limn graphs - https://phabricator.wikimedia.org/T93690#1149557 (Jdlrobson) [17:22:46] Analytics, Gather Sprint E, Mobile-Web, Patch-For-Review: Update main menu schema to include collections for limn graphs - https://phabricator.wikimedia.org/T93690#1143524 (Jdlrobson) [18:06:07] Analytics-Engineering, Analytics-EventLogging, Analytics-Kanban: Tungsten deprovision, substitute host in Eventlogging setup - https://phabricator.wikimedia.org/T93920#1149730 (Nuria) NEW [18:18:27] Analytics-Engineering, Analytics-EventLogging, Analytics-Kanban: Tungsten deprovision, substitute host in Eventlogging setup - https://phabricator.wikimedia.org/T93920#1149822 (Nuria) Looks like it i s been substituted by graphite1001 [18:21:56] halfak: you should have an invite [19:08:57] Analytics-Engineering, Analytics-EventLogging, Analytics-Kanban: Tungsten deprovision, substitute host in Eventlogging setup - https://phabricator.wikimedia.org/T93920#1150159 (Ottomata) [19:09:17] Analytics-Engineering, Analytics-EventLogging, Analytics-Kanban: Tungsten deprovision, substitute host in Eventlogging setup - https://phabricator.wikimedia.org/T93920#1149730 (Ottomata) Filippo can you comment? [19:09:30] qchris: question for ya, if you are there [19:09:43] * qchris is there [19:10:18] qchris: i am trying to clean up a local commit on limn1 [19:10:25] k [19:10:50] qchris: and have added a new template to apache to be used by the local commit: https://gerrit.wikimedia.org/r/#/c/198782/ [19:11:15] qchris: similar to the zero stuff you did the other day [19:11:21] k. [19:12:00] qchris: what is the best way to bring that commit to the limn1 instance (last update of puppet prod branch there is from another era, like sep or something) [19:13:17] Since it's merged in operations/puppet: Just cherry-pick the commit. [19:13:34] And if possible make it that at the point [19:14:06] qchris: ok, just that one commit , very well, will do that [19:14:10] where you diverge from operations/puppet, first all the commits that you cherry-pick from operations/puppet follow, and [19:14:19] then the local customizations on top. [19:14:23] ok [19:14:38] cool. [19:14:40] will break for lunch and clean up after that [19:14:50] Cool. Ping me if you run into problems. [20:05:50] Analytics-Tech-community-metrics, Google-Summer-of-Code-2015: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T93937#1150479 (Sarvesh.onlyme) NEW a:Sarvesh.onlyme [20:12:50] milimetric, yt? [20:12:57] Analytics-Tech-community-metrics, Google-Summer-of-Code-2015: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T93937#1150525 (Sarvesh.onlyme) Open>Invalid [20:12:59] Analytics-Tech-community-metrics, Possible-Tech-Projects, ECT-March-2015, Epic, and 2 others: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#1150527 (Sarvesh.onlyme) [20:13:19] mforns: yes sir [20:13:23] hi! [20:13:30] I'm looking at your changeset [20:13:45] cool [20:14:12] qq: when gulping, it asks me to chose the version of d3, which one should I chose? d3#~3.3.13 or d3#3.4.10 [20:14:14] ? [20:14:31] oh, 3.4.10 is fine [20:14:50] and milimetric, that was a huge work, lots of changes :] [20:14:51] it doesn't actually matter, that happens from time to time, just a version conflict in their packaging specs [20:14:52] ok [20:14:59] oh I see [20:15:03] oh yeah, as I was saying, no reason to review it [20:15:09] because i don't want it merged [20:15:24] you can just grab it, run it, check it out in general [20:15:44] well I read all the code, because I want to know how it works [20:15:46] see if it breaks in some bad ways. There are no tests, there's hardcoded config, etc. lots of reasons we shouldn't merge [20:16:02] cool, yeah, hope it didn't give you a headache :) [20:16:41] xD, no. Also, did not go too deep to understand each line... [20:23:21] hey mforns, yt? [20:23:26] yes ottomata [20:23:43] how are you feeling? [20:25:09] sniffly, it is nice to only kinda work on a sick day [20:25:20] but, i want to poke around at stuff for a bit, so i was wondering how your stuff was going [20:25:26] have you been doing spark stuff? [20:25:43] today not yet, I was looking at Dan's code [20:26:05] the druid stuff? [20:26:18] ottomata, but I have 100 lines of scala-spark code that pretty much work [20:26:29] no, Dan's VE stuff [20:26:36] dashiki [20:26:40] ah ok, cool [20:26:55] mforns: was there something you said you wanted my help with today? [20:27:05] so ottomata if you want to have a look at the scala code I can show it to you in batcave [20:27:16] sure [20:27:18] ottomata, yes, I'd like to know where to put the code [20:27:29] and how to integrate with oozie [20:27:37] and how to test it properly [20:27:42] ah, i need to do that. [20:27:44] hmmm [20:27:46] yeah show me what you got [20:27:52] but just an overview on that, ottomata [20:28:00] i need to add that stuff to maven, i got it working locally, but never committed it [20:28:15] ok, gimme 1 minute, I'll be in the batcave [20:28:48] k [20:47:59] qchris: yt? [20:48:07] ottomata: yup. [20:51:18] What's up? [20:51:36] oh, just need someone to argue about refinerh source hierarchy [20:51:53] marcell is writing some spark/scala code that is a job [20:52:07] ah :-) [20:52:08] this is the first time we have had JVM code that is job specific, and not librarly code [20:52:17] so, i'm not sure where it should go! [20:52:22] refinery-source/refinery-wmf? [20:52:30] refinery/refinery-spark? [20:52:42] i mean: refinery-source/reifnery-spark [20:52:44] Mhmm... how does the code get built? [20:52:46] whatcha think? [20:52:48] mvn package [20:53:22] mvn package ... that sources like refinery-source. Yes. [20:53:36] ja, i think so too [20:53:52] Is the code somewhere to look at? [20:53:55] but it is weird now that there is a distinction, because now the jobs will live in two repositoires [20:53:57] mforns: ? [20:54:21] the only distinctino between the repositories is that now one is compiled, and the other isn't [20:54:29] before the compiled one only contained library code [20:54:41] True. [20:54:57] But I mean ... maybe the spark code can be seen as library? [20:55:05] naw, he's writing a job that will be launced by oozie [20:55:08] it will use refinery-core stuff [20:55:08] If one has to compile it ... it sort of is. [20:55:23] naw, by library, i mean that it won't be imported by anything else [20:55:43] pretend he was writing a MapReduce job to do some analysis [20:55:44] its the same [20:55:46] but refinery-tools isn't imported by anything either. [20:55:48] where would we put that? [20:55:54] aye true! that is not library code. [20:55:59] refinery-jobs [20:56:00] ? [20:56:26] Maybe ... I guess I'd want have to look at the code to roughly understand what would make sense to me. [20:56:28] refinery-job [20:56:52] Is the code like an jar that is launched in an action? [20:57:03] mforns: is your code online somewhere? maybe you can just put a gist real fast? qchris, i think you will not like it, i think it has the characters u-u-i-d in it [20:57:05] yes [20:57:13] maybe you shoudl stop helping me [20:57:15] :p [20:57:15] Hahaha :-) [20:57:33] hehehe [20:57:37] Let's just pretend it's a black-box jar for an action then. [20:57:41] right :) [20:58:06] This seems to be coupled to oozie (not sure)... so refinery-oozie? [20:58:18] naw, def not. it could be run on its own, or scheduled by anything [20:58:29] k. [20:58:39] given input and output paths, it will do stuff [20:58:44] refinery-action? [20:58:48] refinery-job? [20:59:02] job sounds hadoopy [20:59:15] there might be spark streaming stuff in here in the future, who knows, but job is still relevant i think [20:59:20] well, it is pretty hadoopy [20:59:29] that is true, but job is pretty generic, no? [20:59:30] cron job [20:59:31] oozie job [20:59:34] hadoop job [20:59:44] Yes, it's overloaded so many times already. [20:59:58] I would not know where I can/cannot use refinery-job classes. [21:00:00] ja, but it possibly fits all three of those descriptions [21:00:23] refinery-job classes are meant to only be used as applications of other stuff. [21:00:30] if this were a MVC framework [21:00:31] Is there an english term for a part of a refinery? [21:00:33] this would be the app [21:00:36] haha [21:01:10] refinery-pipe? [21:01:12] Naw. [21:01:38] smelting [21:01:59] That's not bad. [21:02:33] On the other hand ... if it is generic an can be used outside of oozie ... can it live in refinery-core? [21:02:33] it is kinda [21:02:46] i think it might be good to be out of core, i think core should be just libraries [21:02:52] that are meant to be used [21:02:57] true. [21:02:58] rather than applications of those libraries [21:03:03] so far i like either [21:03:11] refinery-job[s] [21:03:11] refinery-app[s] [21:03:38] leaning towards job[s] [21:04:10] That doesn't sound too bad ... but job[s] is a bit vague ... mhmm [21:04:32] mhmmm [21:04:55] refinery-work-unit [21:04:56] Naw. [21:05:30] refinery-process? [21:05:34] Not good either. [21:05:43] job [21:05:46] I guess I cannot come up with a convincing alternative. [21:05:50] job :-) [21:05:51] i think its gonna be job, i think it is good [21:05:59] Yup. [21:06:16] yup yup yup. [21:08:48] thank you! [21:25:31] (CR) Mforns: [C: 1] "I looked at all the code, although not 100% deep." (2 comments) [analytics/dashiki] - https://gerrit.wikimedia.org/r/198169 (owner: Milimetric) [21:27:07] milimetric, there were 2 tasks in phab to be reviewed, they were both inside https://gerrit.wikimedia.org/r/#/c/198169/ ? [21:28:00] mforns: yes [21:28:01] milimetric, well it's a retoric question, I know they were [21:28:04] thanks for the review! [21:28:14] but just wanted to check if there were other thing to review [21:28:47] ok [21:29:44] (PS1) Ottomata: Add refinery-job maven module and directory hierarchy [analytics/refinery/source] - https://gerrit.wikimedia.org/r/199744 [21:30:59] mforns: ^ [21:31:05] milimetric: yt? [21:33:24] ottomata, yay, thanks! [21:34:15] gimme that +1 [21:37:10] (CR) Nuria: Add refinery-job maven module and directory hierarchy (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/199744 (owner: Ottomata) [21:37:39] milimetric: I have cleaned up limn1 a bit so now there is only 1 local commit that includes usage of templates (but not the templates themselves) , I will not be doing more changes there. [21:37:42] nuria: do i gotta do that for this patch? i kinda wanted to give marcell something to work in [21:38:03] ottomata: of course not, actually my comment is more for mforns [21:38:10] ah :) [21:38:16] gimme that sweeeeet +1 then! [21:38:24] nuria, ottomata, yes I got it :] [21:38:41] ottomata: k, let me re-read- everything again [21:38:45] k [21:40:09] Analytics-Cluster: Link refinery-source with cdh5.3.1 - https://phabricator.wikimedia.org/T93952#1150960 (Ottomata) NEW a:Ottomata [21:40:48] (CR) Nuria: [C: 1] Add refinery-job maven module and directory hierarchy [analytics/refinery/source] - https://gerrit.wikimedia.org/r/199744 (owner: Ottomata) [21:41:42] (CR) Ottomata: [C: 2 V: 2] Add refinery-job maven module and directory hierarchy [analytics/refinery/source] - https://gerrit.wikimedia.org/r/199744 (owner: Ottomata) [21:41:46] yeehaw, there you go mforns [21:42:02] ottomata, :] [21:46:54] Analytics-Kanban, Analytics-Visualization, Patch-For-Review: Provide temporary home for VE visualizations - https://phabricator.wikimedia.org/T89255#1151019 (Nuria) [21:51:18] Analytics-Kanban: Environment for Visual Editor visualizations in labs that can report usage metrics [13 pts] {lion} - https://phabricator.wikimedia.org/T93954#1151040 (Nuria) [21:51:35] Analytics-Kanban, Analytics-Visualization, Patch-For-Review: Provide temporary home for VE visualizations - https://phabricator.wikimedia.org/T89255#1151047 (Nuria) Open>Resolved [22:19:37] ottomata: still there? [22:19:56] yup [22:20:19] ottomata: on the topic of logster [22:20:47] ottomata: i was going to use the LineCountLogster [22:21:06] ottomata: https://github.com/wikimedia/operations-debs-logster/blob/master/logster/parsers/LineCountLogster.py [22:21:16] ja? [22:21:40] ottomata: but rather that counting lines on apache and returning pageviews [22:21:54] ottomata: it returns lines/duration. see: [22:21:57] https://www.irccloud.com/pastebin/OhhvpF1A [22:22:19] ah yes [22:22:21] ottomata: should i change that so it returns both [22:22:24] https://www.irccloud.com/pastebin/w4b97KGm [22:22:52] that would tell the number of lines it saw since it last run [22:22:53] ottomata: or rather create another parser? [22:22:57] right? [22:23:02] why do you want that instead of the rate? [22:23:22] right. cause i though for pageviews makes more sense right? [22:23:46] report hourly number of "lines" (or ... ahem.. pageviews) in the hour [22:23:54] ottomata: does that sound too strange? [22:24:02] btw, we are talking about reqeuests, not pageviews, but ja [22:24:13] we report pageviews as a rate, don't we? [22:24:14] ottomata: ya yaa, requests [22:24:23] number of pageviews in an hour [22:24:23] ottomata: not really [22:24:38] ah well [22:24:39] pagviews per hour [22:24:47] number of pageviews in a certain period [22:24:54] nuria, where are you going to report these? [22:24:55] yes yes [22:25:09] but ja, i see what you are saying [22:25:14] graphite labs, they are already there [22:25:21] nuria, i forget what returning multiple metric objects does, i guess it handles that? [22:25:35] hm, i think that will be fine with graphite, if i remember correctly [22:25:38] ottomata: it does ... [22:25:40] the reason i used a rate, is because ganglia is dumb [22:26:00] so, nuria, sure i think that should be fine [22:26:02] if that works [22:26:14] ottomata: ya i think so, let me check yesterday's test [22:26:17] *tests [22:28:19] ottomata: ahem, no stuff on graphite from yesterday.. why? cause nuria was sending it to stdout, nice, i shall correct that [22:28:23] ottomata: and let you know [22:28:52] ha, nice :) [22:41:36] Analytics-Tech-community-metrics, Possible-Tech-Projects, ECT-March-2015, Epic, and 2 others: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#1151352 (Qgil)