[00:05:32] Mobile-Apps, Analytics, Wikipedia-App-Android-App, Wikipedia-App-iOS-App, MediaWiki-extensions-UniversalLanguageSelector, Language-Engineering, Mobile-Web: there should be a comparison of clicks count on interlanguage on different platforms - https://phabricator.wikimedia.org/T78351#843122 (Amire80) NEW [00:09:27] interesting on the event.gif filter. [00:34:49] hey milimetric you there? [00:35:28] or maybe ori nuria__ can help [00:35:36] I need reminding how i can see data logged on beta labs.. where does it get stored? [00:36:05] deployment-eventlogging02 [00:36:19] there's a mysql database there [00:36:52] deployment-eventlogging02.wikimedia.org? [00:37:24] no, wmflabs [00:37:36] deployment-eventlogging02.eqiad.wmflabs [00:37:47] on the beta cluster [00:38:10] Analytics-Engineering: find a better way to identify events that fail validation as early as possible - https://phabricator.wikimedia.org/T78355#843193 (ggellerman) NEW [00:38:29] Analytics-Engineering: find a better way to identify events that fail validation as early as possible - https://phabricator.wikimedia.org/T78355#843193 (ggellerman) from Toby: A +1 from me -- this will make integration a lot easier. Let's see if we can address this in the Q3 project about dashboarding. [00:39:06] Analytics-Engineering: find a better way to identify events that fail validation as early as possible - https://phabricator.wikimedia.org/T78355#843201 (ggellerman) from Ori: wrote: is there a way to inspect invalid events in near real time without having access to vanadium? Ther... [00:39:27] ori: looks like i don't have access :/ [00:39:44] seriously? [00:39:45] i'll fix [00:39:57] <3 ori [00:40:08] * jdlrobso_ gives ori a phabricator token despite not understanding what they are and what they do [00:40:37] :) [00:40:40] ori: you did it :D [00:41:32] jdlrobso_: you were able to log in? [00:41:38] i added you to the project admins [00:41:48] ori: i've sshed in but now i need to work out how to log in to sql [00:41:57] jdlrobson@deployment-eventlogging02:~$ mysql [00:41:57] ERROR 1045 (28000): Access denied for user 'jdlrobson'@'localhost' (using password: NO) [00:42:40] Analytics-EventLogging: Engineer reads documentation on Wikitech to set up a dashboard from EL data [3 pts] - https://phabricator.wikimedia.org/T76364#843212 (kevinator) [00:42:47] Analytics-EventLogging: Engineer knows by when to expect a dashboard from EL data [1 pts] - https://phabricator.wikimedia.org/T76365#843213 (kevinator) [00:42:55] Analytics-Dashiki: Story: Dashiki uses Mediawiki for storage [13 pts] - https://phabricator.wikimedia.org/T70448#843215 (kevinator) [00:43:03] Analytics, Analytics-Wikimetrics: Ops has feedback from initial vetting of data in Data Warehouse [13 pts] - https://phabricator.wikimedia.org/T76480#843216 (kevinator) [00:43:45] jdlrobso_: um, let me see if i can find the password [00:46:06] jdlrobso_: mysql -h 127.0.0.1 --user=eventlogging --password=68QrOq220717816UycU1 [00:46:59] (it's labs, the password is not really a secret.) [00:59:10] (CR) Jdlrobson: [C: -1] "I'm a bit worried you didn't test the sql/graphs locally... if this is the case best to say so in the commit." (4 comments) [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/178381 (owner: Bmansurov) [01:52:08] Analytics-Refinery: Eng uses Mahout installed on Hadoop cluster - https://phabricator.wikimedia.org/T78016#843309 (ellery) Thanks Andrew! That was speedy. I just ran my first mahout job in hadoop. [02:07:48] ori: was jon able to find the info he wanted from beta eventlogging? [02:09:06] yes, i think so [02:26:21] (CR) Nuria: Add cohort membership page (3 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/178746 (owner: Mforns) [02:43:35] Analytics-Wikimetrics, Analytics-Engineering: Uploading cohort by copy-pasting breaks if names contain special characters [8 pts] - https://phabricator.wikimedia.org/T76105#843387 (Nuria) a:Nuria [02:48:08] Analytics, MediaWiki-extensions-MultimediaViewer, Multimedia: Compare file page edit counts pre- and post-MediaViewer - https://phabricator.wikimedia.org/T78364#843392 (Tgr) NEW [06:22:39] Analytics-Refinery: Geo-coding UDF - https://phabricator.wikimedia.org/T77683#843623 (Ironholds) I've spent the week integrating the new maxmind API. If we're going to build geolocation UDFs, we should absolutely be using the new API, not the old one: the V2 files have muuuuuch better IPV6 resolution! See htt... [14:57:37] morning milimetric :) [14:57:57] ottomata, milimetric: so I've spent part of my week testing out libmaxminddb (this is going to be useful for the geolocation UDFs, I imagine) [14:58:13] coool [14:58:20] it has a tremendously weird API, but it's got /awesome IPv6 lookup/. [14:58:21] yeah agree we should use the new api [14:58:29] Ironholds: I'm just preparing for standup but I'll read up right after [14:58:32] like, 99% country-location, as opposed to about 40% for the old one. [14:58:35] so yeah, we should use it. [14:58:37] milimetric, cool! [14:59:04] (one important factor; I was unable to detect any slowdown. Which is important.) [15:08:06] Analytics-Dashiki: Order concatenation of css files on build should be deterministic - https://phabricator.wikimedia.org/T78109#844312 (Nuria) Open>Resolved [15:23:23] Ironholds: that sounds great [15:23:35] how weird is this tremendously weird api? [15:23:48] it's written in C and has non-null-terminated char strings [15:23:55] which is tremendously WTF-WHY-WOULD-YOU-EVEN-DO-THIS inducing [15:24:08] the lookup path for retrieving a record varies depending on what record you want to retrieve [15:24:21] and it inconsistently has or does not have explicit methods for freeing object types [15:24:41] my advice: ignore the C API. Use the Java one. It cannot possibly be as mad. [15:24:55] If you really want to use the C API, use the linked list option, it's more of a pain to traverse but nowhere near as stupid. [15:25:06] sounds like someone with my level of C skills wrote it :) [15:25:22] actually, reading the code I have a strong suspicion that it's less you and more Toby [15:25:37] I checked my suspicions with Brian, who knows the guy, and he confirms [15:25:41] I've got good money Toby is better at C than me [15:25:48] oh no, I meant "he's a perl programmer" [15:25:52] anytime I try to pick up C I'm just like aaarghh [15:26:13] it contains idioms I've only ever seen when I've arghed at some code problem I'm dealing with and Toby's gone "see, in Perl we'd just do [something horrifying involving default values and constants that exist in all caps]" [15:26:14] gotcha, java api should be better anyway [15:26:26] lol [15:26:30] C lacks some features, it's true. [15:26:38] R is just C-like enough that C disappoints me on occasion [15:26:52] like, c'mon! You can't pass functions except as pointers? And you can't parse variadic function arguments at ALL? [15:27:01] but it's so useful! [15:27:15] ottomata: I think there should be an larm in icinga about EL that we should claim and close but i cannot find it [15:27:35] ? [15:28:27] i got an alert (you probably have it too) [15:28:31] https://www.irccloud.com/pastebin/5iV4yIk7 [15:28:45] ottomata: that looks to be resolved now [15:29:43] ottomata: it lasted a few minutes after midnite (UTC) but now it is resolved [15:29:48] ja looks ok [15:29:48] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=tungsten&service=Throughput+of+event+logging+NavigationTiming+events [15:30:27] ottomata: ah, so it ok-ed itself, i thought i had to do it [15:30:32] mforns: do i need to run populate_tags after this deploy? [15:30:59] milimetric, no I don't think so, the tags will be still there [15:31:42] nuria__ / mforns: and did anything change in puppet? [15:32:07] not from my side... [15:32:09] milimetric: no, it did not [15:33:49] ok, i did a gentle deploy just restarting the web [15:35:04] milimetric: i think you shoud re-start the queue /scheduler as to make sure everything is running of the same code [15:35:18] nuria__: nothing changed except html and js [15:35:44] milimetric: not even in the prior sprint? cause we have not deployed for couple weeks [15:35:54] when i did git pull, that's what it said [15:36:14] hm... i'll restart just in case we look at when the processes last restarted though - that's a good enough reason [15:37:22] and nuria__: for dashiki I should do another "build" release right? [15:38:05] mmmm, let me see, [15:38:26] i guess i can tell from the suffix on the files, it's the git commit [15:39:59] no, i guess not :) [15:40:52] (PS1) Milimetric: Build to deploy [analytics/dashiki] - https://gerrit.wikimedia.org/r/179475 [15:42:04] (CR) Milimetric: [C: 2 V: 2] Build to deploy [analytics/dashiki] - https://gerrit.wikimedia.org/r/179475 (owner: Milimetric) [15:44:22] milimetric: yes, you need to do 1 more build because my last fix of grabbing latest from mediawiki storage [15:44:35] hm... i did and deployed but the https issue is still there - looking [15:44:42] that's why [15:45:16] did you do bower install in your checkout before [15:45:37] milimetric: you need to grab latest from master/bower install/build [15:45:58] milimetric: makes sense? so you are building with teh most up-to-date sets of dependencies [15:46:28] k, my bad [15:46:37] good i did this though [15:48:54] so nuria__ are we keeping the pageview files in staging for now? [15:49:01] the pageviewApi points at staging [15:49:34] milimetric: i thought we were not showing pageviews for now [15:50:01] kevinator: are we deploying pageviews with dashiki? [15:50:09] once we showcase stuff, I thought we always deploy [15:50:31] unless like people hate it or something :) [15:50:51] They don [15:50:56] :-) [15:51:13] don is ambiguous it could be "do" or "don't" :) [15:51:32] Research does have concerns that we are using the webstats collector definition of pageviews and not explicitely saying so. [15:51:50] what is the link to the metric definition? [15:52:05] https://meta.wikimedia.org/wiki/Research:Pageviews [15:52:14] that's bad :) should update to a page on WSC [15:52:21] *don’t* [15:52:39] k - so... this is configurable from the wiki anyway [15:52:50] cool [15:52:51] feel free to do as you wish, i'm going to be bold and deploy it so it's available [15:52:58] ok, deploy [15:53:16] I’ll look at the links and get that fixed. [15:53:40] Is there a page for the WSC definition [15:53:42] ? [15:53:44] (PS1) Milimetric: Build to deploy [analytics/dashiki] - https://gerrit.wikimedia.org/r/179477 [15:53:58] not sure, don't know of one good one [15:54:12] (CR) Milimetric: [C: 2 V: 2] Build to deploy [analytics/dashiki] - https://gerrit.wikimedia.org/r/179477 (owner: Milimetric) [15:54:46] kevinator: https://metrics.wmflabs.org/static/public/dash/#projects=ruwiki,itwiki,dewiki,frwiki,enwiki,eswiki,jawiki/metrics=DailyPageviews [15:55:14] kevinator: http://meta.wikimedia.org/wiki/Dashiki:CategorizedMetrics [15:55:17] wow, that was a fast deploy! [15:55:34] that's why we love dashiki : [15:55:49] that categorized metrics link is where you can change where the pageview definition points to and all that [15:55:57] you can also, of course, remove that metric completely [15:56:18] but for now, the pageview files will be grabbed from our staging server as the data part of it hasn't been put in prod yet [15:56:41] ok, cool [15:57:26] I need to head to office now… I’ll be off-line for an hour [16:07:20] milimetric: i think teh only puppet change we had as of late is the one that checks out pageview depot [16:07:32] that is the depot with pageview data [16:07:47] gotcha [16:07:56] so if we re-run puppet and check it out pageview api can point to prod machine [16:08:01] i didn't deploy that - i figure we let Christian fix up all the stuff with pageviews and then we push it to prod [16:08:22] he's gotta do the 0 -> undefined thing and the weekly rollup [16:11:12] milimetric: maybe is something our team should do next sprint [16:18:42] dear god, this is horrifying [16:18:58] functions can only be passed as pointers? ARGH. [16:25:18] nuria__, one question: when testing the cohort ownership from the controller, I had an app context with the current user on it [16:25:42] now, when testing from the service, there's no current_user available, and I get a: RuntimeError: working outside of application context [16:26:54] is there something in the fixtures I have to use to create the context? or there's another approach? [16:27:33] mforns: you need to pass the user explicitily to the method [16:27:59] ok, so I'll have to pass it to delete_cohort_wikiuser right? [16:28:05] mforns: yes [16:28:11] ok thanks! [16:50:31] cahnging locations, lunching bbl [16:55:26] (PS3) Mforns: Add cohort membership page [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/178746 [16:56:27] (CR) Mforns: Add cohort membership page (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/178746 (owner: Mforns) [17:11:44] Analytics-Refinery: productionizing xmldump -> avro jobs - https://phabricator.wikimedia.org/T78404#844421 (ggellerman) NEW [17:14:49] Analytics-Refinery: SPIKE: experimenting with importing Revision history from XML dumps into an easier to use format, Avro - https://phabricator.wikimedia.org/T78405#844429 (ggellerman) NEW [17:17:18] Analytics-Wikimetrics, Analytics-Engineering: Story: WikimetricsUser deletes user from cohort [21 pts] - https://phabricator.wikimedia.org/T75350#844438 (Nuria) {F21014} [17:19:48] nuria__, having a look at the screenshot [17:20:01] mforns: you are a ninja! [17:20:07] mforns: i just uploaded 2 secs ago [17:20:26] xD I received the wikibugs notification [17:24:01] nuria__, could you paste the usernames you used, please? [17:24:14] mforns: i used random text from ar.wikipedia [17:24:17] (CR) Yuvipanda: [C: 2] CORS all /meta/ queries [analytics/quarry/web] - https://gerrit.wikimedia.org/r/178702 (owner: Merlijn van Deen) [17:24:25] ok [17:24:30] (Merged) jenkins-bot: CORS all /meta/ queries [analytics/quarry/web] - https://gerrit.wikimedia.org/r/178702 (owner: Merlijn van Deen) [17:25:04] (CR) Nuria: Add cohort membership page (2 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/178746 (owner: Mforns) [17:26:57] nuria__, I think that happened because the validation did not take place [17:27:12] do you have the queue service running? [17:27:26] well, anyway that's a problem [17:27:57] mforns: no, you know what i bet I had teh queue still stop cause i had just run tests [17:28:00] it should read something like: 'validation pending' [17:28:23] mforns: ya, that too [17:29:05] mforns: i think it highlites teh need of us having some "placeholder" text [17:29:08] but that's it [17:29:41] nuria__, what do you mean with placeholder text? [17:30:04] mforns: what you just said, i was just agreeing to " it should read something like: 'validation pending'" [17:32:57] nuria__, ok fine [17:37:55] (CR) Mforns: Add cohort membership page (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/178746 (owner: Mforns) [18:12:17] ottomata: Your client appears to be having a seizure. [18:12:59] its the network [18:13:13] i'm at nyu, and whatever fancy schmangly thing they do here is not happy with my computer [18:13:25] maybe because I am using my friend's student wifi account? iunno! [18:13:35] i just switched to phone's wifi [18:19:37] Analytics-Engineering: find a better way to identify events that fail validation as early as possible - https://phabricator.wikimedia.org/T78355#844534 (ggellerman) from Dario: thanks for the quick turnaround. On Dec 11, 2014, at 4:28 PM, Ori Livneh wrote: There's this graph: https://gra... [18:19:55] Analytics-Engineering: find a better way to identify events that fail validation as early as possible - https://phabricator.wikimedia.org/T78355#844535 (ggellerman) from Nuria: Besides the ability of testing in beta labs and the monitoring that ori highlited the incoming raw stream of events is available in... [18:28:50] Analytics: Move stat1002 and stat1003 into Analytics VLAN - https://phabricator.wikimedia.org/T76346#844544 (Ottomata) Correction , stat1001 will be moved! :) We are ready for this. [19:02:28] (PS1) Nuria: Fixes for cohort uploading via copy-pasting on box [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/179517 [19:04:29] kevinator, milimetric , ggellerman: meeting? [19:04:47] can you not hear us nuria__ ? [19:04:49] I see you [19:09:57] Analytics-Wikimetrics, Analytics-Engineering: Wikimetrics auditor has read-only login to Wikimetrics DB [3 pts] - https://phabricator.wikimedia.org/T76109#844651 (Nuria) a:Nuria [19:22:44] (PS4) Mforns: Add cohort membership page [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/178746 [19:30:22] (CR) Mforns: Add cohort membership page (2 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/178746 (owner: Mforns) [19:52:33] (CR) Mforns: "Looks good to me!" (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/179517 (owner: Nuria) [20:18:21] can anyone point me to the etherpad for the combined staff meeting? [20:25:29] (CR) Nuria: "Looks great, let's just deploy to staging & test it and in the absence of any issues let's merge." [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/178746 (owner: Mforns) [20:30:47] (PS2) Nuria: Fixes for cohort uploading via copy-pasting on box [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/179517 [21:08:54] (PS5) Mforns: Add cohort membership page [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/178746 [21:13:35] (CR) Mforns: Add cohort membership page (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/178746 (owner: Mforns) [21:16:41] (CR) Nuria: [C: 2] Add cohort membership page [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/178746 (owner: Mforns) [21:21:24] (PS3) Mforns: Fixes for cohort uploading via copy-pasting on box [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/179517 (owner: Nuria) [21:33:56] (CR) Mforns: [C: 2] "LGTM" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/179517 (owner: Nuria) [21:37:47] Phabricator, Analytics-Tech-community-metrics, Engineering-Community: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#844983 (kevinator) When I saw the wiki with [[ https://www.mediawiki.org/wiki/Talk:Community_metrics#Bugzilla_stats | number of active bugzilla use... [21:42:59] Analytics-Wikimetrics, Analytics-Engineering: Uploading cohort by copy-pasting breaks if names contain special characters [8 pts] - https://phabricator.wikimedia.org/T76105#845004 (Nuria) Open>Resolved [21:43:53] Analytics-Wikimetrics, Analytics-Engineering: Story: WikimetricsUser deletes user from cohort [21 pts] - https://phabricator.wikimedia.org/T75350#845005 (mforns) Open>Resolved [21:43:55] Analytics-Wikimetrics, Analytics-Engineering: Epic: WikimetricsUser deletes user from cohort - https://phabricator.wikimedia.org/T76421#845006 (mforns) [22:13:05] Hi all, can someone please give me access to deployment-eventlogging02.eqiad.wmflabs ? I'm trying to test my patch with tools labs data. thank you [22:25:17] Analytics-EventLogging: find a better way to identify events that fail validation as early as possible - https://phabricator.wikimedia.org/T78355#845066 (kevinator) p:Triage>Normal [22:25:48] hi bmansurov, checking [22:25:55] not sure i have access to that [22:26:01] milimetric: cool thanks [22:27:18] bmansurov: you don't happen to know what "project" it's under in labs, do you [22:27:24] (I never found a way to figure that out quickly) [22:27:38] milimetric: I'm not sure, let me ask jon [22:27:59] not sure he'd know [22:29:07] yes, he probably doesn't know either, he hasn't replied yet [22:30:16] bmansurov: found it: https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep [22:30:22] but - bad news - i have no rights there [22:30:36] aaand... i spoke too soon, I'm an admin [22:30:37] lol [22:30:50] ;) [22:32:08] bmansurov: ok, i added you as a member - ping me directly if that's not enough and you need admin [22:32:23] milimetric: thanks, let me try connecting [22:33:15] milimetric: I'm in and I can access mysql, that's all I need for now. thanks again [22:35:01] np, good [23:13:37] (PS3) Bmansurov: Update click tracking graphs [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/178381 [23:14:21] Analytics-Engineering: set up mtg w/Visual Editor (Ron) to talk about what we did months back - https://phabricator.wikimedia.org/T76798#845134 (ggellerman) Open>declined a:ggellerman No longer need to do this per 12/12/14 EL Evangelism mtg [23:14:22] Analytics-Engineering: EPIC EL documentation & evangelism - https://phabricator.wikimedia.org/T76795#845137 (ggellerman) [23:14:40] Analytics-EventLogging: set up mtg w/Visual Editor (Ron) to talk about what we did months back - https://phabricator.wikimedia.org/T76798#845138 (ggellerman) [23:15:06] Analytics-EventLogging: think through capacity management & Information Architecture - https://phabricator.wikimedia.org/T76803#845141 (ggellerman) [23:16:09] Analytics-EventLogging, Analytics-Engineering: think through capacity management & Information Architecture - https://phabricator.wikimedia.org/T76803#845142 (kevinator) [23:18:07] Analytics-EventLogging, Analytics-Engineering: Product Instrumentation and Visualization - https://phabricator.wikimedia.org/T76795#845143 (kevinator) [23:46:36] Analytics-Engineering: EL office hours [23:47:56] Analytics-Engineering: EL office hours - https://phabricator.wikimedia.org/T76796#819860 (ggellerman) [23:48:37] Analytics-Engineering: EL office hours - https://phabricator.wikimedia.org/T76796#845220 (ggellerman) [23:48:45] Analytics-EventLogging, Analytics-Engineering: Analytics Eng has capacity & monitoring for EventLogging - https://phabricator.wikimedia.org/T76803#845221 (kevinator) [23:56:41] Analytics-EventLogging, Analytics-Engineering: EL office hours - https://phabricator.wikimedia.org/T76796#845228 (kevinator) [23:58:50] Analytics-EventLogging, Analytics-Engineering: WMF Engineers have harmonized schemas - https://phabricator.wikimedia.org/T78442#845229 (kevinator) NEW