[00:01:07] Analytics-Engineering: EPIC: data warehouse - https://phabricator.wikimedia.org/T76382#1013285 (mforns) [00:04:40] nuria: meeting? [00:08:27] springle, https://wikitech.wikimedia.org/wiki/Analytics/DataWarehouse/DataVerifications/2015-02-03 [00:08:32] mforns: tnx [00:19:28] nuria_: http://en.wikipedia.org/wiki/Universally_unique_identifier#Variants_and_versions [00:47:48] nite everyone [01:19:05] (CR) Bmansurov: [C: 2] Update scripts in light of recent changes [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/181428 (owner: Jdlrobson) [01:19:12] (Merged) jenkins-bot: Update scripts in light of recent changes [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/181428 (owner: Jdlrobson) [02:10:48] Analytics-Engineering: Browser report for mobile folks from hadoop data in refined tables (Weekly or Daily) - https://phabricator.wikimedia.org/T88504#1013502 (Nuria) NEW [02:11:15] Analytics-Engineering: Browser report for mobile folks from hadoop data in refined tables (Weekly or Daily) - https://phabricator.wikimedia.org/T88504#1013509 (Nuria) [02:21:29] Analytics-Wikistats: Provide total active editors for December 2014 - https://phabricator.wikimedia.org/T88403#1013514 (Tbayer) Thanks, Toby! We also need the December number for these two: http://reportcard.wmflabs.org/graphs/edits http://reportcard.wmflabs.org/graphs/new_editors I'll assume for now that t... [04:20:18] Analytics-Wikistats: Provide total active editors for December 2014 - https://phabricator.wikimedia.org/T88403#1013614 (Eloquence) [07:38:24] MediaWiki-Developer-Summit-2015, Analytics-Tech-community-metrics: Achievements, lessons learned, and data related with the MediaWiki Developer Summit 2015 - https://phabricator.wikimedia.org/T87514#1013839 (Qgil) a:Rfarrand [07:38:47] jgage: You around for some namenode help? [07:47:38] MediaWiki-Developer-Summit-2015, Analytics-Tech-community-metrics: Achievements, lessons learned, and data related with the MediaWiki Developer Summit 2015 - https://phabricator.wikimedia.org/T87514#1013846 (Qgil) [07:49:56] !log Manual failover of Hadoop namenode from analytics1002 to analytics1001, as analytics1002 had Heap space errors [07:50:16] Wikimedia-Git-or-Gerrit, Analytics-Tech-community-metrics: Active code review users on a monthly basis - https://phabricator.wikimedia.org/T86152#1013849 (Qgil) a:Dicortazar [07:54:47] Analytics-Tech-community-metrics: Remove the filter for key Wikimedia software projects in korma.wmflabs.org - https://phabricator.wikimedia.org/T86154#1013858 (Qgil) a:Dicortazar [08:10:19] halfak: You around by any chance? [08:11:26] qchris: A quick question on the spring-security-web library that you recommend for IpUtil. Good time? [08:11:55] ananthrk: HDFS is currently having issues, so the time is not ideal. [08:11:57] 30mins? [08:12:11] np [08:12:13] k [08:16:32] ananthrk: hdfs is back in shape. [08:16:40] So what's the question about that library? [08:17:37] when I add 3.2.5.RELEASE of springframework-security, mvn complains of not being to find the corresponding pom in repository [08:18:59] Ah, yes. [08:19:08] [ERROR] Failed to execute goal on project refinery-core: Could not resolve dependencies for project org.wikimedia.analytics.refinery.core:refinery-core:jar:0.0.6-SNAPSHOT: Failure to find org.springframework.security:spring-security-web:jar:3.2.5.RELEASE in http://archiva.wikimedia.org/repository/mirrored/ was cached in the local repository, resolution will [08:19:08] not be reattempted until the update interval of wmf-mirrored has elapsed or updates are forced [08:19:18] That is sort of expected, as the jars are not in archiva. [08:19:39] $SOMEONE_WITH_ARCHIVA_CREDENTIALS can upload them. [08:19:57] However, I guess only ottomata qualifies for $SOMEONE_WITH_ARCHIVA_CREDENTIALS [08:19:59] oops..I thought that was going to be you :) [08:20:15] But there is an easy workaround for now. [08:20:25] Just enable central in the parent pom.xml. [08:20:44] That should allow you to build. [08:21:07] hm..so when I check-in pom.xml should I leave it enabled or disable it back? [08:21:11] Once ottomata is back, he can put the jars into archiva and disable central again. [08:21:30] While it's wip, just have it disabled. [08:21:36] That will help people test it. [08:22:01] However, I guess that also means that one can only build the jar locally (not on stat1002). [08:22:14] okay [08:22:41] But if you find a better library, feel free to use it. [08:22:56] I just picked the spring one as it can do IPv4 and IPv6, plain hostnames [08:23:04] and is maintained by a well established entity. [08:23:35] even during my original search I landed up with 3 options - commons net (no support for Ipv6), spring webframework (not light-weight) and CDRUtils [08:23:49] but as you say CDRUtils looked raw and with no tests [08:24:05] and it is producing false results ;-) [08:24:36] during one of our discussions, we (me & Nuria) decided that we can probably add tests to CDRUtils and bring it in [08:25:16] Ouch. So you are again getting mixed signals from us. Sorry. [08:25:21] but if we are okay bringing in spring, then ofcourse, it is a better choice [08:26:41] btw, regarding CDRUtils, you say "The implementation is broken, and returns false results as it does not separate IPv4 and IPv6." [08:26:41] Let's bring in spring. [08:27:01] Yes. I think CIDRUtils is broken. [08:27:11] is there any value that you tested that returned false results for you? [08:27:28] Yup. [08:27:38] I ask because it would atleast help improve the tests for CDRUtils. Even if we don't end up using it, we can atleast send it upstream [08:28:04] The issue is that both IPv4 and IPv6 are converted to plain numbers, and checking is done on those numbers without checking address families. [08:28:15] Let me see if I locally checked in the tests I did. [08:28:58] There it is. [08:29:08] Use the range: "::a00:0/126" [08:29:20] And check if "10.0.0.1" is in that range. [08:29:32] CIDRUtils will tell you that it is in the range. [08:30:00] But "::a00:0/126" is neither a reserved range, nor is it a IPv4-in-IPv6 range. [08:30:15] Hence "10.0.0.1" should not be in that range. [08:30:45] ha..thanks..TIL [08:32:41] (It is an outlier and constructed example. Agreed. But still, these kind of things are what put maintenance burden on us. So I'd prefer an implementation where others do the maintenance ... like the spring one) [08:33:52] would a simple address family check circumvent the problem? [08:34:21] In this case: Yes. [08:35:07] But I do not want to do maintenance of such code :-) [08:35:13] I want to reuse good existing code. [08:35:16] okay...i see this utility being recommended in several places and hence would help to have an issue logged in the git repo [08:35:39] If you want to log it, please do. [08:35:48] But we cannot fix all code on github. [08:35:50] (not for our needs though) [08:36:05] So ... I would not care. [08:36:21] ok [08:38:07] BTW, I am learning a lot from your detailed CR comments - so thanks for doing them [08:38:51] Meh. I am a nit-picker :-D Feel free to push back on any of my comments ;-) [08:40:20] will do, but not sure whether I am going to anytime sooner :) [08:42:55] (enabling central repo worked..so onto other changes) [08:44:14] :-) [09:01:16] Analytics-Tech-community-metrics: "Contributors new and gone" in korma is stalled - https://phabricator.wikimedia.org/T88278#1013957 (Qgil) @Dicortazar, since this is a piece of functionality that should work, can we add it to the February sprint? [13:09:00] Analytics-Engineering: Hadoop errors - https://phabricator.wikimedia.org/T88537#1014252 (Ironholds) NEW [13:09:15] Analytics-Engineering: Hadoop errors - https://phabricator.wikimedia.org/T88537#1014259 (Ironholds) [14:21:46] (PS1) Jsahleen: Add additional dashboard for manual beta enablements for CX [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/188556 (https://phabricator.wikimedia.org/T88516) [14:37:41] It is 9:37am and I am writing Java. God hates all her children. [14:43:02] (CR) Ananthrk: "Please enable the central repository in pom.xml in order to get spring-security-web package as it is currently not available in archiva." (28 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187651 (owner: Ananthrk) [14:43:20] (PS3) Ananthrk: [WiP] Add new UDF to determine client IP given source IP and XFF header [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187651 [15:15:30] Analytics-Kanban, Analytics-Cluster: Rough estimate of how many users might not have javascript capable/enable browsers - https://phabricator.wikimedia.org/T88560#1014744 (Nuria) NEW a:Nuria [15:28:24] mforns: we're going to talk about the sunburst code if you're interested (batcave) [15:37:11] Analytics-Wikistats: Provide total active editors for December 2014 - https://phabricator.wikimedia.org/T88403#1014918 (Joe) [15:51:55] (CR) Bmansurov: "It would be nice to get this merged. Is the original author going to work on it? Should someone else pick it up?" [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/180828 (https://phabricator.wikimedia.org/T84892) (owner: Rtnpro) [16:46:34] (PS1) Mforns: Update for February meeting [analytics/reportcard/data] - https://gerrit.wikimedia.org/r/188576 [16:47:34] Analytics-Engineering, Analytics-Wikimetrics: EPIC: Productionizing Wikimetrics - https://phabricator.wikimedia.org/T76726#1015136 (kevinator) [16:49:06] (CR) Mforns: [C: 2 V: 2] "SELFMERGE :]" [analytics/reportcard/data] - https://gerrit.wikimedia.org/r/188576 (owner: Mforns) [16:49:46] hi milimetric. I just sent you an invite for a half hour meeting today if you have time. feel free to move it around if it doesn't work for you. [16:50:09] ok, thanks leila [16:50:26] looks good, accepted [16:50:48] I hate to admit it, but I think the costly-and-brittle-regex for host parsing may actually be the least terrible option [16:56:34] thanks milimetric. [16:59:13] nuria, ping :) [16:59:20] Ironholds: yessir [16:59:38] so, I'm writing this hashMap outputting UDF, and looking to write JUnit tests around it [16:59:57] there's the uaParserUDF tests as an example, but otto is trying to move to a "store test examples in a CSV where they can be read consistently" approach [17:00:11] can you think of a decent way of storing a hashMap in a CSV? [17:00:24] the only thing I can think of is storing the values as different fields and then combining them within the test framework [17:02:31] Ironholds: mmmm... [17:02:48] milimetric, you meant that I should CC Erik Moller right? [17:02:50] Ironholds: not all test examples make sense to be stored in a cvs [17:02:53] yes [17:03:01] right [17:03:26] Ironholds: even if you did you would not store the "object" but rather the data that the object holds [17:03:45] Ironholds: what data are you thinking it will go into your tests [17:05:12] nuria, example uri_hosts, example outputs [17:05:26] I guess I could just store them as the ua-parser examples are; that would make sense. [17:06:04] Ironholds: let me look at that for a sec, i have no memory.... [17:06:27] Analytics-Engineering, operations: Puppet Production role class for wikimetrics scheduler/queue - https://phabricator.wikimedia.org/T76791#1015174 (kevinator) [17:06:29] Analytics-Engineering: changing puppetization for scheduler/queue - https://phabricator.wikimedia.org/T76790#1015175 (kevinator) [17:08:36] Ironholds: ok, i see, regardless of format you could read data from cvs file as long as it is text with @FileParameters("src/test/resources/test.csv") [17:09:00] Ironholds: for your test it would work to do: [17:10:28] Ironholds: , {key: 'value'} on the csv I think, that way, as otto requested the test data is not mixed with the test. Since the output of your UDF is a string representation anyways it should work fine [17:10:31] Ironholds: makes sense? [17:11:56] aha, thank you :) [17:12:09] it does! I'll test and work it out. [17:24:26] urgh. how the heck do I return a hashMap?! ;p [17:25:34] Ironholds: letme see your udf is returning a string representation right? [17:25:37] Ironholds: like... [17:26:20] '{"metasyntactic_variable":"foo","meta-joke":"this_is_a_hashmap_string_representation"}'? [17:26:51] so your evaluate will return an "object" correct? [17:26:54] Ironholds: public Object evaluate(DeferredObject[] arguments) throws HiveException { [17:26:54] [17:27:33] and the tests are going to compare teh string representation of that object ('{"metasyntactic_variable":"foo","meta-joke":"this_is_a_hashmap_string_representation"}') [17:27:43] to what it says on the test file [17:27:53] Ironholds: is this clearer? take a look at [17:28:17] https://www.irccloud.com/pastebin/4nf3tiW3 [17:28:25] on https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-hive/src/test/java/org/wikimedia/analytics/refinery/hive/TestUAParserUDFUserAgentMostPopular.java [17:31:44] nuria thanks [17:31:48] * Ironholds will read. Brain. [17:34:08] Analytics-Kanban, Analytics-Cluster: Rough estimate of how many users might not have javascript capable/enable browsers - https://phabricator.wikimedia.org/T88560#1015269 (ggellerman) [17:41:18] Analytics-Kanban, Analytics-Wikimetrics: EPIC: Productionizing Wikimetrics {dove} - https://phabricator.wikimedia.org/T76726#1015283 (kevinator) p:Normal>Low [17:42:50] Analytics-Kanban, Analytics-Wikimetrics: upgrade wikimetrics to trusty [8 pts] {dove} - https://phabricator.wikimedia.org/T76769#1015291 (kevinator) p:Normal>Low [17:43:45] nuria, okay, read! So, would this work for building said UDF, too? [17:43:59] i.e., output a JSONObject consisting of the decoded hashMap? [17:44:19] your UDF outputs an "object" [17:44:25] plain, it's a hashmap [17:44:50] your test will compare the output of your udf with the object is going to load from the csv [17:44:55] Analytics-Wikimetrics: Debianize wikimetrics dependencies not in trusty {ops} {dove} - https://phabricator.wikimedia.org/T76789#1015299 (kevinator) p:Normal>Low [17:45:00] yeah, I'm ignoring the tests for the time being [17:45:10] so it will transform this string: '{"metasyntactic_variable":"foo","meta-joke":"this_is_a_hashmap_string_representation"}' [17:45:21] the question is: can I write the UDF that wraps a java function outputting a hashmap, to believe it's outputting an object? [17:45:28] (I made a stab at it, it responded by pooping itself at runtime) [17:45:45] this would probably be easier if I actually just threw it into gerrit so the code lived somewhere :D [17:45:46] * Ironholds does that [17:46:39] your function can return object directly which is a parent class of hashmap , that is what the "evaluate" method of the udf does: '{"metasyntactic_variable":"foo","meta-joke":"this_is_a_hashmap_string_representation"}' [17:46:56] aha [17:46:58] Ironholds: sorry, "public Object evaluate(DeferredObject[] arguments) throws HiveException {" [17:47:05] Ironholds: we can talk in the batcave if you want [17:47:17] Ironholds: the udf alreday has an object signature [17:47:20] *already [17:48:12] aha! [17:48:14] Analytics-Kanban, Analytics-Wikimetrics: Debianize wikimetrics dependencies not in trusty {ops} {dove} - https://phabricator.wikimedia.org/T76789#1015320 (kevinator) [17:48:17] can we talk later? I'm still sort of revving up [17:48:31] I had this big burst of energy and wrote a ton of stuff and then it went away [17:48:51] Ironholds: sure np [17:50:24] Analytics-Kanban, Analytics-Wikimetrics: change puppetization for scheduler/queue {ops} {dove} - https://phabricator.wikimedia.org/T76790#1015329 (kevinator) p:Normal>Low [17:50:48] aw i missed an exciting namenode outage last night [17:52:00] Analytics-Kanban, Analytics-Wikimetrics: change wikimetrics in labs to use debian packages [13 pts] {dove} - https://phabricator.wikimedia.org/T76770#1015341 (kevinator) p:Normal>Low [17:52:44] jgage: did qchris bring it back up? [17:52:50] what happened? [17:53:35] yeah, qchris revived it. the hadoop namenode ran out of heap space, i'm looking for more info now. [17:55:56] Analytics-Kanban, Analytics-Wikimetrics: Change wikimetrics admin script - no symlinking [8 pts] {dove} - https://phabricator.wikimedia.org/T76780#1015364 (kevinator) p:Normal>Low [17:56:45] did the second namenode step in? [17:57:33] the failover process is manual, so yes it did its job but not until qchris entered the command to fail over :P [17:58:03] (CR) Rtnpro: "Hey, please feel free to pick it up. I am badly stuck in some personal affairs due to which I am not able to devote much time. Things shou" [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/180828 (https://phabricator.wikimedia.org/T84892) (owner: Rtnpro) [17:59:40] Analytics-Kanban, Analytics-Wikimetrics: create new group for devs to sudo run wikimetrics admin script {ops} {dove} - https://phabricator.wikimedia.org/T76792#1015373 (kevinator) p:Normal>Low [18:02:52] Analytics-Engineering, Security-Reviews, Analytics-Wikimetrics: security review of Wikimetrics {dove} - https://phabricator.wikimedia.org/T76782#1015380 (kevinator) p:Normal>Low [18:03:27] Analytics-Kanban, Security-Reviews, Analytics-Wikimetrics: security review of Wikimetrics {dove} - https://phabricator.wikimedia.org/T76782#819706 (kevinator) [18:05:46] Analytics-Kanban, Analytics-Cluster: estimate roughly of how many users might not have javascript capable/enable browsers - https://phabricator.wikimedia.org/T88560#1015393 (kevinator) p:Triage>Normal [18:06:00] Analytics-Kanban, Analytics-Cluster: Estimate roughly of how many users might not have javascript capable/enable browsers - https://phabricator.wikimedia.org/T88560#1014744 (kevinator) [18:19:13] (PS1) OliverKeyes: (WIP) project class/variant extraction and zero-rated request UDFs [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188588 [18:20:32] (CR) OliverKeyes: "As an aside; the use of the utilities functions in the pageviews/webrequest classes makes me feel like we probably want a generic "utiliti" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188588 (owner: OliverKeyes) [18:21:46] (PS2) OliverKeyes: (WIP) project class/variant extraction and zero-rated request UDFs [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188588 [18:33:05] Analytics-Kanban: Troubleshoot data warehouse importing process - https://phabricator.wikimedia.org/T88583#1015449 (mforns) NEW a:mforns [18:33:53] Analytics-Kanban: Troubleshoot data warehouse importing process - https://phabricator.wikimedia.org/T88583#1015458 (mforns) p:High>Normal [18:54:23] Analytics-Kanban: Add time range selection to Limn dashboards (or new Dashiki dashboards) - https://phabricator.wikimedia.org/T87603#1015510 (Milimetric) [18:55:58] Analytics-Kanban: Calculate per-wiki Edit Completion Rate for Visual Editor edits - https://phabricator.wikimedia.org/T87865#1015513 (Milimetric) [18:56:08] Analytics-Kanban: Add ops-reportcard dashboard with analysis that shows the http to https slowdown on russian wikipedia - https://phabricator.wikimedia.org/T87604#1015515 (Milimetric) [19:02:54] (PS1) Milimetric: Add funnel-gathering sql and prototype html [analytics/limn-edit-data] - https://gerrit.wikimedia.org/r/188601 [19:08:08] mforns_brb / nuria: I ended up just putting all the code here: https://gerrit.wikimedia.org/r/#/c/188601/ [19:08:45] milimetric: k, want to merge it? [19:30:51] nuria: if you could +1 it, that'd be great [19:30:54] i'll merge it once I talk to James [19:31:18] (CR) Nuria: [C: 1] Add funnel-gathering sql and prototype html [analytics/limn-edit-data] - https://gerrit.wikimedia.org/r/188601 (owner: Milimetric) [20:22:17] nuria / mforns / kevinator: question from Krenair [20:22:27] he's working on https://phabricator.wikimedia.org/T88027 [20:22:27] aham [20:22:34] ok [20:22:43] which is, implement Schema:Edit for the wiki editor [20:22:58] and he's wondering if we see a problem with the editingSessionId part of the schema [20:23:18] because it would imply a unique token to follow a user across page loads while they edit / preview / etc. [20:24:07] Analytics-Kanban, Analytics-EventLogging: Drop clientValidated and isTruncated fields from event capsule - https://phabricator.wikimedia.org/T88595#1015821 (Nuria) NEW [20:24:27] hmm, more uniques :-( [20:24:31] yes [20:24:38] milimetric, but this id is local to the session right? when the session ends, the id is respawned? [20:24:53] Is there any way to make the token not persist beyond a session? [20:25:00] but, in my opinion, nothing different from the current VE instrumentation, just done "across pages" as opposed to "across ajax requests" [20:25:04] which, technically speaking, is identical [20:25:17] yes, kevinator, that's what i want to suggest / enforce [20:25:22] milimetric: how is getting a unique token? [20:25:35] no plans yet, just talking theory [20:26:34] if they kill previous editingSessionIds at the beginning of each new session, in theory, no id will be back-traceable, right? [20:26:44] milimetric: client side tokens have issues, please send him this: https://gerrit.wikimedia.org/r/#/c/187876/ [20:27:15] milimetric: if they do not "store" them anywhere, there is no need to kill them [20:27:43] milimetric: that being said, with actual issues you can still use the tokens [20:28:03] they would have to store them across pages [20:28:13] so cookies probably [20:28:21] yes that's what I thought [20:28:22] since localStorage would be too limiting to get a good baseline [20:28:52] milimetric: and wouldn't that affect the cache hit ratios or it doesn't matter cause they are on the editing pipeline? [20:29:28] milimetric: cause in their case seems like if there is server side execution they might as well expire/set tokens from php [20:29:31] right? [20:29:43] milimetric: why do they need need to use js? [20:30:01] milimetric: they can set cookies from php (if their hits are not cached) [20:30:02] nuria: i don't think they need to use js [20:30:32] shouldn't be cached as far as I know [20:30:50] Multimedia, Analytics: Track image context and pass information onto X-Analytics - https://phabricator.wikimedia.org/T85922#1015847 (Tgr) So what exactly should be done here? # add a query parameter (say `&x-analytics=fc=mv`?) to all MediaViewer file requests - done in T77882 # add a query parameter (`&x-anal... [20:32:27] milimetric: then easiest would be to set a cookie using the MWCryptRand server side that gets render with the page request upon hitting step #1 of pipeline, after every pipeline step propagates the token. That way is not a cookie and gets automatically discharged at the end of session, makes sense? [20:33:32] milimetric: we can talk in batcave if it is not clear [20:34:18] sorry i'm having parallel discussion in wikimedia-visualeditor [20:34:23] but no what you're saying makes sense [20:34:44] i don't think they need tech support as much as "is this ok" and "how do we expire the cookie etc." because the funnel looks messed up [20:34:47] i'm trying to decipher now [20:34:56] (ex: they say saveFailure could be the first step) [20:35:26] milimetric: what i am saying is that they do not even need a cookie [20:36:06] milimetric: so there is no need to expire it [20:36:12] " then easiest would be to set a cookie using the MWCryptRand " [20:36:24] argh sorry [20:36:26] I don't think you are saying that :) [20:36:39] Sorry [20:36:49] then easiest would be to set a parameyter using the MWCryptRand [20:36:52] *parameter [20:37:11] that parameter gets propagated from step #1 of pipeline [20:37:39] milimetric: to step#2 to step#n (final) [20:38:03] milimetric: at the end of the pipeline there is nothing to do, as it is not persisted it does not need to be discharged [20:38:09] milimetric: makes sense? [20:38:11] Analytics-Wikistats: Provide total active editors for December 2014 - https://phabricator.wikimedia.org/T88403#1015866 (Eloquence) Big thanks to @joe for getting the dumps back on track. Per @ezachte, this unfortunately won't be enough to hit our February 15 publication deadline for the quarterly report, sinc... [20:38:18] milimetric: we can talk in hangout [20:38:35] sorry, that'd be hard 'cause i'm trying to chat in the other channel [20:38:51] but how would the MWCryptRand parameter be propagated over different page loads? [20:38:56] #wikimedia-visualeditor? [20:39:15] milimetric: cause you pass it along with the data of your editing session [20:40:15] looks like that channel is invite only if they add me i can join and explain [20:40:22] ^milimetric [20:40:30] i don't have rights :( [20:41:18] sorry nuria it's #mediawiki-visualeditor [20:43:12] milimetric: two questions about your sql code here https://gerrit.wikimedia.org/r/#/c/188601/1/edit/save_rates_low_noise.sql [20:43:36] I'm assuming timestamp include time as in hhmmss in your data, correct? [20:44:11] scratch it milimetric. sorry. I thought there is a problem with grouping but it's my bad [20:49:32] leila: yes, timestamp is a 14-character field in YYYYMMDDHHmmss format [20:49:47] makes sense milimetric. [20:56:45] Ironholds, around? [20:58:15] yurikR, yup [20:58:16] what's up? [20:58:37] Ironholds, hey, was looking at your udf patch [20:58:55] not sure what you are trying to accomplish - i thought we were doing it by ourselves? [20:58:58] cool! The isZero UDF is the important one [20:59:19] the code is fairly involved, requires calling api - https://zero.wikimedia.org/w/api.php?action=zeroportal&type=analyticsconfig [20:59:27] which you can't call without creds [20:59:41] Well, Dario informed me you wanted some QA on the pageview UDF, which I'm workin' on, but I figured if you were planning on using Hadoop you probably wanted a way to detect Zero requests [20:59:50] if you already have a UDF for that, merging it upstream would be most appreciated. [21:00:28] i have this: http://git.wikimedia.org/blob/analytics%2Fzero-sms.git/HEAD/scripts%2Fzero-counts.hql [21:01:01] Ironholds, which is called by http://git.wikimedia.org/blob/analytics%2Fzero-sms.git/HEAD/scripts%2Fweblogs2.py [21:02:00] ah. Oh. Ah. [21:02:19] yep ) [21:02:25] That's fine then; if there's no interest from the Zero team in UDF-side identification I can just scrap it [21:02:39] the cost, presumably, will be ongoing maintenance of zero identification (and IDIng zero page requests at all!) from your end [21:02:50] that's correct [21:02:59] we keep changing what it means to be zero [21:03:08] so it would be easier to maintain it in this rego [21:03:10] repo for now [21:03:20] gotcha [21:03:22] yeah, makes sense :/ [21:03:31] * Ironholds has a hankerin' for some stability round here. *chews tobacco* [21:03:56] it also deals with uploading those stats to the zero portal [21:03:58] for the graphs [21:04:11] Ironholds, are we able to use the "isPageView" udf yet? [21:04:21] i would love to change the HQL to use it [21:04:37] yurikR, should be! I mean, it's merged. [21:04:42] but Dario told me y'all wanted QA first [21:04:46] I have not finished said QA :P [21:05:01] let me know when its ready, would love to start using it ) [21:05:06] or submit a patch to my HQL [21:05:10] shall do! [21:05:36] also, another thing we have been contemplaiting - we will soon start adding zero=NNN to ALL traffic, not just m|zero.wikipedia [21:05:51] which will allow us to track download size per carrier [21:06:05] simply by running sum(content_size) group by zero [21:06:41] but this will probably have to be done as a separate query - as it shouldn't be filtered by the isPageView() udf [21:07:12] (by "all traffic" i mean only that traffic that came from a partner) [21:13:04] Ironholds, sorry, had a connection problem, if you wrote anything please repeat [21:13:49] Ironholds, sorry, had a connection problem, if you wrote anything please repeat [21:14:34] Didn't! [21:14:52] and, surely adding zero= will make tracking trivial, no? [21:17:23] Ironholds, nope, zero= only identifies if it came from a zero partner, not if it was actually zero rated :) [21:18:04] partners are complicated - they could have multiple networks, could even send us an additional header that identifies if it was a zero or not :) [21:19:49] ahhh [21:22:06] its a pain :) [21:46:07] Analytics-Kanban, Analytics-Cluster: Mobile product managers should have reports on session-related metrics from the Wikipedia Apps - https://phabricator.wikimedia.org/T86535#1016005 (kevinator) [21:49:46] Analytics-Kanban, Analytics-Cluster: Mobile product managers should have reports on session-related metrics from the Wikipedia Apps - https://phabricator.wikimedia.org/T86535#1016026 (Deskana) The high level requirement is that the data be comparable with the old ad-hoc reports. So whatever time period was us... [21:51:20] milimetric: bat-cave? [21:51:46] sure [21:54:57] Analytics-Kanban, Analytics-EventLogging: Add ops-reportcard dashboard with analysis that shows the http to https slowdown on russian wikipedia - https://phabricator.wikimedia.org/T87604#1016034 (kevinator) p:Triage>High [21:56:15] milimetric: whenever you have some time: in the funnel-prototype.html, where do you say what "separator" you've used in actions? I had to use _ instead of - [21:57:26] Analytics-Wikimetrics: Make timezone selector comprehensive and consistent - https://phabricator.wikimedia.org/T88604#1016040 (Capt_Swing) NEW [21:58:09] Analytics-Kanban: Calculate per-wiki Edit Completion Rate for Visual Editor edits - https://phabricator.wikimedia.org/T87865#1016055 (Milimetric) Open>declined Is better captured by the other VE funnel anlaysis tasks. [21:58:55] leila: line 471 if you have not changed it much [21:58:56] var parts = row[2].split('-'); [21:59:35] thanks, milimetric. :-) [22:24:28] Analytics-Wikimetrics: Story: user wants to be able to re-run a failed report more easily - https://phabricator.wikimedia.org/T88610#1016115 (Capt_Swing) NEW [22:25:44] Analytics-Kanban, Analytics-Dashiki: {Epic} User visualizes dashboards in Dashiki {frog} - https://phabricator.wikimedia.org/T88611#1016124 (kevinator) NEW [22:26:48] Analytics-Kanban, Analytics-Dashiki: Add time range selection to Limn dashboards (or new Dashiki dashboards) {frog} - https://phabricator.wikimedia.org/T87603#1016133 (kevinator) p:Triage>Low [22:28:42] nuria: deskana just answered the questions you posted on https://phabricator.wikimedia.org/T86535 [22:29:01] I’ll bring up this task tomorrow during tasking. ^^ [22:33:49] (PS3) OliverKeyes: (WIP) project class/variant extraction UDF [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188588 [22:34:33] sooo, the cluster just crashed on my query again [22:49:56] milimetric, one question.. what hapens in mediawiki db when a user wants to close his/her account? [22:50:17] close? [22:50:29] yes, delete [22:50:33] hm... don't know [22:50:37] their history remains [22:50:42] but i don't know how delete works [22:50:46] only rename [22:50:55] and that's a bit weird [22:51:04] because, we are setting valid_to to NULL for all users [22:51:23] and I do not find any reference to a similar field in mw [22:52:02] ok, thanks anyway! [22:53:04] one sec [22:57:47] so valid_to is meant to track changes through time [22:58:02] and it's exactly the point that this doesn't exist in mediawiki [22:58:11] valid_to will allow us to track as people move in and out of the bot group [22:58:16] or pages as they move in and out of namespaces [22:58:17] etc. [22:58:52] aha [22:59:27] valid_to refers to the period of time when a set of properties for that entity stops being valid [22:59:37] I guess then, that the timestamp for one user that changes in or out the bot group is to be found in the logging table [22:59:40] http://en.wikipedia.org/wiki/Slowly_changing_dimension [22:59:48] no [22:59:54] we have to manufacture the valid_to timestamp [22:59:56] but that's later [22:59:59] i can talk about it more tomorrow [23:00:20] there's a "delete"? [23:00:23] I didn't know we had a delete. [23:00:35] it would be a massive copyright problem [23:01:16] we have to calculate the timestamp based on the state from the last data import and the new state from the current import? [23:02:46] I see, so valid_to does not refer to a user that does not exist any more, but just that it has entered the bot group [23:03:58] then, can a user exit the bot group after entering it? [23:09:35] Analytics-Engineering: Hadoop errors - https://phabricator.wikimedia.org/T88537#1016304 (Ironholds) New instance, error attached.{F37338} [23:22:09] !log Manual failover of Hadoop namenode from analytics1001 to analytics1002, as analytics1001 had Heap space errors [23:24:01] yikes qchris [23:24:11] jgage yes. [23:24:18] The second time I am doing this today :-( [23:24:33] i saw your report :( [23:24:47] it's odd because historically namenodes have been quite stable [23:24:54] so i'm looking for what has changed recently [23:25:05] Yup. No clue. [23:25:11] and also trying to work out why analytics1002 isn't reporting to ganglia [23:25:25] :-D [23:25:41] Generally, stat1002's load allows to detect the issue: [23:25:43] http://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&h=stat1002.eqiad.wmnet&m=cpu_report&s=descending&mc=2&g=load_report&c=Analytics+cluster+eqiad [23:25:53] Both hdfs issues stand out a but. [23:25:57] s/but/bit/ [23:26:16] i wonder whether those fuse mounts could be causing namenodes to work harder [23:26:49] Maybe. Because when we see the issue, there are 20-something check_disk processes hanging. [23:26:51] milimetric: if you're still around, and whenever you have some time, can we chat briefly? [23:29:25] Ironholds: Do the spikes in [23:29:27] http://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&h=stat1002.eqiad.wmnet&m=cpu_report&s=descending&mc=2&g=load_report&c=Analytics+cluster+eqiad [23:29:42] Align with when your UDF job runs? [23:29:54] * Ironholds calculates [23:30:14] the second one does, the first one doesn't - I launched that job Tuesday morning UTC [23:30:27] also they were the same job over the same number of files each time, so there shouldn't be a substantial variation in runtime. [23:30:34] *load size. brain. [23:30:54] Load would not come into play here. [23:31:06] Because ... [23:31:26] Assume the UDF is doing $FOO to cause a heap exhaustion in the active namenode, then [23:31:41] the hdfs mount on stat1002 would get hanging, and [23:31:55] the load on stat1002 would increase. [23:32:01] (Until someone comesa around to fix it) [23:32:14] So the namenode dying is the real issue. [23:32:25] The load on stat1002 is just an indicator that the namenode died. [23:32:50] Mhmmm... How long does your job take to run? [23:36:18] Ironholds: ^ [23:36:41] a day-ish? [23:36:51] it's running over 31 days of text/mobile partitions, twice [23:37:06] and then UNIONing the results. [23:37:08] Aaaw :-/ [23:37:11] ....I guess that could do it huh ;p [23:37:19] Hahaha. [23:37:26] Meh. How big can the output be? [23:37:27] :-P [23:37:35] oh, the output is only 48 lines! [23:37:44] wait. 60 lines [23:37:48] See! [23:37:50] but, you know. That's after the reducing. [23:37:52] 12 more lines! [23:38:03] 12 lines ... of death! [23:38:23] Is any number in them "666"? [23:39:42] Seriously though ... not sure. Could that be related? Totally not sure. [23:39:59] heh [23:40:23] By when would you need the output of the query? [23:40:35] Would it be possible to maybe split the query into smaller chunks? [23:42:08] that's the plan! [23:42:21] I mean, frankly I can halve it and it'll work fine [23:42:37] just run it as two distinct queries and union by way of, well, append piping ;p [23:43:06] I'll work on that tomorrow when my brain is no longer slurry. [23:43:25] * qchris looks up "slurry" [23:43:29] :-D [23:43:37] Hahaha. [23:43:56] Is your query running right now? Do you intend to run it? [23:44:13] (Just want to make sure whether or not to expect something happening) [23:44:27] Poor namenodes and their heap. [23:44:43] naw, I'll throw it in tomorrow morning [23:44:50] k. Cool. [23:44:54] Thanks. [23:44:56] no sleep-deprived driving, query execution or archery [23:44:58] those are my rules [23:45:31] Those are good rules. [23:48:36] they are! you don't make the sleep-deprived archery mistake twice. [23:48:45] (because the first time, they take away your license :()