[00:00:40] milimetric, does the test test_login_meta_mw pass in your machine for the mwoauth changeset? [00:01:20] Analytics-Wikimetrics: enwiki does not show up in my centralauth expanded cohort - https://phabricator.wikimedia.org/T78584#849591 (kevinator) [00:02:43] Analytics-Wikimetrics: enwiki does not show up in my centralauth expanded cohort - https://phabricator.wikimedia.org/T78584#849595 (mforns) This issue is not directly related to cohort user deletion, but with centralauth expansion. When separating the username and the project with a comma followed by a space... [00:04:49] Analytics-Wikimetrics: Story: WikimetricsUser deletes user from cohort [21 pts] - https://phabricator.wikimedia.org/T75350#849603 (kevinator) [00:05:21] Analytics-Wikimetrics: Wikimetrics User reads disclaimer on website [3 pts] - https://phabricator.wikimedia.org/T76107#849604 (kevinator) [00:05:32] Analytics-Wikimetrics: Wikimetrics auditor has read-only login to Wikimetrics DB [3 pts] - https://phabricator.wikimedia.org/T76109#849605 (kevinator) [00:05:43] Analytics-Wikimetrics: Fix oauth and do a quick pre-security review [13 pts] - https://phabricator.wikimedia.org/T76779#849607 (kevinator) [00:05:55] Analytics-Wikimetrics, Analytics-Dashiki: Vital Signs user sees annotations on graphs [13 pts] - https://phabricator.wikimedia.org/T78151#849608 (kevinator) [00:06:19] Analytics-Wikimetrics: Uploading cohort by copy-pasting breaks if names contain special characters [8 pts] - https://phabricator.wikimedia.org/T76105#849609 (kevinator) [00:06:50] Analytics-Engineering, Analytics-Wikimetrics: Re-run Wikimetrics data once Labs issues are fixed [8 pts] - https://phabricator.wikimedia.org/T78305#849610 (kevinator) p:High>Normal [00:12:59] (CR) Mforns: Strip project when uploading cohort (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/180071 (owner: Mforns) [00:21:14] Analytics-Engineering, Analytics-Wikimetrics: Story: WikimetricsUser reports pages edited by cohort [13pts] - https://phabricator.wikimedia.org/T75072#849641 (kevinator) [00:22:28] ottomata, kicking off some test streaming jobs now. [00:22:31] :) [00:22:36] Your file structure is pretty [00:26:08] cool! heh [00:26:28] the parquet one may be a problem...due to some annoying hadoop streaming details (it uses the deprecated API...) [00:26:55] Honestly, I wasn't quite sure how to set up for that one. [00:27:09] Is parquet a compression format? [00:27:49] Oh! It's columnar. [00:28:08] ya, don't worry about that for now then [00:28:09] ui' [00:28:12] i'll have to figure that one out [00:28:15] kk [00:28:24] i was working on that today, but then i spent the day helping oliver instead :) [00:28:46] no worries. Pageviews is important. [00:28:48] halfak: http://grepalex.com/2014/05/13/parquet-file-format-and-object-model/ [00:28:53] I was working on other stuff all day anyway. [00:29:13] ottomata, I'm having trouble imagining how a columnar format would work for streaming. [00:29:34] Given that the scripts will expect to get a whole json thing at a time. [00:30:06] yeah, for streaming it might not make as much sense [00:30:13] but, i'm trying to benchmark more than just streaming [00:30:19] getting an idea for this in other use cases too [00:30:26] you can query this stuff with hive too [00:30:32] Totally. Hive makes a lot of sense. [00:31:31] ok, i'm out for the eve, ttyl [00:36:46] o/ ott [00:36:49] o [00:36:54] too late [00:39:13] I'm outta here too [00:46:54] Analytics-Engineering, Analytics-Wikimetrics: Story: WikimetricsUser reports pages edited by cohort [13pts] - https://phabricator.wikimedia.org/T75072#849730 (kevinator) p:High>Low After much discussion with the dev team, it became apparent that this is not a simple change to Wikimetrics. It is built t... [01:36:23] did we change how many days we stored in hive? [01:43:35] Analytics-EventLogging: find a better way to identify events that fail validation as early as possible - https://phabricator.wikimedia.org/T78355#849763 (kevinator) Resolved>Open reopening task because there are still issues to iron out. [04:00:08] (CR) Nuria: "Wow, code looks so much cleaner! Tests run clean in vagrant" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/179962 (owner: Milimetric) [07:43:08] (CR) Gergő Tisza: "mysql:research@analytics-store.eqiad.wmnet [log]> show index from NavigationTiming_10374055;" [analytics/multimedia] - https://gerrit.wikimedia.org/r/179872 (owner: Gergő Tisza) [08:15:50] Analytics-Tech-community-metrics: Key performance indicator: Bugzilla response time - https://phabricator.wikimedia.org/T63561#849864 (Qgil) [08:15:51] Analytics-Tech-community-metrics: Bugzilla response time: "Longest time without comment" is actually "Longest time without any comment by non-reporter"? - https://phabricator.wikimedia.org/T69589#849862 (Qgil) Open>declined Well, in fact this one refers to the Bugzilla stats page, which now is deprecate... [08:16:39] Analytics-Tech-community-metrics: Key performance indicator: code contributors new / gone - https://phabricator.wikimedia.org/T63563#849865 (Qgil) a:Qgil>None [08:16:51] Analytics-Tech-community-metrics: Key performance indicator: Top contributors - https://phabricator.wikimedia.org/T64221#849866 (Qgil) a:Qgil>None [09:55:56] Multimedia, Analytics, MediaWiki-extensions-MultimediaViewer: Create MediaViewer image varnish hit/miss ratio dashboard - https://phabricator.wikimedia.org/T78205#850032 (Gilles) a:Tgr [10:04:30] (CR) Gilles: [C: -1] "The data is there now, but the graph doesn't render the "cache miss ratio" curve, only the total request count. I think it's a limn bug, b" [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/179778 (owner: Gergő Tisza) [10:06:48] Multimedia, Analytics, MediaWiki-extensions-MultimediaViewer: Investigate if pre-rendering images is having an impact on performance - https://phabricator.wikimedia.org/T76035#788560 (Gilles) Initial poking at the data suggests that prerendering actually worsened performance. I'll work on creating graphs arou... [10:10:53] Multimedia, Analytics, MediaWiki-extensions-MultimediaViewer: Create MediaViewer image varnish hit/miss ratio dashboard - https://phabricator.wikimedia.org/T78205#850050 (Gilles) Note that pre-rendered thumbnails will appear as a varnish miss. The first time they're requested they're in swift, but not in varn... [10:29:33] Analytics-Refinery: Make webrequest partition validation handle races between time and sequence numbers - https://phabricator.wikimedia.org/T71615#850070 (QChris) Happened again for: 2014-12-15T15/2H (on text) [10:30:10] !log Marked raw text webrequest partition for 2014-12-15T15/2H ok (See {{PhabT|71615|850070}}) [10:30:53] (PS1) Gilles: Query image performance by upload time [analytics/multimedia] - https://gerrit.wikimedia.org/r/180136 [11:11:31] Multimedia, Analytics, MediaWiki-extensions-MultimediaViewer: Investigate if pre-rendering images is having an impact on performance - https://phabricator.wikimedia.org/T76035#850128 (Gilles) Actually I've just thought of checking that data by dividing varnish hits (shouldn't be affected by prerendering) and... [11:17:47] (PS2) Gilles: Query image performance by upload time [analytics/multimedia] - https://gerrit.wikimedia.org/r/180136 [11:34:21] Analytics-Engineering: Dedupe data while doing ETL for the Analytics cluster - https://phabricator.wikimedia.org/T76724#850163 (QChris) [11:47:59] Multimedia, Analytics, MediaWiki-extensions-MultimediaViewer: Investigate if pre-rendering images is having an impact on performance - https://phabricator.wikimedia.org/T76035#850186 (fgiunchedi) good question Gilles, it shouldn't happen AFAIK but it is possible, were those also direct fetches from swift? wha... [12:38:21] Phabricator, Analytics-Tech-community-metrics, Engineering-Community: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#850331 (Aklapper) >>! In T1003#844983, @kevinator wrote: > This might be a cheap way to get a visualization of the data. Thanks. We might play wit... [12:38:26] Phabricator, Analytics-Tech-community-metrics, Engineering-Community: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#850332 (Aklapper) Status update: * Daniel triggered a "Phabricator monthly statistics" test email to my account last week. * For unknown reasons (t... [13:01:46] Phabricator, Analytics-Tech-community-metrics, Engineering-Community: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#850358 (Dzahn) >>! In T1003#850332, @Aklapper wrote: > * For unknown reasons (those SQL queries work locally, maybe the //cut -d " " -f3// in the... [13:06:39] (PS1) QChris: Add requirements.txt [analytics/blog] - https://gerrit.wikimedia.org/r/180156 [13:06:41] (PS1) QChris: Fix flake8 errors [analytics/blog] - https://gerrit.wikimedia.org/r/180157 [13:06:43] (PS1) QChris: Allow to specify date to compute the report for [analytics/blog] - https://gerrit.wikimedia.org/r/180158 [13:06:45] (PS1) QChris: Add basic tox setup [analytics/blog] - https://gerrit.wikimedia.org/r/180159 [13:06:47] (PS1) QChris: Move blogreport code behind a __main__ guard [analytics/blog] - https://gerrit.wikimedia.org/r/180160 [13:06:49] (PS1) QChris: Add tests for parsing string to date [analytics/blog] - https://gerrit.wikimedia.org/r/180161 [13:13:26] Multimedia, Analytics, MediaWiki-extensions-MultimediaViewer: Investigate if pre-rendering images is having an impact on performance - https://phabricator.wikimedia.org/T76035#850363 (Gilles) I'm looking at our own data gathered from sampled clients that checks if the image was a varnish hit or not. It's pars... [13:32:47] Multimedia, Analytics, MediaWiki-extensions-MultimediaViewer: Investigate if pre-rendering images is having an impact on performance - https://phabricator.wikimedia.org/T76035#850380 (Gilles) Average performance experienced by users hitting the cp4xxx "varnish2" servers is slower: P160 is this expected? No... [14:25:31] (PS2) Dr0ptp4kt: Adding MCC-MNC mismatch script. [analytics/zero-sms] - https://gerrit.wikimedia.org/r/179926 [14:26:00] (CR) Dr0ptp4kt: [C: 2] Adding MCC-MNC mismatch script. [analytics/zero-sms] - https://gerrit.wikimedia.org/r/179926 (owner: Dr0ptp4kt) [14:27:26] (CR) Dr0ptp4kt: [V: 2] Adding MCC-MNC mismatch script. [analytics/zero-sms] - https://gerrit.wikimedia.org/r/179926 (owner: Dr0ptp4kt) [14:37:43] (PS1) Dr0ptp4kt: Update README [analytics/zero-sms] - https://gerrit.wikimedia.org/r/180182 [14:38:14] (CR) Dr0ptp4kt: [C: 2 V: 2] Update README [analytics/zero-sms] - https://gerrit.wikimedia.org/r/180182 (owner: Dr0ptp4kt) [14:41:43] Phabricator, Engineering-Community, Analytics-Tech-community-metrics: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#850592 (Aklapper) I've added a few more patchsets / revisions to https://gerrit.wikimedia.org/r/#/c/177792/ : * worked around (fixed?) the two empt... [14:46:26] (PS2) Mforns: Strip project when uploading cohort [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/180071 [14:46:37] (CR) jenkins-bot: [V: -1] Strip project when uploading cohort [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/180071 (owner: Mforns) [14:49:16] yay ottomata! [14:49:19] (PS3) Mforns: Strip project when uploading cohort [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/180071 [14:50:31] mornin! [14:51:28] I have comments! comments on the UDF! [14:51:31] and also questions [14:51:35] Phabricator, Engineering-Community, Analytics-Tech-community-metrics: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#850622 (Dzahn) >>! In T1003#850592, @Aklapper wrote: > Works locally. Now waiting for Daniel to trigger another test email to me. yep, works for m... [14:51:48] like, should we also exclude based on GET versus POST, y'think? It seems to catch some silliness. [14:52:44] (CR) OliverKeyes: [C: -1] "Comments!" (2 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180023 (owner: Ottomata) [14:53:52] Ironholds: that is up to the specification committee :) [14:54:16] but sure! and! I would be much obliged if you felt empowered to make new patches to the change :) [14:54:18] the specification committee thinks it's too finnicky a rule to be useful [14:54:22] but wanted to make sure [14:54:35] totally! I...have never actually patched an existing commit throug hgerrit before [14:54:42] do you use git review? [14:55:09] I read Patrick's tutorial to it? That counts. [14:55:31] well, i mean ,you can use straight git, but i use git review, and if you did I would know how to advise you [14:55:36] how do you usually push to gerrit for review? [14:55:40] oh, right. yes, I do! [14:55:46] ok cool [14:55:50] I mean, I've only patched MW a coupla times. [14:55:50] welp, git review makes this easy [14:56:22] okay, I've cloned the refinery project [14:56:25] gimme the tutorial! [14:56:25] so, the uhh the change number is the id in the url [14:56:29] ah, you actually need refinery-source [14:56:31] do you have that? [14:56:43] this one? [14:56:43] https://gerrit.wikimedia.org/r/#/admin/projects/analytics/refinery/source [14:57:23] once you do [14:57:32] you can take the change number from the url [14:57:33] indeed [14:57:37] https://gerrit.wikimedia.org/r/#/c/180023/ [14:57:38] in this ase [14:57:39] case [14:57:43] 180023 [14:57:43] and do [14:57:46] git review -d 180023 [14:57:48] aha [14:57:50] and it will checkout the change for you [14:57:58] then, make whatever changes you want [14:58:00] "no .gitreview file found" [14:58:01] and git commit --amend [14:58:03] ah [14:58:08] yeah i should fix that, id on't use git review files [14:58:08] um [14:58:09] do this [14:58:20] ahhhh, i'll just check in a git review file [14:58:23] kk [14:58:27] will wait and pull, in order [14:59:53] (CR) Mforns: Strip project when uploading cohort (7 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/180071 (owner: Mforns) [14:59:59] (PS1) Ottomata: Add .gitreview file [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180188 [15:00:16] (CR) Ottomata: [C: 2 V: 2] Add .gitreview file [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180188 (owner: Ottomata) [15:00:20] ok Ironholds [15:00:21] pull [15:00:22] and try that again [15:02:58] shall do [15:03:32] We don't know where your gerrit is. Please manually create a remote [15:03:32] named "gerrit" and try again [15:03:45] hm [15:03:50] (sory in standup now) [15:03:51] hm [15:04:00] ok, then fine, don't use a .gitreview file [15:04:00] do [15:04:06] np! [15:04:23] git remote add gerrit ssh://ironholds@gerrit.wikimedia.org:29418/analytics/refinery/source [15:04:44] done and done [15:04:51] ok try again [15:04:56] oh maybe [15:04:57] do [15:04:57] first [15:04:59] git review -s [15:05:05] (setup) [15:05:43] Analytics-Wikimetrics: Fix oauth and do a quick pre-security review [13 pts] - https://phabricator.wikimedia.org/T76779#850637 (Milimetric) security review results: SQL in controllers and some forms / ajax improvements remain to be done. [15:05:46] permission denied [15:05:50] I'll make sure I'm using the right key [15:08:46] aha, wrong key [15:12:07] dammit [15:12:15] how the hell do I specify what key to use? [15:13:16] hm [15:13:22] git config, i'm sure uhm [15:13:48] hm maybe [15:14:33] hm, no? ok Ironholds in your ~/.ssh/config file [15:14:34] add [15:15:02] Host gerrit.wikimedia.org [15:15:02] User ironholds [15:15:02] IdentityFile /path/to/your/ssh/key [15:15:04] ta [15:19:31] aaand it can't find my key. hrm. [15:20:15] urgh [15:20:21] will finish setting up after the combined staff meeting [15:36:00] Phabricator, Engineering-Community, Analytics-Tech-community-metrics: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#850674 (Dzahn) merged PS10 https://gerrit.wikimedia.org/r/#/c/177792/ puppet-compiler: http://puppet-compiler.wmflabs.org/558/change/177792/html/i... [15:36:09] ottomata, got it working! [15:36:17] cool! [15:36:22] * Ironholds starts patching [15:36:24] ok, yeah, so now that you have the change checked out [15:36:29] you can commit new patches by doing [15:36:32] git commit --amend [15:36:44] (make sure you amend, otherwise things will get pretty annoying) [15:36:46] then [15:36:47] just [15:36:50] git review [15:37:18] make sure you only have this one commit that you are working on and amending at the head of this checkedout change-branch when you git review [15:50:07] ottomata, is it okay if I expand the unit tests? [15:50:17] ....I appreciate that's usually a stupid question [15:50:30] please! [15:50:31] haha [15:50:48] that was a quick and get it working proof of concept patch [15:50:52] it will need MANY more unit tests [15:50:59] btw, have you seen christian's webstatscollector tests? [15:51:27] https://github.com/wikimedia/analytics-webstatscollector/tree/master/tests [15:51:57] https://github.com/wikimedia/analytics-webstatscollector/blob/master/tests/test.sh#L265 [15:51:59] (PS4) Milimetric: Use the wonderful mwoauth [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/179962 [15:52:26] we should probably put together some canonical list of tests that can be reusable by different implementations of this [15:53:01] even just text files, or json files, or something [15:53:16] we could make json files with the same schema content of webrequest logs, and then maybe add an extra field in the tests [15:53:22] is_pageview? [15:53:29] and then just iterate throught them all [15:53:34] and assert that [15:53:49] Webrequest.is_pageview(...) == line['is_pageview'] [15:53:51] or whatever [15:54:06] i think qchris_meeting has opinions on this :) [15:54:14] but we are all in the same meeting! [15:54:16] shhhhh [15:55:57] Phabricator, Engineering-Community, Analytics-Tech-community-metrics: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#850735 (Dzahn) needed one more small fix: https://gerrit.wikimedia.org/r/#/c/180193/ but now it works, puppet applied it and i tested it. the co... [15:56:42] Ironholds: if you can put together a big list of test cases, lines that ARE pageviews, and lines that AREN'T pageviews, i can work on that [15:56:58] ottomata, will do! [15:57:00] at the moment I'm duplicating tests into the hive tests and the underlying java tests [15:57:08] this is inefficient although largely trivial with find-and-replace :D [15:57:08] ? [15:57:12] yeah [15:57:25] you have hive tests? [15:59:02] sorry, tests of the resulting UDF [15:59:27] refinery-hive/src/test/java/org/wikimedia/analytics/refinery/hive/TestIsPageviewUDF.java compared to refinery-core/src/test/java/org/wikimedia/mediawiki/TestWebrequest.java [15:59:51] ah yes [15:59:52] yeah [16:00:27] Ironholds: if you want to, for now, just do TestWebrequest.java? we can duplicate to the UDF tests later? or meh, do whatever you want :) [16:00:39] already wrote em! [16:01:04] (PS3) OliverKeyes: [WIP] UDF for classifying pageviews according to https://meta.wikimedia.org/wiki/Research:Page_view/Generalised_filters [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180023 (owner: Ottomata) [16:01:34] https://gerrit.wikimedia.org/r/#/c/180023/ boom [16:01:35] of course, I make no promises that it will actually run, as I am unable to test it locally [16:01:43] but it seems to syntactically match existing code [16:02:52] why can't you run locally? [16:02:54] do: [16:02:57] mvn test [16:03:08] hm, maybe you need maven, hmm? [16:03:39] will get [16:03:56] what OS are you Ironholds? [16:04:17] xubuntu; sudo apt-get install maven [16:04:23] ah easy peasy. [16:05:51] running... [16:06:45] incompatible types! aw man.. [16:07:36] okay, this makes no sense to me. [16:07:52] looking [16:07:58] apparently the new version of source/refinery-hive/src/test/java/org/wikimedia/analytics/refinery/hive/TestIsPageviewUDF.java has incompatible types everywhere [16:08:02] I assume that means I borked the UDF, actually [16:08:11] yes [16:08:19] you are assigning instead of newing [16:08:22] this [16:08:22] Text uriHost = new Text("en.mobile.wikipedia.org"); [16:08:23] vs. [16:08:26] Text uriHost = "en.wikipedia.org"; [16:08:36] in this case, Text is a [16:08:39] import org.apache.hadoop.io.Text; [16:08:39] javascript:; [16:08:41] oops [16:08:53] org.apache.hadoop.io.Text [16:08:58] which is a special Hadoop object [16:09:01] not a default Java String [16:09:02] aha [16:09:04] doy [16:09:38] note that the refinery.core.Webrequest class works with regular Strings [16:09:41] cause it is supposed to be generic [16:09:54] since Hive will always run in hadoop context, it works with Hadoop objects [16:10:02] yeah [16:11:28] this is making my internal C++ programmer really antsy [16:11:39] "you're calling new but there's no equivalent free() or delete! EVERYONE IS GOING TO DIE AND IT WILL BE THIS CODE'S FAULT" [16:12:11] haha [16:12:18] well, at least you feel that way! [16:12:22] okay, fixed, testing again [16:12:24] i think that is a good feeling [16:12:28] I spent all of yesterday tracking down a single memory leak [16:12:33] when people start with java they don't have that! [16:12:40] heh [16:14:18] hmn [16:14:30] ottomata, if I just include "foo" in a match pattern in Java, will it match barfoobar? [16:14:49] Ironholds: i think not, which is strange. [16:14:57] tha's why I prepended the .*s in the regexes [16:15:04] hrrrm [16:15:12] i use this to check: http://www.regexplanet.com/advanced/java/index.html [16:15:13] :) [16:15:38] it wants normal regexes there, not java style [16:15:42] .e.g. you don't need \\. [16:15:43] just \. [16:16:41] ohhh [16:16:55] that'd do it [16:17:55] nope [16:18:01] "illegal escape character" [16:18:55] ....wtf? [16:18:59] ? [16:19:00] ottomata, api.php matches api.php [16:19:09] that is, a regex, api.php, matches a string, api.php [16:19:16] you don't need to escape it at ALL? How can that be right? [16:19:21] . == single character [16:19:24] any character [16:19:35] aha [16:19:44] yet api\.php is an illegal escape, and api\\.php fails. [16:20:04] wait, no, api\\.php works. [16:20:06] ...sod it. [16:20:09] all the tests pass [16:20:12] want the code? :p [16:20:19] api\.php works [16:20:23] in the tester [16:20:31] which means api\\.php should work [16:20:33] aye [16:20:34] (PS4) OliverKeyes: [WIP] UDF for classifying pageviews according to https://meta.wikimedia.org/wiki/Research:Page_view/Generalised_filters [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180023 (owner: Ottomata) [16:20:36] yeah, submit, :) [16:20:39] done! [16:20:45] maven checks pass [16:20:52] Ironholds: if you want to build it and run it in hadoop [16:20:52] do [16:20:57] mvn clean package [16:20:57] then [16:21:31] scp refiney-hive/target/refinery-hive-0.0.3-SNAPSHOT.jar stat1002.e:~/ [16:21:35] then you've got the .jar on stat1002 [16:21:41] and you can ADD JAR like you know how to already :)( [16:21:49] oh [16:21:50] sorry [16:22:00] scp refiney-hive/target/refinery-hive-0.0.3-SNAPSHOT.jar stat1002.eqiad.wmnet:~/ [16:22:04] cool! Thankee :) [16:22:05] but ja [16:22:18] so I can just iterate on this endlessly? [16:22:20] yup [16:22:25] yay! [16:31:39] ottomata, will be in the dev thingy in ~5. Just gimme the hangout URI :) [16:31:41] Ironholds: https://plus.google.com/hangouts/_/wikimedia.org/aotto-jgerard [16:31:44] k [16:31:51] yay! [16:31:55] also, Java is not as hard as I expected. [16:32:01] I've been told it's a wretched hive of scum and villany. [16:32:26] naw, its easier than C++ Ironholds, for sure [16:32:32] but about the same [16:54:12] Analytics-Engineering, Analytics-EventLogging: EL office hours - https://phabricator.wikimedia.org/T76796#850892 (ggellerman) [16:56:58] (CR) Nuria: [C: 2] Use the wonderful mwoauth [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/179962 (owner: Milimetric) [16:58:28] ottomata, thoughts on making tests more exportable - https://gist.github.com/Ironholds/30c09e6b4402bfc0e967 [16:58:40] that's what I was trying to say but failing :D [17:02:29] Analytics-Wikimetrics: Fix oauth and do a quick pre-security review [13 pts] - https://phabricator.wikimedia.org/T76779#850902 (Milimetric) Open>Resolved [17:02:31] Analytics-Wikimetrics, Analytics-Engineering: EPIC: Productionizing Wikimetrics - https://phabricator.wikimedia.org/T76726#850903 (Milimetric) [17:09:05] qchris_meeting: what?! [17:09:16] tnegrin, okay, you may have been right. [17:09:33] "I'm not a java programmer! This is a one-off! Would anyone mind if I do some restructuring of the user-agent parser?" [17:09:34] about what? [17:09:39] heh [17:09:47] welcome what you cannot avoid [17:09:50] It'd make sense to have the actual parser as a Java function and the UDF Just do the type-checking and conversion [17:09:58] at the moment it's all one blob [17:10:15] so I figure, first I split, and then I can add Wikimedia-specific spider UAs to the Java function [17:10:16] * Ironholds nods [17:10:17] Ironholds: Java ain't so bad! come on in! [17:10:28] You say that now [17:10:44] and then in 6 months R&D see me as an engineer with ggplot2, and AnEng see me as a researcher with maven [17:10:59] and I have an existential crisis and flee to Peru, where I live under an assumed name and drink at the beach all day [17:11:04] it'll happen, I guarantee it. [17:11:15] sounds likea good life [17:11:16] (I may or may not have watched too many bad spy movies) [17:12:18] I don't think I could deal with the heat, though :( [17:16:17] gotta run to post office, bbiab [17:16:31] too hot? this looks cold to me :P http://www.weatherbase.com/weather/weather.php3?s=82648&cityname=Lima-Peru [17:26:45] Analytics-EventLogging: EventLogging ValidateSchemaTest::testValidEvent() fails under HHVM - https://phabricator.wikimedia.org/T78680#850963 (hashar) NEW [17:38:18] Analytics-Engineering: Split ua-parser UDF - https://phabricator.wikimedia.org/T78685#851017 (Ironholds) NEW [17:42:45] Analytics-Engineering: Include Wikimedia-specific spiders in ua-parser UDF - https://phabricator.wikimedia.org/T78686#851031 (Ironholds) NEW a:Ironholds [17:42:59] Analytics-Engineering: Include Wikimedia-specific spiders in ua-parser UDF - https://phabricator.wikimedia.org/T78686#851031 (Ironholds) [17:43:54] Analytics-Engineering: Oozie tutorial - https://phabricator.wikimedia.org/T78687#851042 (ggellerman) NEW [18:05:10] Analytics-EventLogging: find a better way to identify events that fail validation as early as possible - https://phabricator.wikimedia.org/T78355#851094 (kaldari) > It's possible (and easy) to set something up that watches invalid events in real-time and does something with them. The question is: what? E-mail... [18:11:04] ironholds: for the user agent parser udf you mean creating a static class that does the parsing? [18:11:28] Ironholds: that is done by UA parser already right? [18:11:42] nuria__, see ottomata|afk 's pageview identifier, where the identifier is one file and the UDF wrapper is another [18:12:02] If people think the current bundling of the tool is cool, that's cool :). I'll just add the WM-specific spiders and limit my changes to that [18:12:10] Ironholds: right, but in that case there is no external library like ua parser that is doing all the lifting [18:12:23] ahh, fair point [18:12:25] Ironholds: we are just wrapping ua parser a bit [18:12:27] okay, let's leave it :) [18:12:37] I'll incorporate the spiders as I would anyway and close that phab ticket [18:12:44] Ironholds: otherwise i feel we are wrapping the wrapper [18:12:51] so we can wrap while we wrap [18:12:54] Ironholds: also check Generic udf versus udf [18:13:01] will do! [18:13:23] Analytics-Engineering: Include Wikimedia-specific spiders in ua-parser UDF - https://phabricator.wikimedia.org/T78686#851222 (Ironholds) p:Triage>Normal [18:13:41] Analytics-Engineering: Include Wikimedia-specific spiders in ua-parser UDF - https://phabricator.wikimedia.org/T78686#851031 (Ironholds) [18:13:42] Analytics-Engineering: Split ua-parser UDF - https://phabricator.wikimedia.org/T78685#851223 (Ironholds) Open>declined a:Ironholds [18:24:46] Phabricator, Engineering-Community, Analytics-Tech-community-metrics: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#851249 (Aklapper) >>! In T1003#850735, @Dzahn wrote: > let me know if you don't receive it daily from now on Uh, //daily//? The idea was monthly (... [18:25:12] phew, man post office is busy! [18:26:14] nuria__: Ironholds, ja i remember this now [18:26:24] we didn't add a refinery.core class because ja, it was just a ua-parser wrapper [18:26:28] ottomata: yessss [18:26:29] but, Ironholds wants to add extra logic, right? [18:26:36] yup [18:26:46] so if that is the case, then there should be a new refinery.core class [18:26:51] just a pattern and then a check against it in the "else" of "if(device.family != NULL)" [18:26:54] hrm [18:26:54] so that tools other than hive can share Ironholds' new logic [18:27:14] Ironholds, ottomata [18:27:18] i see [18:27:37] oh, that makes sense [18:27:56] ottomata: but do all those calsses taht we wnat to share need to be static [18:27:57] ? [18:28:01] and then we have the ua-parser UDF inherit from that as well [18:28:02] sensible [18:28:06] that kind of does not sound so great [18:28:11] *that [18:28:22] (FWIW, in a health check, so may not have the best responsiveness) [18:28:31] Ironholds: no inherit please rather composition [18:28:33] no [18:28:36] gotcha [18:28:41] they do not need to be static necessarily [18:28:49] i just haven't seen a need for anything non-static yet [18:29:06] I created a Webrequest class for pageviews stuff just now, not sure that was a good idea [18:29:17] but, perhaps it will turn into a generic Webrequest object model, iunno [18:29:34] i have that for Revisions in the avro stuff i'm doing [18:29:49] if that were the case [18:29:56] then new Webrequest(..) might make sense [18:30:12] but, for hive UDFs (main use case right now), static functions is all we need [18:31:41] ottomata: last word goes to qchris in CR he can comment on that but unit testing of loads of static functionality might not be so friendly [18:32:11] ottomata: specially if some static functions end up using others, you will not be able to do partial testing [18:32:11]