[00:04:22] arghhhh, just as I was backlog grooming, phab went offline [00:04:48] took a while to figure why everything I clicked on in Phab went to a wiki page on Phab vs RT [00:09:33] (PS10) OliverKeyes: [WIP] UDF for classifying pageviews according to https://meta.wikimedia.org/wiki/Research:Page_view/Generalised_filters [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180023 (owner: Ottomata) [00:10:48] (CR) OliverKeyes: [WIP] UDF for classifying pageviews according to https://meta.wikimedia.org/wiki/Research:Page_view/Generalised_filters (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180023 (owner: Ottomata) [00:23:12] (PS11) OliverKeyes: [WIP] UDF for classifying pageviews according to https://meta.wikimedia.org/wiki/Research:Page_view/Generalised_filters [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180023 (owner: Ottomata) [00:34:59] (CR) QChris: [C: -1] Add overall counts for URLs (1 comment) [analytics/blog] - https://gerrit.wikimedia.org/r/180369 (owner: Ori.livneh) [00:38:29] ori: With all the schema naming stuff merged, I'd like to deploy a new EventLogging tomorrow. [00:38:47] But I am not sure what to do with the "minor tidy" change. [00:39:07] The deprecated function is getting in the way. [00:40:26] Do you think it's worth for the deploy to wait for the tidy (and corresponding speedup)? [01:45:34] kevinator: I tried to ping you about that earlier [01:45:43] sorry I forgot to mention at standup [01:46:10] milimetric: yeah, I saw the email about it going down… then forgot [01:46:28] it only took 2 minutes to figure out what was going on [01:46:47] k, good [01:46:56] have a nice night [01:51:19] good night [01:55:19] hey guys, we made some changes to limn-mobile-data and would like to see them live. Could you please deploy those changes? Thanks https://gerrit.wikimedia.org/r/#/c/178381/ [04:45:23] YuviPanda, o/ [07:31:14] rtnpro: heya! [08:22:54] YuviPanda, what's up? [08:23:04] YuviPanda, I took some long break :\ [08:23:16] YuviPanda, started working on my tasks from today :) [08:23:28] rtnpro: coool! :) [08:23:29] welcome back [08:24:28] :) [08:26:14] YuviPanda, is Github down at your end? [08:26:49] rtnpro: no, works for me... [08:27:40] YuviPanda, may be some network issue in office :\ [08:49:52] MediaWiki-extensions-MultimediaViewer, Analytics, Multimedia: Update the SQL queries for the new versions of the schemas - https://phabricator.wikimedia.org/T78312#931881 (Gilles) Open>Resolved [09:36:29] Analytics, Multimedia, MediaWiki-extensions-MultimediaViewer: Update network performance schema queries - https://phabricator.wikimedia.org/T84852#932045 (Gilles) NEW a:Gilles [09:51:41] (PS1) Gilles: Update network performance schema queries [analytics/multimedia] - https://gerrit.wikimedia.org/r/180744 [10:02:00] Analytics-EventLogging: EventLogging ValidateSchemaTest::testValidEvent() fails under HHVM - https://phabricator.wikimedia.org/T78680#932121 (QChris) I saw that the schema verifier took different code paths when figuring out data types. But HHVM's choice looked saner and right. The fact that the test passed... [10:47:44] YuviPanda, can you review this: https://github.com/wikimedia/analytics-limn-mobile-data/pull/1/files ? [10:48:12] YuviPanda, does the logger integration look flexible? [10:59:59] rtnpro: I do not know much about that code at all, sadly. [11:00:08] rtnpro: you’d have to wait for milimetric... [11:00:23] YuviPanda, no problem, I will ping him :) [11:05:36] Analytics-EventLogging: EventLogging ValidateSchemaTest::testValidEvent() fails under HHVM - https://phabricator.wikimedia.org/T78680#932191 (hashar) Ideally we would have a test highlighting the issue. Your call on whether it is important to investigate, my main point was to have the tests to pass in both Ze... [14:26:37] morning. [14:26:41] (CR) Milimetric: "I deployed the dashboard changes to limn1, everything looks good to me." [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/178381 (owner: Bmansurov) [14:26:49] mornin [14:26:52] how are you Ironholds [14:27:01] same as I always am. You? [14:29:28] I am scatter-minded and distracted [14:29:34] so not the same as always - which is interesting [14:31:29] he [14:31:30] h [14:36:09] (PS12) OliverKeyes: [WIP] UDF for classifying pageviews according to https://meta.wikimedia.org/wiki/Research:Page_view/Generalised_filters [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180023 (owner: Ottomata) [14:36:29] (CR) OliverKeyes: [WIP] UDF for classifying pageviews according to https://meta.wikimedia.org/wiki/Research:Page_view/Generalised_filters (4 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180023 (owner: Ottomata) [14:41:17] Analytics-Refinery: Dedupe data while doing ETL for the Analytics cluster - https://phabricator.wikimedia.org/T76724#932541 (kevinator) p:Triage>Normal [14:44:43] darnit [14:44:51] implicit conversion of booleans to booleanwriteables is a pain. [14:44:52] Analytics-Wikimetrics: Re-run Wikimetrics data once Labs issues are fixed [8 pts] - https://phabricator.wikimedia.org/T78305#932545 (Milimetric) babysitting on 12-18: 99 jobs did not finish out of 10224. That means that labs still can not backfill more than a day. This concludes our babysitting because we... [14:45:05] Ironholds: :) [14:45:50] also, this project is a really good argument of why you write the tests first [14:45:54] it makes experimentation so convenient [14:46:29] Analytics-Wikimetrics: Re-run Wikimetrics data once Labs issues are fixed [8 pts] - https://phabricator.wikimedia.org/T78305#932554 (Milimetric) [14:46:56] Analytics-Wikimetrics: Re-run Wikimetrics data once Labs issues are fixed [8 pts] - https://phabricator.wikimedia.org/T78305#932555 (Milimetric) Open>Resolved Babysitting ended sadly, so this is done. Including the dancing clown. [14:47:23] even if my conclusion is "None of my experimentation is working." [14:47:47] I do that for dashiki and wikimetrics sometimes and it makes me feel like I just took a long relaxing bath [14:47:52] (test first) [14:50:19] Analytics-Dashiki, Analytics-Engineering: Vital Signs user knows to click on metric title to open definition - https://phabricator.wikimedia.org/T76741#932575 (kevinator) p:Normal>High [14:52:17] Analytics-Engineering: Update Gerrit Documentation - https://phabricator.wikimedia.org/T77059#932588 (kevinator) [14:56:00] (PS13) OliverKeyes: [WIP] UDF for classifying pageviews according to https://meta.wikimedia.org/wiki/Research:Page_view/Generalised_filters [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180023 (owner: Ottomata) [14:56:18] (CR) OliverKeyes: [WIP] UDF for classifying pageviews according to https://meta.wikimedia.org/wiki/Research:Page_view/Generalised_filters (5 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180023 (owner: Ottomata) [14:56:28] milimetric, also, writing tests is just FUN [14:57:03] ottomata, qchris_away, comments on that patch are now divided into "Oliver has fixed, or at least broken in a new way that does not cause compile errors or trigger the test cases", and "Oliver is too much of a n00b to fix". [14:57:29] If meestar otto could take a quick spin through tweaking the pom files and one of the regexes I think (?) it'll be done, for the time being. [14:57:49] ahhh! yes. i can do, but it will have to wait a bit, I have to do standup and then I have a stat* box downtime scheduled! [14:57:53] which I should email about as a reminder! [14:58:52] Analytics-EventLogging, Analytics-Engineering: Engineer opens an EventLogging validation error log - https://phabricator.wikimedia.org/T84879#932608 (kevinator) NEW a:kevinator [14:59:46] Analytics-EventLogging, Analytics-Engineering: Engineer opens an EventLogging validation error log - https://phabricator.wikimedia.org/T84879#932608 (kevinator) a:kevinator>None [15:01:24] ...you should because I'm building my presentation for this afternoon! :D [15:11:26] Analytics: Fix Varnishkafka delivery error icinga warning - https://phabricator.wikimedia.org/T76342#932672 (Ottomata) [15:18:28] Analytics-EventLogging, Analytics-Engineering: Engineer opens an EventLogging validation error log - https://phabricator.wikimedia.org/T84879#932681 (Nuria) I feel that this is kind of filed "too fast". On vagrant the validations are already logged by the dev server, which devs should be using. On vanadium va... [15:28:21] Analytics-Wikimetrics: Story: WikimetricsUser deletes user from cohort [21 pts] - https://phabricator.wikimedia.org/T75350#932709 (kevinator) From: Jonathan Date: Wed, Dec 17, 2014 at 12:16 PM Subject: Re: Delete User is on staging To: Kevin Bug is still there for me. Screenshots attached. The cohort is a ra... [15:31:16] Analytics-EventLogging, Analytics-Engineering: EL office hours - https://phabricator.wikimedia.org/T76796#932722 (kevinator) p:Normal>High [15:33:25] Analytics: Move stat1001, stat1002 and stat1003 into Analytics VLAN - https://phabricator.wikimedia.org/T76346#932730 (BBlack) Specifics on the old/new network settings: https://phabricator.wikimedia.org/P169 [15:41:04] Ironholds: tried to run the apps-uniques query for just 15 days and it is taking over a day, was that your experience before? [15:41:19] yup :(. Hence only running it once a week [15:49:38] Analytics-EventLogging: Do analysis for SendBeaconReliability experiment - https://phabricator.wikimedia.org/T78110#932751 (Nuria) First round of data analysis: https://www.mediawiki.org/wiki/Extension:EventLogging/SendBeacon [15:51:07] Analytics-EventLogging: Do analysis for SendBeaconReliability experiment [5 pts] - https://phabricator.wikimedia.org/T78110#932752 (kevinator) [16:07:33] Ironholds: regarding opt-out and appInstallId [16:08:19] Ironholds: if a user has opted out does appInstallID still come with a appInstallID=0 [16:08:46] at the moment I don't think they can opt-out, just opt-out of EL [16:08:51] but that's a Deskana question, I think. [16:10:54] As far as I know, it will still send the app install ID with the read actions [16:11:30] AFAIK the check on whether it's enabled or not happens in the parent class of all of the EventLogging funnels [16:11:37] And read actions don't touch that class at all [16:26:17] Deskana: maybe you can talk to kevinator about opt-out abit more? Cause it seems if a user has opted out we should not have its distinct appinstallid linked to his activity, correct? [16:30:37] Or we should at least clarify the text around the opt-out [16:30:56] I mean: we are using their data, even if they opt-out [16:45:53] Analytics-Dashiki, Analytics-Engineering: Vital Signs user knows to click on metric title to open definition [3 pts] - https://phabricator.wikimedia.org/T76741#932967 (kevinator) [16:47:47] qchris: hey do you know the history of https://gerrit.wikimedia.org/r/#/c/157841/ ? [16:47:55] we were trying to see if there's any work we can do to help it along [16:48:33] The history is that setting the X-Analytics in varnish is tedious and error-prone. [16:48:59] It should be possible for extensions to set X-Analytics without doing crazy things and without overwriting X-Analytics of other extensions, [16:49:08] and with have to duplicate logic. [16:49:16] oh we love the extension [16:49:23] O-ri came around and started the extension. [16:49:26] but see that a required change to MobileFrontEnd is needed [16:49:29] and saw it was done [16:49:31] then reverted [16:50:08] so i was curious if this change is ready to be merged or it's blocked on changes to MobileFrontEnd still [16:50:22] I only knew that the XAnalytics extension and MobileFrontend would get in the way of each other. Hence I flagged it. [16:50:42] But I haven't checked what MobileFrontEnd ended up doing about it. [16:51:23] so Ori commented that this would make MobileFrontEnd play nic: https://gerrit.wikimedia.org/r/#q,Id22eaf5dc,n,z [16:51:25] *nice [16:51:40] Yup. I saw that, but could not check up to now. [16:51:55] and then this was reverted: https://gerrit.wikimedia.org/r/#/c/171188/ [16:52:11] Analytics-EventLogging: find a better way to identify events that fail validation as early as possible - https://phabricator.wikimedia.org/T78355#933005 (Nuria) Kaldari: In vagrant if you use the event logging dev server you will see validation errors there, do you think you will need something beyond that?... [16:53:05] so then you're not familiar with why that got reverted, right? [16:53:15] no, I am not. [16:53:26] thanks, ok [16:54:14] ori: when you're around, maybe let us know why https://gerrit.wikimedia.org/r/#/c/171188/ was done, and if you have plans for https://gerrit.wikimedia.org/r/#/c/157841/ [16:58:45] Analytics-EventLogging: find a better way to identify events that fail validation as early as possible - https://phabricator.wikimedia.org/T78355#933029 (Nuria) Also in beta labs the logs on failed validation are available for anyone to look at: root@deployment-eventlogging02:/var/log/upstart# tail... [16:59:15] Analytics-EventLogging: find a better way to identify events that fail validation as early as possible - https://phabricator.wikimedia.org/T78355#933031 (Nuria) argh, sorry about formatting. [17:00:56] Analytics-EventLogging: find a better way to identify events that fail validation as early as possible - https://phabricator.wikimedia.org/T78355#933035 (Nuria) Also, note these same logs: eventlogging_processor-client-side-events.log, eventlogging_processor-server-side-events.log exist in vanadium (they are... [17:06:03] qchris: Ironholds and nuria__ can explain about the uuid patch i just added you to. but, in a nutshell, this is to support uuids for the app not in the the request path. [17:07:47] dr0ptp4kt, yeah, I know! [17:07:49] I read it [17:07:55] I'll leave reviewing that to others and remove myself again. [17:08:03] have one possible comment about the inclusion of the https=1 logic on desktop, but otherwise don't have an opinion [17:08:16] just grabbing some sampled logs so I can validate/invalidate my suspicion [17:10:22] comment left [17:10:29] (suspicion invalidated ;p) [17:12:46] fyi - rebooting limn1 server because it's acting up and i'm teaching it a lesson [17:13:25] wait, hangon [17:13:44] if HTTPS requests are going to be tagged as such, now... [17:13:51] that is, for all text/mobile requests... [17:14:10] ...this removes a tremendous amount of fuzziness around when to trust internal IPs, no? [17:14:19] or, when to grab x_forwarded_for [17:18:46] Analytics-Visualization, Analytics-Engineering: PM shares a deep link into Limn Dashboard [8 pts] - https://phabricator.wikimedia.org/T78743#933135 (kevinator) [17:19:21] ottomata: uh... problem [17:19:30] limn1 died after I rebooted it [17:19:40] and it won't come up, won't let me ssh, console output is blank [17:21:23] qchris: I just invited you to a meeting with Dario and Nuria this afternoon to talk about troubleshooting event logging. [17:21:41] qchris: you are welcome to join us, let me know if you can [17:21:56] qchris: sorry for the late invite [17:22:16] Yay for late meetings on short notice :-) [17:22:52] milimetric: Instance state is "ERROR" that's a case for the labs channel I guess. [17:23:04] Oh you're there already :-) [17:23:08] :) [17:23:09] yep [17:23:19] qchris: you're optional on that meeting I think [17:23:30] we just thought it nice to invite you since you've been doing so much EL stuff lately [17:23:35] just in case you wanted to attend [17:23:55] milimetric: k. Thanks. [17:24:08] Analytics-Engineering: Improve logging - https://phabricator.wikimedia.org/T84892#933170 (rtnpro) NEW a:rtnpro [17:28:55] milimetric, ping [17:34:58] ottomata: [17:35:02] you busy? [17:35:32] ah [17:35:33] yes [17:35:36] doing stat vlan move [17:35:37] UhMmM [17:36:02] you have time later today tnegrin? or are you your usual in-meeting self? [17:36:22] rtnpro: hi [17:36:47] let's do it tomorrow -- if you have serious tech work to do, please focus on that [17:36:53] I don't have anything urgent [17:36:53] milimetric, I am trying to push my work for review [17:36:55] k? [17:37:07] k [17:37:14] (to rtnpro not tnegrin ) [17:37:36] milimetric: don't cross the streams! [17:37:56] there is actually a puff marshmallow monster though - limn1 is down [17:37:58] milimetric, I am getting this error: http://paste.fedoraproject.org/161037/18924242/ [17:37:58] tnegrin: cool let's talk tomorrow [17:38:06] kk [17:38:11] milimetric, any idea how to get around it [17:38:17] milimetric: need anything? [17:39:10] no, labs folks are helping us [17:39:18] but just fyi - they may be down for a while [17:39:23] why? [17:39:25] it's an OOM error on the virt host limn1 is on [17:39:30] oh my [17:39:32] and we can't reach the instance at all [17:39:44] so we're trying to shut down other unrelated stuff to see if it works [17:39:44] kevinator, milimetric should we send out an email? [17:40:02] rtnpro: I think that means you don't have rights... qchris can I add him or do I have to bother you? [17:40:19] * qchris reads backscroll [17:40:35] tnegrin: it was sent out. I didn’t get to close the task yesterday. [17:40:49] no -- for the limn downtime [17:41:10] oh - my head’s on the task board [17:41:18] yes, I think so… [17:41:19] rtnpro: Looks like you'r trying to create a ref in gerrit, which gerrit does not allow you. [17:41:32] rtnpro: What does your push command look like? [17:41:36] tnegrin: not sure yet, if the problem persists over the next 30 min. maybe [17:41:36] product teams are going to open their dashboards and wonder what’s going on. [17:41:48] milimetric, git push gerrit integrate_logging [17:42:02] rtnpro: Make that: [17:42:12] kevinator: I'd prefer to send an email -- at least that give us something to do and not bug devs :) [17:42:15] git push gerrit integrate_logging:refs/for/master [17:42:18] rtnpro: ^ [17:42:48] qchris, ok, trying [17:42:55] Limn is alive again [17:43:05] tnegrin: looks like we don’t need to send an email [17:43:14] (PS1) Rtnpro: Integrated logging [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/180828 [17:43:15] yay [17:43:36] it's still early in sf [17:43:45] rtnpro: Looks like it worked \o/ [17:43:57] qchris, yes :) [17:43:59] thanks rtnpro! I'll add myself to review [17:44:03] thanks qchris :) [17:44:07] yw [17:44:41] qchris, thanks :) [17:45:32] milimetric, when you get time, have a look at the above review request [17:46:37] i shall! thanks rtnpro - little hectic at the moment but will do soon [17:46:54] milimetric, np :) [17:49:36] (PS1) Milimetric: Fix links, stats1001 hostname is dead [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/180839 [17:49:38] (CR) jenkins-bot: [V: -1] Fix links, stats1001 hostname is dead [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/180839 (owner: Milimetric) [17:51:17] (PS2) Milimetric: Fix links, stats1001 hostname is dead [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/180839 [17:51:33] (CR) Milimetric: [C: 2 V: 2] Fix links, stats1001 hostname is dead [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/180839 (owner: Milimetric) [18:06:14] (CR) QChris: [WIP] UDF for classifying pageviews according to https://meta.wikimedia.org/wiki/Research:Page_view/Generalised_filters (3 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180023 (owner: Ottomata) [18:08:51] Analytics-EventLogging, Analytics-Engineering: Epic: WMF Engineer reads documentation to set up a dashboard from EL data - https://phabricator.wikimedia.org/T76362#933354 (kevinator) [18:10:02] Analytics-EventLogging: find a better way to identify events that fail validation as early as possible - https://phabricator.wikimedia.org/T78355#843193 (kevinator) [18:10:04] Analytics-EventLogging, Analytics-Engineering: Engineer opens an EventLogging validation error log - https://phabricator.wikimedia.org/T84879#933360 (kevinator) Open>declined a:kevinator T78355 already covers the meat of this issue. Closing this ticket. [18:10:16] Analytics-EventLogging: Engineer opens an EventLogging validation error log - https://phabricator.wikimedia.org/T84879#933366 (kevinator) [18:14:17] (PS14) OliverKeyes: [WIP] UDF for classifying pageviews according to https://meta.wikimedia.org/wiki/Research:Page_view/Generalised_filters [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180023 (owner: Ottomata) [18:14:30] (CR) OliverKeyes: [WIP] UDF for classifying pageviews according to https://meta.wikimedia.org/wiki/Research:Page_view/Generalised_filters (2 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180023 (owner: Ottomata) [18:16:52] Ironholds: about the String vs. Text in the UDF. [18:17:06] The errors you get is that during compilation or during testing? [18:17:54] compilation of the tests, looks like. So, yes! :D [18:18:05] [ERROR] /home/ironholds/Code/source/refinery-hive/src/test/java/org/wikimedia/analytics/refinery/hive/TestIsPageviewUDF.java:[52,15] error: method evaluate in class IsPageviewUDF cannot be applied to given types; [18:18:47] Did you update the tests too to pass Strings instead of Text? [18:21:59] Ironholds: ^ [18:22:08] ..*headdesks* [18:38:42] qchris, error is now "/home/ironholds/Code/source/refinery-hive/src/test/java/org/wikimedia/analytics/refinery/hive/TestIsPageviewUDF.java:[59,13] error: boolean cannot be dereferenced" [18:41:24] Ok. Sounds like you fixed the "Text"s to "String"s in the test ... did you also fix the "BooleanWritable" to boolean in the test ;-) [18:41:32] Ironholds: ^ [18:44:29] I can hear you judging me ;p [18:45:12] I am not judging ... I am heading to dinner. That's better :-) [18:46:07] Ironholds: qchris, i am not sure if httpStatus should be int or text [18:46:09] i left it at text [18:46:16] but almost changed it to int yesterday [18:46:33] text [18:46:42] it's a string in the system [18:46:44] ja? [18:46:51] aye, but it is conceptually? [18:46:59] what would it be if we stored it in a binary format? [18:47:36] noo idea [18:47:43] depends on if the lines from kraken ever get borked [18:48:07] while you're here, can you take a look at the remaining notes on patch 5? I'm worried about breaking stuff if I fix them [18:48:16] and the sooner we get this in the sooner we can start using it as a template [18:50:12] ottomata: is the info on how to test oozie jobs in this page still current: https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Oozie [18:50:55] Ironholds: I think I will ahve time today to do that, but don't at the moment, i'm still making sure stat* boxes are healthy [18:51:34] nuria, it isn't irrelevant, but it is probably not up to date [18:51:48] if I rewrote it now there are probably things I would change [18:52:20] ta [18:53:31] ottomata: any docs i can look at? [18:55:47] ottomata: on how to test/run a job in oozie? [18:56:13] um, oozie docs? [18:56:16] not sure what you mean by test/run [18:56:37] nuria [18:56:38] https://oozie.apache.org/docs/3.1.3-incubating/DG_CommandLineTool.html#Job_Operations [18:56:55] (CR) Gergő Tisza: [C: 2] Update network performance schema queries [analytics/multimedia] - https://gerrit.wikimedia.org/r/180744 (owner: Gilles) [18:58:13] (Merged) jenkins-bot: Update network performance schema queries [analytics/multimedia] - https://gerrit.wikimedia.org/r/180744 (owner: Gilles) [18:58:28] ottomata: how can i setup a test job and run it [18:59:40] nuria, if you want just a workflow job. at minimum you need: workflow.xml and workflow.properties [18:59:45] workflow.xml will have to be in hdfs somewhere [18:59:58] workflow.properties will reference where it is [19:00:01] then you can do [19:00:28] oozie job -config workflow.properties -run [19:03:13] (CR) Gergő Tisza: [C: 2] Add scroll metadata open/close events to dashboards [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/180501 (owner: Gilles) [19:10:28] milimetric, did you find out something with wikimetrics? [19:10:54] oh milimetric, now I've seen the email [19:18:25] (CR) Gergő Tisza: [V: 2] Add scroll metadata open/close events to dashboards [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/180501 (owner: Gilles) [19:20:01] Analytics, MediaWiki-extensions-MultimediaViewer, Multimedia: Update network performance schema queries - https://phabricator.wikimedia.org/T84852#933605 (Tgr) [19:36:58] Wikimedia-Logstash, Analytics-Engineering: Kafka logging to Logstash - https://phabricator.wikimedia.org/T84907#933668 (Gage) NEW [19:38:39] Wikimedia-Logstash, Analytics-Engineering: Zookeeper logging to Logstash - https://phabricator.wikimedia.org/T84908#933675 (Gage) NEW [19:41:11] mforns: what do you think? was that the bug? [19:41:25] milimetric, yes seems so [19:41:52] what I don't get is why the second user is not valid? [19:42:16] and the 'invalid_username' reason [19:42:23] but I´ll find out [19:42:42] the first thing - why the second user is not validated - is because the list of wikiusers is turned into a dictionary by validate_users [19:42:59] the second thing - why they're coming up as 'invalid_username' - I don't know, but that may be related too [19:43:24] ok [19:43:33] well, thanks for your help! [19:43:47] I'm going to file a bug in phab [19:45:19] ottomata: how can i put workflow.xml in hdfs? [19:45:48] hdfs dfs -put [19:45:49] :) [19:46:50] ottomata: but "anywhere i like?" [19:47:25] milimetric, in fact, I think I will use the existing bug... The problem is the same, right? My original fix was not exhaustive, so this should be the same task. If it's not reopened, I will reopen it. [19:47:59] mforns: that sounds fine, no strong opinions [19:48:12] ok [19:49:13] nuria__: yeah, yoi have to set the app path in the .properties file [19:49:16] so that is wherever you put it [19:59:07] ottomata: about the int vs string for http_status. [19:59:22] While it's a number conceptually, we do not only see numbers there. [19:59:56] So we need a marker for "Something else" [20:00:17] Currently, that's "-" ... which is not perfect on its own ... but ok. [20:00:17] aye ok, sounds good. [20:00:25] makes sense [20:15:34] Analytics: Move stat1001, stat1002 and stat1003 into Analytics VLAN - https://phabricator.wikimedia.org/T76346#933756 (Ottomata) Open>Resolved Woot! Done! As far as I can tell, all is good. There will surely be some issues that pop up here and there. I will work with them as they happen. [20:30:01] ottomata: so if oozie is running a hql job does the hql file need to be copied somewhere to hdfs too? [20:31:54] Analytics-Dashiki, Analytics-Engineering: Vital Signs user knows to click on metric title to open definition [3 pts] - https://phabricator.wikimedia.org/T76741#933826 (Milimetric) a:Milimetric [20:32:18] (PS1) Milimetric: Add external link icon to metric name [analytics/dashiki] - https://gerrit.wikimedia.org/r/180882 [20:33:51] (CR) Milimetric: "small side note: at first I tried upgrading to the new Semantic as a lot has changed. Turns out too much has changed and the upgrade brea" [analytics/dashiki] - https://gerrit.wikimedia.org/r/180882 (owner: Milimetric) [20:37:21] nuria__: yes [20:37:26] everything is relative to the app path [20:38:28] I'll be gone for a while - nuria__ I'll look at the queries and let you know though [20:38:38] milimetric: k [20:42:31] hey guys, we made some changes to limn-mobile-data and would like to see them live. Could you please deploy those changes? Thanks https://gerrit.wikimedia.org/r/#/c/178381/ [20:42:55] bmansurov: I deployed those this morning [20:42:59] are there more? [20:43:26] milimetric: oh thanks, no that was it. I thought you missed my request yesterday [20:44:02] i did, but saw it this morning [20:44:10] did you see my note there about doing it yourself? [20:44:15] with limn-deploy? [20:44:39] milimetric: no I haven't seen it. was it addressed to me? [20:44:50] oh in the patch? [20:45:00] by the way bmansurov two graphs on the "other" tab look like they're not getting data - maybe the queries didn't work? [20:45:06] yes, the comment was in the patch [20:45:32] milimetric: i see your comment. thanks. [20:45:43] milimetric: maybe, i'll need to check them [20:45:48] cool, i'm gonna step out for a bit but let me know here / email if you have trouble [20:46:27] milimetric: ok thanks [20:48:31] ottomata: thanks for the note on stat boxes. I had a hiccup with mysql connectivity but it reconnected and was fine [20:50:14] cool [20:50:35] ok, i'm off to a care, got a couple more hours of work left in me, gonna use it help iron holds with his udf [20:50:37] back in a bit [20:52:14] Analytics-Wikimetrics: Story: WikimetricsUser deletes user from cohort [21 pts] - https://phabricator.wikimedia.org/T75350#933910 (mforns) We found a bug that causes this behavior. It is an old bug, that was made evident by an improvement on centralauth expansion included in this task. Namely, when there are... [20:52:34] Analytics-Wikimetrics: Story: WikimetricsUser deletes user from cohort [21 pts] - https://phabricator.wikimedia.org/T75350#933911 (mforns) Resolved>Open [20:52:35] Analytics-Wikimetrics, Analytics-Engineering: Epic: WikimetricsUser deletes user from cohort - https://phabricator.wikimedia.org/T76421#933912 (mforns) [21:03:07] (CR) OliverKeyes: "A thought occurs: could we somehow exclude the (True|False) from the test cases? That way we could also use the tests on (e.g.) the legacy" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180023 (owner: Ottomata) [21:20:39] ok Ironholds, what's up with your change now? [21:20:47] there are so many coments! [21:20:49] comments* [21:20:56] ah pom [21:20:57] lgot it [21:21:02] yeah want to roganize that [21:35:29] (CR) Nuria: "Small nit: Can we make it so icon is not "underlined" on hover and lower it a bit so it lines up with bottom line of metric tittle?" [analytics/dashiki] - https://gerrit.wikimedia.org/r/180882 (owner: Milimetric) [21:38:49] qchris: I'm not sure how to remove ${project.version} from the refinery-hive/pom.xml dependency of refinery-core [21:39:26] just put it in the parent pom.xml [21:39:42] Then you can drop the version line from refinery-hive's pom.xml [21:43:56] where in the paretn pom? [21:44:04] make refinery-core a depenency of the parent refinery pom? [21:44:11] qchris: ^ [21:44:47] would seem weird to make the parent depend on the child [21:46:21] The tags are , that's true. But they are within tag. So they are not real effective dependencies, but [21:46:35] that's more a library of dependencies that a child pom can easily pick from. [21:46:49] Only in the child pom, they become effective dependencies. [21:49:29] hm [21:49:30] ok [21:49:31] cool [21:49:45] so we should list all sub modules in the parent po section [21:50:51] Only the ones that we need. [21:51:09] I'd just add them as they come along. [21:51:12] only ones that other sub modules will need to depend on? [21:51:13] ok [22:07:36] (PS15) Ottomata: [WIP] UDF for classifying pageviews according to https://meta.wikimedia.org/wiki/Research:Page_view/Generalised_filters [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180023 [22:07:39] qchris: woot! [22:07:50] (CR) Ottomata: [WIP] UDF for classifying pageviews according to https://meta.wikimedia.org/wiki/Research:Page_view/Generalised_filters (6 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180023 (owner: Ottomata) [22:08:05] :-D [22:08:28] yay! [22:09:06] (CR) Ottomata: "> A thought occurs: could we somehow exclude the (True|False) from the test cases? That way we could also use the tests on (e.g.) the lega" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180023 (owner: Ottomata) [22:09:35] qchris: I did not know hive would be all slick and auto cast the types [22:09:36] ! [22:09:45] s/cast/convert) [22:10:13] For GenericUDF it isn't. [22:10:18] Only for UDF. [22:10:25] That's nice, isn't it :-) [22:12:17] oh. we converted this? [22:12:21] it was GenericUDF, ja? [22:12:47] Not sure what it was initially ... now it is a plain UDF, yes. [22:13:09] oh, sorrry, i got them backwords [22:13:16] plain UDF is the simple one [22:13:17] yeah [22:13:22] cool! [22:13:47] qchris: Ironholds, I split the uriHostPattern into two regexes [22:13:54] Thanks! [22:13:56] one to match wikimedia domains, the other to match project domains [22:14:04] cool [22:14:11] will do the same with the LegacyPageviews class I'm building [22:14:15] what a great collaborative change this is! [22:14:36] got the LP class built, pretty much; that regex was the last bit [22:14:44] my only open question was around speed - regexes versus strings [22:15:02] like, is substringing and comparing to a host of IP ranges, as the hive does it, the optimised thing within Java, too? [22:15:32] or is "(ip|ip|ip|ip|ip)\\..+" as a regex likely to be not noticeably slower? [22:15:44] because the substringing in the hive query makes things sometimes hard to read [22:15:58] Ironholds: Optimize when things are slow. Not beforehand. [22:16:04] I do not know! I tend to think regexes are slower [22:16:06] Do the simplest thing that could possibly work. [22:16:10] qchris, hey, it's the right answer 3% of the time :D [22:16:13] * Ironholds rimshots [22:16:29] Strings are simpler ... start with strings. [22:16:42] ahh...already built it. whoops ;p [22:16:42] Once it gets messy, use a library for it :-P [22:17:09] nuria__: hi sorry, been busy all day! how's oozielife? [22:30:09] qchris: did you end up deploying EventLogging? [22:30:29] yup. [22:30:35] I thought I logged it. [22:30:40] * qchris checks again [22:31:02] oh, I didn't check the logs, sorry. [22:31:17] I deployed 937d804 [22:32:11] I saw the CR-1 on associative array thing :-) [22:32:25]