[05:53:47] YuviPanda, Hey [08:28:59] (PS1) Gergő Tisza: File page loading times are not experimental any more [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/176638 [08:54:55] Analytics-Wikimetrics: Cohort listing going against the database instead of using CohortService - https://phabricator.wikimedia.org/T68126#795945 (Liuxinyu970226) [09:59:23] YuviPanda, ping [11:50:41] Analytics: Network outage on 2014-11-30 affecting analytics infrastructure - https://phabricator.wikimedia.org/T76334 (QChris) NEW p:Triage [11:51:00] Analytics: Network outage on 2014-11-30 affecting analytics infrastructure - https://phabricator.wikimedia.org/T76334#796158 (QChris) a:QChris [12:01:34] rtnpro: heya! [12:01:38] sorry was away [12:01:47] YuviPanda, no problem :) [12:02:09] YuviPanda, are you free now? [12:02:17] well, for some definition of free :) [12:02:18] 'sup [12:02:28] hehe [12:03:07] there's a review request for you, and I wanted some suggestion on slugifying query details page URL [12:03:28] YuviPanda, I commented on the issue, you can take a look when you have time :) [12:03:35] oh, which one? [12:03:36] * YuviPanda looks [12:07:10] YuviPanda, let me fetch you the link [12:07:19] rtnpro: yeah, saw it. thihnking [12:07:29] rtnpro: so thing is titles aren't immutable, and by default the title is 'Untitled query' [12:07:29] YuviPanda, :) [12:07:35] so question is when do you sluggify it [12:07:50] so my thinking is that you would just discard everything after the first - [12:07:53] and just use that as an id [12:07:57] but by default just slugify current title [12:08:25] YuviPanda, so, the link will be only for displaying nice URL [12:08:34] yeah, [12:08:39] YuviPanda, makes sense +1 [12:08:44] but that also means it can be easily spoofed to say *anything* :) [12:08:56] YuviPanda, yes, that's the trade off [12:09:37] YuviPanda, in the approach of saving the slug in a column will be full proof in that case [12:10:08] YuviPanda, we can even slugify the text "Untitled query" to "untitled-querry" [12:10:28] YuviPanda, and whenever the title is updated, we update the slug as well [12:10:36] rtnpro: ah, so look at https://stackoverflow.com/questions/27226854/command-failed-with-exit-code-enoent-in-visual-studio-2013 [12:10:42] rtnpro: you can discard everything after the / [12:10:47] rtnpro: *but* it redirects you to that if it is missing [12:10:52] rtnpro: so I guess that might be a proper solution. [12:10:57] have it be / [12:11:08] and only check id, and if slug is wrong just redirect to correct, *current* slug [12:11:15] with a temp. redirect [12:11:33] rtnpro: yeah, but then *everything* has untitled-query- :) [12:12:09] YuviPanda, no, the slug will be '-some-text' [12:12:22] rtnpro: that is somewhat violating url structures, no? [12:12:29] YuviPanda, yes [12:12:29] hmm, or not [12:12:41] why not be id/text? [12:13:22] YuviPanda, the above approach sounds good too, I mean the redirect one on bad slugs [12:13:28] yeah [12:13:31] we should definitely do that [12:13:38] and that doesn't require a db table :) [12:13:39] Agree +1 [12:13:40] err [12:13:41] db colum [12:13:42] n [12:13:48] sluggifying is fairly quick operation [12:13:54] yes [12:14:30] I'm ok with - or / for separator [12:41:21] YuviPanda, '/' would look better :) [12:41:30] ok then :) [12:42:04] YuviPanda, who can help me with wikimetrics? [12:48:13] rtnpro: milimetric and nuria [12:48:14] I think [12:48:58] YuviPanda, they are not around, any idea when will they be available? [12:49:09] hmm, nope. probably later in the day, both are in the US [12:49:16] YuviPanda, ok [12:49:41] YuviPanda, also, we can include some test db data as well for quarry [12:49:46] YuviPanda, what say you? [12:49:55] sure, but what data exactly? [12:49:59] just random test data? [12:50:06] sure! but it needs to be updated when new things are added, etc [12:50:08] but go ahead :) [12:50:29] YuviPanda, yes, random data [12:50:34] cool. [15:01:12] ottomata: sneeeeaaky meeting time :-) [15:16:47] Analytics: Fix Varnishkafka delivery error icinga warning - https://phabricator.wikimedia.org/T76342 (Ottomata) NEW p:Normal a:Ottomata [15:20:01] Analytics: Increase HADOOP_HEAPSIZE (-Xmx) for hive-server2 - https://phabricator.wikimedia.org/T76343 (Ottomata) NEW p:Normal a:Ottomata [15:21:20] Analytics: Implement stats_print in kafkatee - https://phabricator.wikimedia.org/T76345 (Ottomata) NEW p:Normal a:Ottomata [15:23:12] Analytics: Move stat1002 and stat1003 into Analytics VLAN - https://phabricator.wikimedia.org/T76346 (Ottomata) NEW p:Normal a:Ottomata [15:24:09] Analytics: Import Mediawiki XML dumps into HDFS - https://phabricator.wikimedia.org/T76347 (Ottomata) NEW p:Normal a:Ottomata [15:24:50] Analytics: Upgrade stat1001 to Ubuntu Trusty - https://phabricator.wikimedia.org/T76348 (Ottomata) NEW p:Normal a:Ottomata [15:25:39] (CR) Gilles: [C: 2] File page loading times are not experimental any more [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/176638 (owner: Gergő Tisza) [15:26:23] Analytics: Create Geocode Hive UDF for Refinery - https://phabricator.wikimedia.org/T76349#798416 (Ottomata) [15:26:34] Analytics: Create Geocode Hive UDF for Refinery - https://phabricator.wikimedia.org/T76349#798416 (Ottomata) [15:27:46] Analytics: Explore usage of Spark with Python and streaming - https://phabricator.wikimedia.org/T76351 (Ottomata) NEW p:Triage a:Ottomata [15:27:59] Analytics: Explore usage of Spark with Python and streaming - https://phabricator.wikimedia.org/T76351#798432 (Ottomata) p:Triage>Low [15:28:14] Analytics: Create Geocode Hive UDF for Refinery - https://phabricator.wikimedia.org/T76349#798416 (Ottomata) p:Normal>Low [15:28:21] Analytics: Upgrade stat1001 to Ubuntu Trusty - https://phabricator.wikimedia.org/T76348#798442 (Ottomata) p:Normal>Low [15:28:30] Analytics: Move stat1002 and stat1003 into Analytics VLAN - https://phabricator.wikimedia.org/T76346#798443 (Ottomata) p:Normal>Low [15:28:43] Analytics: Upgrade Analytics Cluster to Trusty, and then to CDH 5.2 - https://phabricator.wikimedia.org/T1200#798444 (Ottomata) p:Normal>High [15:28:50] Analytics: Increase HADOOP_HEAPSIZE (-Xmx) for hive-server2 - https://phabricator.wikimedia.org/T76343#798445 (Ottomata) [15:29:04] Analytics: Implement stats_print in kafkatee - https://phabricator.wikimedia.org/T76345#798446 (Ottomata) p:Normal>Low [15:30:12] I like Phabricator [15:30:13] https://phabricator.wikimedia.org/maniphest/query/DrfAmlJO9y5e/ [15:30:14] :) [15:41:36] Analytics-Wikimetrics: Uploading cohort by copy-pasting breaks if names contain special characters - https://phabricator.wikimedia.org/T76105#798459 (kevinator) [15:44:53] Analytics-Wikimetrics: Epic: Grantmaking User gets reports on Wikimetrics usage - https://phabricator.wikimedia.org/T76106#798464 (kevinator) [15:50:25] Analytics-EventLogging: Epic: Simplify dashboard deployment with EventLogging data - https://phabricator.wikimedia.org/T75836#798466 (kevinator) [15:51:05] Analytics-EventLogging: WMF reads announcement and documentation on how to get a dashboard - https://phabricator.wikimedia.org/T76058#798467 (kevinator) [15:51:27] Analytics-EventLogging: Engineer runs script to create a new dashboard - https://phabricator.wikimedia.org/T75838#798468 (kevinator) [15:51:38] Analytics-EventLogging: Engineer runs script to test SQL for dashboard against a test database - https://phabricator.wikimedia.org/T75837#798469 (kevinator) [15:53:22] Analytics-Wikimetrics: Story: WikimetricsUser reports pages edited by cohort [13pts] - https://phabricator.wikimedia.org/T75072#798473 (kevinator) [15:53:29] Analytics-EventLogging: Engineer runs script to create a new dashboard - https://phabricator.wikimedia.org/T75838#798474 (kevinator) [15:53:39] Analytics-EventLogging: Engineer runs script to test SQL for dashboard against a test database - https://phabricator.wikimedia.org/T75837#783501 (kevinator) [15:53:55] Analytics-EventLogging: Epic: Simplify dashboard deployment with EventLogging data - https://phabricator.wikimedia.org/T75836#798477 (kevinator) [15:57:01] Analytics-EventLogging: Engineer runs script to test SQL for dashboard against a test database - https://phabricator.wikimedia.org/T75837#798483 (kevinator) p:Triage>High [15:57:10] Analytics-EventLogging: Engineer runs script to create a new dashboard - https://phabricator.wikimedia.org/T75838#798484 (kevinator) p:Triage>High [16:05:35] Analytics-Dashiki: Story: User selects breakdown in Vital Signs [21pts] - https://phabricator.wikimedia.org/T74739#798494 (Milimetric) Open>Resolved [16:05:55] Analytics-Dashiki: Story: Vital Signs User selects the Daily Pageviews metrics [34pts] - https://phabricator.wikimedia.org/T74740#798496 (Milimetric) Open>Resolved [16:08:43] (CR) Nuria: [C: 2 V: 2] Remove unused wikimetrics format [analytics/dashiki] - https://gerrit.wikimedia.org/r/175330 (owner: Milimetric) [16:13:46] Analytics-EventLogging: database consumer could batch inserts (sometimes) [34pts] - https://phabricator.wikimedia.org/T69450#798505 (Nuria) [16:15:26] Analytics-EventLogging: database consumer could batch inserts (sometimes) [34pts] - https://phabricator.wikimedia.org/T69450#720228 (Nuria) [16:19:58] Analytics: Analytics User uses CentralNotice cookie in x-analytics field of web-request logs - https://phabricator.wikimedia.org/T75835#798515 (Tnegrin) Megan told me that there is a new "supercookie" used to track banner impressions. Please touch base with Ellery before we implement this. [16:24:59] Analytics: Analytics User uses CentralNotice cookie in x-analytics field of web-request logs - https://phabricator.wikimedia.org/T75835#798527 (kevinator) Open>stalled [17:36:42] Analytics-Wikimetrics: Story: Dashiki uses Mediawiki for storage [13pts] - https://phabricator.wikimedia.org/T70448#798640 (kevinator) [17:57:25] ottomata, pingeth [17:57:46] Ironholds: hiya! about to start an interview [17:58:08] okay! It can wait :) [18:31:11] nuria__: Hi. Do you have a moment? [18:34:43] jsahleen: hello, sure, what's up? [18:35:30] nuria__: Were you able to verify that ContentTranslation events are being collected in beta-labs? Just so we know our instrumentation is working. [18:36:55] jsahleen: i have not looked at things recently, lemme see [18:37:16] Thanks! [18:39:10] Analytics-EventLogging: Epic: WMF Engineer reads documentation to set up a dashboard from EL data - https://phabricator.wikimedia.org/T76362 (kevinator) NEW p:Unbreak! [18:40:23] Analytics-EventLogging: Engineer reads documentation on Wikitech to set up a dashboard from EL data - https://phabricator.wikimedia.org/T76364#798781 (kevinator) [18:43:43] jsahleen: the table select * from ContentTranslation_7146627; [18:43:55] Analytics-EventLogging: Engineer knows by when to expect a dashboard from EL data - https://phabricator.wikimedia.org/T76365 (kevinator) NEW p:Unbreak! [18:43:55] has 22 records in labs [18:44:03] jsahleen: beta-labs taht is [18:44:06] *that is [18:44:19] nuria__: Perfect! That's what I needed to know. Thanks for checking for us. [18:44:53] Analytics-EventLogging: Engineer reads an email announcement about the documentation for creating a dashboard from EL data - https://phabricator.wikimedia.org/T76367 (kevinator) NEW p:Unbreak! [18:56:23] (PS1) Nuria: Report card Dec 2014 [analytics/reportcard/data] - https://gerrit.wikimedia.org/r/176708 [19:04:25] (CR) Nuria: [C: 2 V: 2] Report card Dec 2014 [analytics/reportcard/data] - https://gerrit.wikimedia.org/r/176708 (owner: Nuria) [19:12:36] Analytics-EventLogging: Engineer reads documentation on Wikitech to set up a dashboard from EL data [3 points] - https://phabricator.wikimedia.org/T76364#798905 (kevinator) [19:13:21] Analytics-EventLogging: Engineer knows by when to expect a dashboard from EL data - https://phabricator.wikimedia.org/T76365#798906 (kevinator) [19:14:19] Analytics-EventLogging: Engineer knows by when to expect a dashboard from EL data (1points) - https://phabricator.wikimedia.org/T76365#798907 (ggellerman) [19:16:31] Analytics-EventLogging: Engineer reads an email announcement about the documentation for creating a dashboard from EL data (1 point) - https://phabricator.wikimedia.org/T76367#798923 (ggellerman) a:Tnegrin [19:18:07] Analytics-Dashiki: Story: Vital Signs User selects the Daily Pageviews metrics [34pts] - https://phabricator.wikimedia.org/T74740#798927 (kevinator) UI is showing '0' values for lines that didn't exist (Zero) before a certain date These values need to be changed to 'undefined' This may break the metric. [19:24:19] analytics people: anyone know where I could get an actual log of https://gdash.wikimedia.org/dashboards/jobq/ ? [19:28:01] ottomata, not sure if you're back yet, but: just wanted to check how hard putting the boost iostream stuff on stat2 was. Also if there's a standardised way of doing this, so that I can submit a patch and have people review rather than waste y'all's time writing the patches :) [19:49:34] (CR) Gergő Tisza: [C: 2 V: 2] File page loading times are not experimental any more [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/176638 (owner: Gergő Tisza) [19:53:37] Ironholds: ja just got done with ops meeting [19:53:48] it is pretty easy if there is a deb package from apt repos we already use [19:53:53] should be, yeah [19:53:57] e.g., if you can do aptitude search [19:54:03] libboost-iostreams-dev [19:54:42] Ironholds: https://gerrit.wikimedia.org/r/#/c/174131/3/manifests/misc/statistics.pp [19:54:45] that's an example [19:54:46] but. [19:55:03] there was a problem, since stat1001 also includes this class and hasn't been upgraded to trusty [19:55:04] https://gerrit.wikimedia.org/r/#/c/176656/ [19:55:23] so, for now...i will install it for you, but I need to upgrade stat1001 before i puppetized [19:55:24] puppetize* [19:55:26] gotcha [19:55:27] thankee! [19:55:28] which node do you want this on? [19:55:38] stat1002? [19:55:44] 1002, ideally? [19:55:46] yup [19:55:48] ottomata: I just realized your profile pic on hangouts has a chicken in it. [19:56:18] yes, but what is a chicken, really? [19:56:26] Ironholds: . done. [19:56:32] ta! [19:59:07] Oh, great, updates! https://stats.wikimedia.org/EN_Europe/Sitemap.htm [20:12:29] It's not that bad! is.wiki +11 % https://stats.wikimedia.org/EN_Europe/TablesRecentTrends.htm#C [21:01:16] milimetric: Was https://gerrit.wikimedia.org/r/#/c/175465/ brought up in some Scrum of Scrum meeting? [21:01:38] looking [21:01:52] I do not recall having it seen there. [21:02:08] And it is in the list of changes that I want to look how it impacts us. [21:02:22] qchris: really sorry - my note-keeping on SoS severely limits my ability to pay attention [21:02:32] it sounds familiar but if someone brought it up, it'd be in the etherpad [21:02:35] I can't say for sure I heard it [21:02:42] *etherpad -> wiki where I move the notes [21:02:59] I checked the pages and could not find it there. [21:03:10] k, then most likely no. Is it a big deal? [21:03:21] and if so, how - that way I can try and learn what to listen for [21:03:32] It blows up how webstatscollector is working. [21:04:12] We can account for it in the Hive re-implementation, but I am not so sure about the C variant. [21:04:57] Mhmm. Gonna do some more checking :-/ [21:05:23] qchris, i just heard about this today in the ops meeting [21:06:15] Any idea how to address that in webstatscollector's C implementation? [21:06:33] Meaning ... in the spirit of the webstatscollector pageview definition. [21:06:37] (Not doing the Hive implementation trick) [21:06:52] Because we wanted to give the community a chance to compare the implementations. [21:07:11] not really. wasn't there some mention of an https=true header, or something? [21:07:11] If we change the C implementation to do what the Hive implementation does, [21:07:35] it does not allow them to compare the old webstatscollector to the new Hive variant. [21:07:53] When I last checked, https was only set on the mobile varnishes. [21:08:36] Yup. Just rechecked. No usable https=true. [21:09:03] yep [21:09:05] then we should set it. [21:09:08] Dario wanted it everywhere, but.. [21:09:11] but? [21:09:21] ...but it doesn't exist yet. That's all I know ;p [21:09:23] ok [21:09:26] huh. wait. [21:09:28] * Ironholds raises hand [21:09:35] yes? [21:09:47] It was built for zero. And hence it is expected to only live on the mobile varnishes. [21:09:50] does http=true equate to "this request will appear to come from an SSL terminator"? [21:09:53] *https [21:11:01] that is, do all requests from SSL terminators, appear with HTTPS? Because there is a semi-related problem we have had to deal with around making sure we are excluding internal traffic, but not excluding traffic from the SSL terminators. [21:11:19] and if we could say "any request with https=true that comes from an internal machine, is from the SSL terminators"...that would be nice. [21:11:32] Currently, that's not the way it works. [21:11:49] yeah, currently. But if we're discussing having the desktop varnishes set it too, would it happen then? [21:11:56] or am I crazy and taking too much cold medication :D [21:12:24] If we define it that way then that's what it is :-D [21:12:45] But https=true broke before. I do not trust it. [21:13:04] We have to determine that on our end. Otherwise, we cannot reprocess data. [21:15:56] «Respondents are most likely to have been a victim of online harassment in Sweden (9%), Denmark (7%) and Luxembourg (7%), while the proportion is lowest in Bulgaria (1%), Greece (1%) and Portugal (1%). Respondents in Denmark and the Netherlands are most likely to say they their children have been a victim of online harassment (5% in each country).» http://ec.europa.eu/public_opinion/archives/ebs/ebs_404_en.pdf [21:16:24] Would be curious to see if various editor engagement stats follow any correlation to these numbers [21:19:55] qchris: should we move into ops room and ping bblack? [21:20:21] Not sure. I guess we need to handle the thing on our end. [21:20:29] bblack's change looks sane. [21:20:37] It just came as a surprise to us. [21:21:05] I would not want to roll it back, just because webstatscollector does not like it :-) [21:21:30] aye, but maybe we can get him to add https=true? [21:21:52] The x_analytics headers varnish sets are broken from my point of view. [21:22:11] They are not reliable. And if we build on them, [21:22:23] we cannot reprocess data in case of errors. [21:22:39] As we cannot (easily) re-pass the logs through varnish logic. [21:23:24] How bad would it be to bring your webstatscollector patch that counts internal traffic if it has a x-forwarded-for? [21:23:41] That would kill the issue, but it deviates from the spirit of the definition, and would [21:23:51] make comparing the C and Hive implementation pointless. [21:23:58] yeah, hm [21:24:05] ha, so, is webstatscollector counting double traffic now? [21:24:24] Yup. Some HTTPS traffic is double counted. [21:24:42] But only in the C implementation. [21:24:44] well, it would be all now, since that patch, right? [21:24:46] yes. [21:25:01] No. Ulsfo is not double counted. [21:25:11] (PS1) EBernhardson: Initial reporting for flow [analytics/limn-flow-data] - https://gerrit.wikimedia.org/r/176789 [21:25:18] As we patched that part some months ago. [21:25:19] how do we differentiate there? (forgive me if I have forgotten all of this) [21:25:31] By IP address. [21:25:47] If the request is coming with a client_ip of ulsfo, we consider it a dupe. [21:25:48] Analytics-Wikimetrics: Story: AnalyticsEng has editor_day table in labsdb - https://phabricator.wikimedia.org/T71145#799224 (ggellerman) [21:26:03] The dupe (from the front-end varnish) is thrown away [21:26:06] hm [21:26:17] and the request from the SSL terminator (same machine) is counted [21:26:34] can we do the same for this? just check client_ip for an internal addy? [21:26:41] Back then we checked the consequences and if it buys us false positives. [21:26:45] yes. [21:26:46] sure [21:26:47] monitoring, etc. [21:26:50] but meh? [21:26:58] better than duping everything in eqiad [21:27:06] i gues checking XFF is just as bad [21:27:25] I would not want to do the same thing again, because it's more unneccessary work on a service that we're about to phase out :-( [21:27:57] I guess then I'll again check on the false positives/negatives :-(( [21:28:12] This time for esams :-) [21:28:36] ottomata: Would you be around tomorrow for a deploy? [21:28:41] Analytics-Wikimetrics: Story: AnalyticsEng has editor_day table in labsdb - https://phabricator.wikimedia.org/T71145#799239 (ggellerman) Was 38 points...removing points from subj line because: 1. This ticket had been in progress for some time. Team came to consensus that it was too big and changed scope of... [21:28:48] s/deploy/webstatscollector deploy/? [21:29:26] qchris: , sure [21:29:29] if we do it early enough in the day [21:30:32] Mhmm. Not sure I can finish in time for that. But I guess you're around this week? [21:31:59] yes [21:32:02] will be around [21:32:05] we can deploy [21:32:06] coolio. [21:32:11] Thanks. [21:33:22] seriously, stat1002 no longer appears to have yaml-cpp :/ [21:33:40] hm, they shouldn't have been removed. [21:34:05] unless faidon did it manually [21:34:08] sigh, ok. [21:34:09] maybe he did [21:34:28] huh. I wonder why. [21:34:43] well, he didnt' like my puppet change, because it made stat1001 unhappy [21:34:50] (statistics.pp is not well organized) [21:34:54] so he jsut reverted it [21:34:56] he told me about it first [21:35:01] but I didn't get to it in time because of hte holiday [21:35:44] gotcha [21:35:49] (PS2) EBernhardson: Initial reporting for flow [analytics/limn-flow-data] - https://gerrit.wikimedia.org/r/176789 [21:36:33] Analytics: After switch to local ssl terminators, pagecounts-raw (C implementation of webstatscollector) overcounts HTTPS from some data-centers. - https://phabricator.wikimedia.org/T76390#799265 (QChris) [21:37:18] Ironholds: all the packages you've beenn asking me for, you only need them on stat1002, right? [21:39:16] Analytics: Network outage on 2014-11-30 affecting analytics infrastructure - https://phabricator.wikimedia.org/T76334#799276 (QChris) It seems a recent change of the HTTPS setup makes pagecounts-raw overcount (T76390). Hence, we first put pagecount-raw on a sane footing again and then backfill for 2014-11-30... [21:40:06] Analytics: Network outage on 2014-11-30 affecting analytics infrastructure - https://phabricator.wikimedia.org/T76334#799280 (QChris) [21:40:17] ottomata, yep [21:40:21] Analytics: After switch to local ssl terminators, pagecounts-raw (C implementation of webstatscollector) overcounts HTTPS from some data-centers. - https://phabricator.wikimedia.org/T76390#799282 (QChris) a:QChris [21:40:22] I mean, they /will/ be necessary on 3. [21:40:27] but for production-y tasks, 2 only. [21:41:42] hm, ok [21:41:50] Ironholds: what specifically are these packages for? [21:42:34] they're dependencies for my generalised utilities package. [21:42:55] specifically, the yaml, boost regex and glog libraries are needed for ua-parsing, [21:43:19] and the boost iostreams are needed for in-C gunzipping and processing [21:43:19] (or, sampled log reading by any other name) [21:55:21] ok Ironholdsyou should have packges back now [21:55:24] Ironholds: ^ [21:55:46] ta! [21:59:40] qchris: huhhhmmm. i can't see more than 1000 oozie workflows [21:59:50] Try -len [21:59:56] Or --length [22:00:00] Let me look it up. [22:00:20] "-len" it is [22:00:25] hmm [22:00:51] cool, thank you, that worked [22:01:01] yw. [22:01:05] it is weird for hue too [22:01:10] hue only shows 1000 workflows [22:01:14] ? [22:01:17] Yikes! [22:01:19] yup [22:01:21] looks like a bug [22:01:28] maye cdh 5.2 will be better! [22:01:31] but for now, i have cli [22:01:33] good enough for me [22:01:37] k. [22:10:42] (CR) EBernhardson: [C: 2] "lets test this out" [analytics/limn-flow-data] - https://gerrit.wikimedia.org/r/176789 (owner: EBernhardson) [22:14:55] ottomata: this says "per ottomata": https://gerrit.wikimedia.org/r/173089 [22:15:24] we were just about to test some flow SQL to see if that config worked, and we saw it got reverted [22:15:39] DarTar: btw, we've a new contributor to quarry code! rtnpro has been writing patches :) [22:16:09] oh YuviPanda: he wanted to work on some wikimetrics stuff, so he's got python skillz then? [22:16:41] milimetric: yes, very good as well. I'd say his patches are better code quality than avg. quarry quality :) [22:16:45] they even have documentation! [22:17:06] aweesoome :) [22:17:14] (CR) EBernhardson: [V: 2] Initial reporting for flow [analytics/limn-flow-data] - https://gerrit.wikimedia.org/r/176789 (owner: EBernhardson) [22:17:15] :) [22:17:22] milimetric: dzahn reverted because it wasn't ready, afaik [22:17:30] the script wasn't deployed at the time [22:17:36] which would have just made the cron job fail [22:17:54] YuviPanda: fantastic [22:17:56] or, rather, the repo/script didn't exist [22:17:59] hm, but isn't the script deployed just one time for the mobile repo? [22:19:19] milimetric: all i knew was: [22:19:28] we prepared this change, but were waiting on someone to create the repo, or the script [22:19:37] when dzahn merged this, i hadn't heard that that had happened [22:20:03] ok, cool, so we'll submit the patch again in a moment, after the repo is ready [22:20:17] the "script" is generate.py and it's been there in the limn-mobile-data repo [22:20:25] we'll factor that out at some point, but it's not necessary now [22:20:31] this is for flow, right? [22:20:34] the "repo" is the limn-flow-data repo [22:20:34] yes [22:20:36] doesn't it need a new repo? [22:20:38] k [22:20:42] it exists [22:20:46] k [22:20:51] but it needed config - which erik just submitted [22:20:59] i'll merge it now and re-submit the puppet change [22:20:59] thx [22:21:03] k [22:28:24] ottomata: mind merging: https://gerrit.wikimedia.org/r/#/c/176810/ [22:29:46] k fingers crossed! [22:31:46] thank you muchly :) [22:34:30] milimetric: [22:34:31] ValueError: Can not find SQL or Python for unique-users-posting-messages [22:35:01] File "/srv/limn-mobile-data/generate.py", line 330, in [22:35:01] dg.execute() [22:35:01] File "/srv/limn-mobile-data/generate.py", line 173, in execute [22:35:01] ok = self.generate_graph_full(key, value) [22:35:01] File "/srv/limn-mobile-data/generate.py", line 293, in generate_graph_full [22:35:02] raise ValueError('Can not find SQL or Python for {0}'.format(key)) [22:35:02] ValueError: Can not find SQL or Python for unique-users-posting-messages [22:35:59] cool, useful [22:36:00] looking [22:36:18] ottomata: that's good though - means it's running the code [22:36:31] we'll fiddle with the sql and all that to make it work and submit any puppet changes we need [22:36:40] ok cool [22:36:53] we should probably make the job log stderr to the log file too [22:36:57] right now it is just >> [22:36:59] should prbably be [22:37:01] 2>&1 [22:39:02] (PS1) EBernhardson: Link direct to csv in reportcard.json [analytics/limn-flow-data] - https://gerrit.wikimedia.org/r/176816 [22:39:04] (PS1) EBernhardson: Correct name of unique-users graph [analytics/limn-flow-data] - https://gerrit.wikimedia.org/r/176817 [22:39:08] ottomata: it used to be like that - you changed it I think [22:39:14] when you refactored [22:39:27] q though - where'd you get that error, I couldn't see it in /var/log/limn-data/limn-flow-data.log [22:39:39] (CR) EBernhardson: [C: 2 V: 2] Link direct to csv in reportcard.json [analytics/limn-flow-data] - https://gerrit.wikimedia.org/r/176816 (owner: EBernhardson) [22:39:40] oh - that's what 2>&1 would fix [22:40:00] (CR) EBernhardson: [C: 2 V: 2] Correct name of unique-users graph [analytics/limn-flow-data] - https://gerrit.wikimedia.org/r/176817 (owner: EBernhardson) [22:44:17] yup [22:44:20] i ran the thing manually [22:44:26] i changed it? [22:44:28] hm [22:44:30] why would I do that? [22:44:32] hm [22:51:33] ottomata: I accused you wrongly, it was there when you changed it: https://github.com/wikimedia/operations-puppet/commit/0766e1465a223f0dd30e16de9034f71a00e713b6#diff-8e6b2be32d05dc422cadfc2d0e699cdeR832 [22:51:36] not sure who changed it then [22:53:05] ottomata: does that user have the rights to create all those folders it needs in /a/limn-public-data ? [22:56:01] stats user, yes. [23:00:15] Analytics-Wikimetrics: Story: Wikimetrics compiles target-site breakdown on metrics based on MW tags [34pts] - https://phabricator.wikimedia.org/T74736#799586 (Milimetric) Hi rtnpro, Thanks so much for offering to help. This feature is in limbo right now, we are currently querying the vanilla mediawiki sche... [23:02:23] sweet, it worked-ish (no data, but output files are there) [23:03:14] cool [23:05:37] how do we add Erik to the research group so he can use the research password? [23:05:50] Erik Bernhardson [23:09:30] (PS1) Mforns: Add dist folder to bower package [analytics/mediawiki-storage] - https://gerrit.wikimedia.org/r/176825 (https://bugzilla.wikimedia.org/68448) [23:11:34] bbl [23:15:22] Analytics-Refinery: PageView reports by hive-webstatscollector should return undefined values when data is not available - https://phabricator.wikimedia.org/T76406 (kevinator) NEW p:High a:QChris [23:27:25] (PS1) EBernhardson: Rewrite moderation actions query to break out hide/delete topic/post [analytics/limn-flow-data] - https://gerrit.wikimedia.org/r/176828 [23:29:39] (PS1) EBernhardson: Change default db to flowdb [analytics/limn-flow-data] - https://gerrit.wikimedia.org/r/176831 [23:30:19] Analytics-Refinery: PageView reports by hive-webstatscollector should return undefined values when data is not available - https://phabricator.wikimedia.org/T76406#799733 (ggellerman) [23:36:38] Analytics-EventLogging: Epic: WMF Engineer has simpler process to create a dashboard from EL data - https://phabricator.wikimedia.org/T76413 (kevinator) NEW p:Triage [23:37:09] Analytics-EventLogging: Epic: WMF Engineer has simpler process to create a dashboard from EL data - https://phabricator.wikimedia.org/T76413#799759 (kevinator) [23:38:23] Analytics-EventLogging: Epic: WMF Engineer has simpler process to create a dashboard from EL data - https://phabricator.wikimedia.org/T76413#799759 (kevinator) p:Triage>Normal [23:40:16] Analytics-EventLogging: Epic: Announce how to deploy a dashboard with EventLogging data - https://phabricator.wikimedia.org/T75836#799778 (kevinator) [23:43:01]