[00:00:59] Analytics-EventLogging, MediaWiki-ContentHandler: HTML of Schema pages appears garbled - https://phabricator.wikimedia.org/T86748#977844 (Krinkle) As @Nuria noticed, it only affecting new pages was the case for simple MediaWiki installs. But in production it did actually affect existing pages, too. This becau... [00:08:49] DarTar: Hey, you about! I'm sat outside Mushroom Kingdom. [00:08:55] DarTar: We had a meeting which I was late for. :-) [00:09:18] Deskana: just done with the previous one [00:09:28] DarTar: Cool, then I won't feel guilty. :-) [00:09:30] I’ll be there in 5 [00:34:22] Analytics-EventLogging, MediaWiki-ContentHandler: HTML of Schema pages appears garbled - https://phabricator.wikimedia.org/T86748#978127 (Legoktm) * JsonContent was changed to return objects instead of assoc arrays, it also (unintentionally) broke the ability of extensions to extend JsonContent ("looked" fine... [00:35:17] Analytics-EventLogging, MediaWiki-ContentHandler: HTML of Schema pages appears garbled - https://phabricator.wikimedia.org/T86748#978134 (Legoktm) [00:35:19] Analytics-EventLogging, MediaWiki-ContentHandler: [Regression 1.25wmf14] EventLogging schemas on Meta and other forms of JSON no longer display properly - https://phabricator.wikimedia.org/T86706#978133 (Legoktm) [00:35:42] Analytics-EventLogging, MediaWiki-ContentHandler: HTML of Schema pages appears garbled [1.25wmf14 regression] - https://phabricator.wikimedia.org/T86748#978142 (Legoktm) [04:24:44] (PS1) KartikMistry: Start data from 2014-0114 to get correct stats [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/185123 [05:43:28] Analytics-Engineering: EPIC: Prepare and host Event Logging hackathon at MWDS - https://phabricator.wikimedia.org/T86212#978714 (kevinator) p:Triage>High [05:45:23] Analytics-EventLogging, Analytics-Engineering: Epic: WMF Engineer reads documentation to set up a dashboard from EL data - https://phabricator.wikimedia.org/T76362#978715 (kevinator) Open>Resolved a:kevinator All stories were completed Dec 18. Epic is done! [05:47:11] Analytics-Wikimetrics, Analytics-Engineering: Epic: Grantmaking User gets reports on Wikimetrics usage - https://phabricator.wikimedia.org/T76106#978722 (kevinator) [05:47:14] Analytics-Wikimetrics, Analytics-Engineering: Wikimetrics-l receives email about Lab’s Terms of Use - https://phabricator.wikimedia.org/T76108#978720 (kevinator) Open>Resolved email went out Jan 9th https://lists.wikimedia.org/pipermail/wikimetrics/2015-January/000198.html [05:47:55] Analytics-Wikimetrics, Analytics-Engineering: Epic: Grantmaking User gets reports on Wikimetrics usage - https://phabricator.wikimedia.org/T76106#789667 (kevinator) [05:48:11] Analytics-Wikimetrics, Analytics-Engineering: Epic: Grantmaking User gets reports on Wikimetrics usage - https://phabricator.wikimedia.org/T76106#978724 (kevinator) Open>Resolved a:kevinator [05:49:44] (PS1) KartikMistry: Enable query on enwiki too [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/185126 [05:50:06] (PS2) KartikMistry: Start data from 2014-01-14 to get correct stats [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/185123 [05:50:59] (CR) KartikMistry: [C: 2] Better tabs name and subheading [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/184328 (owner: KartikMistry) [05:51:07] (Merged) jenkins-bot: Better tabs name and subheading [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/184328 (owner: KartikMistry) [05:51:22] Analytics-EventLogging, Analytics-Engineering: EL office hours - https://phabricator.wikimedia.org/T76796#978726 (kevinator) [05:51:43] (CR) KartikMistry: [C: 2] "enwiki is needed!" [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/185126 (owner: KartikMistry) [05:51:49] (Merged) jenkins-bot: Enable query on enwiki too [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/185126 (owner: KartikMistry) [05:53:28] (PS3) KartikMistry: Start data from 2014-01-13 to get correct stats [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/185123 [05:54:22] Analytics-EventLogging, Analytics-Engineering: Product Instrumentation and Visualization - https://phabricator.wikimedia.org/T76795#978729 (kevinator) [05:54:24] Analytics-EventLogging, Analytics-Engineering: EL office hours - https://phabricator.wikimedia.org/T76796#978727 (kevinator) Open>Resolved 2 people attended office hours. One had questions about knowing what to instrument, the other about Limn dashboards. Most of the discussion occurred on IRC and is... [08:25:09] !log Ran kafka leader re-election to bring analytics1021 back into the set of leaders [08:39:12] Analytics, Wikipedia-App-iOS-App, Language-Engineering, Wikipedia-App-Android-App, Mobile-Apps, MediaWiki-extensions-UniversalLanguageSelector, Mobile-Web: there should be a comparison of clicks count on interlanguage on different platforms - https://phabricator.wikimedia.org/T78351#978890 (Amire80) [08:43:10] (CR) Amire80: [C: 1] Start data from 2014-01-13 to get correct stats [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/185123 (owner: KartikMistry) [08:48:00] Analytics, Wikipedia-App-iOS-App, Language-Engineering, Wikipedia-App-Android-App, Mobile-Apps, MediaWiki-extensions-UniversalLanguageSelector, Mobile-Web: there should be a comparison of clicks count on interlanguage on different platforms - https://phabricator.wikimedia.org/T78351#978902 (Amire80) [08:48:18] Analytics, Wikipedia-App-iOS-App, Language-Engineering, Wikipedia-App-Android-App, Mobile-Apps, MediaWiki-extensions-UniversalLanguageSelector, Mobile-Web: there should be a comparison of clicks count on interlanguage on different platforms - https://phabricator.wikimedia.org/T78351#843122 (Amire80) [09:31:19] (CR) QChris: "webrequest/load would break. But acutally, most things are" (15 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/184796 (owner: Ottomata) [09:46:12] (PS1) QChris: Update Oozie diagram for switch to refined webrequest table [analytics/refinery] - https://gerrit.wikimedia.org/r/185140 [09:48:31] (CR) QChris: "Linked diagram update." (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/184796 (owner: Ottomata) [09:50:12] Analytics, operations, ops-core: Deprecate HTTPS udp2log stream? - https://phabricator.wikimedia.org/T86656#978960 (fgiunchedi) on the re-architecturing, I think (newer?) nginx versions can write logs to a pipe so that might be a quick win without patching nginx (?) [10:39:27] (PS2) QChris: Add _SUCCESS done-flag in refined webrequest data directories after successful refinement [analytics/refinery] - https://gerrit.wikimedia.org/r/184804 (owner: Ottomata) [10:39:59] (CR) QChris: Add _SUCCESS done-flag in refined webrequest data directories after successful refinement (2 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/184804 (owner: Ottomata) [10:40:57] (CR) QChris: Add _SUCCESS done-flag in refined webrequest data directories after successful refinement (2 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/184804 (owner: Ottomata) [13:04:18] Analytics-Tech-community-metrics: Consolidating time ranges across tech community metrics - https://phabricator.wikimedia.org/T86630#979332 (Aklapper) For a ten year old project, it makes sense to have curves more normalized (and each data point not being entirely disjunct). The only small downside would be t... [13:17:02] Analytics-EventLogging, MediaWiki-extensions-Popups: FF 35: hovercard conflicts with ad block extensions: doesn't disappear, and new hovercard don't pop up - https://phabricator.wikimedia.org/T86900#979360 (Se4598) this must be caused by a recent (<2 weeks?) change in FF, Adblock and/or MediaWiki (Popups, ..)... [13:33:53] (CR) Gilles: [C: 2] Generate pageview stats [analytics/multimedia] - https://gerrit.wikimedia.org/r/179872 (https://phabricator.wikimedia.org/T78189) (owner: Gergő Tisza) [13:34:01] (Merged) jenkins-bot: Generate pageview stats [analytics/multimedia] - https://gerrit.wikimedia.org/r/179872 (https://phabricator.wikimedia.org/T78189) (owner: Gergő Tisza) [14:03:30] Analytics-Engineering: Request for current data about mobile editing in he.wikipedia - https://phabricator.wikimedia.org/T86793#979442 (Elitre) [14:11:02] Analytics-EventLogging: Recent EventLogging change breaking Echo, Popups on blocked event.gif calls / on Firefox with Adblock Plus - https://phabricator.wikimedia.org/T86918#979465 (Se4598) NEW [14:11:26] Analytics-EventLogging: Recent EventLogging change breaking Echo, Popups on blocked event.gif calls / on Firefox with Adblock Plus - https://phabricator.wikimedia.org/T86918#979475 (Se4598) [14:11:27] Analytics-EventLogging, MediaWiki-extensions-Popups: FF 35: hovercard conflicts with ad block extensions: doesn't disappear, and new hovercard don't pop up - https://phabricator.wikimedia.org/T86900#979110 (Se4598) [14:14:15] (CR) KartikMistry: [C: 2] Start data from 2014-01-13 to get correct stats [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/185123 (owner: KartikMistry) [14:14:21] (Merged) jenkins-bot: Start data from 2014-01-13 to get correct stats [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/185123 (owner: KartikMistry) [14:23:15] Analytics-EventLogging: Recent EventLogging change breaking Echo, Popups on blocked event.gif calls / on Firefox with Adblock Plus - https://phabricator.wikimedia.org/T86918#979495 (Aklapper) Yesterday for me this also broke the rendering of CPB on top of https://www.mediawiki.org/wiki/Special:Notifications?d... [14:31:36] (PS1) QChris: Drop unneeded user property for webrequest refining [analytics/refinery] - https://gerrit.wikimedia.org/r/185178 [14:43:03] (PS1) QChris: Explain why tables that should be external are internal [analytics/refinery] - https://gerrit.wikimedia.org/r/185179 [14:50:26] (CR) Ottomata: "This did not work for me without this property." [analytics/refinery] - https://gerrit.wikimedia.org/r/185178 (owner: QChris) [15:16:34] Analytics, operations, ops-core: Deprecate HTTPS udp2log stream? - https://phabricator.wikimedia.org/T86656#979566 (mark) [15:41:26] qchris: would you have time later to let you know about dropping events on EL and how i calculated that? [15:41:34] Sure. [15:41:47] Just let me know when, and I'll be around. [15:49:57] soo qchris, what's the external insert thing about? [15:50:19] You mean the cdh change, or the refinery change? [15:50:47] (i.e.: me wanting it to turn it on, or me wanting to document the current tables?) [15:51:06] uhm, wanting to turn it on, I didn't know this was a thing, or if I did, I forgot [15:51:49] So since we're on hdfs, I'd allow to insert into external tables. [15:52:03] hey ggellerman, I'm looking at hadoop usage right now. tomorrow morning might be a good time to do this namenode migration. what do you think? [15:52:20] I checked it on my labs cluster some time back, and it worked like a charm. [15:52:21] qchris, that's fine with me, what is the real difference between external and internal then? [15:52:42] DROP TABLE on external tables, does not kill the data. [15:52:47] DROP TABLE on internal tables, kills the data. [15:52:48] just drops? [15:52:49] yeah [15:53:04] That's the main difference I care about. [15:53:15] To me, Hive should not think it owns this data. [15:53:23] i agree with that [15:53:39] so you thikn we should use more external tables then. e.g. for wmf.webrequest? [15:53:39] Analytics-EventLogging: Can't update schemas on meta - https://phabricator.wikimedia.org/T86926#979604 (Gilles) NEW [15:53:55] can it add partitions to external tables during insert? [15:54:33] Yes, I'd want to use them more. But due to that config setting, we cannot. [15:54:47] +ottomata if the sys becomes unavailable, will that affect anyone beyond Christian? [15:54:49] In my tests, I could do everything with it that was needed. [15:55:25] Actually ... there was no difference when working with the data from within Hive. [15:56:14] I also trued with a custom, per-user hive-site.xml (with that setting to true) in production, and it worked just fine. [15:56:40] People on the hive bug also say that it should work just fine in HDFS. [15:57:14] Analytics-EventLogging: Can't update schemas on meta - https://phabricator.wikimedia.org/T86926#979612 (Gilles) Note that null edits aren't affected. It happens when I try to add a property to the schema. [15:57:14] But since you explicitly turned it off at some point, I figured you really want it off. [15:57:57] Hm! i do not rememer! [15:57:59] let's turn it on! [15:58:07] ggellerman: yes, it will affect anyone who wants to use hadoop [15:58:09] \o/ [15:58:16] hadoop will not be available durin gthis time [15:58:30] I actually expect no more than an hour of downtime, maybe even a few minutes [15:58:37] I'd lke to schedule 2 hours just in case. [15:58:41] https://gerrit.wikimedia.org/r/#/c/185176/ [15:58:52] also, I will likely need to schedule a second downtime when we actually upgrade cloudera [15:58:52] ottomata: change against cdh is here ^ [15:59:20] qchris, the hive default is to be true anyway, right? [15:59:33] Is there an email list for "anyone who wants to use hadoop?" [15:59:48] nope, but most folks using it are just researchers [16:00:10] there is actually no good way for me to contact folks I need to often. for example, I am supposed to change the research slave db password again [16:00:16] there is no way for me to notify the users of this password [16:00:21] ottomata, ggellerman: and wikipedia zero. And sometimes Ops IIRC. [16:00:27] yes [16:00:48] Is one day enough warning for that many groups? [16:01:42] well, the downtime should be very small. but maybe not [16:01:54] the trouble is if there are long running jobs, i can't turn the cluster off until they finish [16:01:58] at least without killing them [16:02:22] i just want to get this part done though, seeing as I will be in SF next week, and I doubt i will get much done on this then [16:02:37] could you write to Research, Ops & Zero and summarize the risks, estimated downtime? [16:02:48] there really isn't much going on in the cluster now, i could probably just disable new submission of jobs and restart it without anyone knowing [16:03:01] yes [16:03:16] Then I can make sure that Research responds, and I can try follow up with Zero...and maybe you have an in with Ops? [16:05:45] (It might be easier to just send a heads up to the analytics list. People that care about analytics infrastructure should be on there. But I guess I am just sloppy by assuming that :-/ ) [16:06:53] If sloppy = good enough, then that could be a good thing [16:08:17] Analytics-EventLogging: Can't update schemas on meta - https://phabricator.wikimedia.org/T86926#979650 (Gilles) I presume this is the exception (found in logstash): P220 [16:08:38] ottomata: About the default ... "HIVE_INSERT_INTO_EXTERNAL_TABLES("hive.insert.into.external.tables", true, [...]" from [16:08:42] https://svn.apache.org/repos/asf/hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java [16:08:47] So it's true by default. [16:08:54] (I could not find it in the Hive wiki) [16:13:39] i was googling aroudn for that too, thanks [16:13:51] qchris, should we just remove it from the xml file then altogether? [16:13:58] i think we should either remove it, or parameterize it in cdh module [16:14:09] ggellerman: probably analytics list + yuri will be enought [16:14:54] ottomata: I'd vote for removing it. [16:15:05] ok, fine with me. [16:15:09] do that and I will merge it [16:15:10] I'll update the change. [16:15:12] k [16:19:01] Ha! Could you email Analtyics and Yuri? Then I'll work on getting the researchers to respond [16:19:20] (CR) QChris: "> This did not work for me without this property." [analytics/refinery] - https://gerrit.wikimedia.org/r/185178 (owner: QChris) [16:20:29] yup, writing email now [16:21:22] (CR) QChris: [C: -1] "Since we're about to allow inserting into external tables," [analytics/refinery] - https://gerrit.wikimedia.org/r/185179 (owner: QChris) [16:21:49] (CR) Ottomata: "But for other parameters, like queue.name, we choose default values such that the job will work for non production setups. Why not make t" [analytics/refinery] - https://gerrit.wikimedia.org/r/185178 (owner: QChris) [16:28:12] (CR) QChris: "I think the custom hive setup is easier." [analytics/refinery] - https://gerrit.wikimedia.org/r/185178 (owner: QChris) [16:28:28] Analytics-EventLogging, MediaWiki-ContentHandler: HTML of Schema pages appears garbled [1.25wmf14 regression] - https://phabricator.wikimedia.org/T86748#979693 (Jdforrester-WMF) See also {T86270} caused by the same patch. [16:29:13] qchris, I dunno. we are also using these jobs as examples for other to copy when developing their own oozie stuff [16:29:24] they dont 'have sudo -u hdfs privs [16:29:26] (CR) QChris: "On second thought ... maybe you're right." [analytics/refinery] - https://gerrit.wikimedia.org/r/185178 (owner: QChris) [16:29:31] haha [16:29:39] :-D [16:29:56] I still don't like that artificial cruft. [16:30:10] One doesn't need sudo to run those files. [16:30:45] (Abandoned) QChris: Drop unneeded user property for webrequest refining [analytics/refinery] - https://gerrit.wikimedia.org/r/185178 (owner: QChris) [16:41:07] Anyone seen milimetric today? [17:02:59] nuria: +2 on mobile apps :) [17:03:12] you think I should merge now, and then we make a new change to make it work with my refined refactor? [17:03:19] or shoudl we wait and fix before we merge this? [17:05:20] ottomata: we can merge now as your refactor only changes teh nature of datasets [17:05:34] ottomata: right??? [17:05:47] ottomata: as in now they have a done flag blah blah [17:06:00] ottomata: maybe i need to look at your changes... [17:06:37] it changes the name too [17:06:43] and files [17:07:53] lemme see [17:09:12] ottomata: what would you prefer? [17:09:14] qchris: uHHH, when I added these column comments to the refined table, i thought there weren't any real comments in the create for the raw table, just the 'from deserializer' ones. uHHH [17:09:18] weird. i will use the same ones. [17:09:43] nuria: doesn't matter to me, i think we should wait on starting the job before we get my refactor merged [17:09:45] just less to do later [17:09:58] but we can merge now and then fix the dataset names etc. as a separate patch [17:10:08] hm, actually. [17:10:10] hmmm [17:10:13] i dunno, whatever :) [17:10:35] ottomata: ok, let's go with your criteria then. [17:10:47] ottomata: let's merge the refined refactor 1st [17:10:49] ottomata: Fine by me to merge now and clean up later. [17:10:55] naw i want to clean up now [17:10:58] k. [17:10:58] ok [17:11:02] yesssir [17:12:10] qchris, i wonder what the output format of the raw table shoudl be if we now allow external table intputs [17:12:21] i guess [17:12:21] SequenceFileOutputFormat [17:12:36] ottomata: We currently do not insert there ... so it does not matter. [17:12:50] well, if we allow inserts, then who knows what's gonna happen :p [17:12:57] :-D [17:12:59] but aye [17:13:00] Right. [17:13:28] I have not tested SequenceFileOutputFormat. [17:13:37] But it exists, and looks right. [17:13:52] yeah haven't tested either [17:14:01] qchris: re Deprecated parquet thing [17:14:01] But ... not sure .. the change already contains so many things. [17:14:08] yes when we ahve 0.13 it will look more normal [17:14:13] Is it ok to switch output type in a separate chaneg. [17:14:15] it will use the new mapreduce api [17:14:26] yeha lets' do the raw output format in a different change [17:14:31] cool. [17:14:36] i will add comments about it [17:14:39] Ja, about the deprecated. [17:14:46] Ok. It was mostly about the comments. [17:14:48] Thanks. [17:28:20] (PS3) Ottomata: Refactor webrequest dataset names [analytics/refinery] - https://gerrit.wikimedia.org/r/184796 [17:28:26] (CR) Ottomata: Refactor webrequest dataset names (14 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/184796 (owner: Ottomata) [17:29:03] (CR) Ottomata: [C: 1] Add _SUCCESS done-flag in refined webrequest data directories after successful refinement [analytics/refinery] - https://gerrit.wikimedia.org/r/184804 (owner: Ottomata) [17:30:40] qchris, I am sure there will need to be some follow up commits to this stuff. i have a 1:1 with toby and then need to eat some food. How do you feel about me merging it and then working on it this afternoon after the merge, I'dd change the refined table to be external (and recreate partitions), and then work with nuria to apply the change to her mobile-apps stuff, and then get it working with that first. [17:30:50] then i'll work on restarting existing jobs that use the new dsataset [17:31:06] Sounds simply amazing! [17:31:31] haa, ok [17:31:34] The one thing is the missing coordinator change. [17:31:40] That might be worth adding from the start. [17:31:59] ? [17:32:06] Let me grab the link. [17:32:25] ottomata: sounds great [17:33:00] Oh. There is a new patchset. [17:33:05] I guess just run wild then :-) [18:18:04] qchris: can I tell you about how i was looking into us dropping events to see if things sound right? [18:18:10] Sure. [18:22:25] qchris: I looked at one server side event: pagecontentsavecomplete as the server side logs are smaller than client logs and as such faster to parse [18:23:00] Ok. [18:23:16] qchris: 1st i looked at a full day of data w/o outage events on the db and the logs to see how close those two agree (from 6 am to 6am) [18:24:47] qchris: for the pagecontentsavecomplete event the db and log two matched up to 0.01% [18:25:00] ok. [18:26:32] after i looked at the 2 hours of the alarms being raised and the number of db events of those two hours [18:27:09] qchris: i went into vanadium, got the server log for those two hours for page-content-save-complete and compared number of events with db events [18:27:45] qchris: i used a script like : [18:27:48] https://www.irccloud.com/pastebin/g2JMipip [18:27:53] * qchris looks [18:28:38] k [18:30:49] qchris: and it seems that yes we were dropping events [18:31:17] Do you roughly know how big the difference was for you? [18:31:23] qchris: but it's 111746 on log vs 59013 [18:31:25] on db [18:31:57] I checked the schema, and cannot find this difference. [18:32:00] Mhmmm [18:32:09] ya exactly [18:32:09] Let we double check in graphite. [18:32:16] hmm, seems pentaho is unresponsive [18:32:38] it finaly broke [18:33:08] nuria: I cannot find such a drop in graphite for that schema. [18:33:14] qchris: cause i also got the sequence minute by minute on the db for this schema and there are no big fluctuations [18:33:48] qchris: ya, so general idea looks right but i have a bug somewhere that would be the diagnosis right? [18:34:01] I think so. [18:34:07] But let's double check. [18:34:18] How did you create page-content-save-complete.txt [18:34:20] ? [18:35:43] qchris: i found the problem! [18:35:53] qchris: this is rubber duck debugging! [18:35:53] \o/ [18:36:26] qchris: my per sec counts are fine, i calculated badly my per minute counts [18:36:31] qchris: thank you ! [18:36:33] Ah. Ok. [18:36:35] yw [18:36:40] Glad you found the issue. [19:09:23] of course I've been accidentally disconnected all day :( sorry [19:20:03] qchris: hm, how about I relaunch the load job first, then recreate refined the table and partitions as external, then relaunch the refine job, then get nuria's job working [19:20:27] ottomata: FIne by me :-) [19:20:34] okeydokey [19:23:25] awesome presentation, halfak. :) [19:26:49] Thanks jgage :) [19:45:20] nuria, milimetric: eventlogging-devserver seems to be finally working... pushed for review. Thanks ottomata!! [19:45:59] sho thang! [19:46:33] ah i see, mforns I guess the proxy modules wer just not there, right?https://gerrit.wikimedia.org/r/#/c/184794/1..2/puppet/modules/role/manifests/eventlogging.pp [19:47:55] nuria, yes, they were there at the beggining, then I removed them. And as it continued working for me, I thought they were not needed. [19:48:09] *beginning [19:48:11] k [19:49:03] and nuria, apache log level in vagrant is set to error, that's why we could find no access logs... [19:49:51] mforns: that doesn't sound right in both counts 1) that the absence of a module will not trigger an error 2) that the log is set to error [19:49:54] so, qchris, i'm testing this, want to check with you to see if you are thinking the same thing [19:50:02] if I add a partition to an external table without specifiying the location [19:50:23] it will default to the part1=value1/part2=value2 scheme, yes? [19:50:48] i guess as a subdir in location defined in the create table? [19:50:59] you can set the log level to 'info' at: /etc/apache2/site-confs/devwiki/00-default.conf [19:57:50] nuria, ^ and yes, when provisioning vagrant without the module's includes, they should be disabled, agree [19:57:50] FYI, I"m going to do a little refactoring of the webrequest refined table.  i don't see any queries currently using it, but it will be funky for a few minutes [20:09:14] wmf.webrequest table should be back and jsut fine [20:13:49] is there a phabricator task for the EL presentation? [20:13:59] coool, ok, first refactored job (load) just succeeded, proceeding to launch the refine job [20:17:03] milimetric: was there a phabricator task for the EL presentation? [20:18:34] I don't know / think so? [20:18:52] kevinator said he'd make a skeleton one [20:19:05] but i might have disagreed with that (sorry for the confusion I caused in that case) [20:22:09] milimetric: no worries [20:22:09] ottomata: Yes. That's what partition adding should do for external tables. [20:22:23] I guess it worked as the other jobs seem to work? [20:26:43] yes, refined didn't work because uhhh i didn't merge the cdh change! [20:26:46] doing now and testing something [20:29:43] (CR) Ottomata: [C: 2 V: 2] "Merging this, then will change nuria's mobile_apps job to use this, then merge that, then launch that job with this refactor. Then will r" [analytics/refinery] - https://gerrit.wikimedia.org/r/184796 (owner: Ottomata) [20:36:34] (PS1) Milimetric: Speed up verification with scripts [analytics/data-warehouse] - https://gerrit.wikimedia.org/r/185239 [20:42:50] Analytics-EventLogging: sendBeacon throws an exception in Firefox when event.gif is adblocked - https://phabricator.wikimedia.org/T86680#980221 (Tgr) [20:44:00] cool, refine job + refactor working too, woot! [20:44:21] nuria: , let's do yours! gonna submit a last patch on top of yours to use hte new dataset [20:44:32] oh! [20:44:37] no, first, we want the done flag... [20:44:43] doh. should have done that first. [20:44:44] doh. [20:44:53] (PS3) Ottomata: Add _SUCCESS done-flag in refined webrequest data directories after successful refinement [analytics/refinery] - https://gerrit.wikimedia.org/r/184804 [20:45:19] (CR) Ottomata: [C: 2 V: 2] Add _SUCCESS done-flag in refined webrequest data directories after successful refinement [analytics/refinery] - https://gerrit.wikimedia.org/r/184804 (owner: Ottomata) [20:45:55] ottomata: ok, let me know if you want me to look at something, i can do CR if qchris is gone [20:46:00] k np [20:46:07] i'm just chatting to the room really [20:46:10] :) [20:47:22] wikimedia/mediawiki-extensions-EventLogging#331 (master - ddaa55a : Timo Tijhof): The build passed. [20:47:22] Change view : https://github.com/wikimedia/mediawiki-extensions-EventLogging/compare/d4e34cf5e7a2...ddaa55af41de [20:47:22] Build details : http://travis-ci.org/wikimedia/mediawiki-extensions-EventLogging/builds/47166343 [20:51:54] (PS10) OliverKeyes: Generalised class of UDFs for handling the webrequests table [analytics/refinery/source] - https://gerrit.wikimedia.org/r/181939 [20:52:16] ottomata, patched! [20:54:07] Analytics-EventLogging: sendBeacon throws an exception in Firefox when event.gif is adblocked - https://phabricator.wikimedia.org/T86680#980264 (ori) >>! In T86680#980071, @Rillke wrote: > IMHO failed tracking due to a vigilant anti-tracking software should not cause this kind of issues; the error should be s... [21:01:08] Analytics-Dashiki: PM renames metric in Vital Signs - https://phabricator.wikimedia.org/T86963#980298 (kevinator) NEW [21:01:23] Analytics-Dashiki: PM relables metric in Vital Signs - https://phabricator.wikimedia.org/T86963#980305 (kevinator) [21:03:28] Analytics-Dashiki: PM relables metric in Vital Signs - https://phabricator.wikimedia.org/T86963#980316 (kevinator) This could be an issue for bookmarks since the name of the metric can be specified in the URL. In that case, don't touch the URL, just the visible labels in vital signs (on the tab & drop down m... [21:03:40] Analytics-Dashiki: PM relabels metric in Vital Signs - https://phabricator.wikimedia.org/T86963#980317 (kevinator) [21:05:36] (PS2) Ottomata: Update Oozie diagram for switch to refined webrequest table [analytics/refinery] - https://gerrit.wikimedia.org/r/185140 (owner: QChris) [21:05:41] (CR) Ottomata: [C: 2 V: 2] Update Oozie diagram for switch to refined webrequest table [analytics/refinery] - https://gerrit.wikimedia.org/r/185140 (owner: QChris) [21:08:53] Analytics-Dashiki: Analyst bookmarks Vital Signs showing multiple metrics - https://phabricator.wikimedia.org/T86966#980353 (kevinator) NEW [21:11:08] (PS22) Ottomata: Mobile apps oozie jobs [analytics/refinery] - https://gerrit.wikimedia.org/r/181017 (owner: Nuria) [21:11:47] (CR) Ottomata: [C: 2 V: 2] Mobile apps oozie jobs [analytics/refinery] - https://gerrit.wikimedia.org/r/181017 (owner: Nuria) [21:20:57] (PS2) Milimetric: Speed up verification with scripts [analytics/data-warehouse] - https://gerrit.wikimedia.org/r/185239 [21:29:11] Ironholds, so the cube is OK, if so, I'll mark my task as done [21:29:20] mforns, thanks! [21:29:42] sorry, that was a question, forgot the question mark... the data in the cube is OK? [21:29:51] Ironholds, ^ [21:30:06] it looks good, yeah! Thanks so much for your work on it :) [21:30:26] np at all, thank *you! [21:30:49] Analytics-Engineering, Analytics-Visualization: Analysts visualize Pageview 0.4 cube in Pentaho - https://phabricator.wikimedia.org/T86540#980512 (mforns) Open>Resolved [21:36:18] (PS1) Ottomata: Use daily frequency for mobile apps daily uniques coordinator [analytics/refinery] - https://gerrit.wikimedia.org/r/185256 [21:36:33] (CR) Ottomata: [C: 2 V: 2] Use daily frequency for mobile apps daily uniques coordinator [analytics/refinery] - https://gerrit.wikimedia.org/r/185256 (owner: Ottomata) [21:41:55] (CR) Nuria: "Thanks for catching this!" [analytics/refinery] - https://gerrit.wikimedia.org/r/185256 (owner: Ottomata) [21:42:40] nuria, i submitted the job, and have it starting for tomorrow's day. so i guess we wait on that and see :) [21:43:32] ottomata: are you planning to backfill your datasets? [21:43:51] ? [21:44:01] which are my datasets? [21:44:12] ottomata: haha, i mean the "refined" [21:44:19] ah, yes, its got everything it had before [21:44:52] Analytics-EventLogging, Mobile-Web: Many of the mobile report cards are broken - https://phabricator.wikimedia.org/T86972#980586 (Jdlrobson) NEW [21:44:55] ottomata: I will make (for testing purposes) a version of teh job that just goes through 1 hour of data just to make sure it runs. this would be after lunch. [21:45:17] ottomata: thanks for doing the changes. vetting the data shoudl be teh last task. [21:45:21] *the last task [21:45:51] ok [21:50:27] Analytics: Firewall changes on 2015-01-13 affect udp2log - https://phabricator.wikimedia.org/T86973#980599 (mforns) NEW a:mforns [21:50:40] Analytics: Firewall changes on 2015-01-13 affect udp2log - https://phabricator.wikimedia.org/T86973#980607 (mforns) Open>Resolved [22:09:40] milimetric: yt? [22:10:03] kevinator: yes [22:10:14] did you know Pentaho was down? [22:10:43] yes, but not because monitoring worked [22:10:57] but because Christian told me [22:11:08] ah, I was wondering if monitoring worked [22:11:09] (p.s. I gave up trying to not use his name 'cause he reads the logs too carefully anyway :)) [22:11:31] :-) [22:11:36] sadly no, shinken told me nothing [22:11:48] btw, i sent a rough draft of what I see as the changes to the dashboarding pipeline [22:11:59] oh, cool, I’ll look at that [22:12:03] not very specific but it gives us something graphic to point at so we can put the stuff from this morning into context [22:27:25] Analytics, Multimedia: Per-file view stats - https://phabricator.wikimedia.org/T77541#980692 (Tgr) [22:27:34] Analytics, Multimedia: Per-file view stats - https://phabricator.wikimedia.org/T77541#829103 (Tgr) [22:33:18] hey folks. any idea why http://pentaho.wmflabs.org/ shows a blank white page? Deskana tells me i should be able to see stats there [23:26:46] (PS1) Gergő Tisza: Show page view / image view comparison [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/185336 (https://phabricator.wikimedia.org/T78189)