[00:06:10] 06Analytics-Kanban, 13Patch-For-Review: Add unique devices dataset to pivot - https://phabricator.wikimedia.org/T159471#3188464 (10JKatzWMF) >>! In T159471#3172456, @JAllemandou wrote: > Note: Only daily uniques are imported into druid. Monthly don't work because of druid not allowing for monthly granularity q... [02:48:26] 10Analytics, 10Analytics-General-or-Unknown: Provide regular cross-wiki reports on flagged revisions status - https://phabricator.wikimedia.org/T44360#3188761 (10Zache) Still relevant. Current Special:ValidationStatistics is also broken; see bug T163107 . [03:52:13] 10Analytics, 10Analytics-Cluster: can't compile numpy on stat1004 - https://phabricator.wikimedia.org/T163177#3188807 (10EBernhardson) [03:56:04] 10Analytics, 10Analytics-Cluster: can't compile numpy on stat1004 - https://phabricator.wikimedia.org/T163177#3188822 (10EBernhardson) I was going to just write up a puppet patch, but it's not clear what the right soution is. stat1002 has all kinds of useful stuff via the puppet class `::statistics::packages`.... [08:47:59] 10Analytics, 10Analytics-Cluster, 13Patch-For-Review: can't compile numpy on stat1004 - https://phabricator.wikimedia.org/T163177#3188807 (10JAllemandou) Hi @EBernhardson, thanks for doing this patch ! I've used stat1002 so far to build python venvs, it'll be awesome to be able to do it on stat1004. [08:50:32] 10Analytics: upgrade druid to 0.9.2 - https://phabricator.wikimedia.org/T157977#3022179 (10JAllemandou) This is a blocker for https://phabricator.wikimedia.org/T159471, let's move forward fast if we can. [08:51:29] 06Analytics-Kanban, 13Patch-For-Review: Add unique devices dataset to pivot - https://phabricator.wikimedia.org/T159471#3189247 (10JAllemandou) @JKatzWMF: We want to import monthly (code is ready: https://gerrit.wikimedia.org/r/#/c/348052/), but our version of druid and pivot can't handle the daily query granu... [08:55:26] (03CR) 10Joal: [C: 031] "Minor comments, looks good to me !" (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/346755 (https://phabricator.wikimedia.org/T157362) (owner: 10Fdans) [08:57:21] (03CR) 10Joal: [C: 031] "LGTM !" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/348461 (https://phabricator.wikimedia.org/T157362) (owner: 10Fdans) [10:05:40] going afk for ~1hour people (early lunch + errand) [10:05:43] brb! [11:50:54] while installing piwik on my VM [11:50:55] "From time to time, Piwik database schema must be upgraded so Piwik can continue to work correctly. You have the ability to automate the process so the database is upgraded automatically. Though, you should ensure you have proper backup in place." [11:51:06] ............................ [11:51:10] ......................... [12:03:14] 10Analytics, 10Analytics-Dashiki, 13Patch-For-Review: Create dashboard for upload wizard - https://phabricator.wikimedia.org/T159233#3189534 (10matthiasmullie) @Milimetric: The patch for the data is https://gerrit.wikimedia.org/r/#/c/340720/ I can have it merged, but I want to be sure it's in a format/config... [12:47:33] elukey: systems built for non-engineers :( [12:48:06] taking a break now - I finally managed to get all my beginning-of-year-administrative stuff done ! Yay ! [12:49:41] 06Analytics-Kanban: Security Upgrade for piwik - https://phabricator.wikimedia.org/T158322#3189599 (10elukey) Today I've set up a Debian Jessie VM, installed 2.16 and tried to upgrade to 2.17-1. This is the result of the db upgrade script: ``` vagrant@debian-jessie:~$ sudo php /usr/share/piwik/console core:upda... [12:52:37] milimetric: tried to migrate piwik and everything went fine (on my VM), we are good to do in my opinion :) [12:52:54] the LTS database changes to get to 2.17 seems very small (contained in the task --^) [13:04:39] 06Analytics-Kanban: Piwik improvements - https://phabricator.wikimedia.org/T163000#3189631 (10elukey) [13:17:49] (03CR) 10Fdans: [C: 032] Change name of field to revision_seconds_to_identity_revert [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/348461 (https://phabricator.wikimedia.org/T157362) (owner: 10Fdans) [13:19:55] (03PS4) 10Fdans: Update jobs to remove is_productive and update time to revert [analytics/refinery] - 10https://gerrit.wikimedia.org/r/346755 (https://phabricator.wikimedia.org/T157362) [13:26:43] elukey: where's the DB upgrade script? I'll take a look just in case [13:27:49] milimetric: o/ [13:27:57] they just say to run sudo php /usr/share/piwik/console core:update [13:28:08] and before starting they ask you if you are ok with the diff [13:28:11] (in the task) [13:28:25] it seems that there is only one alter table to run [13:28:29] in our use case [13:28:38] oh ok, so they have some table versioning thing [13:28:45] sounds good then, elukey [13:33:43] (03CR) 10Fdans: [V: 032 C: 032] Update jobs to remove is_productive and update time to revert [analytics/refinery] - 10https://gerrit.wikimedia.org/r/346755 (https://phabricator.wikimedia.org/T157362) (owner: 10Fdans) [13:33:46] milimetric: who should we alert for this upgrade? [13:34:56] elukey: I think the mobile team is the only one that relies on any kind of accuracy here, so maybe cc them [13:35:02] the other users are us and comms [13:35:58] if we have a longer outage we can let comms know so they can read the numbers correctly [13:38:19] milimetric: I hope for maximum 30mins [13:38:28] the alter table should be quick [13:38:38] yeah, I wouldn't worry then [13:39:15] nobody would actually do anything different if we told them, only take the numbers with a grain of salt if they drop a lot [13:39:26] and in 30 minutes nobody's numbers are going to drop at all [13:42:30] 10Analytics: Metrics and Dashboards for Piwik - https://phabricator.wikimedia.org/T163204#3189718 (10elukey) [13:43:06] 06Analytics-Kanban, 15User-Elukey: Piwik puppet configuration refactoring and updates - https://phabricator.wikimedia.org/T159136#3189732 (10elukey) [13:44:32] 10Analytics, 10Analytics-Cluster: Update SSH fingerprints page for stat1004 - https://phabricator.wikimedia.org/T162972#3189739 (10Ottomata) 05Open>03Resolved Huh, I didn't even no that we tracked server fingerprints on wiki. OooooooK! [13:52:04] 10Analytics, 10Analytics-Cluster, 13Patch-For-Review: can't compile numpy on stat1004 - https://phabricator.wikimedia.org/T163177#3189752 (10Ottomata) BTW, python-numpy and python3-numpy are installed on all hadoop workers. I think we should stall this ticket on T152712. We'll need to sort out and clean u... [14:08:11] (03Abandoned) 10Hashar: uild: run karma test with just "npm test" [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/315659 (https://phabricator.wikimedia.org/T148019) (owner: 10Hashar) [14:08:17] 10Analytics, 10Analytics-Dashiki, 10Continuous-Integration-Config, 13Patch-For-Review: Add CI job for Dashiki - https://phabricator.wikimedia.org/T148019#3189797 (10hashar) 05Open>03declined [14:11:52] 10Analytics, 10Analytics-Dashiki, 13Patch-For-Review: Create dashboard for upload wizard - https://phabricator.wikimedia.org/T159233#3189801 (10Milimetric) oh great, I didn't see this change until today, apologies if it was in my incoming. Reviewing now. [14:14:32] joal: hiii, scala q if you are around [14:18:56] hi team! :] [14:20:59] hiii [14:22:37] o/ [14:25:21] (03CR) 10Milimetric: [C: 04-1] Query the most common UploadWizard exceptions & errors (038 comments) [analytics/limn-multimedia-data] - 10https://gerrit.wikimedia.org/r/340720 (https://phabricator.wikimedia.org/T156694) (owner: 10Matthias Mullie) [14:41:17] 10Analytics, 10Analytics-Dashiki, 10Continuous-Integration-Config, 13Patch-For-Review: Add CI job for Dashiki - https://phabricator.wikimedia.org/T148019#3189884 (10mforns) Thanks @hashar anyway! [14:43:45] Hey ottomata :) [14:43:50] scalaaaaaa ! [14:46:05] HyeyYyy [14:46:21] was going to ask you a sorting question, but i think i changed my mind... [14:46:39] okey :) [14:47:08] ottomata: sorting is very tricky in scala, since it can often rely on implicits [14:47:12] yeah [14:47:26] i was going to try to make a custom sorting function that took partitions into account, and didn't sort those [14:47:36] but then, i realized i only need to be careful with that when creating the table [14:56:24] a_team: i will be at QR, missing standup [14:56:44] nuria: I'll do the same if you don't mind [14:58:28] a-team, then cancel standup? [14:59:31] ok w me [15:00:16] a-team: standup canceled, everyone should have 2 invites for QR [15:02:19] (03PS6) 10Nuria: Add oozie job loading daily uniques in druid [analytics/refinery] - 10https://gerrit.wikimedia.org/r/347611 (https://phabricator.wikimedia.org/T159471) (owner: 10Joal) [15:06:40] joal: teh daily uniques druid job seems good to go correct? You did test it , right? [15:07:04] *the [15:07:40] milimetric: did we get anywhere yesterday with the issues of dashiki extension? [15:07:51] Hi nuria - I tested it, and if naming / conventions etc are good to you, should be good to go :) [15:08:00] (03CR) 10Nuria: [V: 032 C: 032] Add oozie job loading daily uniques in druid [analytics/refinery] - 10https://gerrit.wikimedia.org/r/347611 (https://phabricator.wikimedia.org/T159471) (owner: 10Joal) [15:11:00] nuria: yes, we got to me finding out about this super important and yet not really clearly documented rule: [15:11:06] ay ay [15:11:09] mediawiki-config must always be deployed immediately after merge [15:11:27] (I was trying to merge my config changes but we're in a deployment freeze) [15:11:52] so I have to wait until we're done with the codfw experiment, meaning we're stuck for another couple of weeks, going to put it in paused [15:12:07] right, cause if they are so coupled why are there 2 different deploys? ... so many things about mw i do not understand [15:12:56] 2 deploys make sense, they're not really coupled by design, just how people do some extensions [15:43:56] 06Analytics-Kanban, 10Analytics-Wikistats, 06Community-Liaisons: Create and monitor Round2 consultation page - https://phabricator.wikimedia.org/T162155#3190103 (10Elitre) [15:50:15] (03CR) 10Mforns: Add annotations to tabs layout (031 comment) [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/348227 (https://phabricator.wikimedia.org/T162482) (owner: 10Mforns) [16:10:04] joal: got a min for brain bounce? [16:10:14] sure ottomata [17:01:21] 10Analytics: upgrade druid to 0.9.2 - https://phabricator.wikimedia.org/T157977#3190378 (10Nuria) In order to actually take advantage of the upgrade pivot needs to be upgraded too. [17:16:01] * elukey off!! o/ [18:02:59] 06Analytics-Kanban: Review by legal department of text on wikistats site - https://phabricator.wikimedia.org/T163229#3190638 (10Nuria) [18:04:22] 06Analytics-Kanban: Review by legal department of text on wikistats site - https://phabricator.wikimedia.org/T163229#3190638 (10Nuria) A suggestion from legal: There is some draft legal language in the footer. I think we can help improve and simplify. Instead of saying: "To the extent possible under law, The W... [18:07:26] ottomata: I need to help Melissa with Lino [18:07:36] ottomata: can we take it back tomorrow? [18:08:58] ya [18:09:06] ttyt joal [18:26:47] 10Analytics, 10ChangeProp, 10EventBus, 06Services (later): Make EventBus service support wildcards in schema definitions - https://phabricator.wikimedia.org/T157091#3190784 (10Pchelolo) @Ottomata would you have any spare cycles to work on this or to guide me a bit through the event logging code to see how... [18:36:05] 10Analytics, 10ChangeProp, 10EventBus, 06Services (later): Make EventBus service support wildcards in schema definitions - https://phabricator.wikimedia.org/T157091#3190818 (10Ottomata) Sure! If you want to take a look, this is handled by [[https://github.com/wikimedia/eventlogging/blob/master/eventloggin... [18:37:24] 10Analytics, 10ChangeProp, 10EventBus, 06Services (later): Make EventBus service support wildcards in schema definitions - https://phabricator.wikimedia.org/T157091#3190821 (10Pchelolo) All right @Ottomata, I'll try to remember python which I didn't use in like 5 years, let's see where it gets us :) [19:01:35] 10Analytics, 06Collaboration-Team-Triage, 10Edit-Review-Improvements-ReviewStream, 10EventBus, and 3 others: Add fields needed by ERI to mediawiki.revision-create - https://phabricator.wikimedia.org/T145164#3190956 (10jmatazzoni) [19:02:17] 10Analytics, 10ChangeProp, 06Collaboration-Team-Triage, 10Edit-Review-Improvements-ReviewStream, and 4 others: Set up the foundation for the ReviewStream feed - https://phabricator.wikimedia.org/T143743#3190958 (10jmatazzoni) [19:28:54] 10Analytics-EventLogging, 06Analytics-Kanban: Implement EventLogging Hive refinement - https://phabricator.wikimedia.org/T162610#3191152 (10Ottomata) [19:29:26] 10Analytics-EventLogging, 06Analytics-Kanban: Implement EventLogging Hive refinement - https://phabricator.wikimedia.org/T162610#3168526 (10Ottomata) a:03Ottomata [19:31:19] 06Analytics-Kanban: Review by legal department of text on wikistats site - https://phabricator.wikimedia.org/T163229#3190638 (10Slaporte) Thanks Nuria. Let me know when you're ready for review. [19:36:19] 06Analytics-Kanban: Security Upgrade for piwik - https://phabricator.wikimedia.org/T158322#3191206 (10JMinor) Nope, that should be no problem from out side. Thanks for checking! [19:52:30] 10Analytics-EventLogging, 06Analytics-Kanban: Implement EventLogging Hive refinement - https://phabricator.wikimedia.org/T162610#3191312 (10Ottomata) Working on this. It's getting funky! Talked with Joseph for an hour today about an issue, and I've got things to write down before they fall out of my brain.... [19:52:39] 10Analytics-EventLogging, 06Analytics-Kanban: Implement EventLogging Hive refinement - https://phabricator.wikimedia.org/T162610#3191330 (10Ottomata) Or, we could flatten :( [19:52:59] 10Analytics: Add templating support to reportupdater scripts - https://phabricator.wikimedia.org/T163252#3191334 (10Milimetric) [20:30:31] 10Analytics-EventLogging, 06Analytics-Kanban: Implement EventLogging Hive refinement - https://phabricator.wikimedia.org/T162610#3191490 (10Ottomata) OOf, actually: > 4. inputDF = sql.read.schema(uberSchema).json(path) // This re-reads the input data, this time with our Hive table's uberSchema This doesn't w... [21:54:01] 10Analytics, 10Analytics-EventLogging, 10MobileFrontend, 06Reading-Web-Backlog, and 3 others: Should be possible to override sampling in EventLogging schemas for development purpose - https://phabricator.wikimedia.org/T125122#3191905 (10Jdlrobson) [23:07:09] 10Analytics, 10Analytics-EventLogging, 10MobileFrontend, 06Reading-Web-Backlog, and 3 others: Should be possible to override sampling in EventLogging schemas for development purpose - https://phabricator.wikimedia.org/T125122#1978941 (10Nuria) Not sure we understand. Development eventlogging requests do no... [23:09:49] musikanimal: did you see my latest round of comments on Xhamster issue? [23:10:46] musikanimal: I am going to close ticket but you are correct that a portion of traffic comes from some sort of proxy, not the majority on the hours i looked at but a significant portion [23:11:08] I did, thank you for the deep analysis! [23:12:28] I'm probably going to keep it excluded in Topviews, and The Signpost will probably do the same [23:13:24] it's just too weird to have it topping the charts =P [23:13:52] this one is tricky because as you say most does look organic, so it's hard to decide where to draw the line [23:14:06] musikanimal: ya, when we do the bot marking of traffic this will be one of the things we look to; the histogram of ips/requests on say, hourly basis, we are not there yet though [23:14:33] musikanimal: but in this case the Bulk of it is organic traffic, likely the "boss button" of some app [23:19:48] 06Analytics-Kanban: Check abnormal pageviews for XHamster - https://phabricator.wikimedia.org/T158071#3192266 (10Nuria) 05Open>03Resolved [23:21:17] 10Analytics, 10Analytics-EventLogging, 10MobileFrontend, 06Reading-Web-Backlog, and 3 others: Should be possible to override sampling in EventLogging schemas for development purpose - https://phabricator.wikimedia.org/T125122#3192270 (10Jdlrobson) 05Open>03declined I think we can decline this on second... [23:21:48] 10Analytics, 10Analytics-EventLogging, 10MobileFrontend, 06Reading-Web-Backlog, and 3 others: Should be possible to override sampling in EventLogging schemas for development purpose - https://phabricator.wikimedia.org/T125122#3192273 (10Jdlrobson) See also: https://gerrit.wikimedia.org/r/348877 [23:30:52] 10Analytics, 10Analytics-EventLogging, 10MobileFrontend, 06Reading-Web-Backlog, and 3 others: Should be possible to override sampling in EventLogging schemas for development purpose - https://phabricator.wikimedia.org/T125122#3192312 (10Nuria) Development meaning loical box and you wanted to have no sampli... [23:33:55] 10Analytics, 10Analytics-EventLogging, 10MobileFrontend, 06Reading-Web-Backlog, and 3 others: Should be possible to override sampling in EventLogging schemas for development purpose - https://phabricator.wikimedia.org/T125122#3192323 (10Jdlrobson) Development as in localhost yup exactly. This dates back to... [23:40:59] 10Analytics, 10Analytics-EventLogging, 10MobileFrontend, 06Reading-Web-Backlog, and 3 others: Should be possible to override sampling in EventLogging schemas for development purpose - https://phabricator.wikimedia.org/T125122#3192343 (10Nuria) Ahhh, it is *last year's*!!! Duh ! Closing. [23:41:18] 10Analytics, 10Analytics-EventLogging, 10MobileFrontend, 06Reading-Web-Backlog, and 3 others: Should be possible to override sampling in EventLogging schemas for development purpose - https://phabricator.wikimedia.org/T125122#3192344 (10Nuria) 05declined>03Invalid