[04:13:12] Analytics, Wikipedia-iOS-App-Product-Backlog: Fix iOS uniques in mobile_apps_uniques_daily after 5.0 launch - https://phabricator.wikimedia.org/T130432#2139727 (Tbayer) Thanks Madhu, that sounds plausible; I'll let the iOS team confirm (Monte and I were looking at this very briefly on Friday, but that exp... [07:02:58] Analytics-Kanban: Get jenkins to automate releases {hawk} - https://phabricator.wikimedia.org/T130122#2139853 (madhuvishy) a:madhuvishy [07:05:30] Analytics-Kanban: [Spike] Figure out how to automate releases with jenkins - https://phabricator.wikimedia.org/T130576#2139858 (madhuvishy) [07:05:39] Analytics-Kanban: [Spike] Figure out how to automate releases with jenkins {hawk} - https://phabricator.wikimedia.org/T130576#2139858 (madhuvishy) [08:22:05] o/ [09:29:43] Hi elukey \o [10:43:22] (PS2) Joal: Add unique_devices endpoint to AQS [analytics/aqs] - https://gerrit.wikimedia.org/r/277784 (https://phabricator.wikimedia.org/T129518) [10:44:23] (CR) Joal: "Tests pass, tested locally with dockerised cassandra." [analytics/aqs] - https://gerrit.wikimedia.org/r/277784 (https://phabricator.wikimedia.org/T129518) (owner: Joal) [11:02:57] Analytics, Datasets-General-or-Unknown, Operations, Traffic, Patch-For-Review: http://dumps.wikimedia.org should redirect to https:// - https://phabricator.wikimedia.org/T128587#2140256 (ArielGlenn) I remember there was a discussion and I don't remember why it was wontfixed then. But we can s... [11:09:58] (PS5) Joal: Add cassandra unique devices load job [analytics/refinery] - https://gerrit.wikimedia.org/r/277507 (https://phabricator.wikimedia.org/T129519) [11:12:56] (PS6) Joal: Add cassandra unique devices load job [analytics/refinery] - https://gerrit.wikimedia.org/r/277507 (https://phabricator.wikimedia.org/T129519) [11:24:21] Analytics, Datasets-General-or-Unknown, Operations, Traffic, Patch-For-Review: http://dumps.wikimedia.org should redirect to https:// - https://phabricator.wikimedia.org/T128587#2079753 (hashar) Some past tasks: * {T83675} * Declined: {T60292} The later had: > https://download.wikimedia.org/m... [11:53:44] * elukey afk for lunch [12:54:38] o/ milimetric & joal [12:54:50] I think we should cancel this week's meeting. [12:55:01] I wasn't able to make any progress since the last one :( [12:55:46] halfak: no problem for me [12:56:35] If you're curious, the reason that I've sort-of been checked out from this work is ORES. We're right on the cusp of properly being in production. [12:57:05] halfak: That's indeed a great reason :) [12:58:10] halfak: analytics next quarter goals include having historical edits in HDFS formatted in EventBus schema [12:58:26] Oooh! That's awesome! [12:58:30] halfak: I think those could be our future discussion topics ;) [12:58:37] That sounds great :) [12:59:08] halfak: I also don't forget wiki-hadoop automation, but I have not dedicated time lately [12:59:35] Totally reasonable. I think that the next step is on me anyway. [12:59:43] Gotta run a test with the frwiki data. [13:00:05] halfak: I can always try to produce automation scripts and offer PRs ;) [13:02:45] joal, once I can get some basic productivity measures for frwiki and some of the emerging communities, I think it might be a good idea to start looking at event production. [13:02:53] We'll probably not work from the dumps though [13:03:00] Only edits to non-deleted pages there. [13:03:27] halfak: hmmm [13:03:52] halfak: we need to have that discussion on historical events (later, after your production rush is passed) [13:06:45] joal / halfak: cool with me but let's ping schana too since he's invited to the meetings [13:06:54] schana: probably good to add -analytics to your channel list [13:07:22] Woops. Forgot that schana has been joining us. Thanks milimetric [13:07:34] schana is also helping us move ORES to prod. [13:07:39] Thanks milimetric [13:07:40] :) [13:08:54] Analytics, Reading-Admin, Zero: Country mapping routine for proxied requests - https://phabricator.wikimedia.org/T116678#2140497 (BBlack) @dr0ptp4kt - yes, it's possible/likely that the AS32934 info you linked could be used to verify internet.org proxying. Probably the best method for approaching this... [13:09:15] (PS7) Joal: Add cassandra unique devices load job [analytics/refinery] - https://gerrit.wikimedia.org/r/277507 (https://phabricator.wikimedia.org/T129519) [13:09:39] hm, for ottomata when he joins: http://www.meetup.com/Chicago-Area-Kafka-Enthusiasts/events/228620608/ [13:09:47] (CR) Joal: "Tested on cluster using test-keyspace in cassandra." [analytics/refinery] - https://gerrit.wikimedia.org/r/277507 (https://phabricator.wikimedia.org/T129519) (owner: Joal) [13:19:32] ooh joal you should tell me how you do that test! [13:19:53] milimetric: Sure ! [13:20:15] milimetric: I ran a test version locally using dockerised cassandra [13:20:16] oh wait no I got excited, I thought you tested the AQS module :) [13:20:25] wait... [13:20:28] now I'm confused :) [13:20:30] milimetric: I did, locally ;) [13:20:53] sorry, I heard cassandra and thought client, you tested the load job [13:20:54] cool [13:20:58] I tested AQS on local with dockerised cassandra, and tested data loading on cluster on test keyspace [13:21:03] should I review this? [13:21:11] Please feel free :) [13:21:42] I renamed fields as we discussed yesterday, everything seems ready to go (except restbase front) [13:21:47] milimetric: --^ [13:22:21] sweet, ok, I'll +1 it but we should wait to merge until we coordinate, right? Because if we merge it would be un-deployable [13:22:38] milimetric: which one ? [13:22:51] the AQS change [13:23:16] hm, not sure I understand why, but I trust you :) [13:23:34] I thought we could have an aqs endpoint not used by restbase front [13:23:52] But if they ask us for changes, then it's changes to deploy on our side again ... [13:23:57] hm [13:24:21] joal: but you changed pageviews.yaml too, removing the /pageviews [13:24:37] milimetric: RIIIIIGHT ! Forgot about that, good catch [13:24:44] milimetric: let's wait indeed [13:24:46] that's all it is, otherwise you're right [13:27:41] (CR) Milimetric: [C: -1] "This is actually a +2, but one of the changes requires a change in the front-end restbase config, so we have to coordinate before we merge" [analytics/aqs] - https://gerrit.wikimedia.org/r/277784 (https://phabricator.wikimedia.org/T129518) (owner: Joal) [13:28:02] sweet, code's so clean I could kiss it [13:28:54] milimetric: I never kiss code, you never whose hands has played with it before ! [13:29:47] but it's so clean! :) [13:30:05] :D [13:36:32] joal, hi [13:36:38] hi mforns [13:36:43] :] one question [13:37:01] are requests to the API counted as pageviews? [13:37:19] mforns: depends which [13:37:24] aha [13:38:00] mforns: API requests made by mobile apps (or more precisely identified as so by our formula) are in [13:39:31] I see, all other are out? [13:39:31] mforns: I think they are [13:39:31] ok [13:39:53] thanks joal! [13:46:08] np mforns :) [14:10:31] milimetric: just realized something about the need to wait ofr restbase: I removed "pageviews" in module finition, but added it in project definition, URL doesn't change [14:12:57] * elukey pictures milimetric like http://vignette4.wikia.nocookie.net/a-team/images/f/f6/Orginal_John_Hannibal_Smith.jpg/revision/latest?cb=20100923000610 [14:13:25] +1 elukey :) [14:13:29] joal: but isn't projects/aqs_default.yaml what's configured in the front-end restbase as well? [14:13:32] in puppet? [14:14:11] milimetric: I would have thought it's a copy of pageviews.yaml, but since I have looked into it, can't say [14:14:18] lol [14:14:56] just one step closer to Liam Nieson, elukey, the ultimate achievement :) [14:15:36] joal: yeah, I'm fuzzy on all that too, after the shuffling around of configs I have too many old memories in my head [14:15:53] xD [14:16:13] milimetric: I'll make sure I kinda understand, then send a PR on github for services team to review with links to our changes [14:16:35] And then maybe decide on what to do afer :) [14:17:03] ok [14:17:14] you might be right though, I'm just not sure [14:17:24] np at all :) [14:17:30] PR and all needs to be done [15:14:28] (PS2) Joal: Change webrequest load to fail in data loss case [analytics/refinery] - https://gerrit.wikimedia.org/r/278325 (https://phabricator.wikimedia.org/T130187) [15:15:09] (CR) Joal: "Tested on cluster." [analytics/refinery] - https://gerrit.wikimedia.org/r/278325 (https://phabricator.wikimedia.org/T130187) (owner: Joal) [15:15:32] a-team: I am currently re-imaging the last Redis Job Queue host and I'd need to use my "meeting delete" ticket to avoid long delays for the jobs that are "suspended" at the moment [15:15:48] k, np [15:15:49] elukey: Good luck mate :S [15:30:59] a-team: standdduppp [15:45:21] (CR) Milimetric: "tiny nit, but otherwise I'm ok to merge this" (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/277507 (https://phabricator.wikimedia.org/T129519) (owner: Joal) [15:54:18] joal, ottomata: still in the middle of the re-image, I need to apply the backup to redis and then put the host in service asap. :( [16:03:29] Analytics, Operations, hardware-requests, Patch-For-Review: eqiad: (3) AQS replacement nodes - https://phabricator.wikimedia.org/T124947#2141036 (Ottomata) Bump. [16:03:35] Analytics-Cluster, Operations, hardware-requests: eqiad: New Hive / Oozie server node in eqiad Analytics VLAN - https://phabricator.wikimedia.org/T124945#2141038 (Ottomata) Bump. [16:06:49] (PS1) Mforns: [WIP] Link to the new browser-reports [analytics/wikistats] - https://gerrit.wikimedia.org/r/278917 (https://phabricator.wikimedia.org/T129101) [16:09:47] ok np! [16:09:51] oh joal, meant to ask [16:09:59] what' up with the _partitioned dataset name? [16:10:02] i see you are removing it [16:10:13] but i'm not sure why, since i don't see it being used in this patchset...isn't that used by refine? [16:11:41] ottomata: I'm not removing it, I'm changing the dataset dependency in refine job to be the core one (based on _SUCCESS flag) [16:13:17] ah yes, ok does anything depend on _partitioned now then? [16:14:03] also quick q, why the change in date formats in https://gerrit.wikimedia.org/r/#/c/278325/2/oozie/webrequest/load/coordinator.xml ? [16:14:42] ottomata: nothing depends on partition now ... I kept it for historical reasons, but we can remove it if you prefer [16:15:30] ottomata: date change in coordinator is to have padded folders. Can be changed back (just aesthetic reason) [16:19:48] hm, that won't be a problem? [16:19:53] i think there was a choice to not have padded..... [16:19:57] um [16:20:08] ottomata: I tested it, no issue [16:20:39] Only issue is if we backfill: manual data deletion is needed (partition name is hive is written as string even if int) [16:21:26] joal: i think there is a reason [16:21:30] a tricky hive reason [16:21:34] looking for qchris' docs [16:21:44] ottomata: sure [16:27:19] joal: i cannot find the doc! [16:27:26] ottomata: https://wikitech.wikimedia.org/wiki/File:Hive_partition_formats.png [16:27:29] YES! [16:27:41] haha, its an image that's why no searching helped [16:27:59] Wait. You cannot search text in images? [16:28:00] :-P [16:28:05] heheh [16:28:13] :) [16:28:22] Depends on who hosts the image ;) [16:28:30] Yes, the image sucks. [16:28:44] Feel free to Wiki-encode it and kill the image. [16:29:04] thanks qchris! [16:29:08] yw. [16:29:27] IIRC the issue was the usage in HQL commands. [16:29:34] Like: [16:29:38] joal: hmmm, so maybe what I remembered isn't relevant, it looks like it matters more with what the partition was added with [16:29:41] and used in query commands [16:29:46] not the location path in hdfs [16:30:05] 06 > '5' ... maybe such issues are gone in newer Hives. [16:30:10] but [16:30:16] i think we chose to use the non padded path [16:30:25] ottomata: Trick is that the way partition is added (here using padding 0 with my change) also applies to partition path on hdfs [16:30:27] because the best use was to add partitions like '6' [16:30:37] just to keep the partition matching the path [16:30:47] When using ints to select, no issues (hive succesfully converts 05 to 5, YEAH) [16:31:01] When using text matching, can be troublesome [16:31:53] so ummm, i think we should probably leave it as is, no? [16:31:56] I also just thought that the files I generate will not be read: we'll go to statistics table - So never minf is they are not in padded format, I'll revert that part of the change [16:32:00] ottomata: --^ [16:32:05] maybe link to https://wikitech.wikimedia.org/wiki/File:Hive_partition_formats.png in comments so we remeber? [16:32:19] ok cool [16:32:29] ottomata: I don't really think the link is needed either [16:32:35] I'll just revert that [16:32:45] ok, maybe write a comment about why we chose to not do padded thenn [16:32:46] so we remember [16:33:09] k [16:33:13] we choose not padded because hive behaves better when the partition is not added as padded string, and we wanted to keep partitions and locations consistent [16:44:41] (CR) Ottomata: Change webrequest load to fail in data loss case (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/278325 (https://phabricator.wikimedia.org/T130187) (owner: Joal) [16:44:46] joal: cool, aside from that, only one comment [16:44:47] :) [16:44:49] looks awesome! [16:46:33] so ottomata just to confirm: Do you want me to remove partitioned dataset? [16:46:38] oh [16:46:41] no that's fine, you can leave it [16:46:46] might be handy for something [16:46:46] ok cool :) [16:51:10] any objections to be doing a rolling restart of the AQS Cassandra cluster? [16:51:37] probably unnecessary, but to be sure that https://gerrit.wikimedia.org/r/#/c/278330/ and https://gerrit.wikimedia.org/r/#/c/277265/ are applied [16:51:54] s/be/me/ [16:52:03] if someone else wants to do it that is fine too [16:52:16] urandom: In staff meeting in a minute [16:52:30] urandom: Please go for it, I'll keep an eye on monitoring [16:52:38] urandom: Thanks :) [16:53:19] (CR) Joal: Change webrequest load to fail in data loss case (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/278325 (https://phabricator.wikimedia.org/T130187) (owner: Joal) [16:53:28] (PS3) Joal: Change webrequest load to fail in data loss case [analytics/refinery] - https://gerrit.wikimedia.org/r/278325 (https://phabricator.wikimedia.org/T130187) [16:53:57] joal: sure thing [16:58:04] (CR) Joal: Add cassandra unique devices load job (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/277507 (https://phabricator.wikimedia.org/T129519) (owner: Joal) [16:58:12] (PS8) Joal: Add cassandra unique devices load job [analytics/refinery] - https://gerrit.wikimedia.org/r/277507 (https://phabricator.wikimedia.org/T129519) [16:58:47] Analytics-Kanban, EventBus, Patch-For-Review: Deploy mediawiki/event-schemas with scap separately instead of using submodule in eventlogging - https://phabricator.wikimedia.org/T127099#2032407 (Ottomata) This was too hard to do via scap. Am using git::clone in puppet instead. [17:01:09] a-team: batcave or other hangout? [17:01:39] Some of us are in the other [17:01:41] madhuvishy: [17:01:45] yup ok :) [17:02:01] oop [17:02:02] batcave! [17:02:04] nuria: could you change the staff invite to the batcave after this, it's always confusing [17:02:08] always batcave! [17:02:09] right? [17:02:12] YES [17:02:12] god [17:02:13] yes [17:02:13] :) [17:02:28] i have changed that already but i cannot chnage teh "series" [17:02:30] *the [17:02:36] do you folks need me today? I'm going to skip out otherwise, have a crazy day :/ [17:02:42] what? really? google calendar /me shakes fist [17:02:52] k, np ori [17:02:53] i just fixed it! [17:03:07] uhhh maybe i didn't [17:03:21] ottomata: yeah, that's what I thought I did too but it doesn't save [17:03:26] nope [17:03:26] An error has occurred. Please try again later. [17:03:28] maybe we just need to delete and re-create [17:03:29] yeah [17:05:47] urandom: Thank you for the restart! [17:05:59] joal: no worries [17:22:12] joal: question for youuu - tomorrow do you mind if we stop HDFS to add automatic failover? [17:23:04] elukey: np :0 [17:23:28] Analytics: Wikistats 2.0. Edit Reports: Setting up a pipeline to source Historical Edit Data into hdfs {lama} - https://phabricator.wikimedia.org/T130256#2141492 (Nuria) [17:29:34] mforns: you dropped erik z. on "to" line on your replies [17:29:44] mforns: on teh wikistats threads [17:29:46] *the [17:29:57] nuria, !!! [17:30:07] mforns: it's gmail [17:31:38] thanks nuria [17:32:23] (CR) Nuria: [C: 1] "Looks good, waiting for Erik's feedback" [analytics/wikistats] - https://gerrit.wikimedia.org/r/278917 (https://phabricator.wikimedia.org/T129101) (owner: Mforns) [17:32:24] Analytics, MediaWiki-API, Pageviews-API, RESTBase: RFC: Update profile URLs in content types to point to format documentation - https://phabricator.wikimedia.org/T128609#2141549 (GWicke) a:GWicke [17:41:47] ottomata: when do you have time to merge/test https://gerrit.wikimedia.org/r/#/c/269696/ with me? [17:42:01] I'm writing the email, and I'll need to finish that first [17:42:06] but just asking so I know to plan [17:45:30] milimetric, joal: any of you that has patience/time to explain AQS to me in the next days? [17:46:20] sure! [17:46:23] whenever elukey [17:48:30] milimetric: thanks! I guess that we have some docs that I should read to avoid stupid questions first, I'll try to find them [17:49:00] https://wikitech.wikimedia.org/wiki/Analytics/AQS [17:49:05] my goal would be to be able to respond to "AQS is having trouble!" or just to do some maintenance [17:51:23] wikimedia/mediawiki-extensions-EventLogging#544 (wmf/1.27.0-wmf.18 - 7b0c058 : thcipriani): The build has errored. [17:51:23] Change view : https://github.com/wikimedia/mediawiki-extensions-EventLogging/commit/7b0c058711c5 [17:51:24] Build details : https://travis-ci.org/wikimedia/mediawiki-extensions-EventLogging/builds/117772039 [17:53:17] milimetric: for example https://phabricator.wikimedia.org/T123629 - is it something that I can do with the appropriate knowledge or should it be handled by some other team? [17:54:23] elukey: that upgrade should be identical with the restbase upgrade, aqs doesn't add any dependencies so there should be no problem [17:54:29] so yes, you can totally do that one [17:54:45] I wasn't aware of it, it probably should've been tagged analytics [17:55:08] yeah I just remember I had it in my backlog :P [17:57:11] http://config-master.wikimedia.org/conftool/eqiad/aqs all managed with conftool, love i t [18:04:56] Analytics: Upgrade aqs* to nodejs 4.3 - https://phabricator.wikimedia.org/T123629#2141684 (elukey) [18:07:48] ---^ Cc joal, ottomata [18:08:42] elukey: eh? [18:10:11] ottomata: just wanted to let you know, it was in my backlog [18:11:14] oh write docs? [18:12:27] milimetric: looks pretty simple, ja just adds a page and changes links? [18:15:15] ottomata: no no I meant me upgrading nodejs on AQS for https://phabricator.wikimedia.org/T123629 [18:16:03] ahhh ok [18:16:03] cool [18:16:16] nuria: you ok if I just make one public event streams tracking task for now? [18:16:24] not sure i'm ready to make individual tasks [18:20:34] oooooookkkkkk team, going offline! [18:20:36] laters! [18:20:37] talk with you tomorrow o/ [18:25:12] Analytics, Wikimedia-Stream: Public Event Streams - https://phabricator.wikimedia.org/T130651#2141764 (Ottomata) [18:26:40] team, see you tomorrow! [18:27:33] Analytics, EventBus, Wikimedia-Stream: Public Event Streams - https://phabricator.wikimedia.org/T130651#2141801 (Ottomata) [18:32:38] ottomata: yes, just page + links changes [18:32:48] milimetric: shall we merge? [18:32:50] lots of work on the phrasing but I never know if that pays off :) [18:33:05] ottomata: not yet, I was in a meeting and have some follow-up and then I'll finish the email but then we can merge, yes [18:34:25] ok [18:34:28] you lemme know [18:35:00] hmmmm [18:35:49] coreyfloyd: annoyingly there's no "disable" feature for piwik websites [18:35:57] coreyfloyd: but there's this: "Only track visits and actions when the action URL starts with one of the above URLs." [18:36:25] coreyfloyd: does your app mock "URL"s such that if I change the URLs in that setting it'll stop tracking? [18:36:46] milimetric: doh… thats stupid… [18:36:47] hmm [18:37:03] so we have the url hard coded… so if the url changes, that would be that I suppose [18:37:06] milimetric: [18:37:24] the only thing i need to look out for is when you do that to make sure it fails silently on the device [18:37:27] milimetric: ^ [18:37:43] coreyfloyd: nothing like testing it live right? [18:37:47] fuck it, we're doing it live [18:37:56] milimetric so maybe you can do that and I can keep an eye on the app for a minute to make sure that no crazy alerts pop up [18:37:58] I'll change it now, can you check? [18:38:02] milimetric: nice lol [18:38:05] yep [18:38:29] k, coreyfloyd, I changed it to https://disabled.m.en.wikipedia.org [18:38:41] lemme know if stuff blows up [18:38:43] milimetric: one sec… downloading the production version [18:38:47] just to make sure [18:38:56] i'm looking at it too [18:39:28] milimetric: ok - it won’t send unless you do some actions and then it takes i think 2 minutes before it will send events [18:41:20] coreyfloyd: what specific actions should I do? The top pages thing? [18:41:31] yeah just page views [18:41:32] will owrk [18:41:48] not seeing anything yet [18:42:21] looks like it's working ok to me too, I reinstalled it [18:43:02] cool - i think we are good [18:43:18] ok, let me know right away if anything looks weird, I'll pm you my cell [18:43:29] milimetric: i’ll keep an eye on it and let you know if i see anything [18:45:01] !log changed the piwik URLs setting for the iOS website to https://disabled.m.en.wikipedia.org from https://m.en.wikipedia.org to try and restore the piwik server [18:48:15] coreyfloyd: ok, so it looks like the UI is working ok now, but I'm not seeing data for any day other than https://piwik.wikimedia.org/index.php?module=CoreHome&action=index&idSite=3&period=day&date=2016-03-15#?module=Dashboard&action=embeddedIndex&idSite=3&period=day&date=2016-03-15&idDashboard=1 [18:48:37] it's possible the indexing jobs just die with the size of the data since then [18:48:54] (I'm gonna poke in the db to see what's there) [18:58:10] ottomata: our berlin house is booked so we check out friday afternoon? [18:59:44] Analytics-Kanban: Add piwik reporting to browser reports - https://phabricator.wikimedia.org/T130653#2142012 (Nuria) [19:03:35] madhuvishy: ja, but i think we have to be out in the morning [19:03:40] we are allowed to leave bags there though [19:03:46] ottomata: okay cool [19:07:07] (PS1) Nuria: Adding piwik reporting to browser-reports [analytics/dashiki] - https://gerrit.wikimedia.org/r/278960 (https://phabricator.wikimedia.org/T130653) [19:11:40] Analytics-Kanban, Patch-For-Review: dashiki should report metrics to piwiki in production - https://phabricator.wikimedia.org/T128866#2142107 (Nuria) Open>Resolved [19:11:54] Analytics-Kanban: browser_general table should have documenting page in wikitech - https://phabricator.wikimedia.org/T130060#2142109 (Nuria) Open>Resolved [19:12:07] Analytics-Kanban: Parse User-Agent strings with OS like "Windows 7" correctly into the user agent map {hawk} - https://phabricator.wikimedia.org/T127324#2142111 (Nuria) [19:12:09] Analytics-Kanban, Patch-For-Review: Update UA-Parser with latest definitions - https://phabricator.wikimedia.org/T129672#2142110 (Nuria) Open>Resolved [19:12:22] Analytics-Kanban, Patch-For-Review: Clean up Event Logging server side forwarder - https://phabricator.wikimedia.org/T129402#2142112 (Nuria) Open>Resolved [19:13:01] Analytics-Kanban: Build a Dashiki dashboard that shows basic OS and browser request breakdowns - https://phabricator.wikimedia.org/T129102#2142118 (Nuria) Open>Resolved [19:13:04] Analytics, Analytics-Cluster: Story: Community has periodic browser stats report generated from Hadoop data - https://phabricator.wikimedia.org/T69053#2142119 (Nuria) [19:13:19] Analytics-EventLogging, Analytics-Kanban, Patch-For-Review, WMF-deploy-2016-03-08_(1.27.0-wmf.16), and 2 others: EventLogging needs to be ready for codfw failover - https://phabricator.wikimedia.org/T127209#2142122 (Nuria) Open>Resolved [19:13:36] Analytics-Kanban: Build a Dashiki dashboard that shows basic OS and browser request breakdowns - https://phabricator.wikimedia.org/T129102#2095150 (Nuria) [19:13:38] Analytics-Kanban, Patch-For-Review: Deploy new browser dashboard to 'browser-reports.wmflabs.org' - https://phabricator.wikimedia.org/T130069#2142126 (Nuria) Open>Resolved [19:14:00] Analytics-Kanban, Editing-Analysis, Patch-For-Review: Re-enable the edit analysis dashboard - https://phabricator.wikimedia.org/T126058#2142129 (Nuria) Open>Resolved [19:14:15] Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Ensure file.encoding is UTF-8 for all JVMs in the Analytics Cluster - https://phabricator.wikimedia.org/T128607#2142130 (Nuria) Open>Resolved [19:14:29] Analytics-Kanban, Patch-For-Review: Clean up Client IP and hashing related code on Eventlogging {oryx} {mole} - https://phabricator.wikimedia.org/T128408#2142133 (Nuria) Open>Resolved [19:14:42] Analytics-Kanban, Research-and-Data-Backlog, Patch-For-Review: Remove Client IP from Eventlogging capsule {mole} - https://phabricator.wikimedia.org/T128407#2142134 (Nuria) Open>Resolved [19:14:44] Analytics-Kanban, Patch-For-Review: Clean up Client IP and hashing related code on Eventlogging {oryx} {mole} - https://phabricator.wikimedia.org/T128408#2073520 (Nuria) [19:15:00] Analytics-Kanban: Add better error emails to oozie jobs - https://phabricator.wikimedia.org/T128749#2142136 (Nuria) Open>Resolved [19:15:24] Analytics-Kanban, Patch-For-Review: Invalid page titles are appearing in the top_articles data - https://phabricator.wikimedia.org/T117346#2142137 (Nuria) Open>Resolved [19:15:28] nuria / madhuvishy: do you have a main link I can use when we announce the Unique Devices dataset? (Since it's going out with the dumps reorg email, I'm drafting this now at the top of https://etherpad.wikimedia.org/p/analytics-notes) [19:15:41] Analytics-Kanban, Patch-For-Review: Add regexps that match the bots that follow the User-Agent policy {hawk} - https://phabricator.wikimedia.org/T125731#2142138 (Nuria) Open>Resolved [19:16:14] milimetric: hmmm https://meta.wikimedia.org/wiki/Research:Unique_Devices? [19:16:29] nuria: you wanna just paste in your announcement draft that you sent earlier? [19:16:45] or maybe you can send that separately and I can link to the pipermail archive when we deploy this? [19:17:08] thx madhuvishy, I'm thinking that draft that nuria worked on is best. [19:17:24] milimetric: ya, give me asec, cause part of it is outdated [19:17:38] ah yeah okay [19:19:24] thx nuria, so what do you think? Make this all one big email or send it in pieces? [19:20:00] milimetric: one e-mail, let me just expand a bit your unique devices piece [19:20:23] cool, thx [19:25:55] Analytics: Stop generating pagecounts-raw and pagecounts-all-sites - https://phabricator.wikimedia.org/T130656#2142207 (Milimetric) [19:35:07] milimetric: mostly done take a look and edit as needed [19:35:25] thanks nuria, I'll send it to analytics-internal when I'm done drafting [19:37:27] Analytics, Developer-Relations, MediaWiki-API, Reading-Admin, and 4 others: Metrics about the use of the Wikimedia web APIs - https://phabricator.wikimedia.org/T102079#2142280 (bd808) [19:47:36] milimetric: I made a small spelling correction in one place in the etherpad [19:47:44] s/Unquie/Unique [19:47:53] ah, sorry for deleting that [19:48:10] np! just letting you know :) [19:48:13] thx [19:48:52] milimetric: also would be good to clean bg colors - but i'm sure you already did that [19:49:15] oh i wasn't gonna send until yall said it was ok [19:49:30] but sending background colors would be funny :) ultimate transparency comes in rainbow-flavor [19:49:47] :D [19:57:56] jajaja [20:02:00] joal, ottomata: Do I need to wait for some jars to be updated on the stat100* servers before I try to enable the MediaWiki side of the ApiAction logging pipeline again? [20:04:17] bd808: I don't think so! But joal did mention he dropped the table just to be sure and would recreate it after your refactoring was done [20:05:24] *nod* All the patches are merged except the MediaWiki config and backport. [20:05:45] I was trying to guess when to schedule those to happen [20:06:04] bd808: you can enable the mediawiki side anytime [20:06:07] that will just produce to kafka [20:06:20] right. Ok [20:06:43] camus will already import it [20:06:46] into hdfs [20:06:55] the oozie jobs will then need started [20:07:03] and we should coordinate that with discovery folks [20:07:09] since we are taking over there job with an abstraction :) [20:07:12] their* [20:07:28] oh right. the oozie part just partitions the data that has already been dropped into hdfs [20:07:53] yeah discovery should surely make sure that the refactored stuff still works [20:37:54] ottomata: our hadoop cluster is not cross-dc right? [20:38:59] madhuvishy: right [20:39:15] right [20:39:18] and the analytics mysql stuff?/ [20:43:28] madhuvishy: ? [20:44:07] ottomata: i mean - our the mysql shards (m4 is the only one we use?) cross dc? [20:44:10] are* [20:44:31] madhuvishy: i don't know, but i would assume so [20:44:45] ottomata: okay :) [20:51:29] (CR) Ottomata: [C: 1] "Looks good, merge away!" [analytics/refinery] - https://gerrit.wikimedia.org/r/278325 (https://phabricator.wikimedia.org/T130187) (owner: Joal) [20:52:11] madhuvishy: wheever you find some time for a review https://gerrit.wikimedia.org/r/#/c/278979/ [20:52:30] oh i gotta fix some tox stuff [20:52:33] but ja [20:52:49] ottomata: sure! [21:18:05] Analytics, Hovercards: Capture hovers as previews in analytics - https://phabricator.wikimedia.org/T129425#2142578 (Jdlrobson) [21:18:17] Analytics, Hovercards: Capture hovers as previews in analytics - https://phabricator.wikimedia.org/T129425#2105269 (Jdlrobson) [21:18:36] Analytics, Hovercards: Capture hovers as previews in analytics - https://phabricator.wikimedia.org/T129425#2105269 (Jdlrobson) [21:19:01] Analytics, Hovercards: Capture hovers as previews in analytics - https://phabricator.wikimedia.org/T129425#2105269 (Jdlrobson) Can you update the description with what the header name and value needs to be? [21:20:49] yaya there we go, tests pass [21:23:20] milimetric: is pagecounts-ez derived from new pageviews? [21:24:00] ottomata: it is now, but didn't used to be [21:24:14] should we rename it then for consistency? [21:24:17] pageviews-ez [21:24:17] ? [21:24:39] I thought about it but didn't for the same reason as keeping the other links the same [21:24:51] hmm [21:24:55] but, its a new dataset, no? [21:25:13] for the same reasons we aren't replacing pagecounts links with pageviews, we shouldn't just use the same link for a new dataset? [21:25:19] right? [21:25:24] pagecounts-ez isn't new, it's very old [21:25:29] it used to be based on pagecounts-raw [21:25:32] it now contains both? [21:25:34] and then pagecounts-all-sites [21:25:36] and now pageviews [21:25:37] ah [21:25:38] hm [21:25:41] i see [21:25:45]