[04:35:34] Analytics, Pageviews-API, RESTBase-API: Start date of unique devices counts endpoint is always off by one day - https://phabricator.wikimedia.org/T134840#2279443 (MusikAnimal) [06:07:19] Analytics, Pageviews-API: qqq.json is invalid - https://phabricator.wikimedia.org/T134843#2279521 (Nikerabbit) [10:06:03] Analytics-Tech-community-metrics, Developer-Relations, Community-Tech-Sprint: Investigation: Can we find a new search API for CorenSearchBot and Copyvio Detector tool? - https://phabricator.wikimedia.org/T125459#2279903 (Earwig) I've got Yandex up and running for now. I set up a proxy on a personal s... [10:26:54] Analytics, Revision-Slider, TCB-Team: Data need: User Behaviour when comparing article revisions - https://phabricator.wikimedia.org/T134861#2279974 (Jan_Dittrich) [11:03:33] o/ [11:04:01] I spent this morning working on memcached, I'll probably start working on the partman receipe for aqs100[456] after lunch! [11:08:39] * elukey lunch! [11:37:09] Analytics-Tech-community-metrics, Developer-Relations, Community-Tech-Sprint: Investigation: Can we find a new search API for CorenSearchBot and Copyvio Detector tool? - https://phabricator.wikimedia.org/T125459#2280109 (Compassionate727) I actually have a question. Regarding Bing's terms of service,... [12:35:45] Analytics, Pageviews-API, RESTBase-API: Start date of unique devices counts endpoint is always off by one day - https://phabricator.wikimedia.org/T134840#2280389 (JAllemandou) @MusikAnimal : Thanks for catching this ! @Milimetric : This is the exact reverse issue of the one we solved the other day wh... [12:36:00] Analytics-Kanban, RESTBase-API: Start date of unique devices counts endpoint is always off by one day - https://phabricator.wikimedia.org/T134840#2280391 (JAllemandou) a:JAllemandou [12:57:48] Analytics: Augment oozie load SLA - https://phabricator.wikimedia.org/T134876#2280472 (JAllemandou) [12:58:53] milimetric, mforns: let's meet in batcave (messed up my invite call)! [13:02:48] Analytics: Augment oozie load SLA + Add URL to oozei error messages - https://phabricator.wikimedia.org/T134876#2280505 (JAllemandou) [13:44:25] Analytics-Kanban, Operations, ops-eqiad, Patch-For-Review: rack/setup/deploy aqs100[456] - https://phabricator.wikimedia.org/T133785#2280671 (elukey) @Cmjohnson: I tried to make a very simple partman recipe (https://phabricator.wikimedia.org/P3025) but I am sure that it is wrong on some many leve... [13:44:59] I give up, partman is really obscure to me [13:45:15] will ask to Chris some help [14:00:42] joal, milimetric, having problems joining [14:00:54] Arff. [14:01:01] mforns: We wait for you :) [14:05:21] elukey: it is obscure to everyone :p [14:10:09] :D [14:25:50] ottomata, joal: any reason why we have aqs::cassandra_defaultConsistency: localOne ? [14:26:30] elukey: in meeting, later [14:27:57] git blame seems to point to.. mobrovac :P [14:40:02] elukey: iirc, timeouts from other nodes [14:40:19] plus the fact that you have only a read pattern [14:40:28] so that's the only consistency you need [14:40:40] mobrovac: ah that one is read? [14:40:43] okok sorry :) [14:41:20] elukey: no, that's the default consistency (read and write) on a per req basis issued by AQS [14:41:26] via the driver [14:42:03] mobrovac: we had an issue with the auth keyspace that had local one and didn't like one cassandra node down [14:42:26] this is why I am asking, I thought that it would have been safer to have quorum and then specify local when necessary [14:42:36] auth? [14:42:41] that's not part of AQS [14:42:46] that's cassandra [14:42:57] so, a different problem [14:43:31] you shouldn't be looking at AQS' config for that [14:43:42] the auth keyspace has its own consistency [14:43:52] and your problem is that you have three nodes [14:44:07] so if one is down and another times out, you're out of luck [14:44:26] yeah but what do you mean it is not AQS but cassandra? [14:44:30] and AQS reqs have been timing out a lot lately [14:44:41] nono it was an old problem already fixed [14:44:58] anyhow, nevermind [14:45:12] ah ok [14:52:12] Analytics-EventLogging, TimedMediaHandler, Wikimedia-Video: Record and report metrics for audio and video playback - https://phabricator.wikimedia.org/T108522#2280967 (TheDJ) What do we want to measure ? 1: How often are players that are inserted into pages actually interacted with. 2: How often do... [14:55:37] mobrovac: also, whenever you have time, I'd like to have a chat with you about spinning up a test AQS service with cassandra multi-instance [14:56:03] I am checking what is the best way to handle it in puppet [14:56:28] sure elukey [14:56:31] just not today i hope [14:56:32] :) [14:56:36] or maybe it might be enough just to spin up cassandra, etc.. [14:56:38] nono! [14:56:42] whenever you have time :) [16:03:46] Analytics-Kanban: Generate test data for Cassandra - https://phabricator.wikimedia.org/T134899#2281310 (JAllemandou) [16:05:48] Analytics-Kanban: Puppetize druid - https://phabricator.wikimedia.org/T131974#2281333 (JAllemandou) a:Ottomata [16:21:29] Analytics-Kanban, RESTBase-API: Start date of unique devices counts endpoint is always off by one day - https://phabricator.wikimedia.org/T134840#2281423 (JAllemandou) a:JAllemandou>Milimetric [16:25:01] makin lunch [16:25:45] show tables; [16:25:48] oops [16:25:49] :) [16:32:43] +-----------------------+ [16:32:49] | Tables_in_joseph's_db | [16:32:54] +-----------------------+ [16:32:58] | ... | [16:33:05] +-----------------------+ [16:33:39] Mwhahahaha mforns ! [16:33:45] hehe [16:37:51] a-team: done with standup? [16:37:57] yes [16:37:57] nuria_, yes [16:38:12] in 8 mins, whattt? [16:38:20] 38 :) [16:38:28] ah wait, ahem... [16:39:08] Analytics: reportupdater to support multiple output folders per config - https://phabricator.wikimedia.org/T126549#2281531 (Milimetric) Invalid>Open That's good that there's now support for this, but I still need to clean up the bad setup that I did before, for example: https://github.com/wikimedia/a... [16:39:28] hehe [16:40:37] Analytics-Kanban: Bring jobs back to limn-ee-data and limn-flow-data from limn-language-data - https://phabricator.wikimedia.org/T126549#2281535 (Milimetric) a:Milimetric [16:42:14] milimetric: did you talked to Deskana about permits on depo? [16:42:30] I sent an email, but maybe we can just ping him here: Deskana you around? [16:50:08] Hello :) [16:50:16] I was on holiday yesterday [16:50:50] ottomata: do we need any swap partition on aqs nodes? I am not seeing how 1GB swap partion could help (like it was configured for restbase) [16:52:27] elukey: +1, better to fail fast than to slow down before failing [16:52:38] in normal operation, no swap is used [16:53:12] gwicke: thanks :) [16:53:16] Deskana: we have noticed you have +2 on refinery, would you mind changing that to +1? [16:53:34] cc milimetric [16:53:49] a-team: does everyone have retro on their calendars today at 10:00am [16:53:49] To answer the questions I was asked [16:53:54] nuria_: yep [16:53:55] 1) No, I don't know how I got it [16:54:00] nuria_: yop [16:54:01] yes [16:54:08] 2) I don't really mind if you remove it [16:54:45] I'm pretty sure I have global +2 though, so I don't know that it's really worth chasing down exactly why I have these permissions [16:55:16] I'd prefer to address what I perceive to be the real question here, which is... did I do something wrong? [16:55:30] There's some subtext here I think :-) [16:58:57] Deskana: no you did nothing wrong, we were just surprised that anyone besides us had +2 on that repository [16:59:17] in the case that you used your +2, it was totally fine [17:01:14] Okay :-) [17:01:22] Feel free to remove it if it's somehow problematic [17:02:25] elukey: coming to retro [17:02:47] ? [17:03:04] Deskana: no, nothing was wrong [17:07:48] Analytics-EventLogging, MediaWiki-extensions-MultimediaViewer, Patch-For-Review, WMF-deploy-2016-05-08_(1.28.0-wmf.1): MultimediaViewerNetworkPerformance schema validation errors - https://phabricator.wikimedia.org/T134165#2281684 (Jdlrobson) Open>Resolved [17:25:42] Analytics-Kanban: Bring jobs back to limn-ee-data and limn-flow-data from limn-language-data - https://phabricator.wikimedia.org/T126549#2281776 (mforns) Cool, good catch. [17:33:54] going offline!! [17:33:56] byeeeeee [17:34:08] * elukey is still angry with partman [17:36:02] elukey, bye! [17:36:23] nuria_, one question about what I'm doing now [17:36:46] (CR) Milimetric: [C: 2 V: 2] Add support for unique devices in pageview api [analytics/dashiki] - https://gerrit.wikimedia.org/r/287192 (https://phabricator.wikimedia.org/T122533) (owner: Mforns) [17:36:59] I started a mini-audit on EventLogging's new schemas / new revisions of old schemas since august 2015 [17:37:11] that's the date we negotiated with the schema owners. [17:37:46] I wanted to minimize the reaction from new schema owners when we tell them the purging is going to start now [17:38:45] I guess this will take a couple days... not full time, but calendar time [17:39:51] nuria_, or do you think we should just switch on the auto-purging and send a reminder email? [17:41:45] mforns: given that purging was not happening due to technical difficulties I think that we should take no action or ad most send an e-mail but that's it. [17:43:08] nuria_, don't you think that new schema owners that have been enjoying their data since august 2015 will react when we delete a good part of it? [17:43:36] mforns: all that looked great, you got a little time for some quick style updates? [17:43:44] I was thinking it'd be good to push them out at the same time [17:43:57] milimetric, sure [17:43:58] ok, I've gotta do one more and I'll push [17:44:26] milimetric, let me know if I can help [17:44:39] Analytics-Tech-community-metrics, Developer-Relations, Community-Tech-Sprint: Investigation: Can we find a new search API for CorenSearchBot and Copyvio Detector tool? - https://phabricator.wikimedia.org/T125459#2281834 (Earwig) @Compassionate727 It's funny, I asked nearly the exact same question...... [17:45:56] nuria_, what we can do is send an email to all owners explaining that the purging will start, and that the privacy-sensitive data older than 90 days will be sanitized or purged [17:46:18] whoever responds to that, we can negotiate with [17:46:56] mforns: sounds good, please send e-mail to analytics@ and cc neilpquinn and HaeB [17:47:01] mforns: thank yu [17:47:10] will do :] [17:50:27] mforns: the dashiki patch was merged: https://gerrit.wikimedia.org/r/#/c/287192/3 [17:52:01] nuria_, milimetric, thanks! so... I can deploy that to vital-signs and we'll have unique devices there, hopefully [17:52:15] mforns: does it have the monthly and daily metrics ? [17:52:22] mforns: yeah, but take a look at this patch, submitting in 10 seconds [17:52:28] nuria_, it is configurable via wiki [17:52:41] but you can choose only one granularity per metric [17:52:45] (PS1) Milimetric: Improve style and formatting [analytics/dashiki] - https://gerrit.wikimedia.org/r/288007 [17:52:49] nuria_, so we should choose [17:53:08] nuria_, BTW, pageviews.js already supported both granularities [17:53:16] nuria_ / mforns: we should make two metrics, one for daily and one for monthly [17:53:24] milimetric: yes [17:53:27] the daily one can be called Unique Devices, because all other metrics have daily granularity [17:53:28] i was going to say that [17:53:33] and the monthly one can be called Monthly Unique Devices [17:53:57] nuria_, milimetric, mmmmm [17:54:02] sure, that is fine. in this case, for this metric, [17:54:04] there's one problem, though [17:54:24] (btw, take a look at https://gerrit.wikimedia.org/r/#/c/288007/ which I just added you to, good to have style updates) [17:54:35] mforns: ya, i did not see in the code hwo you would do that [17:54:36] batcave? [17:54:43] mforns, milimetric omw [17:54:47] milimetric, nuria_ sure [17:54:48] omw [18:05:58] !log restarting eventlogging after broker stop [18:06:51] Analytics-Tech-community-metrics, Developer-Relations, Community-Tech-Sprint: Investigation: Can we find a new search API for CorenSearchBot and Copyvio Detector tool? - https://phabricator.wikimedia.org/T125459#2281972 (Compassionate727) >>! In T125459#2281834, @Earwig wrote: > @Compassionate727 It'... [18:29:36] (CR) Mforns: [C: 2 V: 2] "LGTM!" (1 comment) [analytics/dashiki] - https://gerrit.wikimedia.org/r/288007 (owner: Milimetric) [18:40:53] hmm, joal, yt? [18:44:56] ah nm, joal, wasdoing something dumb, i got it [18:46:35] wikimedia/mediawiki-extensions-EventLogging#556 (wmf/1.28.0-wmf.1 - 10bd87b : Chad Horohoe): The build has errored. [18:46:35] Change view : https://github.com/wikimedia/mediawiki-extensions-EventLogging/commit/10bd87b11129 [18:46:35] Build details : https://travis-ci.org/wikimedia/mediawiki-extensions-EventLogging/builds/129226525 [18:54:47] Hey ottomata, how is kafka going ? [18:56:07] good! [18:56:14] custom camus run works just fine [18:56:23] going to run a manual hdfs one [18:56:27] uhh prod one :) [18:56:34] awesome :) [18:57:35] ottomata: logging off for tonight, but might double check later [18:58:09] ok thanks joal, see ya! [18:59:20] (PS9) Amire80: Add sorted errors [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/282228 [19:02:06] HaeB: FYI that the "pageview=1"header is coming from the apps into x-analytics [19:11:28] !log reenabling camus and puppet on analytics1027 [19:15:56] Analytics, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown, and 2 others: Implement Schema:ExternalLinkChange - https://phabricator.wikimedia.org/T115119#2282349 (Sadads) @Legoktm and @kaldari do we have a sense of why the data isn't logging yet? @Milimet... [19:24:39] HaeB: and ditto for preview [20:05:09] Analytics-Kanban: Add support to 'displayName' param in dashiki config - https://phabricator.wikimedia.org/T134924#2282512 (mforns) [20:05:58] (PS1) Mforns: Add support for displayName config parameter [analytics/dashiki] - https://gerrit.wikimedia.org/r/288051 (https://phabricator.wikimedia.org/T134924) [20:08:35] (CR) Milimetric: [C: 2 V: 2] Add support for displayName config parameter [analytics/dashiki] - https://gerrit.wikimedia.org/r/288051 (https://phabricator.wikimedia.org/T134924) (owner: Mforns) [20:09:01] I didn't test that but it looks like it would work :) [20:09:18] ninja [20:09:19] hehehe [20:09:25] thanks [20:10:18] milimetric, so can I deploy the unique devices changes? [20:10:33] good for me [20:10:49] nicely done, one more check mark [20:10:51] nuria_, ^? [20:11:04] mforns: deploy? sure [20:11:08] cool [20:11:26] mforns: let me know when you are done deploying to testing and i will click arround [20:11:34] nuria_, sure [20:15:44] nuria_, it's there! [20:15:50] mforns: k testing [20:17:39] Analytics-Tech-community-metrics, Developer-Relations, Community-Tech-Sprint: Investigation: Can we find a new search API for CorenSearchBot and Copyvio Detector tool? - https://phabricator.wikimedia.org/T125459#2282595 (kaldari) @Compassionate727: If only Coren's bot were using Google, I think it wo... [20:17:41] mforns: you get an exception at getting data [20:17:49] mmm [20:18:03] nuria_, when does it happen? [20:18:03] mforns: try this and let me know: https://vital-signs-test.wmflabs.org/#projects=eswiki,frwiki/metrics=UniqueDevices [20:18:49] nuria_, how did you manage to add the same metric twice? [20:19:11] mforns: i did not, check url [20:19:17] mmm [20:22:02] nuria_, oh, I deployed to production directly [20:22:16] mforns: k let me check production [20:26:00] mforns: i see, the only things i see are 1) wiki links should like to metric definition: https://meta.wikimedia.org/wiki/Research:Unique_Devices 2) homework for later: we need to handle better projects for which there is no data and [20:26:53] mforns: wait, there is other thing [20:26:56] nuria_, aha [20:27:31] mforns: https://www.dropbox.com/s/5g0mykkdlupct83/Screen%20Shot%202016-05-10%20at%201.26.39%20PM.png?dl=0 [20:27:52] mforns: breakdowns are duplicated [20:27:59] nuria_, mmmm [20:28:09] nuria_, do you know how to repoduce this? [20:29:37] nuria_, ok, I did repoduce it [20:29:37] mforns: yes comes from having selected a project but not breakdown: https://vital-signs.wmflabs.org/#projects=eswiki/metrics= [20:30:19] mforns: selecting pageviews or unique devcies after empty state renders double breakdowns [20:31:41] nuria_, I got it, will submit a patch quickly [20:31:48] mforns: k [20:34:13] Analytics, Pageviews-API: API incorrectly complains about missing data instead of wrong wiki name - https://phabricator.wikimedia.org/T134926#2282666 (Ijon) [21:12:35] Analytics-Wikistats, Regression: [Regression] wikistats: Safari version detection broken ("Safari 0.0") - https://phabricator.wikimedia.org/T55881#2282837 (Krinkle) Open>Resolved a:Krinkle The SquidReportClients.htm viewer was discontinued in favour of https://browser-reports.wmflabs.org/ - w... [21:32:53] Analytics-Tech-community-metrics, Developer-Relations, Community-Tech-Sprint: Investigation: Can we find a new search API for CorenSearchBot and Copyvio Detector tool? - https://phabricator.wikimedia.org/T125459#2282909 (Earwig) @kaldari Probably—it's not a big deal to implement—but what about the API? [21:49:40] (PS1) Mforns: Fix unique devices bugs [analytics/dashiki] - https://gerrit.wikimedia.org/r/288104 (https://phabricator.wikimedia.org/T122533) [21:52:53] (PS2) Mforns: Fix unique devices bugs [analytics/dashiki] - https://gerrit.wikimedia.org/r/288104 (https://phabricator.wikimedia.org/T122533) [21:55:04] (CR) Mforns: Fix unique devices bugs (3 comments) [analytics/dashiki] - https://gerrit.wikimedia.org/r/288104 (https://phabricator.wikimedia.org/T122533) (owner: Mforns) [23:10:23] Analytics, DBA: Reportupdater does not commit changes after each query - https://phabricator.wikimedia.org/T134950#2283328 (Neil_P._Quinn_WMF) [23:48:14] (PS1) Neil P. Quinn-WMF: Make SQL connections use autocommit [analytics/reportupdater] - https://gerrit.wikimedia.org/r/288133 (https://phabricator.wikimedia.org/T134950)