[00:04:28] 10Analytics, 10Android-app-feature-Feeds, 10Mobile-Content-Service, 10Pageviews-API, and 4 others: Why top views data of different sources is not the same? - https://phabricator.wikimedia.org/T172379#3510994 (10Liuxinyu970226) So those diffs are probably the reasons that both can't shown on feeds, but show... [00:06:36] 10Analytics-Kanban, 10User-Elukey: Archive PageContentSaveComplete in hdfs while we continue collecting data - https://phabricator.wikimedia.org/T170720#3510996 (10Nuria) mmm, java missmatch? cluster is running 7 but local version is set to 8? https://community.cloudera.com/t5/Data-Ingestion-Integration/Oracl... [00:20:23] 10Analytics-Kanban, 10User-Elukey: Archive PageContentSaveComplete in hdfs while we continue collecting data - https://phabricator.wikimedia.org/T170720#3511002 (10Nuria) [00:21:25] 10Analytics-Kanban: vet metrics calculated from the data lake - https://phabricator.wikimedia.org/T153923#3511003 (10Nuria) Ping @ezachte for after wikimanina is done [00:27:25] 10Analytics-Kanban, 10RESTBase-API, 10Services (later), 10User-mobrovac: Expose pageview data in each project's REST API - https://phabricator.wikimedia.org/T119094#3511005 (10Nuria) 05Open>03Resolved [00:27:32] 10Analytics-Kanban, 10RESTBase-API, 10Services (later), 10User-mobrovac: Expose pageview data in each project's REST API - https://phabricator.wikimedia.org/T119094#1817629 (10Nuria) 05Resolved>03declined [00:29:02] 10Analytics-EventLogging, 10Analytics-Kanban: EventLogging tests fail for python 3.4 in Jenkins - https://phabricator.wikimedia.org/T164409#3232541 (10Nuria) Was @hashar working on this? I *think* i saw a code change in passing [00:29:35] 10Analytics-Kanban, 10User-Elukey: Archive PageContentSaveComplete in hdfs while we continue collecting data - https://phabricator.wikimedia.org/T170720#3511010 (10Nuria) Job to create avro files takes 10 minutes for a 60G table. [00:39:05] 10Analytics-Kanban, 10User-Elukey: Archive PageContentSaveComplete in hdfs while we continue collecting data - https://phabricator.wikimedia.org/T170720#3511021 (10Nuria) [00:47:22] 10Analytics-Data-Quality, 10Analytics-Kanban, 10Readers-Web-Backlog (Tracking): Pageview drop in ro.wikipedia hu.wikipedia and fr.wikipedia - https://phabricator.wikimedia.org/T170845#3511023 (10Nuria) {F8991531} {F8991545} Agreed with @Tgr that decrease started earlier but it is more pronounced from April... [00:57:57] 10Analytics-Data-Quality, 10Analytics-Kanban, 10Readers-Web-Backlog (Tracking): Pageview drop in ro.wikipedia hu.wikipedia and fr.wikipedia - https://phabricator.wikimedia.org/T170845#3511026 (10Nuria) {F8991585} Without bot traffic picture is similar [00:58:45] 10Analytics-Data-Quality, 10Analytics-Kanban, 10Readers-Web-Backlog (Tracking): Pageview drop in ro.wikipedia hu.wikipedia and fr.wikipedia - https://phabricator.wikimedia.org/T170845#3511027 (10Nuria) {F8991585} Without bot traffic picture is similar [01:07:15] 10Analytics-Data-Quality, 10Analytics-Kanban, 10Readers-Web-Backlog (Tracking): Pageview drop in ro.wikipedia hu.wikipedia and fr.wikipedia - https://phabricator.wikimedia.org/T170845#3511029 (10Nuria) {F8991593} The drop seems to de on desktop not mobile, @Tgr and once filtered appears to a higher degree o... [02:12:30] 10Analytics, 10Analytics-Wikistats, 10Operations, 10Wikidata, and 6 others: Create Wikiversity Hindi - https://phabricator.wikimedia.org/T168765#3511056 (10Jayprakash12345) [http://tools.wmflabs.org/siteviews/?platform=all-access&source=pageviews&agent=user&start=2017-08-03&end=2017-08-08&sites=hi.wikivers... [03:26:20] 10Analytics-Kanban, 10RESTBase-API, 10Services (later), 10User-mobrovac: Expose pageview data in each project's REST API - https://phabricator.wikimedia.org/T119094#3511099 (10mobrovac) 05declined>03Open >>! In T119094#3489446, @Nuria wrote: > Sorry, we disagree, where does the unique devices data go w... [03:55:15] 10Analytics, 10Analytics-Wikistats, 10Operations, 10Wikidata, and 6 others: Create Wikiversity Hindi - https://phabricator.wikimedia.org/T168765#3511130 (10Reedy) Why would it work for July 2017? The wiki wasn't created till August? [05:07:23] 10Analytics, 10Android-app-feature-Feeds, 10Mobile-Content-Service, 10Pageviews-API, and 4 others: Why top views data of different sources is not the same? - https://phabricator.wikimedia.org/T172379#3511168 (10RazeSoldier) Is there such this situation in other languages? [06:58:27] 10Analytics, 10Analytics-Wikistats, 10Operations, 10Wikidata, and 6 others: Create Wikiversity Hindi - https://phabricator.wikimedia.org/T168765#3511265 (10gh87) In case that links don't work: [[http://tools.wmflabs.org/siteviews/?platform=all-access&source=pageviews&agent=user&start=2017-08-03&end=2017-08... [06:59:42] 10Analytics, 10Analytics-Wikistats, 10Operations, 10Wikidata, and 6 others: Create Wikiversity Hindi - https://phabricator.wikimedia.org/T168765#3511266 (10Liuxinyu970226) >>! In T168765#3511265, @gh87 wrote: > In case that links don't work: [[http://tools.wmflabs.org/siteviews/?platform=all-access&source=... [07:00:18] 10Analytics, 10Android-app-feature-Feeds, 10Mobile-Content-Service, 10Pageviews-API, and 4 others: Why top views data of different sources is not the same? - https://phabricator.wikimedia.org/T172379#3511268 (10Liuxinyu970226) >>! In T172379#3511168, @RazeSoldier wrote: > Is there such this situation in ot... [07:44:43] 10Analytics, 10Android-app-feature-Feeds, 10Mobile-Content-Service, 10Pageviews-API, and 4 others: Why top views data of different sources is not the same? - https://phabricator.wikimedia.org/T172379#3511287 (10Shizhao) > Also @shizhao your comments are still foggy, at least >>>It removes pages where we th... [10:57:38] elukey: nice [11:04:36] milimetric: o/ - now I need to wait for our Kafka master Ottomata to confirm :) the fix would need a deploy of eventstreams [11:04:45] err sorry a restart [11:04:51] (puppet change + service restart) [11:08:27] * elukey lunch! [11:09:51] he won't be around for a while, but we can brainbounce after lunch if you want [11:37:41] milimetric: sure! [11:40:41] the issue is not big but it concerns me since afaics eventstreams is causing some timeouts in unrelated producers like varnishkafka [11:58:50] helloooooo [12:41:52] o/ [13:59:31] 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad, and 2 others: rack/setup/install new kafka nodes kafka-jumbo100[1-6] - https://phabricator.wikimedia.org/T167992#3512185 (10elukey) [14:36:31] elukey: I was waiting for you to ping but I figure you must've finished lunch by now? [14:36:32] :) [14:36:35] cave? [14:37:55] milimetric: ah my ping was the "sure!" a couple of hours ago, I thought you were busy and didn't want to disturb :) [14:38:05] bah [14:38:06] :) [14:38:58] grabbing my headphones [14:43:45] 10Analytics, 10Analytics-Wikistats, 10Operations, 10Wikidata, and 6 others: Create Wikiversity Hindi - https://phabricator.wikimedia.org/T168765#3512393 (10Reedy) 05Open>03Resolved The second link is never gonna work. The wiki didn't exist in July, so there are not going to be any stats to use [15:07:26] elukey: hiii [15:08:48] ottomata: hheellloooooooo [15:09:06] dan pinged me waassuuup [15:09:09] something about eventstreams? [15:09:44] ottomata: I think so! https://phabricator.wikimedia.org/T172681 has all the details.. nothing uber urgent but I wanted to know your thoughts [15:11:30] OH interesting! [15:11:33] setting API version can't hurt [15:11:36] so we should probably do that [15:11:40] but huh. [15:11:49] that is strange, i guess because there are enough consumers? [15:11:50] hmmmmmmm [15:11:54] yeah we might have to increase partitions! [15:13:01] ottomata: what do I need to do to deploy? Merge + restart? [15:15:44] ya [15:15:47] that would do it [15:15:52] actually i think scap deploy will restart [15:16:44] elukey: this sounds like it might also be responsible for the occasional eventlogging burrow alarms on just a few partitions on certain brokers [15:19:51] ottomata: yep it could! [15:19:57] ottomata: do you mind if we deploy now? [15:20:06] (not sure if you have time or not now) [15:20:36] because I need to merge the puppet change [15:20:42] that will not restart the service [15:20:51] then maybe depool, restart, re-pool and check? [15:21:01] (one host at the time) [15:21:21] or even in an hour, not really in a hurry [15:21:24] :) [15:25:07] nuria_, PageContentSaveComplete does also have editCount, which is not white-listed [15:25:14] apart from userAgent [15:25:20] event_editCount [15:25:40] mforns: so should i drop that one too? [15:25:50] nuria_, yes I would say so [15:26:08] mforns: ok, that is going to be my next try, exporting just some columns into hive table [15:26:32] nuria_, although... it's edit information [15:26:37] it's already public [15:26:51] mforns: ok, then keep it? [15:26:57] elukey: go ahead! i'm only in and out to help [15:26:58] not sure why this editCount field is not white-listed [15:27:02] mentoring session is about to start here [15:27:05] and i signed up to be a mentor [15:27:12] i dunno what i am going to mentor on, but i did it [15:27:13] ! [15:27:26] elukey: the eventstreams scap stuff should auto depool repool [15:27:31] so, you could deploy to just one host [15:27:32] and see [15:27:34] nuria_, yes, we can keep it [15:28:28] 10Analytics: upgrade druid and pivot - https://phabricator.wikimedia.org/T157977#3512661 (10Ottomata) [15:28:30] 10Analytics-Kanban, 10Patch-For-Review: Upgrade Druid to 0.9.2 as a temporary measure - https://phabricator.wikimedia.org/T170590#3512659 (10Ottomata) 05Resolved>03Open I don't want this task to disappear yet! I still need to upload the 0.9.2 .debs to our apt. I hadn't done it yet because I wanted to mak... [15:29:27] ottomata: but the change is in puppet, not in scap [15:29:46] so I'd need to depool/restart/pool manually on scb noes [15:30:45] 10Analytics-Kanban, 10Patch-For-Review: Upgrade Druid to 0.9.2 as a temporary measure - https://phabricator.wikimedia.org/T170590#3512669 (10Nuria) Ah, sorry, did not know that. [15:30:47] 10Analytics, 10Operations, 10Research: Phase out and replace analytics-store (multisource) - https://phabricator.wikimedia.org/T172410#3512670 (10Ottomata) Correct me if I'm wrong, but a multi-instance MySQL will likely not be very useful for many of the analysis use cases. Folks want to join cross wiki, no... [15:30:57] elukey: ah, ok [16:06:02] urandom: o/ - dropping from the hangouts, I assume we are skipping it but let me know otherwise [16:09:42] 10Analytics-Kanban, 10RESTBase-API, 10Services (later), 10User-mobrovac: Expose pageview data in each project's REST API - https://phabricator.wikimedia.org/T119094#3512995 (10Nuria) >This ticket is about the pagewview data, so the fact of having one more API now does not invalidate it. Again, I disagree,... [16:26:29] elukey: gah, i'm sorry [16:26:36] elukey: i thought i declined that [16:26:48] no problem!! [16:27:48] in fact, it looks like i did, and android did that pending-net-connectivity-thing they added lately [16:27:50] grr [16:27:56] :D :D [16:39:22] urandom: user friendly [16:39:37] urandom: as in "you will never know if that update was really sent" [16:42:51] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats 2.0. - https://phabricator.wikimedia.org/T130256#3513113 (10Nuria) [16:43:03] 10Analytics-Kanban, 10Patch-For-Review: Add Licensing info to analytics.wikimedia.org - https://phabricator.wikimedia.org/T172814#3513116 (10Nuria) 05Open>03Resolved [16:46:43] nuria_: exactly [16:47:04] nuria_: you mash send, and it's all like... [16:47:05] ¯\_(ツ)_/¯ [16:47:37] "i'll do this for you... [16:47:44] "eventually..." [16:47:48] "probably..." [16:48:01] urandom: but users love it cause it looks like your message was sent despite you being on teh subway [16:48:37] this user doesn't love it :) [16:51:51] 10Analytics-Kanban, 10Analytics-Wikistats: Cleanup Routing code - https://phabricator.wikimedia.org/T170459#3513143 (10mforns) [18:19:16] * elukey off! [18:49:25] 10Analytics, 10Analytics-Cluster: Productionize Tranquility (or shut it off) - https://phabricator.wikimedia.org/T168550#3513627 (10Ottomata) Hm! I think we may be able to push realtime stats to druid with the Tranquility API. It's got a Spark interface: https://github.com/druid-io/tranquility/blob/master/d... [19:00:20] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 4 others: Investigate use-cases for delayed job executions - https://phabricator.wikimedia.org/T172832#3513723 (10daniel) Wikidata has potential need for delayed and/or periodic jobs. This would replace maintenance/dispatchChanges.php, which... [19:01:57] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 4 others: Investigate use-cases for delayed job executions - https://phabricator.wikimedia.org/T172832#3513729 (10mobrovac) @hoo, @daniel let's talk about it tomorrow during the hackathon day 2? [19:04:42] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 4 others: Investigate use-cases for delayed job executions - https://phabricator.wikimedia.org/T172832#3513738 (10daniel) @mobrovac sounds good! [19:14:50] 10Analytics-Kanban, 10Patch-For-Review: Add QuickSurvey schemas to EventLogging white-list - https://phabricator.wikimedia.org/T172112#3513786 (10mforns) Hi @leila :] We'd like to avoid stopping the EL purging process now. It's a several-week-long process and it is an Analytics (carry-on) goal for this quarte... [19:15:29] byeeeeee a-team see you tomorrow :] [19:15:57] seee ya mforns ! [19:16:05] * fdans wishes team europe was here [19:16:09] xD [19:21:48] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 4 others: Investigate use-cases for delayed job executions - https://phabricator.wikimedia.org/T172832#3513822 (10hoo) Sounds good to me too. [20:35:47] 10Analytics, 10Android-app-feature-Feeds, 10Mobile-Content-Service, 10Pageviews-API, and 4 others: Why top views data of different sources is not the same? - https://phabricator.wikimedia.org/T172379#3514125 (10Nuria) I doubt the feed algorithm considers pageviews when removing a page and very likely pages... [20:47:35] 10Analytics-Data-Quality, 10Analytics-Kanban, 10Readers-Web-Backlog (Tracking): Pageview drop in ro.wikipedia hu.wikipedia and fr.wikipedia - https://phabricator.wikimedia.org/T170845#3514203 (10Nuria) {F9012616} See seasonal pattern for hu.wikipedia and ro.wikipedia [20:49:26] 10Analytics-Data-Quality, 10Analytics-Kanban, 10Readers-Web-Backlog (Tracking): Pageview drop in ro.wikipedia hu.wikipedia and fr.wikipedia - https://phabricator.wikimedia.org/T170845#3514208 (10Nuria) @Strainu I really do not see anything major here and if ;look at series in its entirety the oscillations do... [20:55:54] 10Analytics-Data-Quality, 10Analytics-Kanban, 10Readers-Web-Backlog (Tracking): Pageview drop in ro.wikipedia hu.wikipedia and fr.wikipedia - https://phabricator.wikimedia.org/T170845#3514248 (10Strainu) @Nuria, are the stats you show the pageviews for the whole project or for the main page? They don't look... [20:58:15] 10Analytics-Kanban, 10User-Elukey: dbstore1002 /srv filling up - https://phabricator.wikimedia.org/T168303#3514253 (10Marostegui) I will be removing around 120G from m3 instance from dbstore1002 (T156758#3514245) [21:00:29] 10Analytics-Data-Quality, 10Analytics-Kanban, 10Readers-Web-Backlog (Tracking): Pageview drop in ro.wikipedia hu.wikipedia and fr.wikipedia - https://phabricator.wikimedia.org/T170845#3514259 (10Nuria) @Stranu: pageviews are for the whole project, equivalent on pageview tool: http://tools.wmflabs.org/sitevie... [21:08:09] 10Analytics-Data-Quality, 10Analytics-Kanban, 10Readers-Web-Backlog (Tracking): Pageview drop in ro.wikipedia hu.wikipedia and fr.wikipedia - https://phabricator.wikimedia.org/T170845#3514272 (10Nuria) @Strainu : argh i deleted part of my comment but we had an issue with organic but not intentional traffic o... [21:09:47] 10Analytics-Data-Quality, 10Analytics-Kanban, 10Readers-Web-Backlog (Tracking): Pageview drop in ro.wikipedia hu.wikipedia and fr.wikipedia - https://phabricator.wikimedia.org/T170845#3514301 (10Strainu) So are you saying that it's just "random" browser traffic that used to be targeted at the MainPage and no... [21:10:00] 10Analytics-Data-Quality, 10Analytics-Kanban, 10Readers-Web-Backlog (Tracking): Pageview drop in ro.wikipedia hu.wikipedia and fr.wikipedia - https://phabricator.wikimedia.org/T170845#3514302 (10Nuria) @Strainu: pageviews are for the whole project, equivalent on pageview tool: http://tools.wmflabs.org/sitevi... [21:10:57] 10Analytics-Data-Quality, 10Analytics-Kanban, 10Readers-Web-Backlog (Tracking): Pageview drop in ro.wikipedia hu.wikipedia and fr.wikipedia - https://phabricator.wikimedia.org/T170845#3514308 (10Nuria) @Strainu it was not a script, it is a connection failure from the client when stablishing an https connect... [21:37:11] 10Analytics-Data-Quality, 10Analytics-Kanban, 10Readers-Web-Backlog (Tracking): Pageview drop in ro.wikipedia hu.wikipedia and fr.wikipedia - https://phabricator.wikimedia.org/T170845#3514370 (10Strainu) I'm sorry, but I'm a bit skeptic about this explanation for several reasons: - ro.wp did not have that pe... [22:17:39] 10Analytics, 10Operations, 10Research: Phase out and replace analytics-store (multisource) - https://phabricator.wikimedia.org/T172410#3514474 (10Tbayer) >>! In T172410#3512670, @Ottomata wrote: > Correct me if I'm wrong, but a multi-instance MySQL will likely not be very useful for many of the analysis use... [22:50:13] 10Analytics-Kanban, 10User-Elukey: Calculate how much Popups events EL databases can host - https://phabricator.wikimedia.org/T172322#3514505 (10Tbayer) >>! In T172322#3509840, @JKatzWMF wrote: > @elukey @nuria my query is either failing or returning no results due to an error in the logic. My querying skills...