[05:05:47] Analytics, Analytics-Wikimetrics: Usernames with commas not supported - https://phabricator.wikimedia.org/T129422#2105766 (madhuvishy) [05:44:05] Analytics, Pageviews-API, I18n: [[Wikimedia:Pageviews-select2-max-chars/en]] i18n issue - https://phabricator.wikimedia.org/T129442#2105822 (Liuxinyu970226) [05:45:04] Analytics, Pageviews-API, I18n: [[Wikimedia:Pageviews-select2-max-chars/en]] i18n issue - https://phabricator.wikimedia.org/T129442#2105824 (Macofe) Same with 'item' in Wikimedia:Pageviews-select2-max-items/en [05:51:30] Analytics, Pageviews-API, I18n: [[Wikimedia:Pageviews-select2-max-chars/en]] i18n issue - https://phabricator.wikimedia.org/T129442#2105865 (Liuxinyu970226) [07:28:18] (PS9) Joal: Add initial oozie job for ApiAction [analytics/refinery] - https://gerrit.wikimedia.org/r/273557 (https://phabricator.wikimedia.org/T108618) (owner: BryanDavis) [07:37:23] (CR) Joal: "One last minor comment, looks good :)" (4 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/273557 (https://phabricator.wikimedia.org/T108618) (owner: BryanDavis) [08:44:47] o/ [09:18:15] Analytics-Kanban, Operations, Traffic, Patch-For-Review: varnishkafka integration with Varnish 4 for analytics - https://phabricator.wikimedia.org/T124278#2106071 (elukey) Update: 1) completed the porting of the new tags. After a chat with Brandon and Ema we decided to use only the "client" tags... [09:18:33] Hi elukey :P) [09:41:25] (CR) DCausse: "If I understood correctly:" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/274307 (https://phabricator.wikimedia.org/T128530) (owner: EBernhardson) [10:41:12] (PS1) Addshore: Fix path in metrics.php [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/276430 [10:41:35] (CR) Addshore: [C: 2 V: 2] Fix path in metrics.php [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/276430 (owner: Addshore) [11:32:08] Analytics, Pageviews-API, I18n: [[Wikimedia:Pageviews-select2-max-chars/en]] needs PLURAL support - https://phabricator.wikimedia.org/T129442#2106345 (Aklapper) [12:39:56] !log restarted eventlogging to fix consumption of server side events [13:16:11] https://gerrit.wikimedia.org/r/#/c/276439 if anybody wants to review varnish-kafka! [14:37:42] mforns_brb: what's up with el? [14:41:13] (CR) Ottomata: "Indeed! I think it would also be cool to add maven steps to event-schemas to build not only .avsc file, but also class files and artifact" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/274307 (https://phabricator.wikimedia.org/T128530) (owner: EBernhardson) [14:53:34] ottomata, I saw yesterday's alerts when I arrived [14:53:58] and couldn't find anything strange neither in grafana, nor in the db [14:54:02] nor in the logs [14:54:12] oh, maybe server side events are just empty now? [14:54:17] what alerts? [14:54:18] ottomata, no no [14:54:44] I saw that the eventlogging-valid-mixed metric was strange [14:55:05] if you look at grafana, you'll see that this metric is below sum-of-all-schemas [14:55:51] they should be equal in theory, and this started Feb 23 at 17 UTC [14:56:12] I looked into it and found that it is because of the CentralNoticeBannerHistory schema [14:56:21] that started at that date [14:56:29] it is being blacklisted [14:56:46] so its events make it to sumOfAllSchemas, but do not make it into eventlogging-valid-mixed [14:57:17] that's why eventlogging-valid-mixed is below expected, it's not a problem, but maybe we can fix it [14:58:09] or add a new metric to graphite: blacklisted [14:58:27] ahh ok [14:58:33] hm, but, the raw - valid should be ok, right? [14:58:37] that's why we have that metric [14:58:43] ohhh [14:58:46] no because valid doesn't have is [14:58:47] sorry [14:58:48] right [14:58:48] hm [14:59:08] ja, maybe we should make it raw - sum of all schemas [15:07:04] ottomata, currently it is raw - eventlogging-valid-mixed? [15:10:11] ja htink so [15:10:28] aha [15:10:46] if so, this may be the origin of some false alarms [15:24:20] joal: o/ [15:24:27] if you have a min for https://wikitech.wikimedia.org/wiki/User:Elukey/Analytics/PageViewDumps#Example: [15:24:36] let me know :) [15:29:51] Hi elukey :) [15:29:58] How can I help ? [15:30:18] I was thinking about moving forward with one hour in the proper folders on hdfs [15:30:53] removin -Darchive_directory=hdfs://analytics-hadoop/tmp/elukey/pageviewdumps/archive basically from the command line [15:31:11] elukey: feel free :) If you go and monitor, you can even start the full thing :) [15:31:31] Meaning: if the thing fails, you're here to stop it, if not, it goes on [15:31:45] The only risk is stuff failing [15:32:22] and I need to start the work with sudo -u hdfs I suppose [15:32:27] indeed [15:32:35] * elukey begins [15:32:44] elukey: I'll follow wth you [15:45:14] maybe if I run it from analytics1015.eqiad.wmnet instead of stat1002 is better [15:45:17] :P [15:45:55] elukey: not really :) [15:46:01] elukey: doesn't change anythong :) [15:46:29] elukey: when adding sudo-u hdfs, you also need to add --oozie $OOZIE_URL [15:46:49] This variable is set for us users, but bot for hdfs, so we need to explicitely pass it [15:49:37] ahhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh [15:49:42] :) [15:51:08] and $OOZIE_URLjob: 0001239-160309204929436-oozie-oozi-C \o/ [15:52:07] elukey: I meant to tell you every time I saw that and forgot: would have been good to rename the coordinator ;) [15:52:23] no need to restart, but now that I see it, I tell you :) [15:56:18] /wmf/data/archive/pageview/legacy/hourly/2015/2015-05/pageviews-20150501-010000.gz [15:56:47] loooooks good for the moment [15:57:23] elukey: If it has worked for 1 or 2 of them, then you good :) [15:57:53] You can keep a tab open on the coordinator to monitor every once in a while, but globally it's done :) [15:58:24] I just fired one hour, if it works fine I'll start one day [15:59:14] elukey: not true, still computing [15:59:41] elukey: you actually forgot to specify stop time I think [15:59:50] elukey: from config [15:59:56] :) [16:00:11] The value is not "end_time", but "stop_time" [16:01:22] * elukey is angry with oozie [16:30:41] a-team: stadddupppp [16:31:45] Analytics-Kanban: Make TimeseriesData reactive so it can support a legend talking to a graph - https://phabricator.wikimedia.org/T129497#2107463 (Milimetric) [16:33:21] (PS1) Milimetric: Make datasets api more flexible [analytics/dashiki] - https://gerrit.wikimedia.org/r/276492 (https://phabricator.wikimedia.org/T129497) [16:35:39] Analytics-Kanban: Write Unique devices blogpost - https://phabricator.wikimedia.org/T129498#2107488 (Nuria) [16:45:31] Analytics-Tech-community-metrics: top-contributors.html empty due to 404s for several JSON files - https://phabricator.wikimedia.org/T126971#2107572 (Lcanasdiaz) @Aklapper, shouldn't we reopen this bug? [16:45:46] Analytics-Tech-community-metrics, DevRel-March-2016: gerrit_review_queue.html: List of Repositories has not been updated recently - https://phabricator.wikimedia.org/T128170#2107573 (Lcanasdiaz) >>! In T128170#2099085, @Aklapper wrote: > @Lcanasdiaz: It is! Thank you! (Now hoping for a fix for top-contrib... [17:11:06] milimetric: can you check later on (hdfs dfs -ls /wmf/data/archive/pageview/legacy/hourly/2015/2015-05) to make sure that we are ok? :) [17:11:30] checking [17:13:53] thanks! [17:36:48] Analytics-Kanban: Integrate new browser visualization into wikistats - https://phabricator.wikimedia.org/T129101#2095134 (Milimetric) Let's replace the links to wikimedia/squids/SquidReportClients.htm with the new dashiki dashboard. [17:44:23] (CR) BryanDavis: "> You have moved some properties definition ... Any particuliar reason ?" [analytics/refinery] - https://gerrit.wikimedia.org/r/273557 (https://phabricator.wikimedia.org/T108618) (owner: BryanDavis) [17:45:31] Analytics-Kanban: Clean up Event Logging server side forwarder - https://phabricator.wikimedia.org/T129402#2107893 (Milimetric) [17:52:46] Analytics: Add AQS endpoint for uniques - https://phabricator.wikimedia.org/T129518#2107979 (Milimetric) [17:52:58] Analytics: Fill new AQS endpoint with data from Hadoop - https://phabricator.wikimedia.org/T129519#2107994 (Milimetric) [17:53:38] Analytics: Document the new AQS endpoint and launch it - https://phabricator.wikimedia.org/T129520#2108010 (Milimetric) [17:57:06] elukey: those files look good but it seems to have stopped after it got to 20150501-180000 [17:57:47] milimetric: the job is still running :) [17:58:18] elukey: right, but each hour only took 2 minutes, and I noticed that in the last hour it didn't add any more files to that directory (the May 1st one) [17:58:49] ahhhhh so this might be a problem, I'll check! [18:08:11] milimetric: due to rsync cron happening once an hour ? [18:08:18] elukey: --^ [18:08:19] ? [18:08:56] so I checked on hdfs and all the files are there for May 1st [18:09:09] not sure about the rsync joal [18:09:11] Analytics: Add AQS endpoint for uniques - https://phabricator.wikimedia.org/T129518#2108121 (Milimetric) [18:09:32] helllOOoo [18:09:55] Hi ottomata [18:10:24] elukey: files are rsynced from HDFS to the endpoint everyhour [18:11:41] mmmm joal I thought that Dan was checking HDFS directly [18:11:50] oh, don't know :( [18:13:48] ottomata: https://www.varnish-cache.org/installation/debian this might be handy (if you don't have it already) for vk testing [18:14:53] ohh ok great, thanks elukey [18:17:34] (CR) Joal: "Thanks for the answer :)" [analytics/refinery] - https://gerrit.wikimedia.org/r/273557 (https://phabricator.wikimedia.org/T108618) (owner: BryanDavis) [18:18:09] Analytics: Visualize unique devices data in dashiki - https://phabricator.wikimedia.org/T122533#2108147 (Milimetric) [18:22:23] Analytics-Kanban, Commons, Multimedia, Wikidata, Community-Wishlist-Survey: Allow tabular datasets on Commons (or some similar central repository) (CSV, TSV, JSON, XML) - https://phabricator.wikimedia.org/T120452#2108162 (Milimetric) a:Milimetric [18:25:17] ahhh joal sorry, the SLA change didn't make it because i didn't update cdh submodule [18:25:23] doing now [18:26:12] Thx ottomata [18:32:06] joal: can you tell if its there now? [18:32:12] ottomata: checking [18:32:28] YAY ! [18:32:39] Not sure it's working, but at least the UI is here [18:32:41] Thanks ottomata [18:32:57] the UI? in hue? [18:33:03] Yessir [18:33:46] Oozie dashboards --> Workflows Coordinators Bundles SLA oozie [18:34:47] ah interesting [18:36:26] elukey: correct setup on the coordinator ;) [18:36:40] elukey: You'll soon be an oozie master ! [18:36:49] \o/ [18:37:42] joal quick question: how do I check the job's paramters? [18:37:49] I use hue [18:37:52] ahhh okok [18:38:07] make sense, I thought you were using the oozie cli [18:38:08] super [18:38:16] :) [18:38:43] I'll need to offer you some beers in Berlin :) [18:38:51] a-team: going offline! [18:38:53] byyyyeeeeeeeeeeeeeeeeeeeeeeeee [18:38:56] Bye elukey ! [18:42:56] byYYey [18:49:05] bye a-team! see you tomorrow! [19:01:15] (PS1) Joal: Add SLA to webrequest load oozie job [analytics/refinery] - https://gerrit.wikimedia.org/r/276534 [19:01:57] A-team, I'm off for today ! [19:01:59] See you [19:07:11] milimetric: hangout went kaput [19:07:15] not connection [19:11:55] nuria: I was done, I'll have some lunch and we can talk more later if you want [19:12:23] milimetric: k, once you have worked on problem a bit and i can look at code + viz we can touch base again [19:12:25] joal / elukey: yes, I was checking hdfs directly, lemme check again [19:12:50] nuria: k, cool, will do [19:13:22] elukey: joal: I see all the files for the 1st and some for the 2nd of May [19:13:24] so we're good [19:13:36] before I didn't, it looked like it stalled for like an hour [19:13:42] but I'm not worried about it :) [19:15:27] ottomata: around? [19:15:52] he's in the meeting with researchers madhuvishy [19:15:55] aah [19:15:57] okay [19:16:11] i should go to that meeting in the future [19:16:29] you can go now, it just started [19:16:44] hmmm i'm just heading to office :/ [19:43:07] madhuvishy: ja am around [20:05:06] Analytics-Kanban, Commons, Multimedia, Wikidata, Community-Wishlist-Survey: Allow tabular datasets on Commons (or some similar central repository) (CSV, TSV, JSON, XML) - https://phabricator.wikimedia.org/T120452#1854102 (Bawolff) >We already have pretty robust support for pages containing text.... [20:06:51] Analytics-Kanban, Wikipedia-Android-App-Backlog: Count requests to RESTBase from the Android app - https://phabricator.wikimedia.org/T128612#2108900 (Dbrant) [20:09:00] Analytics-Kanban: Create system user that has access to analytics-privatedata-user group owned files for automated hadoop jobs - https://phabricator.wikimedia.org/T129551#2108917 (Ottomata) [20:09:02] milimetric: if the pageview tagging looks good, can you merge? [20:22:01] Analytics-Kanban, Commons, Multimedia, Wikidata, Community-Wishlist-Survey: Allow tabular datasets on Commons (or some similar central repository) (CSV, TSV, JSON, XML) - https://phabricator.wikimedia.org/T120452#2108996 (ekkis) I think there's a very valuable use-case in data sets well below the... [20:32:08] (CR) Catrope: [C: 2] Add more wikis to the cross-wiki beta feature report [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/276356 (owner: Catrope) [20:32:29] (CR) jenkins-bot: [V: -1] Add more wikis to the cross-wiki beta feature report [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/276356 (owner: Catrope) [20:33:20] (CR) Catrope: [C: 2] Add more wikis to the cross-wiki beta feature report [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/276356 (owner: Catrope) [20:33:41] (CR) jenkins-bot: [V: -1] Add more wikis to the cross-wiki beta feature report [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/276356 (owner: Catrope) [20:55:54] this looks good! [20:55:54] http://kafka-summit.org/schedule/ [20:58:28] Analytics-Kanban, Commons, Multimedia, Wikidata, Community-Wishlist-Survey: Allow tabular datasets on Commons (or some similar central repository) (CSV, TSV, JSON, XML) - https://phabricator.wikimedia.org/T120452#2109124 (Bawolff) > I think there's a very valuable use-case in data sets well below... [20:58:49] milimetric, mforns, fyi, changed the reportupdater stuff on stat1002 to use hdfs user [20:59:26] Analytics-Kanban, Patch-For-Review: Create system user that has access to analytics-privatedata-user group owned files for automated hadoop jobs - https://phabricator.wikimedia.org/T129551#2109130 (Ottomata) a:Ottomata [21:00:08] Analytics-Kanban, Patch-For-Review: Create system user that has access to analytics-privatedata-user group owned files for automated hadoop jobs - https://phabricator.wikimedia.org/T129551#2108917 (Ottomata) Talked with chase about this for a while. We'd need to look into ACLs in order to really make thi... [21:00:45] Analytics-Kanban, Patch-For-Review: Puppetize reportupdater to be executed in stat1002 and run the browser reports {lama} - https://phabricator.wikimedia.org/T127327#2039936 (Ottomata) [21:00:47] Analytics-Kanban, Patch-For-Review: Create system user that has access to analytics-privatedata-user group owned files for automated hadoop jobs - https://phabricator.wikimedia.org/T129551#2108917 (Ottomata) Open>declined [21:02:13] (CR) Nuria: "Looks good, let's merge if we have tested it. (we can reduce timeout such e-mail is sent if we have not tested it yet)" [analytics/refinery] - https://gerrit.wikimedia.org/r/276534 (owner: Joal) [21:04:24] Analytics: Remove evenetlogging code from blog. Use piwik to count pageviews - https://phabricator.wikimedia.org/T129558#2109181 (Nuria) [21:16:15] Analytics-Kanban, Commons, Multimedia, Wikidata, Community-Wishlist-Survey: Allow tabular datasets on Commons (or some similar central repository) (CSV, TSV, JSON, XML) - https://phabricator.wikimedia.org/T120452#2109268 (ekkis) > Lua tables I know nothing of that technology but if it can be par... [21:16:34] ottomata: yeah! [21:16:38] Analytics-Kanban, Commons, Multimedia, Wikidata, Community-Wishlist-Survey: Allow tabular datasets on Commons (or some similar central repository) (CSV, TSV, JSON, XML) - https://phabricator.wikimedia.org/T120452#2109270 (Yurik) @Bawolff Lua supports mw.text.jsonDecode(), which is great for thes... [21:16:50] oh ottomata I was gonna ask you - we were going to update ua-parser [21:16:51] :) [21:16:54] ja? [21:17:00] and i see the uap-java hasn't changed [21:17:06] but definitions in uap-core have [21:17:19] and i don't understand how we package the yaml definitions [21:20:47] (CR) Catrope: [C: 2] Add more wikis to the cross-wiki beta feature report [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/276356 (owner: Catrope) [21:21:18] (Merged) jenkins-bot: Add more wikis to the cross-wiki beta feature report [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/276356 (owner: Catrope) [21:23:35] uUuhhh [21:24:34] i'm not sure I know (or remember) either madhuvishy [21:24:54] ottomata: okay - I was asking joal and he was saying we should talk to you [21:25:06] i'll bring it up at standup tomorrow :) [21:25:20] haha [21:25:25] did Ironholds used to do this? [21:25:32] do we just import regexes.yaml into ua-parser [21:26:06] ottomata, nope, nuria handled that [21:28:52] wait .. what yaml definitions? [21:34:11] lol [21:34:22] this has come full circle [21:35:30] nuria: https://github.com/ua-parser/uap-core [21:35:43] are the definitions [21:36:04] https://github.com/ua-parser/uap-java is the java implementation [21:36:13] our jar packages both of these together [21:36:27] uap-java hasn't changed since our last update [21:36:49] uap-core has [21:37:00] how we package it - i don't know [21:38:05] madhuvishy: ah, i see, I am not sure we did any custom packaging , lemme look [21:45:53] oo nice madhuvishy no more server side events, eh? [21:46:14] ottomata: uhh yeah i was just going to ask you [21:46:32] i dont see any! [21:46:37] if that change was all that we need [21:46:45] oh coool [21:46:52] its all client side [21:47:00] so i can remove the processor too? [21:48:09] ja! [21:48:13] nice [21:48:27] well, the latest timestamp i see in server side log is from about 30 mins ago [21:48:29] so we should wait to merge [21:48:30] but ja [22:02:43] ottomata: yeah alright [22:16:50] ottomata: I have the changed cherry-picked and deployed on beta [22:20:33] nice! [22:20:48] everything looks good there? [22:23:38] ottomata: yeah no server side processor and forwarder is running [22:23:47] everything else is fine as it is [22:23:52] i dont know what else to check [22:25:16] (CR) Milimetric: "For future reference, since this can be confusing, the job runs at 00:00 daily as configured now. So this merge landed in time for today'" [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/276356 (owner: Catrope) [22:31:33] cool sounds good madhuvishy [22:45:19] (PS10) BryanDavis: Add initial oozie job for ApiAction [analytics/refinery] - https://gerrit.wikimedia.org/r/273557 (https://phabricator.wikimedia.org/T108618) [22:46:10] (CR) BryanDavis: Add initial oozie job for ApiAction (4 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/273557 (https://phabricator.wikimedia.org/T108618) (owner: BryanDavis) [22:47:27] (PS11) BryanDavis: Add initial oozie job for ApiAction [analytics/refinery] - https://gerrit.wikimedia.org/r/273557 (https://phabricator.wikimedia.org/T108618) [22:48:39] (CR) BryanDavis: "patch set 10 accidentally reverted the rebase that was done by Joal in PS #9" [analytics/refinery] - https://gerrit.wikimedia.org/r/273557 (https://phabricator.wikimedia.org/T108618) (owner: BryanDavis) [23:23:21] (PS1) Milimetric: [WIP] Add a legend to each graph in the tabs layout [analytics/dashiki] - https://gerrit.wikimedia.org/r/276649 (https://phabricator.wikimedia.org/T129497) [23:23:47] nuria: I've been playing around with code like that ^ for a bit, and I'm not comfortable it's the right path [23:24:18] but take a look, I think the hierarchy viz is the easiest one to see where the approach breaks [23:24:25] (I'll take a look at your patch and merge now) [23:25:41] (PS5) Milimetric: Requests that come tagged with pageview=1 in x-analytics header are considered pageviews [analytics/refinery/source] - https://gerrit.wikimedia.org/r/274644 (https://phabricator.wikimedia.org/T128612) (owner: BearND) [23:25:50] (CR) Milimetric: [C: 2 V: 2] Requests that come tagged with pageview=1 in x-analytics header are considered pageviews [analytics/refinery/source] - https://gerrit.wikimedia.org/r/274644 (https://phabricator.wikimedia.org/T128612) (owner: BearND) [23:26:42] (CR) Milimetric: "Nuria & Marcel: this is just some scribbling, to see whether we can make a general approach without changing TimeseriesData, based on a co" [analytics/dashiki] - https://gerrit.wikimedia.org/r/276649 (https://phabricator.wikimedia.org/T129497) (owner: Milimetric)