[03:38:46] Analytics, Pageviews-API: Wikimedia pageviews API blocked by ad blockers - https://phabricator.wikimedia.org/T126947#2027714 (MusikAnimal) NEW [03:53:35] (CR) Milimetric: [C: -1] "minor questions and tweaks" (7 comments) [analytics/dashiki] - https://gerrit.wikimedia.org/r/269713 (https://phabricator.wikimedia.org/T124296) (owner: Mforns) [08:23:03] is anyone using the mwviews.api over here? [11:18:16] dennyvrandecic: maybe you know this. When I use the mwviews.api like this: https://nl.wikipedia.org/wiki/Gebruiker:Edoderoo/nl-stats-gisteren ... then yesterdays views are still zero for todays date. Is this a bug, or a feature? [12:29:03] Analytics-Tech-community-metrics: top-contributors.html empty due to 404s for several JSON files - https://phabricator.wikimedia.org/T126971#2028494 (Aklapper) NEW [13:01:27] reminder! Today is a vacation, happy presidents' day (aka furniture sales day) [13:01:40] Enjoy milimetric :) [13:04:36] !log Deploying refinery [13:19:06] milimetric, joal, oh! cool! so, it will be a european stand-up today? [13:21:43] Analytics-Kanban: Use a new approach to compute monthly top 1000 articles (brute force probably works) {slug} [8 pts] - https://phabricator.wikimedia.org/T120113#2028578 (JAllemandou) a:JAllemandou [13:22:25] !log Failed deploying refinery: deploy from tin failed, need ottomata help [13:23:32] Actually, elukey, you might have root on stat1002 ? [13:24:01] And by the way: HEEEEELLO a-team :) [13:24:18] hi joal ! [13:25:48] o/ [13:26:08] joal: all yours, how can I help? [13:26:27] deploying from tin failed :( [13:27:15] So I wonder how we should move on that [13:27:18] elukey: --^ [13:28:34] joal: Giuseppe was working on tin, but now it should be ok.. have you re-tried? [13:28:49] elukey: tried a few minutes ago [13:28:53] will retry [13:36:00] joal: are you deploying the refinery's code right? [13:36:09] trying to at least [13:36:22] elukey: still failing [13:36:41] elukey: deploy start then sync --> 0/2 minions completed fetch [13:38:32] mmmm [13:38:56] checking [13:44:41] can see a lot of output: error: could not lock config file .git/config: Permission denied [13:45:03] hm [13:45:28] elukey: never had to sudo or anything ... [13:45:50] elukey: I managed, on tin, to git fetch --all, then git pull [13:45:59] It fails at deploy [13:48:48] super ignorant about the procedure, reading :) [13:49:10] elukey: I don't know about the internals, I just use a procedure :) [13:55:12] joal: can I try following the guide in https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Refinery ? [13:55:22] or is there any specific thing that I should do beforehand? [13:56:30] elukey: just reviewed --> seem to be a command order error [13:56:41] ahhh [13:56:50] well, can I do it anyway? [13:56:55] just to learn :) [13:57:40] please, I'll abort mu currently running (failing)deploy [13:58:01] elukey: you're all set [13:58:31] thankss [13:58:43] proceeding [14:00:14] ahhh I am stupid, now I got the complete error message [14:00:22] the minions OF COURSE are failing on the target hosts [14:00:26] * elukey needs coffee [14:01:36] ResponseError: READONLY You can't write against a read only slave. [14:01:37] mmmmm [14:03:09] I think it is because of the switch [14:11:21] joal: working with Giuseppe, I'll keep you posted [14:11:31] Thanks a million elukey [14:11:38] :) [14:52:43] joal, yt? [14:52:52] yup [14:52:59] hey o/ one question [14:53:00] wassup mforns ? [14:53:58] to use hive to append to an exeisting report file, is there any method apart from loading the data in the file and doing a union and overwriting the file with the new data? [14:55:25] That's how I've done it so far [14:55:44] joal, ok, will do it like this then :] [14:56:04] mforns: it can depend on how you want the data to look like [14:56:22] joal: the error logged by the minions is now different, still looking.. it is really weird :D [14:56:30] mforns: The reason why it's better in my opinion, it's because it allows you to have a single file, plus to catch recomputation [14:56:43] joal, I'd like all the data to be in a single file with the format: date, os_family, os_major, view_count [14:56:49] elukey: no problemo, it great you try to solve it :) [14:57:28] joal, I see [14:57:34] mforns: ok, then go for a union (and don't forget to remove current date from existing part of the union :) [14:58:12] mforns: wouldn't a file containing (date, os_family, os_major, view_count) end up very big ? [14:58:37] mforns: having everything in a single file is good for publishing outside the cluster [14:58:48] joal, I cut the long tail, so every day contains just a couple rows [14:58:54] ok makes sense :) [14:59:04] joal, and they are weekly files [14:59:21] so, a yearly file will be like couple kB [14:59:23] I guess [14:59:45] *weekly results [15:00:03] k mforns, I trust you ;) [15:00:12] hehe, should you? :P [15:00:51] team spirit mforns :D [15:02:15] joal, so I guess I need to add an .hql file to refinery/hive/browser to create an external table for the query to be able to read the previous results, no? [15:04:31] mforns: I think so [15:04:51] mforns: an example of what you're trying to is mobile_apps uniques (daily or monthly) [15:04:57] You can take example on that :) [15:05:00] joal, OK cool [15:05:04] thx! [15:05:24] see you in a while folks [15:06:35] hiiii [15:07:16] ottomata: hhheeeelloooooooo I need you [15:07:20] whenever you have time [15:07:53] oh elukey hi! [15:07:54] what's up? [15:07:56] i'm just checkin email [15:12:59] ottomata: I am trying to figure out why the refinery deployment from tin doesn't work, both minions are failing [15:13:22] initially the salt log contained errors (probably) related to the switch from mira to tin [15:13:28] happened today [15:13:42] but now it is not, and other deployments are working fine [15:13:54] so I was wondering if you have any wisdom to share :D [15:17:15] elukey: it is probably because we manually deployed last time [15:17:22] and the targets are funky [15:17:35] as far as I can remember, we only deploy to stat1002 and analytics1027, ja? [15:17:48] i'd go to the refinery dir on those hosts, rename them, and then run puppet on both [15:17:49] hmmm [15:17:50] maybe [15:17:51] hmmmm [15:17:54] actually, lemme try first [15:18:24] elukey: , can I try to deploy? [15:18:45] sure, let me check if I aborted [15:19:11] ottomata: go ahead :) [15:24:55] elukey: ok mostly good, although artifacts/org/wikimedia/analytics/camus-wmf/camus-wmf-0.1.0-wmf6.jar is funky on an27 [15:24:58] not totally sure why [15:25:04] its there, but it is modified [15:25:14] and i can't get it back to normal git status [15:25:26] on both stat1002 and an27, i did [15:25:33] git reset --hard [15:25:35] git checkout master [15:25:40] git remote remove gerrit [15:25:54] ^ we created a remote called gerrit pointing at http gerrit to manually deploy last time [15:26:00] then git deploy sync from tin worked [15:27:34] joal: , yt? [15:27:40] yup [15:27:44] Hi ottomata [15:27:47] doing a review for this: [15:27:47] https://gerrit.wikimedia.org/r/#/c/268129/5 [15:27:55] i don't love the way its going, but its fine. buuuut q for you [15:28:09] where should we sync dumps.wm.o http access logs to on stat1002? [15:28:15] they are 'webrequset' logs [15:28:31] but all of our other 'webrequest' logs there in /a/log/webrequest/archive are via varnish and hadoop [15:29:05] maybe we should put them in /a/log/webrequest/archive/dumps.wikimedia.org with a README file in that dir about where thye come from? [15:30:28] ottomata: trying to understand (I'm still sick, slow processor today) [15:30:48] ottomata: ah ok so there was some context that I didn't have :D [15:30:50] awww sorry joal [15:30:52] ignore! [15:30:55] yeah [15:30:57] elukey: [15:31:10] we knew it was gonna be funky until they and we fixed stuff [15:31:13] sorry bout that [15:31:34] ottomata: no need, I learned a lot of things :) [15:32:45] ottomata: dumps.wm.o don't go through varnish, and we still want them somewherre, right ? [15:33:24] Now the question is where... [15:34:33] ok, makes sense ... ottomata: in /a/log/webrequest/archive/, everything is generated from legacy-tsvs job, or is it old sampled logs as well ? [15:35:55] just legacy-tsv jobs [15:36:04] the old sampled logs are in /a/squid/archive [15:36:06] joal: ^ [15:36:11] right ottomata [15:36:40] ottomata: should we have a dedicated partition in /wmf/data/raw with a specific refine job ? [15:37:09] Cause my guess is that, it there are logs, sooner or later, refined dzta will be asked :) [15:37:13] ottomata: --^ [15:37:53] ? [15:38:11] joal: the dumps logs are going to come directly from dumps.wm.org server, dataset1001 [15:38:19] dumps.wm.org is hosted by nginx [15:38:24] they're just going to be rsynced over [15:38:27] via a cron [15:38:36] i'm just trying to figure out where they should put them on stat1002 [15:38:59] ottomata: yessir, I got that --> May be we should rsync + hdfs copy, then refine, then send back (refined) to /a/log/ ? [15:39:29] Longer process, more complex, but fits better in our current pipeline [15:39:40] OR, instead of rsync, straight into kafka ? [15:39:58] Would even be easier [15:40:07] Maybe I'm thinking overkill :) [15:40:43] ottomata: question about kafka restarts.. are we good to start the work? [15:41:00] ? [15:41:06] haha [15:41:10] joal: no i think just rsyncing them is fine [15:41:10] we still need to restart all the brokers :D [15:41:21] elukey: when did we truncate logs last week? [15:41:32] ok ottomata, sorry for overkill thinking :) [15:41:43] ottomata: Wed I think? [15:41:52] elukey: i think we should wait til then [15:41:57] its probably safe to do that [15:42:01] now [15:42:05] but, if there is no real hurry [15:42:09] its even safer just to wait 2 more days [15:42:11] all right, I'll double check the date and update the email thread [15:42:15] k cool [15:42:15] danke [15:42:23] well Moritz is kinda waiting for us :D [15:42:26] heh, i know [15:42:37] but there's no real hurry aside from wanting to do it so we can close a ticket, ja? [15:42:59] ottomata: double checked x_forwarded_for (yesterday day), not empty from what I have seen :( [15:42:59] joal: i'm going to tell them to put it in /a/log/webrequest/dumps.wikimedia.org [15:43:04] Sure ottomata [15:43:21] will put a README in that dir explaining [15:43:24] joal: https://gerrit.wikimedia.org/r/#/c/253474/ [15:43:27] ja, we ahven't merged that, right? [15:43:28] ottomata: a security engineer has been sent to hunt you down, they read these channels! :P [15:43:49] hehe [15:44:35] Thu this week might be good [15:44:38] ottomata: Arf, ok, since your comment said let's verify, I just checked ... Told you ... sloooooooow brain [15:46:18] hehe, ja you wanted to " Joal is looking into some differences between what we tag as client_ips, and what is now coming in on the IP field. " [15:46:31] hehheeh [15:46:32] joal: ! [15:46:39] no if you are sicky just shHHHhHHhh [15:46:44] hehe [15:46:56] ottomata: the checks over IPs being different has been done, e discussed that sayin [15:47:19] that the version computed on varnish was probably better than the one computed by hadoop [15:47:41] Now, I'll check that client_ip = ip for a day :) [15:48:00] right, cause client_ip is from our XFF logic, ja? [15:48:16] ottomata: it was, but now it should be =ip [15:48:26] (double checking job restart did the job) [15:48:45] ottomata: I think we merged the client_ip change stuff [15:48:48] Man ... [15:50:35] ? [15:50:36] :) [15:50:51] Just fighting with my brain :) [15:51:06] So ottomata, ovfer an hour of refine webrequest, 2016-02-15 15:50:10,133 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 1102.13 sec [15:51:10] oops:) [15:51:23] (ip = client_ip) -- 100% [15:51:31] awesome [15:52:40] ottomata: my last access to kafka1012 was on Tue Feb 9 16:30, so I'll start the reboots on Wed afternoon-ish.. is that ok for you? [15:54:07] ottomata: only place where it's still in use: oozie/pagecounts/load [15:55:06] ottomata: In that place, it's assuming that default empty value for x_forwarded_for is '-' [15:55:13] elukey: cool w me! [15:55:18] all right! [15:56:23] hmm, joal oh, so do we need to deploy and restart that one? [15:56:41] ottomata: I think we should modify and restart yes [16:01:05] ok! [16:07:24] ottomata: trying to join [16:26:27] actually ottomata, I have re-read the code involving x_forwarded_for --> should be fine even without restart [16:26:57] Analytics, Analytics-Kanban: Back-fill pageviews data for dumps.wikimedia.org to May 2015 - https://phabricator.wikimedia.org/T126464#2028982 (elukey) [16:27:14] oh [16:27:17] ja? [16:27:56] Analytics-Kanban, Operations, Traffic: varnishkafka integration with Varnish 4 for analytics - https://phabricator.wikimedia.org/T124278#2028985 (elukey) [16:28:51] joal, put on your headphones :] [16:30:14] ottomata: standuppp [16:30:53] aye [16:32:46] Analytics-Kanban, Wikimedia-Mailing-lists: Home page for the analytics mailing list should link to gmane [1 pts] - https://phabricator.wikimedia.org/T116740#2029005 (Nuria) Open>Resolved [17:30:04] ebernhardson: we're getting ready to release an update to the Android app that contains the CirrusSearch a/b test (with opening_text). I just wanted to triple-check that these are the correct parameters: cirrusMltUseFields=yes&cirrusMltFields=opening_text&cirrusBoostLinks=no [17:30:13] dcausse: ^ [17:30:58] dbrant: sounds good, let me double check [17:31:55] dbrant: yes it's perfect [17:36:52] dcausse: cool, thanks [17:51:47] Analytics-Kanban: Move archive/pageview and archive/projectview into archive/pageview-aggregates {hawk} - https://phabricator.wikimedia.org/T127000#2029349 (mforns) NEW a:mforns [18:08:53] Analytics-Kanban: Move archive/pageview and archive/projectview into archive/pageview-aggregates {hawk} - https://phabricator.wikimedia.org/T127000#2029428 (mforns) [18:09:28] ottomata, can you please review the description of the renaming to archive/pageview-aggregates task? https://phabricator.wikimedia.org/T127000 [18:14:51] Analytics-Kanban: Move archive/pageview and archive/projectview into archive/pageview-aggregates {hawk} - https://phabricator.wikimedia.org/T127000#2029448 (Ottomata) Looks good! We don't really have consistent naming guidelines, but I think in this case I'd like to use `pageview_aggregates` rather than `pag... [18:14:59] mforns: commented [18:15:35] ottomata, thx [18:16:57] Analytics-Kanban: Move archive/pageview and archive/projectview into archive/pageview_aggregates {hawk} - https://phabricator.wikimedia.org/T127000#2029452 (mforns) [18:35:53] (PS1) Mforns: Move location of pageview and projectview archives [analytics/refinery] - https://gerrit.wikimedia.org/r/270789 (https://phabricator.wikimedia.org/T127000) [20:49:07] Analytics-Kanban, Wikipedia-Android-App: Database not updated for beta event logging and all-events.log reports 8x for each event [3 pts] - https://phabricator.wikimedia.org/T125423#2029771 (Niedzielski) Nice, it looks like the duplicate events have been fixed. Unfortunately, now I don't see any events in... [21:01:23] Analytics-Kanban, Wikipedia-Android-App: Database not updated for beta event logging and all-events.log reports 8x for each event [3 pts] - https://phabricator.wikimedia.org/T125423#2029822 (Nuria) I rebooted and reinstalled to make sure there was no debug code from our tests. As far as i can see logs are... [21:11:48] Analytics-Kanban, Wikipedia-Android-App: Database not updated for beta event logging and all-events.log reports 8x for each event [3 pts] - https://phabricator.wikimedia.org/T125423#2029835 (Niedzielski) @Nuria, my mistake! I was using the wrong app build flavor. Thank you for fixing this issue! [21:12:14] Analytics-Kanban, Wikipedia-Android-App: Database not updated for beta event logging and all-events.log reports 8x for each event [3 pts] - https://phabricator.wikimedia.org/T125423#2029837 (Nuria) Open>Resolved [22:02:18] (PS1) Nuria: [WIP] Dashiki gets pageview data from pageview API [analytics/dashiki] - https://gerrit.wikimedia.org/r/270867 (https://phabricator.wikimedia.org/T124063)