[00:00:43] madhuvishy: madhuvishy.last_access_uniques_daily_new and madhuvishy.last_access_uniques_monthly_new are empty but let's talk when you submit your next patch. [00:02:06] nuria: aah, okay, must be something minor with the path - the data got written [00:03:09] madhuvishy: wait ... how do you see that? [00:04:15] nuria: hadoop fs -ls -R /user/madhuvishy/data/last_access_uniques_new/ [00:09:38] nuria: all good now - tables have data [00:09:54] madhuvishy: and teh country column? [00:09:56] *the [00:10:09] nuria: what about country? [00:10:22] the uniques are reported per country? [00:10:48] oh - well - uniques are always per domain - you can of course breakdown by country, but only within the domain [00:11:28] nuria: as long as you have domain fixed, you can break it down by any of the other dimensions here - https://gerrit.wikimedia.org/r/#/c/216341/8/hive/last_access_uniques/create_last_access_uniques_daily_table.hql [00:13:00] madhuvishy: right, ok, i do not think we need the country yet but i do not want to distract you further [00:13:13] nuria: :) we have it anyway [00:13:34] just important that we don't aggregate only by country [00:14:17] madhuvishy: but to give an interval that includes true users w/o cookies we would need to calculate those by country too [00:14:58] nuria: hmmm i'm missing something - are you around for longer, we can chat in a bit [00:15:07] madhuvishy: yes, until 5 [00:15:12] ok [00:15:20] madhuvishy: will look at data until then, thank you! [00:15:42] madhuvishy / nuria: I created an index on the user column in the report table [00:15:49] I have 0 idea why I didn't think of that before [00:15:58] but the report results screen is super fast now [00:16:30] !log wikimetrics1 and limn1 are having serious puppet issues because they're self-hosted [00:16:59] milimetric: nice [00:17:29] milimetric: i was telling madhuvishy that we need to test non ascii chars on those forms [00:17:50] milimetric: cause otherwise that will be teh 1st issue we get reported [00:18:13] nuria: I think the paste_usernames is the only one that allows free input [00:18:19] and that form inherits from the other one that supports all that [00:18:22] so it should be good [00:18:27] but worthwhile testing [00:18:48] milimetric: i am happy to do that tomorrow ( i have given up on fixing encoding on flask but we can fix it for our use case I think) [00:19:38] yep [00:32:44] (PS16) Madhuvishy: [WIP] Setup celery task workflow to handle running reports for the ProgramMetrics API [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/253750 (https://phabricator.wikimedia.org/T118308) [00:32:54] milimetric: i left some minor feedback on your pages_edited thing [00:33:05] nuria: done, wanna talk? [00:33:10] sure [00:33:16] batcave? [00:33:21] yup omw [01:50:32] Analytics-Backlog, Analytics-Wikimetrics, Puppet: Cleanup Wikimetrics puppet module so it can run puppet continuously without own puppetmaster {dove} - https://phabricator.wikimedia.org/T101763#1864407 (yuvipanda) Hello! After every time we change any fundamental settings (DNS, LDAP, puppetmaster, etc)... [09:24:18] hi a-team [09:24:23] Morning mforns [09:24:31] Had a good weekend ? [09:24:33] hey joal, how are things? [09:24:37] yes :] [09:24:38] not bad :) [09:25:01] could slow down a bit [09:25:21] That's good :) [09:25:39] When christmas arrives, social pressure goes up :) [09:25:45] hehehe [09:25:46] right [09:26:00] looots of things to do... [09:26:08] This kind of reasons have had me movign to contry-side ;) [09:26:19] Indeed, too many things [09:26:38] yes, we also don't like them, hehe [09:27:14] mforns: If you could have a quick look here : https://wikitech.wikimedia.org/wiki/Analytics/Data/Pageview_hourly/Sanitization I'd be greatful :) [09:27:34] mforns: There are two broken links on pages for details, I'm on it [09:27:43] * mforns looks [09:28:15] joal, is there another thing I can do for that task? [09:28:24] mforns: not really :) [09:28:27] I mean, in addition to that [09:28:34] ok [09:29:17] did you guys finish the scala code? [09:29:36] mforns: I wanted to play with your scala code but had no time (spent the last two days writing, how difficult) [09:30:03] mforns: I'd like to be around when you run that scala thing for testing :) [09:30:10] the docs? yes I imagine it has been really a challenge [09:30:14] * joal loves seeing stuff working [09:30:19] because it's non-easy stuff [09:31:03] joal, so the scala code still needs to be worked on, to add the parametrization and stuff, no? I can continue with that. [09:31:18] mforns: you know better than I do :) [09:31:33] Before parameterization, I think you should go for test runs using spark-shell [09:31:45] joal, oh yes, I forgot [09:31:48] ok [09:31:56] Also mforns, let's batcave for a minute, I have had an idea yesterday about the whole thing [09:32:06] ok [09:32:18] I'm in the school, but we can try [09:32:20] :] [09:32:37] mforns: We can do IRC as well, it will just be less easy :) [09:32:51] maybe better yes [09:32:58] thx [09:33:01] So, we have been the algorithm as if we only had the current version of pageview_hourly fields [09:33:05] np [09:33:08] mmmmmm [09:33:10] I see [09:33:31] BUT, in fact, we could have IP in the non-anonymized pageview hourly ! [09:34:07] Meaning, we could asctually check for finger-print groups that only have a small number of IPs for anonymization [09:34:20] Instead of using K [09:34:35] did the non-anonymized pageview_hourly have other fields that aren't there any more now? [09:34:58] ??? [09:35:14] Sorry, me no comprendo [09:35:23] :) [09:35:30] in the history of pageview_hourly, have we removed fields at some point? or we have only added fields? [09:35:45] We have only added fields [09:35:56] I see, then I don't get the IP thing [09:36:00] :] [09:36:11] So, for historical purposes, the algo using K is usefull (we don't have IPs in the historical data) [09:36:23] I see [09:36:32] But for new data, we could say that we add IP in the non-anonymized PV [09:36:53] you suggest adding the set of unique ips per row in the pageview_hourly non-anonymized? [09:37:27] ok, makes sense [09:37:59] so, and instead of using K over the page_counts, use K over the unique ips? [09:38:36] hm, we don't even need K : we anonymize finger-print groups having less than X distinct IPs [09:38:46] Whatever the number of requets [09:39:14] Ohhh, that's what you meant I get it now :) So let's not use K, it's misleading :) [09:39:34] I mean, in your suggestion: K == X, no? [09:39:58] hm, I don't think so [09:40:07] I think it's still a k-anonymization algorithm, no? [09:40:41] I don't thinkl it is exactly [09:40:58] Cause K is about the number of requests [09:41:13] well, it was, because we chose it like that [09:41:27] but we can choose it to affect distinct ips instead [09:41:35] Here if you we use distinct IPs, it means that we actually anonymize fp-groups having a lot of requests (many more than K) [09:42:00] You're right [09:42:16] it's just a definition problem [09:42:19] It's just that it feels bizarre to me to affect K to IPs :) [09:42:33] But it's the same thing [09:42:37] but I still think we are implementing k-anonymity on distinc ips [09:42:45] You're right [09:43:03] so K makes sense to me, but if you think we should use another symbol, I'm also ok :] [09:43:14] I think it's really better than using K over requests [09:43:27] Naao it's find :) [09:43:28] sure! that was a great idea [09:43:47] and one question [09:44:07] you imagined we would store the actual ips, or just the number of distinct ips? [09:44:42] mforns: we actually need the actual IPs, since the groups will be different [09:44:50] We need the IP set [09:45:14] I see, so when you aggregate fp-groups you can calculate the union of the ips [09:45:21] cool [09:45:23] correct [09:45:40] Actually we need the distinct IPs set :) [09:45:56] is that feasible in terms of space? [09:46:11] But, since we don't have it for historical data, let's still test what you have on current data :) [09:46:18] ok [09:46:46] mforns: Yeah, that's fine in space term: we'll only keep one or two month, and it only adds a few ints poer row [09:46:52] we might want to store the set of distinct ips just for rows that have less than 10 or something [09:47:03] mforns: you're right [09:47:39] ok, joal I need to leave now, I'll be working from 12 to 14 later on [09:47:48] sounds good [09:47:54] I'll kepp documenting :) [09:48:03] laters mforns [09:48:05] see you later then :] [09:48:10] thanks for the updates! [09:48:15] np :) [09:48:25] Thanks for listening :) [10:32:47] Analytics-Kanban, DBA, Patch-For-Review: Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) - https://phabricator.wikimedia.org/T120187#1864911 (jcrespo) From the gerrit comments, I assume it did work, finally? What was the issue (so we check it on production)? [14:35:46] joal: hi! can't remember the reason why you told that bundle.xml was not needed for our cirrus request oozie jobs. In which case a bundle.xml is needed? [14:35:57] Hi dcausse :) [14:36:21] bundle.xml is need when you want to start multiple coordinators with different parameters [14:37:25] Like imagine you'd like, in the Score aggregator job, aggregate over 3 days, 7 days, and 14 days --> bundle having 3 coordinators defined, passing them the number of days to aggregate on as paramter parameter [14:37:44] If you have only one coordinator to start, having a bundle is overkill [14:37:50] hm ok got it thanks! [14:38:05] here it makes sense then: https://gerrit.wikimedia.org/r/#/c/257436/9/oozie/transfer_to_es/bundle.xml [14:38:17] ASlso dcausse I didn't tell you : you have re-partitioned data from November 4th :) [14:38:34] wow, thank you! [14:38:38] :) [14:38:47] In the example you give it does make sense :) [14:38:55] ok perfect thanks :) [14:39:14] Be careful when implementing loading data into ES to have a measured number of reducers [14:39:27] You don't want to DDOS your es cluster :) [14:39:38] dcausse: --^ [14:39:39] :) [14:39:43] yes :) [15:14:46] (CR) Milimetric: Add pages edited metric (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/174773 (https://bugzilla.wikimedia.org/73072) (owner: Mforns) [15:14:54] (PS5) Milimetric: Add pages edited metric [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/174773 (https://bugzilla.wikimedia.org/73072) (owner: Mforns) [15:43:43] Analytics-Kanban, EventBus: Refactor kafka puppet roles with hiera for more generic use than analytics - https://phabricator.wikimedia.org/T120957#1865563 (Ottomata) NEW a:Ottomata [15:46:12] ottomata: should we deploy the toku db stuff [15:46:45] yeah! [15:47:03] ok, let's do it, let me see if i still have permits to tin [15:47:04] go right ahead, i'm here if you need me :) [15:47:06] i think you do [15:49:12] ottomata: can you answer jaime's question here: https://phabricator.wikimedia.org/T120187 [15:49:28] ottomata: why was it that we could not create tables 1st? and they were aria? [15:49:38] ya... [15:49:41] one sec.. [15:50:13] ??? [15:50:17] i can't get into deployment-eventlogging03? [15:50:21] Permission denied (publickey). [15:50:21] Killed by signal 1. [15:50:25] i can get into deployment-eventlogging04 [15:50:26] hmmMm [15:50:27] ? [15:50:42] nuria, can you run puppet on deployment-eventlogging-3 [15:50:43] 03 [15:50:44] please? [15:51:01] ottomata: the labs machine? [15:51:20] yes [15:51:26] sudo puppet agent -t [15:51:49] cannot get either, labs is hoosed [15:51:53] HmmMH [15:51:59] welp, i can't answer that question right now then, hehe [15:52:03] jaja [15:52:06] need to look at error message in logs [15:52:09] let's tell them on labs though [15:52:15] k [15:59:29] joal: I just messed with your page: https://wikitech.wikimedia.org/wiki/Analytics/Data/Pageview_hourly/Sanitization [15:59:43] as I say in my edit summary, feel free to un-mess with it [16:00:03] nuria: coren fixed it [16:00:05] it needed a puppet run [16:00:20] Thanks milimetric :) [16:00:26] * joal will read :) [16:01:55] Analytics-Kanban, DBA, Patch-For-Review: Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) - https://phabricator.wikimedia.org/T120187#1865615 (Ottomata) Yeah, apparently transparent huge pages and TokuDB don't like each other: ``` Transparent huge pages a... [16:03:03] ottomata: go ascii art go... [16:03:07] ottomata: just deployed [16:03:30] cool [16:04:06] ottomata: let me try to create a table.. ahem... it is going to take a bit as i need to put a schema together [16:04:23] Analytics-Kanban, DBA, Patch-For-Review: Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) - https://phabricator.wikimedia.org/T120187#1865616 (jcrespo) Oh yes, we have that puppetized on production. [16:05:06] aye [16:08:49] ottomata: ah no, i can do it with test events bumping up schema [16:08:56] ottomata: give me a sec [16:10:19] nuria: make sure that schema isn't blacklisted [16:10:37] ottomata: k [16:10:41] eventlogging_valid_mixed_schema_blacklist: ^Analytics|CentralNoticeBannerHistory$ [16:10:58] k [16:12:27] ottomata: where was the scripts we had to generate load? ... [16:13:38] nuria: it is bin/eventlogging-load-tester [16:22:28] PROBLEM - Overall insertion rate from MySQL consumer on graphite1001 is CRITICAL: CRITICAL: 27.27% of data under the critical threshold [10.0] [16:23:32] heyYyy nuria ^ :) [16:23:49] ottomata: argghhhhh [16:23:54] ottomata: rolling back [16:24:03] hang on lemm elook [16:24:43] HMmmm [16:24:51] dunno, logs look good, i see inserted messages [16:25:00] maybe just an artifact of a restart? [16:25:02] ya me too [16:25:04] wait [16:25:32] ja i think it is an artifact maybe [16:25:50] ottomata: maybe is the restart? [16:26:00] ottomata: ah sorry [16:26:05] ottomata: i see your message [16:26:31] ottomata: let's wait 15 [16:26:59] aye [16:26:59] k [16:55:43] o/ joal [16:55:54] Hey halfak :) [16:55:55] Can you remind me where to find out work on cloud9? [16:55:59] *our [16:56:04] Sure :) [16:56:17] hey nuria, i think the events are making it into mysql just fine, they are just so spikey! [16:57:04] average inserted is good [16:57:04] halfak: https://c9.io/jobar/wmf_dumps [16:57:04] Thanks! [16:57:04] I think if you go to c9.io and log in you have access to your workspaces [16:57:04] :) [16:57:56] Gotcha. [16:58:05] Was searching my history for urls containing "cloud9" [16:58:07] :/ [16:58:12] huhu :) [16:58:33] (PS1) Addshore: Remove references scripts [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/257929 [16:58:48] (CR) Addshore: [C: 2] Remove references scripts [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/257929 (owner: Addshore) [16:59:27] (Merged) jenkins-bot: Remove references scripts [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/257929 (owner: Addshore) [17:04:54] Analytics-Kanban: Backfill EL data for 2015-11-27 incident {oryx} [5 pts] - https://phabricator.wikimedia.org/T119981#1865872 (mforns) [17:04:56] Analytics-Kanban: Eventlogging devserver needs maintenance [3 pts] - https://phabricator.wikimedia.org/T120245#1865873 (Milimetric) [17:05:04] Analytics-Cluster, Analytics-Kanban: Estimate number of users (or requests) that have cookies off (due to fresh session or incognito mode) [3 pts] - https://phabricator.wikimedia.org/T119653#1865874 (Milimetric) [17:10:08] Analytics-Kanban, DBA, Patch-For-Review: Make EL create tables using TokuDB by default [5] - https://phabricator.wikimedia.org/T120967#1865885 (Nuria) NEW a:Nuria [17:10:58] Analytics-Kanban, DBA, Patch-For-Review: Make EL create tables using TokuDB by default [5] - https://phabricator.wikimedia.org/T120967#1865885 (Nuria) s://gerrit.wikimedia.org/r/257738 [17:15:14] Analytics-Kanban: Troubleshooting limn1 and wikimetrics1 self-hosted puppet woes - https://phabricator.wikimedia.org/T120968#1865900 (Milimetric) NEW a:Milimetric [17:31:56] kevinator, 1x1? [17:32:09] jsut a sec, I want to hear this [17:32:13] sure :] [17:32:16] then yes, 1x1 [17:32:21] ok [17:39:55] mforns: are you backlog grooming too? [17:45:03] Analytics: Upgrade daily/monthly aggregations of pageview dumps to new data files - https://phabricator.wikimedia.org/T90203#1865991 (Milimetric) Open>Invalid Closing this task in favor of the other work that we already finished, which was to create the dumps now hosted at: http://dumps.wikimedia.org/o... [17:45:23] (Draft1) Addshore: Initial commit [analytics/limn-wikidata-data/vendor] - https://gerrit.wikimedia.org/r/257941 [17:45:30] (CR) Addshore: [C: 2 V: 2] Initial commit [analytics/limn-wikidata-data/vendor] - https://gerrit.wikimedia.org/r/257941 (owner: Addshore) [17:46:37] Analytics: Substantial amount of ip addresses in 1:1000 sampled squid logs does not resolve into geo data, from Nov 2013 onwards - https://phabricator.wikimedia.org/T90235#1865999 (Milimetric) Do the new UDF functions for handling X-Forwarded-For take care of this problem, Erik? [17:46:40] Analytics: Substantial amount of ip addresses in 1:1000 sampled squid logs does not resolve into geo data, from Nov 2013 onwards - https://phabricator.wikimedia.org/T90235#1866000 (Milimetric) [17:49:50] Analytics: Review EL requests from Mediaviewer - https://phabricator.wikimedia.org/T88849#1866018 (Milimetric) a:kevinator>Nuria [17:51:05] Analytics, Multimedia: Add mediacounts to pageview API - https://phabricator.wikimedia.org/T88775#1866021 (Milimetric) [17:53:59] Analytics, Fundraising-Backlog, Wikimedia-Fundraising: Public dashboards for CentralNotice and Fundraising - https://phabricator.wikimedia.org/T88744#1866043 (Milimetric) Can we get more information on this task? What kind of dashboards, what kind of data are we visualizing, how often is it updated,... [17:54:48] (PS1) Addshore: Count references using dumps [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/257945 [17:55:13] (CR) jenkins-bot: [V: -1] Count references using dumps [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/257945 (owner: Addshore) [17:56:14] (PS2) Addshore: Count references using dumps [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/257945 [17:56:37] Analytics: Add pageview counter to wikimetrics/dashiki/dashboards/any tool we own on labs - https://phabricator.wikimedia.org/T87682#1866060 (Milimetric) Open>Resolved a:Milimetric this is done through the piwik instance running on labs. Pageview numbers are very low in general :) We will improve... [17:57:44] Analytics: write a short document directing people who want us to use their predictive analytics tools ..... - https://phabricator.wikimedia.org/T76617#1866068 (Milimetric) Open>Invalid a:Milimetric I don't think this is valid any more [17:59:59] Analytics, Wikimedia-General-or-Unknown: Cache multimedia limn JSON datasources - https://phabricator.wikimedia.org/T75611#1866084 (Milimetric) Open>declined a:Milimetric at this point we're very unlikely to spend more effort making limn better. We made sure dashiki deals with caching very well... [17:59:59] Analytics: Evaluate ABLincoln A/B testing framework - https://phabricator.wikimedia.org/T86853#1866089 (Milimetric) Open>declined a:Milimetric [18:00:20] (PS1) Addshore: Add datavalues libs [analytics/limn-wikidata-data/vendor] - https://gerrit.wikimedia.org/r/257948 [18:00:30] (CR) Addshore: [C: 2 V: 2] Add datavalues libs [analytics/limn-wikidata-data/vendor] - https://gerrit.wikimedia.org/r/257948 (owner: Addshore) [18:03:54] Analytics: Research's daily pagecounts aggregates not increasing date since 2014-12-25 - https://phabricator.wikimedia.org/T85723#1866127 (Milimetric) Open>Invalid a:Milimetric no longer relevant, probably [18:07:10] Analytics, Epic: Per-file view stats - https://phabricator.wikimedia.org/T77541#1866153 (Milimetric) This is technically available as part of the mediacounts dumps: http://dumps.wikimedia.org/other/mediacounts/ [18:07:22] Analytics, Epic: Per-file view stats - https://phabricator.wikimedia.org/T77541#1866154 (Milimetric) Open>Resolved a:Milimetric [18:08:38] Analytics: Cannot permalink easily to a single graph - https://phabricator.wikimedia.org/T76670#1866169 (Milimetric) Open>declined a:Milimetric we're not going to get to any more limn changes, we'll be migrating to dashiki which has good bookmarking support [18:11:58] Analytics: Epic: Create pipeline EL -> DW -> Pentaho - https://phabricator.wikimedia.org/T75841#1866187 (Milimetric) Open>declined a:Milimetric too generic, we'll be working on this pipeline soon though [18:14:52] Analytics: Understand performance & monitor to current capacity- set up tests to identify performance bottleneck (alarm for no writes) - https://phabricator.wikimedia.org/T75825#1866192 (Milimetric) Open>Resolved a:Milimetric We're up on kafka and the future's so bright we gotta wear shades [18:16:32] Analytics: stats.grok.se about a week behind - https://phabricator.wikimedia.org/T89326#1866213 (Milimetric) Open>declined a:Milimetric stats.grok.se seems to not be maintained any more. See the new pageview API and data dumps for the way forward. [18:19:53] Analytics: Page view statistics looks not updated since 2015-08-13 at 18:00:00 - https://phabricator.wikimedia.org/T109070#1866237 (Milimetric) Open>Resolved a:Milimetric All data that could have been recovered was recovered here. Any data missing is forever lost sadly. [18:20:20] Analytics: Pagecount-raw files missing since 27th at 22.00 - https://phabricator.wikimedia.org/T113931#1866241 (Milimetric) Open>Resolved a:Milimetric all recoverable data has been recovered here [18:20:53] Analytics, Continuous-Integration-Config: add CI for repos analytics/limn-*-data - https://phabricator.wikimedia.org/T117416#1866246 (Milimetric) Open>declined a:Milimetric we will focus on phasing out limn next year, as part of the project code named {frog} [18:22:15] milimetric: should we also change the route to /reports/program-global-metrics/create? I wonder if create is really necessary too [18:23:04] madhuvishy: yeah, I think reports/program-global-metrics [18:23:08] ok cool [18:30:40] Analytics, Epic: Per-file view stats - https://phabricator.wikimedia.org/T77541#1866297 (Gilles) [18:31:18] joal, I had a couple comments on the anonymization docs, do you have 5 minutes today to batcave them? [18:31:30] sure mforns ! [18:31:55] joal, ping me when you've got time :] [18:32:20] mforns: joining :) [18:32:24] ok [18:32:26] : [18:32:28] ] [18:35:43] (PS17) Madhuvishy: Setup celery task workflow to handle running reports for the ProgramMetrics API [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/253750 (https://phabricator.wikimedia.org/T118308) [18:36:32] joal, any protips for getting the code I'm working on out of cloud9? [18:36:36] milimetric: ^ removed WIP tag, think it can start getting reviewed. [18:36:38] Should I be copy-pasting? [18:36:45] Is it associated with a repo? [18:36:54] in the big meeting, madhuvishy (5th floor) [18:36:54] halfak: none :( [18:36:58] but I'll look at it right after [18:37:06] Copy-paste it is! [18:37:12] halfak: I usually git commit there [18:37:19] milimetric: no problem, any time :) (after your oozie stuff too ;)) [18:37:21] Oh! I see there is a term [18:37:33] halfak: I'll ask you the same question ;) [18:37:43] halfak: yes, on the terminal, just git commit [18:37:52] sounds great ! [18:39:02] Seems like we should have a repo specifically for the code used to maintain hive on the Research Cluster(TM) [18:40:32] agreed halfak ! [18:41:00] OK. I'll make that and add this script. [18:49:08] joal, https://github.com/wikimedia-research/research-cluster [18:49:09] :) [18:51:55] \o/ halfak ! [18:52:14] I also just CC'd you on a card for this. [18:52:24] I'll update that card as I run tests on the cluster [18:52:41] Awesome :) [19:10:10] Analytics-EventLogging, Analytics-Kanban: EventLogging Kafka consumer stops consuming after Kafka metadata change. See if upgrade fixes it. {oryx} [13 pts] - https://phabricator.wikimedia.org/T118315#1797263 (Ottomata) I have just reproduced this in labs, and can confirm that upgrading to pykafka 2.1.0 fi... [19:25:55] Quarry: Login to somenody's account - https://phabricator.wikimedia.org/T120988#1866469 (IKhitron) NEW [19:26:04] Quarry: Login to somebody's account - https://phabricator.wikimedia.org/T120988#1866476 (IKhitron) [19:27:05] Quarry: Login to somebody's account - https://phabricator.wikimedia.org/T120988#1866479 (yuvipanda) How long ago did this happen? [19:28:03] Quarry: Login to somebody's account - https://phabricator.wikimedia.org/T120988#1866480 (IKhitron) A minute ago. [19:29:04] Quarry: Login to somebody's account - https://phabricator.wikimedia.org/T120988#1866482 (yuvipanda) Probably a blip from the redis session stuff I was just doing. I'll clear out everyone's sessions to make sure :) [19:30:17] Quarry: Login to somebody's account - https://phabricator.wikimedia.org/T120988#1866490 (IKhitron) Hope you are right. Otherwise it's security problem. :-) [19:31:45] Quarry: Login to somebody's account - https://phabricator.wikimedia.org/T120988#1866492 (yuvipanda) I've just cleared out everyone's sessions, which should clear up any other missing issues! Hopefully I won't have to mess with redis again for a while :) Can you confirm that your session has been logged out? [19:33:15] Quarry: Login to somebody's account - https://phabricator.wikimedia.org/T120988#1866500 (IKhitron) Indeed, it has. Thank you. [19:35:13] Quarry: Login to somebody's account - https://phabricator.wikimedia.org/T120988#1866505 (yuvipanda) Open>Resolved a:yuvipanda Ok :) Closing it for now! :) Thanks for reporting! <3 [19:36:00] Quarry: Login to somebody's account - https://phabricator.wikimedia.org/T120988#1866508 (IKhitron) No problem. [19:39:47] Analytics-EventLogging, Analytics-Kanban, Patch-For-Review: EventLogging Kafka consumer stops consuming after Kafka metadata change. See if upgrade fixes it. {oryx} [13 pts] - https://phabricator.wikimedia.org/T118315#1866514 (Nuria) >I have just reproduced this in labs, and can confirm that upgrading... [19:53:10] Quarry: Login to somebody's account - https://phabricator.wikimedia.org/T120988#1866567 (Edgars2007) I'm everywhere :) [19:53:48] Quarry: Login to somebody's account - https://phabricator.wikimedia.org/T120988#1866569 (yuvipanda) As the wise poets from 'The Beatles' once said, 'here, there, everywheeeerreeeeeee' [19:54:33] Quarry: Login to somebody's account - https://phabricator.wikimedia.org/T120988#1866576 (IKhitron) You're everywhere, @Edgars2007, or anyone can use your account? :-) [20:18:33] Quarry: Login to somebody's account - https://phabricator.wikimedia.org/T120988#1866675 (Edgars2007) >>! In T120988#1866576, @IKhitron wrote: > You're everywhere, @Edgars2007, or anyone can use your account? :-) I trust you :) [20:20:10] milimetric: do you have permits to log into the master for EL? [20:27:37] milimetric: nm [20:31:52] Analytics-Kanban, DBA, Patch-For-Review: Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) - https://phabricator.wikimedia.org/T120187#1866755 (Nuria) @jcrespo: Chnaged configuration but as far as i can see tables in prod are not created. How to repro:... [20:40:41] madhuvishy: numbers for last access are about 150 million on desktop + 270+ million on mobile + offsets I need to add [20:41:01] madhuvishy: which checks out with the 500 million number, let see about offsets [20:46:54] nuria: that's what i got too [20:47:54] madhuvishy: Let me calculate the offset (which will always be an increase) and i will pass it along [20:56:33] nuria: is the eventlogging service running on beta? [20:58:36] madhuvishy: it was yesterday, today andrew has deployed an update to pykafka, lemme see [20:59:16] nuria: how can i hit it? [21:00:11] see: [21:00:29] https://wikitech.wikimedia.org/wiki/Analytics/EventLogging/TestingOnBetaCluster [21:00:43] https://wikitech.wikimedia.org/wiki/Analytics/EventLogging/TestingOnBetaCluster#How_to_log_a_client-side_event_to_Beta_Cluster_directly [21:00:54] madhuvishy: service is running fine [21:01:07] (just looked at ps -aux) [21:01:09] oh nuria not this - more like the new service.py that andrew worked on [21:01:24] madhuvishy: ah the new endpoint? [21:01:28] yes [21:01:35] i am not sure it is running there though, let me see [21:01:53] yeah, i want to know how i can run it - if it's not already [21:02:28] madhuvishy: i do not think it is [21:02:41] madhuvishy: let me see again [21:02:53] nuria: okay, i can probably figure it out [21:02:56] no problem [21:03:49] madhuvishy: let me give you some pointers, argh, gerrit so slowww [21:04:05] nuria: sure [21:04:30] nuria: basically, I want to test a change i'm making to the service [21:04:45] s/service/service.py [21:04:48] madhuvishy: ah, for that i would run it on your desktop [21:05:06] madhuvishy: start kafka bin/kafka-server-start.sh config/server.properties [21:05:06] nuria: i can't because to test i want to hit statsd [21:05:14] don't have a local statsd setup [21:05:19] ah ok [21:05:24] so i want to use the labs one [21:08:23] Analytics-EventLogging, Analytics-Kanban, Patch-For-Review: EventLogging Kafka consumer stops consuming after Kafka metadata change. See if upgrade fixes it. {oryx} [13 pts] - https://phabricator.wikimedia.org/T118315#1866900 (madhuvishy) a:Ottomata [21:08:46] madhuvishy: https://github.com/wikimedia/eventlogging/commit/8b31f1206f903a7490da4168a6bec88e5c55411a [21:09:00] nuria: oh coool, thanks [21:10:42] madhuvishy: let me know if you cannot make it work that way, it bypasses puppet though, ... i am not sure whether we have puppet for proxy endpoint as i never reviewed that [21:10:45] mmmm [21:10:58] can find out if needed [21:29:38] nuria: no i got it to work [21:29:44] it's fine :) all works [21:29:55] madhuvishy: with puppet ? or low tech? [21:32:07] nuria: low tech - i cannot go through puppet now because andrew needs to package a library first [21:32:11] but this is fine [21:32:17] madhuvishy: ok, ya, great [21:45:03] Analytics-EventLogging, Analytics-Kanban, EventBus: Send HTTP stats about eventlogging-service to statsd - https://phabricator.wikimedia.org/T118869#1867050 (madhuvishy) a:Ottomata>madhuvishy [21:48:58] Analytics-EventLogging, Analytics-Kanban, Analytics-Visualization: {lion} VE Support - https://phabricator.wikimedia.org/T89924#1867078 (kevinator) Open>Resolved a:kevinator Closing this task as it was a 2014-2015 FQ4 goal which carried into Q1. Any new work for the Editing team will fall und... [22:04:06] Analytics: Upgrade daily/monthly aggregations of pageview dumps to new data files - https://phabricator.wikimedia.org/T90203#1867155 (ezachte) Invalid>Open [22:05:18] Analytics: Upgrade daily/monthly aggregations of pageview dumps to new data files - https://phabricator.wikimedia.org/T90203#1867162 (ezachte) Open>Resolved Actually this was done already some two weeks ago as a subtask of https://phabricator.wikimedia.org/T114379 [22:08:22] Analytics: Upgrade daily/monthly aggregations of pageview dumps to new data files - https://phabricator.wikimedia.org/T90203#1867184 (ezachte) @Milimetrics FYI the dumps use http://dumps.wikimedia.org/other/pageviews/ What makes them still usefull is that they contain page views for all articles (with 5 or m... [22:19:57] Analytics-EventLogging, Editing-Department, MobileFrontend: Schema:MobileWebEditing: What are commons sorts of errors? - https://phabricator.wikimedia.org/T118366#1867237 (Jdlrobson) [23:10:01] (PS2) Milimetric: [WIP] Oozie-fy Country Breakdown Pageview Report [analytics/refinery] - https://gerrit.wikimedia.org/r/256355 [23:32:28] madhuvishy: yt? [23:32:35] nuria: yes [23:32:43] irccloud went away i while back [23:32:58] madhuvishy: for heap space errors in cluster like this one: return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Java heap space [23:33:12] do we increase teh heap just liek we do on the CLi [23:33:14] ? [23:33:18] *the [23:34:03] nuria: yeah, how are you running it? [23:34:10] hive -f [23:34:26] it's the offsets calculation [23:34:29] https://www.irccloud.com/pastebin/bvYGWah9/ [23:34:30] oh yeah, just export HADOOP_HEAPSIZE=1024 before running job [23:34:54] k, will try and let you know [23:35:01] thank youuu [23:35:52] madhuvishy: while this runs i am moving to look at wikimetrics tests [23:36:00] nuria: cooool! [23:38:06] madhuvishy: let me rebase your patch right? [23:38:15] https://gerrit.wikimedia.org/r/#/c/253750/ [23:38:29] yeah sure! [23:38:33] (PS18) Nuria: Setup celery task workflow to handle running reports for the ProgramMetrics API [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/253750 (https://phabricator.wikimedia.org/T118308) (owner: Madhuvishy)