[00:09:41] hey, quick question: are fundraising banners ever shown to people who are logged in automatically? [00:11:26] hi wctaiwan! I don't know really, sorry... I think you have more chances to get a satisfactory answer in the #wikimedia-fundraising channel [00:11:36] thanks. [00:21:14] bye a-team! [00:22:33] bye mforns, good night! [00:29:01] Analytics-Tech-community-metrics, Gerrit-Migration: Make MetricsGrimoire/korma support gathering Code Review statistics from Phabricator's Differential - https://phabricator.wikimedia.org/T118753#1843392 (greg) "Bitergia to check out the the API of Differential." Is that a commitment from them? If so, wha... [01:11:42] madhuvishy: milimetric: hi! quick question: just wanted to check on how our analyitics cluster is feeling this evening... No evidence of things falling over due to the banner history log onslaught? [01:12:48] Anecdotally, I've had a fairly simple EventLogging query running on stat1003 for several hours now. [01:13:02] I expected it to be done in 10 minutes. [01:13:21] I was just wondering if there was something up with the cluster. [01:25:47] madhuvishy: hmm, it actually doesn't seem to be anything wrong with the cluster. I tried a couple other queries and they work fine. But still, I've tried this query twice and it just won't complete. Let me troubleshoot a bit more. [01:36:53] madhuvishy: also the none of the eventlogging dbs have anything newer than 13:00 UTC or so today. So nothing within the last 12 hours. Is that normal replication lag? [01:39:22] neilpquinn: no not normal [01:39:45] yeah, I didn't think so :/ [01:40:05] AndyRussG: hmmm let me check. [01:43:19] https://grafana.wikimedia.org/dashboard/db/eventlogging [01:44:48] Ah heh we're only at 11% of total EL usage [01:44:51] Nice! [01:48:36] madhuvishy: thanks so much! [01:49:36] no problem! [02:05:17] madhuvishy: any thoughts about what might have caused the CentralNoticeBannerHistory eventlogging schema dip around 20:21 UTC today? [02:05:50] (CR) Nuria: Add UDF that turns a country code into a name (3 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/256354 (https://phabricator.wikimedia.org/T118323) (owner: Milimetric) [02:09:54] neilpquinn: I'm not sure, but there's data in hadoop upto the last hour [02:10:48] Analytics-Backlog, Wikimedia-Developer-Summit-2016: Developer summit session: Pageview API overview - https://phabricator.wikimedia.org/T112956#1843670 (Nuria) >@Akeron: yes, sorry, Erik's totally right here. That data has spiders filtered out, but doesn't have the breakdown as you requested it. Please... [02:11:42] AndyRussG: no, I can look at the error logs in logstash [02:12:17] I think it was something with CentralNotice, but it's funny 'cause there was no dip in donations around that time or anything [02:16:19] neilpquinn: think its just the lag [02:16:48] https://www.irccloud.com/pastebin/0CBpADwm/ [02:17:30] Well, the lag's not the main issue. It's just my query that won't finish for no reason I can understand :( [02:18:01] But don't worry too much about it today unless it seems to be bringing down the cluster :) it's not super super urgent. [02:18:33] neilpquinn: are you quryinh hadoop or db? [02:18:38] mysql [02:18:40] mysql [02:18:41] neilpquinn: teh db right? [02:18:46] yep [02:19:05] neilpquinn: do take a look at lag, there is a lot of it, which means that likely there is db mainatenance going on. [02:19:13] neilpquinn: https://tendril.wikimedia.org/host/ [02:19:15] I think some work is happening on the db [02:19:29] neilpquinn: if queries are real slow operation channel is a bette resource than us [02:19:45] nuria: okay, good to know. Thanks. [02:19:54] neilpquinn: we use the infrastructure but it is ops who sets it up and maintains it [02:20:20] Makes sense. I'll just leave it overnight and if it's still having trouble in the morning I'll check with ops. [02:20:21] neilpquinn: and maintenance normally impacts speed of queries [02:20:42] neilpquinn: (see number of bytes on graphs i just sent) [02:21:40] James_F: we would love to make vital signs not sad [02:22:14] nuria: I don't think I have access to that graph? [02:22:32] neilpquinn: you need to use your wikitech credentials [02:22:49] nuria: :-) [02:23:03] James_F: but until we compute that data with something other than labs db .. [02:23:16] nuria: Can't we throw more boxes at the problem? [02:23:47] * Deskana grabs a cardboard box [02:23:52] James_F: at labs? [02:23:57] Sure. [02:24:06] Divide and conquer. Inelegant but… [02:24:07] James_F: since you have all kinds of tools there accessing db [02:24:27] nuria: I tried that, but it still doesn't work. It might be I'm not in the nda group [02:24:33] James_F: I am not sure more boxes will fix it. [02:24:40] :-( [02:24:44] neilpquinn: you can request access to ops [02:25:20] James_F: More and more I think that data should be computed w/o teh db [02:25:35] James_F: since it is only counters for most of it [02:25:45] OK, but right now it's our only dashboard for most of this data. :-( [02:26:13] James_F: we will get there when we replace editing reports in stats.wikimedia.org [02:26:50] Are we doing that? When? [02:26:58] ("we". ;-)) [02:27:06] James_F: Do write a ticket, we are focusing on pageviews first and after editing metrics [02:27:26] s/pageviews/reading metrics [02:27:32] for wikistats [02:27:41] James_F: When we are done replacing pageview reports which is our current focus [02:27:42] nuria: how would you compute it if not using the db? [02:28:08] neilpquinn: with counters when events take place [02:28:16] ah, using eventbus? [02:28:17] neilpquinn: it does not work for every metric [02:28:25] neilpquinn: but for some it does [02:28:35] neilpquinn: OR SIMILAR, YES [02:28:38] sorry! [02:29:08] haha, no problem. I was wondering why you were suddenly foaming at the mouth :) [02:29:36] neilpquinn: my typing skills are pityful [02:30:27] neilpquinn: this is a reasonable overview of stream processing -- http://www.infoq.com/articles/stream-processing-hadoop [02:30:56] nuria: haha. So is it a mistake that you typed "teh db" twice? I assumed it was just the way cool people referred to it :) [02:31:12] neilpquinn: jajaa [02:31:40] bd808: thanks, I'll have to have a look [02:31:45] neilpquinn: leaving now, do get access to those graphs to be able to check lag [02:32:15] nuria: okay, will do [04:29:54] James_F|Away: / neilpquinn: throwing more boxes at labs won't solve this particular problem sadly. Those metrics, as defined, take orders of magnitude longer for large wikis than we can reasonably steal from that cluster. Yes, order*S* plural in some cases. So even if we double the cluster it wouldn't help much. The problem could be partially fixed if [04:29:54] the indices matched the ones on analytics-store. But we've been told that's not a good idea in the past by Sean. Revisiting that would be probably our best bet. The next best bet is to re-organize the data with Event Bus, but still store it in a DB. I measured the performance of such a re-organization and some queries run 1000 times faster. Yes, that [04:29:54] says *one thousand* :) hope that helps [09:28:39] * mforns tests [09:32:15] * elukey tries to read all the past day's correspondence of the channel [09:32:21] o/ [10:08:16] (PS1) Addshore: Switch IRC metric to use wm-bot [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/256390 (https://phabricator.wikimedia.org/T116005) [10:08:24] joal, yt? [10:08:44] (CR) Addshore: [C: 2 V: 2] Switch IRC metric to use wm-bot [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/256390 (https://phabricator.wikimedia.org/T116005) (owner: Addshore) [10:08:48] hi elukey :] [10:13:55] Analytics-Backlog, WMDE-Analytics-Engineering: Provide machine readable directory indexes on http://datasets.wikimedia.org/aggregate-datasets/ - https://phabricator.wikimedia.org/T117480#1844207 (Addshore) [10:40:46] (PS1) Addshore: Track EU distinct page count [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/256391 (https://phabricator.wikimedia.org/T120074) [10:41:26] (Draft2) Addshore: Remove old WDQS triple and lag metrics [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/256339 [10:41:33] (CR) Addshore: [C: 2 V: 2] Remove old WDQS triple and lag metrics [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/256339 (owner: Addshore) [10:41:47] (CR) Addshore: [C: 2 V: 2] Track EU distinct page count [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/256391 (https://phabricator.wikimedia.org/T120074) (owner: Addshore) [10:43:06] (PS1) Addshore: Add dbname for EU page count query [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/256392 [10:43:25] (CR) Addshore: [C: 2 V: 2] Add dbname for EU page count query [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/256392 (owner: Addshore) [10:45:09] (PS1) Addshore: fetchAll EU page query result [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/256393 [10:45:21] (CR) Addshore: [C: 2 V: 2] fetchAll EU page query result [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/256393 (owner: Addshore) [10:51:45] Analytics-Tech-community-metrics, DevRel-December-2015, Patch-For-Review: Review/update mailing list repositories in korma - https://phabricator.wikimedia.org/T116285#1844298 (Aklapper) I'll go for removing them in this case - no strong feelings... [11:21:01] Hey mforns :) [11:21:07] hi joal [11:21:14] How are you ? [11:21:24] good :] and you? [11:21:31] Good as well :) [11:22:01] I wanted to ask you one thing about our spark code [11:22:52] I got some difficulties in handling hierarchies in fields, the dimension code was getting complex... [11:23:15] yeah, I can imagine that [11:23:34] so I thought that we could initially transform the rows into maps with the useragent flattened [11:24:02] maybe you mentioned that before, but I'm not sure [11:24:26] the question is, do you think that is a performance drawback? [11:24:34] mforns: I think it's a godd idea, but maybe not to transform, but to add ? [11:24:53] Like a RDD[Row, flattened_stuf) [11:25:12] joal, aha I thought that also, but then we have the same problem when anonymizing no? [11:25:40] and anyway, when anonymizing, we have to create another row, we can not update the existing one.... [11:25:42] concern with transformation is: how do you not change the things ? [11:25:58] batcave? [11:26:01] sure [12:10:35] (PS1) Addshore: Add script tracking wikidata open tasks [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/256402 (https://phabricator.wikimedia.org/T119183) [12:10:44] (CR) Addshore: [C: 2 V: 2] Add script tracking wikidata open tasks [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/256402 (https://phabricator.wikimedia.org/T119183) (owner: Addshore) [13:13:27] Analytics-Tech-community-metrics, DevRel-December-2015: OwlBot seems to merge random user accounts in korma user data - https://phabricator.wikimedia.org/T119755#1844570 (Aklapper) [13:13:29] Analytics-Tech-community-metrics, Developer-Relations, DevRel-December-2015: Check whether it is true that we have lost 40% of (Git) code contributors in the past 12 months - https://phabricator.wikimedia.org/T103292#1844569 (Aklapper) [13:47:15] Analytics-Tech-community-metrics, Developer-Relations, DevRel-December-2015: Check whether it is true that we have lost 40% of (Git) code contributors in the past 12 months - https://phabricator.wikimedia.org/T103292#1844585 (Aklapper) On http://korma.wmflabs.org/browser/scm-contributors.html I don't s... [13:51:35] Analytics-Tech-community-metrics, Developer-Relations, DevRel-December-2015: Check whether it is true that we have lost 40% of (Git) code contributors in the past 12 months - https://phabricator.wikimedia.org/T103292#1844591 (Aklapper) >>! In T103292#1386502 on June 22, 2015, @Qgil wrote: > January 20... [14:23:12] Analytics-Backlog, Wikimedia-Developer-Summit-2016: Developer summit session: Pageview API overview - https://phabricator.wikimedia.org/T112956#1844637 (Milimetric) >>! In T112956#1843670, @Nuria wrote: >>@Akeron: yes, sorry, Erik's totally right here. That data has spiders filtered out, but doesn't ha... [14:40:44] Analytics-Tech-community-metrics, Developer-Relations, DevRel-December-2015: Check whether it is true that we have lost 40% of (Git) code contributors in the past 12 months - https://phabricator.wikimedia.org/T103292#1844687 (Qgil) January 2014 went from 414 to 303. May 2014 went from 382 to 270. OK.... [14:51:07] Analytics-Backlog, Wikimedia-Developer-Summit-2016: Developer summit session: Pageview API overview - https://phabricator.wikimedia.org/T112956#1844715 (Akeron) > @Akeron: yes, sorry, Erik's totally right here. That data has spiders filtered out, but doesn't have the breakdown as you requested it. Goo... [15:05:57] Analytics-Cluster: Kafka partition leader elections causing a drop of a few log lines - https://phabricator.wikimedia.org/T72087#1844766 (Ottomata) [15:05:58] Analytics-General-or-Unknown, Patch-For-Review: Kafka broker analytics1021 not receiving messages every now and then - https://phabricator.wikimedia.org/T71667#1844763 (Ottomata) Open>Resolved a:Ottomata analytics1021 is gone, and we haven't had problems with this since! Closing. [16:12:37] Analytics, Deployment-Systems, Services, operations, Scap3: Use Scap3 for deploying AQS - https://phabricator.wikimedia.org/T114999#1844955 (greg) [16:12:48] Analytics, Deployment-Systems, Services, operations, Scap3: Deploy AQS with scap3 - https://phabricator.wikimedia.org/T114999#1844957 (greg) [16:13:17] Analytics, Deployment-Systems, Services, operations, Scap3: Deploy AQS with scap3 - https://phabricator.wikimedia.org/T114999#1712059 (greg) [16:14:27] Analytics, Deployment-Systems, Services, operations, Scap3: Deploy AQS with scap3 - https://phabricator.wikimedia.org/T114999#1712059 (greg) (Off-topic-ish: There's no AQS project in Phab? Do all development tasks for it live in.... analytics? services? where?) [16:20:27] Analytics-Kanban: Remove client_ip computation from refine - https://phabricator.wikimedia.org/T120105#1844964 (Nuria) NEW a:JAllemandou [16:30:59] Analytics, Deployment-Systems, Services, operations, Scap3: Deploy AQS with scap3 - https://phabricator.wikimedia.org/T114999#1845008 (mobrovac) >>! In T114999#1844957, @greg wrote: > (Off-topic-ish: There's no AQS project in Phab? Do all development tasks for it live in.... analytics? services?... [16:33:49] Analytics, Deployment-Systems, Services, operations, Scap3: Deploy AQS with scap3 - https://phabricator.wikimedia.org/T114999#1845028 (Nuria) @mobrovac: Shouldn't we create a AQS project if you expect to have other users of the framework besides analytics team? [16:34:03] Analytics, Analytics-Backlog, Deployment-Systems, Services, and 2 others: Deploy AQS with scap3 - https://phabricator.wikimedia.org/T114999#1845029 (Nuria) [16:36:03] Analytics, Analytics-Backlog, Deployment-Systems, Services, and 2 others: Deploy AQS with scap3 - https://phabricator.wikimedia.org/T114999#1845045 (mobrovac) >>! In T114999#1845028, @Nuria wrote: > @mobrovac: Shouldn't we create a AQS project if you expect to have other users of the framework besi... [16:36:57] ....channel is so quiet.... [16:43:06] joal: I'm seeing something really weird [16:44:05] Analytics-Tech-community-metrics, DevRel-December-2015: What is contributors.html for, in contrast to who_contributes_code.html and sc[m,r]-contributors.html and top-contributors.html? - https://phabricator.wikimedia.org/T118522#1845064 (Aklapper) I'm tempted to kill who_contributes_code and contributors.... [16:44:15] my UDF unit tests agree that if I pass "--" I get back "Unknown". But when I run it through Hive on stat1002, passing in "--" results in "--" coming back out. [16:45:32] nuria: o/ (Luca) [16:45:55] (everything else works as normal) [16:47:05] milimetric: cause your unit tests are running against a "fake" maxmind db right? [16:48:21] I think IRC is borked [16:48:39] milimetric: if you see tests for other geo coded udfs you see stuff like " assertEquals("ISO country code check", "--", result.toString());" for invalid ips [16:51:05] milimetric: see on Geocode.java: [16:51:06] https://www.irccloud.com/pastebin/U8VIiNwx/ [16:53:28] milimetric: see Geocode.java [16:53:36] milimetric: [16:53:54] https://www.irccloud.com/pastebin/nmjBYqW9/ [16:53:57] elukey: hola! [16:54:43] milimetric: Hi! [16:56:34] milimetric: Hi! [16:56:46] uhhh [16:56:47] sup with irc [17:01:50] * mforns tests [17:03:42] * madhuvishy too [17:05:44] Analytics, Analytics-Backlog, Deployment-Systems, Services, and 2 others: Deploy AQS with scap3 - https://phabricator.wikimedia.org/T114999#1845141 (JAllemandou) >>! In T114999#1845008, @mobrovac wrote: >>>! In T114999#1844957, @greg wrote: >> (Off-topic-ish: There's no AQS project in Phab? Do all... [17:36:53] (PS2) Milimetric: Add UDF that turns a country code into a name [analytics/refinery/source] - https://gerrit.wikimedia.org/r/256354 (https://phabricator.wikimedia.org/T118323) [17:44:16] any opionions, should popularity_score table in hive go into the wmf database, or create a new discovery database? Most likely we will also be adding a page rank table in the future. maybe others, unknown at this time [17:45:10] ebernhardson: hmmMmMMM [17:45:22] popularity score comes from pageview? [17:45:27] ottomata: yes [17:45:31] HmMmmM [17:45:33] page rank will come from completely external data though [17:45:41] from elasticsearch? [17:45:52] or mysql, but probably elasticsearch for simplicity [17:45:55] hmmm [17:45:59] it requires a graph of all the links in the pedia's [17:46:26] both of those sound really useful for more than just discovery, but if you put them in wmf database, you will probably have to be more diligent in maintaining and documenting them than if you would keep them to yourselves [17:46:39] we'll help of course :) [17:46:44] hm. [17:46:53] but this is what you were asking about oozie and hive code, right? [17:46:55] where that should live? [17:47:27] well, it's related :) since i was starting a new repo and plugging stuff into it thats why i thought to start using a new database name too, so there is a relationship of on repository = one database [17:47:37] or at least, one database isn't filled from multiple repositories [17:48:03] ja so, i think here's a good rule of thumb: [17:48:04] but...there is no great reason for that other than simplifying the mental model of how to find things [17:48:34] if you are generating data for more than just your team, and you want data to exist beyond whatever project you are currently doing for general purpose use, then use refinery and wmf database [17:48:37] otherwise, do your own thing [17:48:39] eh? [17:49:07] i just meant its nice if you can think 'this is the wmf database, so it must come from the refinery repo' [17:49:18] but not required [17:49:24] yeah, i don't want to make that a rule, but i would lean towards that in general [17:49:45] but, if you are going to use wmf db & refinery, then the data should be a higher class citizen than just something one team maintains for themselves [17:50:03] but i don't want to force teams into doing that extra work [17:50:09] dunno [17:50:09] :) [17:50:10] :) [17:50:16] you can start separate and then join later if it looks awesome? [17:50:25] dunno [17:50:28] yea that makes sense [17:50:32] i mean, both of those datasets sound juicy though :) [17:54:23] nuria: cave for am inunte ? [17:54:39] nuria: You were gone before I could ask :) [17:56:27] (PS3) Milimetric: Add UDF that turns a country code into a name [analytics/refinery/source] - https://gerrit.wikimedia.org/r/256354 (https://phabricator.wikimedia.org/T118323) [17:58:19] (CR) Milimetric: Add UDF that turns a country code into a name (3 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/256354 (https://phabricator.wikimedia.org/T118323) (owner: Milimetric) [18:00:25] nuria: joal: oh! irc is back [18:00:38] lol, ok, it worked, I built it two more times for good measure and scp-ed and it was fine [18:01:11] feel free to code review ^ /me is gone to the doctor's now [18:01:22] nuria: can you fix that patch? i'd like to merge that and then rebase what I'm working on before I submit it [18:02:45] kevinator: you there ? [18:02:51] eys [18:02:53] yes [18:02:58] cool :) cave for a minute ? [18:03:03] yup [18:03:10] batcave? [18:03:17] yes please :) [18:12:12] milimetric: are you away 1-4 Eastern or Pacific time? [18:12:12] milimetric: are you away 1-4 pst or eastern? [18:12:42] madhuvishy: eastern, but i might be back a lot sooner, just blocked it off in case [18:12:43] what's up? [18:12:54] Analytics-Backlog: Use a new approach to compute monthly top 1000 articles (brute force in hive doesn't work) - https://phabricator.wikimedia.org/T120113#1845294 (JAllemandou) NEW [18:12:57] oh! I'll miss scrum of scrums, a-team, so if anyone wants to go instead of me you're welcome to [18:12:59] oh so now? [18:13:25] at 10:30 [18:13:30] yep, just about to run out the door. You can text me if you need me [18:13:40] o/ [18:13:40] milimetric: just wanna chat about wikimetrics, ping when back :) [18:13:41] it's in the engineering calendar [18:16:22] Analytics-Cluster, Analytics-Kanban, Easy: Update client IP in webrequest table to use IP [5 pts] {hawk} - https://phabricator.wikimedia.org/T116772#1845319 (JAllemandou) a:Ottomata>JAllemandou [18:19:35] Analytics-Cluster, Analytics-Kanban, Easy: Update client IP in webrequest table to use IP [5 pts] {hawk} - https://phabricator.wikimedia.org/T116772#1845331 (JAllemandou) Tested wiht hive: Over a few hours, there are only a very small amount of requests on which the client IP sent by varnish differs fr... [18:20:44] Analytics-Backlog: Change the Pageview API's RESTBase docs for the top endpoint - https://phabricator.wikimedia.org/T120019#1845340 (kevinator) @mforns, we should remove references to all-days and all-months entirely. The top articles for a month is not available because there is no optimal way to calculate... [18:21:50] (PS1) Joal: Correct CirrusSearchRequestSet table creation hql [analytics/refinery] - https://gerrit.wikimedia.org/r/256466 [18:22:26] Analytics-Backlog, Beta-Cluster-Infrastructure, Services, Scap3, WorkType-NewFunctionality: Set up AQS in Beta - https://phabricator.wikimedia.org/T116206#1845351 (thcipriani) This morning @mobrovac and I setup `deployment-aqs01` on beta. Deployed via Scap3 successfully and without issue. The AQ... [18:22:46] (PS2) Joal: Remove client_ip computation from refine [analytics/refinery] - https://gerrit.wikimedia.org/r/256027 (https://phabricator.wikimedia.org/T116772) [18:24:40] (CR) DCausse: [C: 1] Correct CirrusSearchRequestSet table creation hql [analytics/refinery] - https://gerrit.wikimedia.org/r/256466 (owner: Joal) [18:25:32] (CR) Joal: "Nuria, I had done that: https://gerrit.wikimedia.org/r/#/c/256024/ but we agreed with Andrew that this code is mostly valid, so shouldn't " [analytics/refinery] - https://gerrit.wikimedia.org/r/256027 (https://phabricator.wikimedia.org/T116772) (owner: Joal) [18:26:19] (CR) Joal: [C: 1] "Looks good to me ! Nuria for the merge :)" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/256354 (https://phabricator.wikimedia.org/T118323) (owner: Milimetric) [18:46:17] Analytics-Kanban: Add LRUCache for user agent parsing in refinery [3 pts] {hawk} - https://phabricator.wikimedia.org/T120015#1845432 (Nuria) Open>Resolved [18:46:48] Analytics-Kanban, Patch-For-Review: Add page_id to pageview_hourly when present in webrequest x_analytics header - https://phabricator.wikimedia.org/T116023#1845435 (Nuria) Open>Resolved [18:47:03] Analytics-Kanban, Patch-For-Review: Add avro schema to refinery-camus jar [3 pts] {hawk} - https://phabricator.wikimedia.org/T117885#1845437 (Nuria) Open>Resolved [18:47:34] Analytics-Kanban: Encapsulating the retrieval of schemas from local depot from KafkaSchemaRegistry [3 pts] - https://phabricator.wikimedia.org/T119211#1845438 (Nuria) Open>Resolved [18:49:35] Analytics-Kanban, Patch-For-Review: Investigate / Correct timestamp and data not being properly read from Camus for CirrusSearchRequestSet logs. - https://phabricator.wikimedia.org/T117873#1845444 (Nuria) Open>Resolved [18:50:09] Analytics-Kanban, CirrusSearch, Discovery, Discovery-Cirrus-Sprint, Patch-For-Review: Setup oozie task for adding and removing CirrusSearchRequestSet partitions in hive - https://phabricator.wikimedia.org/T117575#1845445 (Nuria) Open>Resolved [18:50:31] Analytics-Kanban, Patch-For-Review: Wikimedia Analytics Refinery Jobs TestCamusPartitionChecker test failure when running as bd808 on stat1002 {hawk} - https://phabricator.wikimedia.org/T119101#1845446 (Nuria) Open>Resolved [18:50:46] Analytics-Kanban: Investigate cassandra daily top job [5 pts] {slug} - https://phabricator.wikimedia.org/T118449#1845447 (Nuria) Open>Resolved [18:51:52] Analytics-Kanban, CirrusSearch, Discovery, operations, and 2 others: Delete logs on stat1002 in /a/mw-log/archive that are more than 90 days old {hawk} - https://phabricator.wikimedia.org/T118527#1845459 (Nuria) Open>Resolved [18:52:07] Analytics-Kanban, Datasets-General-or-Unknown: Wikimedia "top" pageviews API has problematic double-encoded JSON [8 pts] {melc} - https://phabricator.wikimedia.org/T118931#1845461 (Nuria) Open>Resolved [18:52:24] Analytics-Kanban, Patch-For-Review: Exclude MobileMenu from Pageviews - https://phabricator.wikimedia.org/T117345#1845462 (Nuria) Open>Resolved [18:52:55] cool, alert looking good nuria [18:52:55] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=graphite1001&service=Overall+insertion+rate+from+MySQL+consumer [18:55:53] moving to cafe... [19:03:55] when the oozie docs say `YEAR=org.apache.oozie.coord.CoordELConstants#SUBMIT_YEAR` thats saying the $YEAR variable is defined by when it's running, and not the timeframe the job is running for? [19:20:12] ebernhardson: me no compredou [19:20:32] ebernhardson: that * ahem* I do not get your question [19:21:21] nuria: in oozie there are variables ${YEAR} ${MONTH} ${DAY}, i'm trying to figure out what they refer to exactly [19:21:35] ebernhardson: partition you are running your job on [19:21:37] i'm mostly unsure because in the pageview_hourly you explicitly pass around your own ${year} ${month} ${day} vairables [19:21:51] instead of using the built in ones, so i'm trying to figure out if i need to do that too [19:23:35] nuria: ok that would imply they are the same then. i can probably simplify in this new one and just use ${YEAR}, etc. then. Or at least i'll try it until I run into issues :) [19:24:05] ebernhardson: i am not sure i understand [19:24:15] ebernhardson: but i guess you let us know if things are not working [19:24:37] ebernhardson: those time variable are defined in corrdinator IIRC [19:25:15] nuria: i mean oozie has built in ${YEAR}, but in pageview/hourly/coordinator.xml a new ${day} vairables is defined as ${coord:formatTime(coord:nominalTime(), "d") [19:25:24] err, "y" [19:25:52] * ebernhardson should learn to type better...trying again [19:26:06] nuria: i mean oozie has built in ${YEAR}, but in pageview/hourly/coordinator.xml a new ${year} vairable is defined as ${coord:formatTime(coord:nominalTime(), "y") [19:27:05] i just wasn't sure why the duplication so started looking through docs, and i still can't figure out why. but it's probably not a big deal [19:28:00] (mostly i'm basing my new job off the existing configuration of pageview_hourly to have a starting off point) [19:29:13] ebernhardson: I didn't know about the default time value in oozie (did the same as you, started copying existing jobs [19:30:05] ebernhardson: Looks like you could use the default ones provided by ozie :) [19:30:31] nuria: unless you are about to submit something, mind if i go ahead and fix up that batch change? [19:30:53] joal: ok, makes sense. thanks! oozie is turning out to be rather complicated and i'm trying to simplify because my head isn't big enough for all these properties :) [19:31:44] I did that a bit too --> Before, every property was explicitely passed from bundle to coord to workflow - Now we use implicits :) [19:32:04] ebernhardson: --^ And I didn't know about the time variables, will definitely try them :) [19:32:17] i only happened to run into them because pageview/datasets.xml uses them :) [19:32:53] but pageview/datasets.xml does something else weird about double parsing...i don't understand how that happens either yet but for now leaving it as an almost direct copy with s/pageview_hourly/popularity_score/ [19:33:41] ebernhardson: Double parsing is to ensure leading 0 would be removed [19:33:51] if any [19:34:33] By the way ebernhardson, before I leave: I'm fighting a bit with avro compression, but I think I'll manage to save your november data :) [19:34:33] oh ok [19:35:04] joal: if it's too much work it's really not a big deal. our analysts are already clammoring to add more data to it and that the existing data, while nifty, doesn't have enough for them [19:35:09] ebernhardson: But since those variables are defined by oozie, I expect they don't have 0 ? Don't know though ... [19:35:18] (they want us to record query features, as in which code paths were hit) [19:35:19] :d [19:35:21] :D [19:35:42] I know the thing: analysts view: Why not more data? [19:36:03] I'll do it nontheless ebernhardson, interesting learning on avro :) [19:36:27] (better to do it now while it's not needed and know how to do it next time when it'll be important :) [19:36:31] ebernhardson: --^ [19:36:47] Guys, I'm off for today ! A-team, see you tomorrow :) [19:37:10] latesr! [19:37:53] Analytics-Kanban: Backfill daily-top-articles in cassandra [2015-09-01 - 2015-11-16 (included)] [5 pts] {melc} - https://phabricator.wikimedia.org/T118991#1845619 (Nuria) Open>Resolved [19:38:11] nuria: fyi, I am updating the batch patch :) [19:38:11] :) [19:38:14] Analytics-Kanban: Backfill cassandra pageview data - August [5 pts] {slug} - https://phabricator.wikimedia.org/T118845#1845621 (Nuria) Open>Resolved [19:38:46] i dunno if its related, but stas ran into a segfault related to compression yesterday in pyspark reading parquet files [19:39:15] segfaulted inside # C [libz.so.1+0x8e6d] inflate+0x41d [19:39:25] awesome! [19:39:31] libz? [19:39:39] wonder why its using that... [19:39:47] what are you compressing with? not snappy? [19:40:02] i didn't set any variables for what to compress with when writing the data he's reading [19:40:14] so, whatever is default [19:40:19] default should be snappy, lemme look, what's the path? [19:40:38] /user/ebernhardson/popularity_score_test/year=2015/month=11/day=11/ [19:40:48] hive reads it just fine though [19:42:31] hm, i am not seeing any compression headers on those files, so i guess they aren't compressed. [19:42:36] maybe parquet does some internal compression? [19:42:40] for columns? [19:42:41] dunno [19:43:32] also, kinda related, spark ended up outputing 200 3MB files for each of the internal partitions it used, should i adjust that so it makes bigger files? [19:43:42] does it matter? [19:44:32] ottomata: looks like it was a one-off error that kinda resolved itself. weird [19:46:35] hm ok [19:46:44] ebernhardson: yes, you should [19:47:20] you want files bigger than 256 as that is the default block size [19:47:22] 256M [19:47:25] ok [19:50:49] hmm, nuria somethign went weird with that merge, dunno what [19:50:53] i see that commit in the history twice now... [19:50:59] plus a merge commit :/ [19:53:32] hmm, maybe its just me... [19:53:58] (CR) Nuria: "Thanks for doing changes, 1 small nit." (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/254452 (owner: EBernhardson) [19:56:15] never mind, its fine [19:59:20] (CR) Nuria: [C: 2] Correct CirrusSearchRequestSet table creation hql [analytics/refinery] - https://gerrit.wikimedia.org/r/256466 (owner: Joal) [20:00:01] (CR) Nuria: [V: 2] Correct CirrusSearchRequestSet table creation hql [analytics/refinery] - https://gerrit.wikimedia.org/r/256466 (owner: Joal) [20:29:10] (CR) Nuria: "Right, saw that after *ahem* this change. Let's merge this patch and use the other one to remove the udf code usage of iputil." [analytics/refinery] - https://gerrit.wikimedia.org/r/256027 (https://phabricator.wikimedia.org/T116772) (owner: Joal) [20:29:42] (CR) Nuria: [C: 2] Remove client_ip computation from refine [analytics/refinery] - https://gerrit.wikimedia.org/r/256027 (https://phabricator.wikimedia.org/T116772) (owner: Joal) [20:30:01] (CR) Nuria: [V: 2] Remove client_ip computation from refine [analytics/refinery] - https://gerrit.wikimedia.org/r/256027 (https://phabricator.wikimedia.org/T116772) (owner: Joal) [20:46:54] Analytics-Backlog, Research-and-Data: Historical analysis of edit productivity for English Wikipedia - https://phabricator.wikimedia.org/T99172#1845785 (Halfak) https://meta.wikimedia.org/wiki/Research_talk:Measuring_edit_productivity/Work_log/2015-12-02 [20:47:02] Analytics-Backlog, Research-and-Data: Historical analysis of edit productivity for English Wikipedia - https://phabricator.wikimedia.org/T99172#1845786 (Halfak) [20:49:50] (PS4) Nuria: Add UDF that turns a country code into a name [analytics/refinery/source] - https://gerrit.wikimedia.org/r/256354 (https://phabricator.wikimedia.org/T118323) (owner: Milimetric) [20:50:38] (CR) Nuria: "I have addressed these two comments on the patch I sent." (2 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/256354 (https://phabricator.wikimedia.org/T118323) (owner: Milimetric) [21:00:15] (PS5) Nuria: Add UDF that turns a country code into a name [analytics/refinery/source] - https://gerrit.wikimedia.org/r/256354 (https://phabricator.wikimedia.org/T118323) (owner: Milimetric) [21:02:56] ottomata: missed your pings due to irc being kaput [21:03:10] ottomata: sorry not to get to the patch sooner [21:03:19] nuria: np, we are good [21:03:30] i'm deploying the Event /schema_uri one in beta now [21:03:42] gonna make sure it works there, then ask for review [21:07:14] k [21:07:22] ottomata: i will be working on dan's udf patch [21:07:55] k [21:16:06] Analytics-Backlog, Datasets-Webstatscollector, Language-Engineering: Investigate anomalous views to pages with replacement characters - https://phabricator.wikimedia.org/T117945#1845878 (Nuria) p:Triage>Normal [21:18:34] nuria: that looks fine to me, so you can merge if you like [21:19:03] milimetric: no, i wanted to see if there were any issues so iam testing on 1002 [21:19:04] the null test is the same as joal's I think though, so I'll add a ticket to clean up / standardize the UDFs [21:19:21] nuria: ok, but the "Unknown" was showing up for me last time I deployed [21:19:22] milimetric: sounds good [21:19:32] nuria: latest EL service patch is deployed [21:19:41] sevice is running on deployment-eventlogging04 and working just fine [21:19:49] ottomata: and is the world still all happy [21:19:55] i've also installed and restarted eventlogging services on eventlogging03 [21:20:12] and am watching events be inserted into mysql, etc. [21:20:12] so all is looking good [21:20:17] ottomata: nice [21:20:55] do you think you'll have time to review today before you head out? [21:21:05] milimetric: so expected behaviour is that for anything "not good" we return "unknown" or "--"? [21:22:07] milimetric: sounds like "unknown" is what we want [21:22:42] ottomata: i wanted to look at logs and stuff on eventlogging03, will do that once i test a bit milimetric 's udf [21:22:51] k cool [21:23:12] Analytics-Backlog: Standardize logic, names, and null handling across UDFs in refinery-source {hawk} - https://phabricator.wikimedia.org/T120131#1845910 (Milimetric) NEW [21:23:16] no hurry for today, but I am anxious to merge this stuff to master and deploy to eventlog1001 [21:23:24] nuria: "Unknown" yes [21:23:51] so UNKNOWN_VALUE was correct in your change [21:24:33] milimetric: then [21:24:37] https://www.irccloud.com/pastebin/D7BOkqDj/ [21:24:45] nuria: yep, that's right [21:24:53] milimetric: ok, so it is working on hive [21:25:05] nuria: yeah, I sent an IRC message before I left for the doctor [21:25:23] it worked after the second time I built and deployed, so something must've not saved / copied properly the first time [21:25:25] milimetric: ah sorry, my irc *just* started working [21:25:38] milimetric: ok, there is 1 more thing to remove [21:25:52] milimetric: [21:25:53] oh ok, that makes sense. I sent an email to internal too, but I was afraid freenode was gonna cause problems today :/ [21:26:21] the args.length check on evaluate, other than that is good. great, one less thing to worry about [21:26:46] nuria: did you see my comment on those arg lengths? [21:26:53] milimetric: ah , no, wait [21:27:04] I was saying that the first arg length check was on the semantic analysis basically (in initialize) [21:27:21] and the second arg length check is technically different and should probably most likely not be needed [21:27:35] but just in case there's some weird problem, it feels a little safer to do it [21:28:02] milimetric: right, it is not needed as what the evaluate does is to execute 1 per row with the row values [21:28:15] milimetric: so args length will not change [21:28:42] milimetric: there is a bunch of code arround doing that check but I am pretty sure is not needed [21:28:51] but could in some weird case the argument be like something that generates an array or something weird like that? [21:30:15] milimetric: i do not think so cause the initialize is really generating java code like function some(sometype arg, sometype arg2) {} [21:30:38] and evaluate does "some(x, y)" [21:30:56] right, but say the raw data is something weird like one row has a string for country_code and another row for some reason has a map [21:30:58] more like some( (sometype) x, (sometype) y) [21:31:16] would that fail before it calls the UDF or pass a map on to the UDF? [21:31:22] milimetric: but column types prevent those two from being different types [21:31:38] and can't someone mess with column types like a file is corrupt or messed with manually? [21:32:03] and have different types for a column? i doubt it [21:32:14] I know in the normal flow nothing bad can happen if we only check on initialize, I'm just wondering if somehow we can have a weird state [21:32:36] ok, that's cool, let's remove it and if we run into trouble we can use this UDF to test for such things [21:32:48] nuria: you wanna do it or should I? [21:32:58] i just did it, was building, lemme push [21:33:41] milimetric: let me know when you have some time to chat [21:33:44] milimetric: well, let me package too [21:33:45] no hurry [21:34:02] k, cool nuria, thx! [21:34:06] madhuvishy: that means I'm free :) [21:34:09] batcave? [21:34:14] argh ottomata , i need to fill in feedback for an interview so i might not get to your CR today [21:34:46] milimetric: sure [21:35:20] (PS6) Nuria: Add UDF that turns a country code into a name [analytics/refinery/source] - https://gerrit.wikimedia.org/r/256354 (https://phabricator.wikimedia.org/T118323) (owner: Milimetric) [21:36:17] ok [21:37:59] (CR) Ottomata: [C: 1] "Yeah, I see no reason to remove those functions since they totally work fine." [analytics/refinery] - https://gerrit.wikimedia.org/r/256027 (https://phabricator.wikimedia.org/T116772) (owner: Joal) [21:41:09] madhuvishy: got dropped out of the cave, but I'm back [21:41:09] (PS7) Nuria: Add UDF that turns a country code into a name [analytics/refinery/source] - https://gerrit.wikimedia.org/r/256354 (https://phabricator.wikimedia.org/T118323) (owner: Milimetric) [21:41:20] oh me too [21:42:18] madhuvishy: i can't hear you in the hangout if you're talking... [21:42:20] or see video [21:44:16] (CR) Nuria: [C: 2 V: 2] "Merging per @joal 's message on patch #3" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/256354 (https://phabricator.wikimedia.org/T118323) (owner: Milimetric) [21:56:27] Analytics-Backlog, Beta-Cluster-Infrastructure, Services, Scap3, WorkType-NewFunctionality: Set up AQS in Beta - https://phabricator.wikimedia.org/T116206#1846067 (Milimetric) @mobrovac: those sound like things I can do? Let me know if you've started on them yet, and if not please assign this to... [23:11:16] Analytics-Tech-community-metrics, Gerrit-Migration: Make MetricsGrimoire/korma support gathering Code Review statistics from Phabricator's Differential - https://phabricator.wikimedia.org/T118753#1846331 (greg) >>! In T118753#1843392, @greg wrote: > "Bitergia to check out the the API of Differential." Is... [23:12:59] Analytics-Backlog, operations, HTTPS: EventLogging sees too few distinct client IPs - https://phabricator.wikimedia.org/T119144#1846335 (atgo) [23:13:10] Analytics-Kanban, Patch-For-Review: Bug: client IP is being hashed differently by the different parallel processors {stag} [13 pts] - https://phabricator.wikimedia.org/T112688#1846337 (atgo) [23:14:10] Analytics-Kanban, Patch-For-Review: Bug: client IP is being hashed differently by the different parallel processors {stag} [13 pts] - https://phabricator.wikimedia.org/T112688#1642642 (atgo)