[00:18:41] (PS1) QChris: Allow to mark day changes when dumping webrequest status [analytics/refinery] - https://gerrit.wikimedia.org/r/188003 [00:18:43] (PS1) QChris: For webrequest status dump script, update status meanings [analytics/refinery] - https://gerrit.wikimedia.org/r/188004 [00:18:45] (PS1) QChris: Prepare webrequest status dump script for daily datasets [analytics/refinery] - https://gerrit.wikimedia.org/r/188005 [00:18:47] (PS1) QChris: Show status of daily datasets, even if only part of day is dumped [analytics/refinery] - https://gerrit.wikimedia.org/r/188006 [00:18:49] (PS1) QChris: Add legacy_tsv dataset to webrequest status dump script [analytics/refinery] - https://gerrit.wikimedia.org/r/188007 [00:18:51] (PS1) QChris: Separate dataset status between checked "ok" and unchecked "ok" [analytics/refinery] - https://gerrit.wikimedia.org/r/188008 [00:22:55] (PS1) QChris: Grant more heap for maps of legacy_tsvs computation [analytics/refinery] - https://gerrit.wikimedia.org/r/188009 [00:22:57] (PS1) QChris: Geocode glam_nara files [analytics/refinery] - https://gerrit.wikimedia.org/r/188010 [00:25:04] (PS1) QChris: Fix casing of variable name in Geocoding [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188011 [00:25:06] (PS1) QChris: Fix overriding MaxMind database location in Geocoding [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188012 [00:25:08] (PS1) QChris: Fix NPE when Country Geocoding fails to detect country [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188013 [05:15:46] ezachte's zip is killing stat1002 (( [08:05:31] Analytics-Tech-community-metrics: Consolidating time ranges across tech community metrics - https://phabricator.wikimedia.org/T86630#1007600 (Qgil) a:Qgil [08:09:12] Analytics-Tech-community-metrics, Wikimedia-Git-or-Gerrit: Active code review users on a monthly basis - https://phabricator.wikimedia.org/T86152#1007633 (Qgil) [08:09:14] Analytics-Tech-community-metrics, Wikimedia-Git-or-Gerrit: Basic metrics about contributors exercising +2/-2 permissions in Gerrit - https://phabricator.wikimedia.org/T59038#1007635 (Qgil) [08:09:15] Analytics-Tech-community-metrics: Remove the filter for key Wikimedia software projects in korma.wmflabs.org - https://phabricator.wikimedia.org/T86154#1007637 (Qgil) [08:18:40] Analytics-Tech-community-metrics: Key performance indicator: Gerrit review queue - https://phabricator.wikimedia.org/T39463#1007699 (Qgil) [08:56:17] Project-Creators, Phabricator, Engineering-Community: Analytics-Volunteering and Wikidata's Need-Volunter tags - https://phabricator.wikimedia.org/T88266#1007773 (Qgil) NEW [09:03:33] Engineering-Community, Analytics: Metrics about the Wikimedia APIs usage - https://phabricator.wikimedia.org/T88267#1007786 (Qgil) NEW [10:30:42] Phabricator, Engineering-Community, Analytics-Tech-community-metrics: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#1007960 (Nemo_bis) Trying to get some comparable numbers pre and post-migration. It seems that the sum of MediaWiki (including VisualEditor and Mob... [10:41:12] (PS2) QChris: Add legacy_tsv dataset to webrequest status dump script [analytics/refinery] - https://gerrit.wikimedia.org/r/188007 [10:41:14] (PS2) QChris: Separate dataset status between checked "ok" and unchecked "ok" [analytics/refinery] - https://gerrit.wikimedia.org/r/188008 [11:56:34] Analytics-Tech-community-metrics: Key performance indicator: analyze who contributes code - https://phabricator.wikimedia.org/T55485#1008068 (Qgil) a:Qgil>None [12:03:49] Analytics-Tech-community-metrics: Instructions to update user data in korma - https://phabricator.wikimedia.org/T88277#1008079 (Qgil) NEW [12:04:56] Analytics-Tech-community-metrics: "Contributors new and gone" in korma is stalled - https://phabricator.wikimedia.org/T88278#1008088 (Qgil) NEW [12:11:53] (CR) Ananthrk: [C: 1] Fix overriding MaxMind database location in Geocoding [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188012 (owner: QChris) [12:12:23] (CR) Ananthrk: [C: 1] Fix NPE when Country Geocoding fails to detect country [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188013 (owner: QChris) [12:13:12] (CR) Ananthrk: [C: 1] Fix casing of variable name in Geocoding [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188011 (owner: QChris) [12:16:43] (CR) Ananthrk: "Not relevant thanks to subsequent patches by Chris." [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187906 (owner: QChris) [12:43:59] ananthrk: hey! [12:44:04] heard you are in Chennai [12:44:28] yuvipanda: yup. very much [12:44:57] ananthrk: I'm in Bangalore atm, might be in Chennai for a bit next month [12:45:36] yuvipanda: Oh..let me know when you are here. We can catch up in person [12:46:45] ananthrk: sure [12:46:54] ananthrk: madhuvishy is also in Chennai, is joining the team shortly [12:48:00] yuvipanda: is it? didn't know that... [12:48:14] ananthrk: yeah. recent thing :) [13:18:11] Analytics-Tech-community-metrics, MediaWiki-Developer-Summit-2015: Achievements, lessons learned, and data related with the MediaWiki Developer Summit 2015 - https://phabricator.wikimedia.org/T87514#1008162 (Qgil) [13:58:35] ooooh, a milimetric! [13:58:47] hi yuvi :) [14:05:17] milimetric: this graduate labs thing... what is it? :) [14:05:22] milimetric: also was really nice meeting you in SF :) [14:06:07] sorry, 1/1 with aaron now, but will write back after [14:06:17] milimetric: sure! [14:10:58] Analytics-Engineering, Analytics-Cluster: PageView reports by hive-webstatscollector should return undefined values when data is not available - https://phabricator.wikimedia.org/T76406#1008208 (QChris) Open>Resolved Since the above merge, the hive webstatscollector data distinguishes between 0 and n/a.... [15:26:07] Analytics-Engineering: Data Warehouse manages schema migrations with alembic - https://phabricator.wikimedia.org/T76829#1008324 (Milimetric) Open>Resolved [17:13:59] nuria, milimetric, you wanna get into batcave and we chat about sampling? [17:14:16] I gave it some thought, asked halfak's input, and I have some directions for you. [17:14:47] ok, one second [17:15:12] sure [17:43:47] Analytics-EventLogging: Change timestamp fields to reduce DB storage size [8 pts] - https://phabricator.wikimedia.org/T87660#1008559 (ggellerman) [17:45:32] Analytics-Kanban, Analytics-EventLogging: Change timestamp fields to reduce DB storage size [8 pts] - https://phabricator.wikimedia.org/T87660#1008562 (kevinator) [17:52:27] Analytics-Kanban, Analytics-EventLogging: Change timestamp fields to reduce DB storage size [5 pts] - https://phabricator.wikimedia.org/T87660#1008574 (ggellerman) [17:52:55] Analytics-Kanban, Analytics-EventLogging: Change timestamp fields to reduce DB storage size [8 pts] - https://phabricator.wikimedia.org/T87660#996303 (ggellerman) [17:53:12] Analytics-EventLogging: Remove autoincrement id from tables [5 pts] - https://phabricator.wikimedia.org/T87661#1008576 (ggellerman) [17:57:00] Analytics-Kanban, Analytics-EventLogging: Sanity check changes to timestamp fields and remove autoincrement id from tables & deploy to Prod [8 pts] - https://phabricator.wikimedia.org/T88297#1008583 (ggellerman) NEW [17:59:22] Analytics-Kanban, Analytics-EventLogging: Change timestamp fields to reduce DB storage size [8 pts] - https://phabricator.wikimedia.org/T87660#1008592 (ggellerman) [17:59:23] Analytics-Kanban, Analytics-EventLogging: Sanity check changes to timestamp fields and remove autoincrement id from tables & deploy to Prod [8 pts] - https://phabricator.wikimedia.org/T88297#1008593 (ggellerman) [18:00:38] Analytics-EventLogging: Remove autoincrement id from tables [5 pts] - https://phabricator.wikimedia.org/T87661#1008595 (ggellerman) [18:00:39] Analytics-Kanban, Analytics-EventLogging: Sanity check changes to timestamp fields and remove autoincrement id from tables & deploy to Prod [8 pts] - https://phabricator.wikimedia.org/T88297#1008583 (ggellerman) [18:01:20] Analytics-EventLogging: Remove autoincrement id from tables [5 pts] - https://phabricator.wikimedia.org/T87661#996313 (ggellerman) [18:01:21] Analytics-Kanban, Analytics-EventLogging: Sanity check changes to timestamp fields and remove autoincrement id from tables & deploy to Prod [8 pts] - https://phabricator.wikimedia.org/T88297#1008583 (ggellerman) [18:01:41] Analytics-Kanban, Analytics-EventLogging: Sanity check changes to timestamp fields and remove autoincrement id from tables & deploy to Prod [8 pts] - https://phabricator.wikimedia.org/T88297#1008615 (ggellerman) [18:01:42] Analytics-Kanban, Analytics-EventLogging: Change timestamp fields to reduce DB storage size [8 pts] - https://phabricator.wikimedia.org/T87660#1008614 (ggellerman) [18:03:13] are there stats for skin usage? [18:18:58] nuria: whenever you have some time can you communicate with kaldari the problem we observed on Friday with the userToken and taskToken. [18:19:08] he's up for looking into it to see what's going on [18:20:44] (PS1) Ottomata: Add Maxmind CC BY-SA:3.0 attribution in README.md [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188085 (https://phabricator.wikimedia.org/T88252) [18:22:48] (CR) OliverKeyes: "Are we including the free MMDB files?" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188085 (https://phabricator.wikimedia.org/T88252) (owner: Ottomata) [18:28:22] (CR) Ottomata: "They are committed to the refinery/source repository and used for unit tests." [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188085 (https://phabricator.wikimedia.org/T88252) (owner: Ottomata) [18:29:31] leila: this is the bug: https://phabricator.wikimedia.org/T78449 [18:29:56] leila: i do not see him on #mobile [18:31:25] mforns: let me know if you run into issues with DW stuff [18:31:36] nuria, sure [18:31:54] nuria, I'm reading the docs carefully [18:32:18] leila: i have submitted a patch that will 'mitigate' the problem , but note, i would also look at the way they are instrumenting the code [18:33:32] (CR) Ottomata: [C: 2 V: 2] Allow to mark day changes when dumping webrequest status [analytics/refinery] - https://gerrit.wikimedia.org/r/188003 (owner: QChris) [18:34:13] (CR) Ottomata: [C: 2 V: 2] For webrequest status dump script, update status meanings [analytics/refinery] - https://gerrit.wikimedia.org/r/188004 (owner: QChris) [18:35:12] (CR) Ottomata: [C: 2 V: 2] Prepare webrequest status dump script for daily datasets [analytics/refinery] - https://gerrit.wikimedia.org/r/188005 (owner: QChris) [18:35:30] (CR) Ottomata: [C: 2 V: 2] Show status of daily datasets, even if only part of day is dumped [analytics/refinery] - https://gerrit.wikimedia.org/r/188006 (owner: QChris) [18:36:33] nuria: Hello! [18:36:46] (PS2) Ottomata: Grant more heap for maps of legacy_tsvs computation [analytics/refinery] - https://gerrit.wikimedia.org/r/188009 (owner: QChris) [18:37:02] (CR) Ottomata: [C: 2 V: 2] Grant more heap for maps of legacy_tsvs computation [analytics/refinery] - https://gerrit.wikimedia.org/r/188009 (owner: QChris) [18:37:08] (PS2) Ottomata: Geocode glam_nara files [analytics/refinery] - https://gerrit.wikimedia.org/r/188010 (owner: QChris) [18:41:33] (PS3) Ottomata: Add legacy_tsv dataset to webrequest status dump script [analytics/refinery] - https://gerrit.wikimedia.org/r/188007 (owner: QChris) [18:41:59] nuria: You asked me to speak to you? [18:42:00] (CR) Ottomata: [C: 2 V: 2] Add legacy_tsv dataset to webrequest status dump script [analytics/refinery] - https://gerrit.wikimedia.org/r/188007 (owner: QChris) [18:42:04] (PS3) Ottomata: Separate dataset status between checked "ok" and unchecked "ok" [analytics/refinery] - https://gerrit.wikimedia.org/r/188008 (owner: QChris) [18:42:31] (CR) Ottomata: [C: 2 V: 2] Separate dataset status between checked "ok" and unchecked "ok" [analytics/refinery] - https://gerrit.wikimedia.org/r/188008 (owner: QChris) [18:43:28] (CR) OliverKeyes: [C: 1] "Gotcha!" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188085 (https://phabricator.wikimedia.org/T88252) (owner: Ottomata) [18:43:49] Deskana: yes, so we are on the same page. I was about to create a phabricator item to get you guys a monthly report but note, the 1st data was a rough approximation [18:44:11] Deskana: thus data - from the refined datasets- will differ, it could differ as much as 20% [18:45:09] Deskana: item here: https://phabricator.wikimedia.org/T88308 [18:50:27] (CR) Ottomata: "I don't think we should tie this refinery-core Geocode class to Hadoop Job Confs. Especially not mapred.JobConf, as this is the legacy Ma" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188012 (owner: QChris) [18:51:09] (CR) Ottomata: [C: 2 V: 2] Fix NPE when Country Geocoding fails to detect country [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188013 (owner: QChris) [18:51:22] nuria: Okay. Thanks. [18:52:50] (CR) Ottomata: [C: 2 V: 2] Fix casing of variable name in Geocoding [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188011 (owner: QChris) [18:54:00] (CR) Ottomata: Geocode glam_nara files (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/188010 (owner: QChris) [19:35:44] nuria, here if you need me :). Sorry, was in -research [19:36:06] Ironholds: just sent you e-mail but yeahhhh, let's go ahead and kill those crontabs [19:36:07] (crontabs killed) [19:36:10] snap! [19:36:52] Ironholds: great, one thing less to worry about, thank you! [19:37:02] np! Thanks for your work on this :D [19:37:08] More productionising! [19:37:58] Ironholds: ya, next is the monthly report, which should be pretty easy I think. Also would be nice to move the stuff to the web endpoint too. [19:38:11] agreed! [19:43:00] Analytics-Dashiki: Commons page views in webstatscollector drop precipitously in 2015 - https://phabricator.wikimedia.org/T87589#1009105 (kevinator) [20:13:04] nuria, when were the last corrections to the DW applied? And did they affect all data in the warehouse? I mean, the DW was repopulated from scratch after changes? [20:13:31] yes, that was while back mid dec, i think [20:15:25] nuria, ok, so the data verification from 2015-01-14 must be completed, right? [20:15:45] mforns: no, that is the one dan was doing that is not yet finished [20:16:02] mforns: your work will be the last part of it [20:16:13] mforns: makes sense? [20:16:23] nuria, yes, I was asking if it should be completed ^ [20:16:35] mforns: ahhhh SORRY, [20:16:59] mforns:the data vetting needs to be completed YES! [20:17:08] nuria, OK thanks! [20:17:16] mforns: milimetric will have more details of where did he left it at [20:17:24] ok [20:19:19] mforns: ok, let me know if that makes sense [20:19:41] nuria, ok [20:21:31] Analytics-Engineering, Analytics-Dashiki, Analytics-Cluster: Analytics Engineer has an oozie job to aggreate page views by time - https://phabricator.wikimedia.org/T88125#1009307 (kevinator) [20:24:53] Analytics-Engineering, Analytics-Dashiki, Analytics-Cluster: Analytics Engineer has a python job to aggregate by project - https://phabricator.wikimedia.org/T88127#1009319 (kevinator) [20:26:24] mforns: so you read the verification I did so far? [20:26:56] I linked the code from Gerrit directly so that the page is short and easy to read [20:27:32] milimetric: I'm reading it, along with schema and alembic code and documentation [20:28:03] ok, cool, let me know when you're done and we can talk (but i have my 1/1 with Toby in 30 min.) [20:28:24] milimetric, I'm fine so far, if I have any questions I ping you after Toby [20:28:32] milimetric, thanks! [20:35:14] (PS1) Nuria: Correcting config issue and last build of master [analytics/dashiki] - https://gerrit.wikimedia.org/r/188116 [20:43:52] qchris: hello! [20:44:01] Hi ottomata [20:44:08] Just reading your reviews. [20:44:35] ok cool lemm eknow if you want to discuss the Geocode/Hadoop one, i was thinking about that as I ate lunch, and realized why it might not work [20:44:49] my suggestion i mean [21:02:40] holaaa qchris ... [21:07:04] hola nuria [21:10:16] nuria: Ironholds is going to be submitting some udf stuff, could you be his reviewer for the next week while i'm on vaca? [21:11:35] ottomata: of course [21:14:18] Analytics-Tech-community-metrics: No graphs are displayed on code review queue due to JS error - https://phabricator.wikimedia.org/T88322#1009451 (Qgil) [21:14:28] danke [21:30:32] (PS3) QChris: Drop red herings from documentation of Geocoding UDFs [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187906 [21:30:48] (PS2) QChris: Fix NPE when Country Geocoding fails to detect country [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188013 [21:30:49] (PS2) QChris: Fix casing of variable name in Geocoding [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188011 [21:32:12] (PS3) QChris: Fix NPE when Country Geocoding fails to detect country [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188013 [21:32:14] (PS3) QChris: Fix casing of variable name in Geocoding [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188011 [21:46:31] (Abandoned) QChris: Drop red herings from documentation of Geocoding UDFs [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187906 (owner: QChris) [21:49:36] (CR) QChris: Geocode glam_nara files (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/188010 (owner: QChris) [21:57:41] Analytics-Engineering: Mobile Apps PM has monthly report from oozie about apps uniques - https://phabricator.wikimedia.org/T88308#1009648 (Aklapper) [21:58:47] Analytics-Engineering: Mobile Apps PM has monthly report from oozie about apps uniques - https://phabricator.wikimedia.org/T88308#1009660 (Tnegrin) Agree -- we should have daily uniques and monthly uniques. This matches industry convention. thanks, -Toby [22:00:14] (CR) QChris: Add Maxmind CC BY-SA:3.0 attribution in README.md (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188085 (https://phabricator.wikimedia.org/T88252) (owner: Ottomata) [22:13:07] (CR) QChris: Geocode glam_nara files (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/188010 (owner: QChris) [22:17:00] (PS3) Ottomata: Geocode glam_nara files [analytics/refinery] - https://gerrit.wikimedia.org/r/188010 (owner: QChris) [22:17:07] (CR) Ottomata: [C: 2 V: 2] Geocode glam_nara files [analytics/refinery] - https://gerrit.wikimedia.org/r/188010 (owner: QChris) [22:19:30] Continuous-Integration, Analytics-EventLogging: Set up jsduck test job for EventLogging - https://phabricator.wikimedia.org/T88343#1009758 (Krinkle) NEW [22:20:39] grrr, qchris, i'm having trouble pushing this geocode change [22:20:41] due to rebase [22:20:45] not sure whyyyyYyyy [22:20:51] so, the other patches were merged, ja? [22:20:59] i have rebased against origin/master locally [22:21:02] (after fetching) [22:21:18] but, it still says that I have outstanding commits other than this one [22:21:19] IIRC the two refinery/source ones are still pending your merge [22:21:20] including [22:21:30] Let me check. [22:21:30] oh [22:21:31] these two? [22:21:32] 0dd95ab Fix casing of variable name in Geocoding [22:21:32] 5a82dfb Drop red herings from documentation of Geocoding UDFs [22:21:47] No. [22:21:49] https://gerrit.wikimedia.org/r/#/c/188011/ [22:21:54] https://gerrit.wikimedia.org/r/#/c/188013/ [22:21:56] ottomata: ^ [22:22:00] (CR) Ottomata: [V: 2] Fix casing of variable name in Geocoding [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188011 (owner: QChris) [22:22:15] (CR) Ottomata: [C: 2 V: 2] Fix NPE when Country Geocoding fails to detect country [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188013 (owner: QChris) [22:22:23] I abandoned the "5a82dfb Drop red herings from documentation of Geocoding UDFs" per our chat before. [22:22:36] ergh, but i have that pulled down., hm [22:22:42] hmm [22:22:47] rebasing on master not working because it is in the history [22:22:48] hmmm [22:22:58] "git rebase -i master" [22:23:06] And dropping the line with the change in the editor. [22:23:35] weirdddd ok [22:23:36] (And since you did merges ... "git pull" before that) [22:23:45] yeah did that [22:23:48] k. [22:25:18] (PS2) Ottomata: Fix overriding MaxMind database location in Geocoding [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188012 (owner: QChris) [22:26:01] ok, qchris, do you have an easy way of testing already? [22:26:07] i'm building jar now and will upload for ya [22:26:10] or for me [22:26:12] if you show me how [22:26:16] i guess.. [22:26:18] set [22:26:25] and then check logs of a job for the paths? [22:26:31] Still reading the change ... [22:26:45] ACK, after rebase a test fails..>>>>... [22:28:10] i lost the .configures in the test somehow.... [22:28:18] (If you want to test the jar, and need other maxmind databases ... hdfs://user/qchris/GeoIP* will have some) [22:28:54] oh just one of them [22:28:54] hm [22:28:59] ok [22:32:45] Analytics-Tech-community-metrics: No graphs are displayed on code review queue due to JS error - https://phabricator.wikimedia.org/T88322#1009810 (Aklapper) Same problem happened here in Firefox 35: * mutating the [[Prototype]] of an object will cause your code to run very slowly; instead create the object w... [22:37:24] Ironholds: https://gerrit.wikimedia.org/r/#/c/187304/ [22:37:28] look ok? [22:37:33] I imagine ottomata just pinged you about it too. [22:37:52] (CR) QChris: [C: -1] Fix overriding MaxMind database location in Geocoding (6 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188012 (owner: QChris) [22:37:55] pinged him in #wikimedia-research [22:37:58] aha [22:38:00] too many channels [22:38:02] hehe [22:38:05] (PS3) Ottomata: Fix overriding MaxMind database location in Geocoding [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188012 (owner: QChris) [22:38:11] halfak: come over here and tell us what to do [22:38:22] :D [22:38:56] ottomata: something else to note: HTML for anon views is cached for up to 30 days, and this includes the header, so it will take up to 30 days for this header to be present on all page requests. [22:39:01] So. In recent past, we had decided to stick with the "namespace" == Namespace Number/Identifier thingie [22:39:21] But honestly, I don't feel all that strongly about it. [22:40:34] If we're going to be referencing DB values, I'd like to use the DB names: page_id and page_namespace. [22:40:37] (CR) Ottomata: Fix overriding MaxMind database location in Geocoding (6 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188012 (owner: QChris) [22:41:44] (PS4) Ottomata: Fix overriding MaxMind database location in Geocoding [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188012 (owner: QChris) [22:41:51] Ah :-D [22:42:02] I was wondering about the "Done"s. [22:42:11] haha, ori, +5 more bytes plz? :p [22:42:33] ottomata, 5 bytes for "page_"? [22:42:53] yeah, ori was all "make it ns, save bytes!" [22:43:01] i'm all "NO BUT GOOD NAMES!" [22:43:03] Honestly, we could call it "p_id" and "ns" if it's documented. :) [22:43:08] BOOO [22:43:13] :P [22:43:20] p_id would make me crazy. i could concede on ns [22:43:48] i will likely refine common x-analytics headers like this out into first class named fields in the refined table [22:43:54] so I can rename them then [22:44:39] ottomata, +1 [22:45:02] Ironholds, you concur? [22:45:25] i just so strongly lean towards easily understandable names everywhere. and, in this case, i know it saves a lot of bytes in the long run. buuuuut [22:45:27] i mean [22:45:32] we are already logging via json [22:45:48] and the logs are compressed [22:46:03] so on the backend i'm not worried about it (maybe I should be?) [22:46:09] but not the response body sent to users [22:46:13] livnetata? [22:46:14] aye [22:46:18] who was that? [22:46:57] :-D [22:47:06] halfak, yep [22:47:19] I am hearing "page_namespace" [22:47:51] I'd be cool with page_namespace if those concerned about bytes think it is OK. [22:48:07] ori, if you feel strongly about saving bytes here, then go ahead and do ns, and I will ETL that out later [22:49:53] why does it need ETLing out? [22:49:59] in hive dbs [22:50:02] from my POV; namespace number and any name for that field and I'm happy [22:50:14] i would just die if this field was called ns [22:50:23] ottomata, I agree that field names are important [22:50:23] i'm ok with either: consistency, or fixing names [22:50:29] +1 [22:50:36] a while ago halfak and i decided to stick with consistency as our guidelines [22:50:41] since we won't be changing the mw dbs anytime soon [22:50:43] or ever [22:51:00] I'm only slightly in favor of consistency. If I thought we could effectively change names without muddying the problem further, I'd be down for that. [22:51:04] so, unless there is a very good reason ( like saving bytes) we are going to let consistency be our guide when picking field names [22:51:23] yeah, we talked about this i think and just decided to stick with consistency, unless it really made sense not to [22:51:30] +1 [22:51:37] for example, since our schema was hierarchical, we chose page.namespace [22:51:54] instead of page.page_namespace [22:52:00] NEWAY [22:52:07] ori, ns then? [22:52:31] ok [22:55:41] (CR) QChris: [V: 2] "Code looks good and works as expected." (5 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188012 (owner: QChris) [22:56:11] (CR) QChris: [C: -1] "(Forgot CR-1 on my previous comment.)" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188012 (owner: QChris) [22:58:47] (PS5) Ottomata: Fix overriding MaxMind database location in Geocoding [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188012 (owner: QChris) [22:59:34] (CR) Ottomata: Fix overriding MaxMind database location in Geocoding (5 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188012 (owner: QChris) [22:59:53] ottomata: ok, patch updated. the only blocker is: bblack: if responses from MediaWiki suddenly contain a small 'X-Analytics' header, would anything in varnish freak out? [23:00:07] bblack hasn't responded, but i suspect his answer will be "no" (as in, nothing would freak out) [23:00:19] once that gets the all-clear, feel free to merge and deploy [23:00:36] ori, whered you ask? security? [23:00:42] yeah [23:00:45] k [23:00:58] listening for response...:) [23:00:59] thanks! [23:01:38] (CR) QChris: Fix overriding MaxMind database location in Geocoding (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188012 (owner: QChris) [23:01:46] (CR) QChris: [C: -1] Fix overriding MaxMind database location in Geocoding (3 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188012 (owner: QChris) [23:01:49] bah! [23:01:50] :p [23:02:08] thought i removed that last one [23:02:10] haha [23:02:57] (PS6) Ottomata: Fix overriding MaxMind database location in Geocoding [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188012 (owner: QChris) [23:03:05] (CR) Ottomata: Fix overriding MaxMind database location in Geocoding (3 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188012 (owner: QChris) [23:03:14] take that! [23:03:50] Ouch! [23:03:56] :-P [23:07:29] tnegrin, https://upload.wikimedia.org/wikipedia/commons/b/bf/MW_Core_Review_January_2015.pdf [23:08:19] (CR) Milimetric: [C: 2 V: 2] Correcting config issue and last build of master [analytics/dashiki] - https://gerrit.wikimedia.org/r/188116 (owner: Nuria) [23:10:32] tnegrin, https://phabricator.wikimedia.org/project/board/37/ [23:18:36] (PS7) Ottomata: Fix overriding MaxMind database location in Geocoding [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188012 (owner: QChris) [23:23:14] (CR) QChris: [C: 2 V: 2] Fix overriding MaxMind database location in Geocoding [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188012 (owner: QChris) [23:23:45] ottomata: Wanna try you refinery/source release setup? [23:23:56] qchris: do it! [23:24:00] wait [23:24:03] i have a license file for ya [23:24:51] * qchris is scared. I saw what it did to the source tree the first time you tried :-) [23:24:52] (PS2) Ottomata: Add LICENSE.md with refinery/source license and Maxmind attribution. [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188085 (https://phabricator.wikimedia.org/T88252) [23:24:58] ? [23:25:11] (the release) [23:25:12] better? [23:25:14] ohhh [23:25:15] haha [23:25:18] that was a few times i tried [23:25:21] it should be better now [23:25:33] although, qchris, you will not like this: I made gerrit accept pushes from analytics developers :D [23:25:41] in order to let maven deploy push the releases and tags [23:25:53] (CR) QChris: [C: 2 V: 2] Add LICENSE.md with refinery/source license and Maxmind attribution. [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188085 (https://phabricator.wikimedia.org/T88252) (owner: Ottomata) [23:26:31] Yup. I saw that. :-D [23:26:45] Analytics-Cluster: CC BY-SA:3.0 data committed to refinery? - https://phabricator.wikimedia.org/T88252#1010055 (Ottomata) Open>Resolved [23:27:03] qchris: i say you try it and see! oh, do you have the archiva-deploy credentials? [23:27:11] ottomata: Nonetheless. Could you do ... [23:27:14] Yes. Exactly. [23:27:17] i can do i [23:27:18] it [23:27:32] Thanks. [23:31:52] (PS1) QChris: Bump glam_nara's refinery-hive to version 0.0.5 [analytics/refinery] - https://gerrit.wikimedia.org/r/188231 [23:32:39] (CR) QChris: [C: -1] "Let's wait with this change until refinery-hive-0.0.5 has been released and added." [analytics/refinery] - https://gerrit.wikimedia.org/r/188231 (owner: QChris) [23:36:41] nuria: I think you and I should work together to converge on our proposals, do you have 15-30 min some time today? [23:38:29] WHAT THE [23:38:32] why'd it not prompt me [23:38:33] OHHH [23:39:01] ? [23:43:09] i think it worked. [23:43:31] qchris: pull? [23:43:49] (PS1) Ottomata: Add README.md docs about how to cleanup aborted releases [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188238 [23:44:17] (CR) Ottomata: [C: 2 V: 2] Add README.md docs about how to cleanup aborted releases [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188238 (owner: Ottomata) [23:45:08] qchris: do you get the new tag? [23:45:14] if so, I will deploy the release to archiva [23:45:52] hm, i se eit in github [23:45:55] so I am proceeding :) [23:46:13] * qchris checks [23:46:23] Yup. Tag is ther. [23:46:26] k [23:46:50] this takes a while...but only because archiva is very slow. [23:46:55] i do not know why archiva is so slow :/ [23:47:15] you read chas-mp's link the other day? [23:47:39] no [23:47:42] link? [23:48:15] http://en.wikipedia.org/wiki/The_Scorpion_and_the_Frog [23:48:17] ottomata: ^ [23:48:40] :-/ [23:48:40] oh, ha yes [23:49:22] it seems to just take a long time before it begins to process certain requests [23:49:28] once it starts processing them it is fine [23:49:29] Thanks for the release! [23:49:36] Cool. [23:50:19] I think I have to call it a day. Gonna have to get up early tomorrow. [23:50:25] okey dokey [23:50:29] Thanks for all the merges, CR, and release! [23:50:30] thanks qchris, tty in a week! [23:50:31] tup! [23:50:32] yup! [23:50:38] We have working geocoding! :-D [23:51:42] Analytics-Kanban, Analytics-EventLogging: Remove autoincrement id from tables [5 pts] - https://phabricator.wikimedia.org/T87661#1010105 (kevinator) [23:52:32] Analytics-Kanban, Analytics-EventLogging: Sanity check changes to timestamp fields and remove autoincrement id from tables & deploy to Prod [8 pts] - https://phabricator.wikimedia.org/T88297#1010110 (kevinator) p:Triage>Normal [23:53:00] wheee [23:53:28] Analytics-Kanban, Analytics-EventLogging: Remove autoincrement id from tables [5 pts] - https://phabricator.wikimedia.org/T87661#996313 (kevinator) [23:53:30] Analytics-Kanban, Analytics-EventLogging: Change timestamp fields to reduce DB storage size [8 pts] - https://phabricator.wikimedia.org/T87660#1010117 (kevinator) [23:53:31] Analytics-Kanban, Analytics-EventLogging: Sanity check changes to timestamp fields and remove autoincrement id from tables & deploy to Prod [8 pts] - https://phabricator.wikimedia.org/T88297#1008583 (kevinator)