[00:00:01] Pchelolo: when deploying aqs to beta we are getting an error (there are several and some of them come from cassandra) but there is one i would like some help with [00:00:45] Pchelolo:What we are deploying: https://gerrit.wikimedia.org/r/#/c/333104/ [00:00:59] Pchelolo: the error https://github.com/wikimedia/restbase-mod-table-cassandra/blob/master/lib/schemaMigration.js#L18 [00:01:20] nuria: just bump the schema version [00:01:24] Pchelolo: the change we are deploying is really just the last one on list https://gerrit.wikimedia.org/r/#/q/a7eb80d [00:02:08] Pchelolo: as the others we deployed from a branch [00:02:47] nuria: ye, just put version: 3 on line 35 of pageviews.js [00:03:09] that's a sanity change to protect from unintentional schema changes [00:03:59] Pchelolo: what i would like to understand is where does the schema change come from? is it the change to compression scheme? or rather the changes to pageview yaml? [00:04:54] nuria: yep. when you change the compaction scheme the cassandra module actually detects that and creates a CQL query to alter the tables in cassandra [00:05:15] Pchelolo: but see, that change has ALREADY happen on prod [00:05:27] Pchelolo: as our cluster was rebuild from scratch [00:05:43] 10Analytics, 10MediaWiki-extensions-WikimediaEvents, 10The-Wikipedia-Library, 10Wikimedia-General-or-Unknown, 13Patch-For-Review: Implement Schema:ExternalLinksChange - https://phabricator.wikimedia.org/T115119#2954851 (10Samwalton9) @Legoktm Interesting! Any idea how this could be resolved? [00:05:51] Pchelolo: so i really do not want to be triggering updates on prod , that change has alredy been deployed (from a branch) [00:06:10] Pchelolo: I am just deploying an up to date master branch that also includes those changes [00:06:12] nuria: em, I'm not following.. you're deployinng on beta don't you? [00:07:04] Pchelolo: yes, but some of my changes are already deployed to prod as we did so from a branch , now we are just briging beta up to speed [00:07:29] Pchelolo: so we are deploying what we have in prod plus one extra change [00:07:33] nuria: aha. ok. got it. lemme check something really quick [00:07:48] and that extra change is the one that's changeing the schema? [00:08:20] Pchelolo: that extra change is this one: https://gerrit.wikimedia.org/r/#/c/326545/ [00:08:29] Pchelolo: is this consider a schema change? [00:09:18] nope. Ok. lemme check something quickly [00:09:55] Why is cassandra down on deployment-aqs01 ? is it on purpuse? can I start it? [00:10:15] oh.. jmx reporter is missinng [00:10:15] Pchelolo: if you can sure, we couldn't do it [00:10:25] Pchelolo: that was ahem .. problem #2 [00:11:55] the jmx reporter repo is not ready yet, sorry about that nuria [00:12:06] but I think i have a work-around for it [00:12:09] gimme 5 mins [00:12:32] mobrovac: this is an issue on beta only right? [00:14:55] at this point, yes [00:25:53] mobrovac: ok, just let us know when you have a workarround [00:25:59] mobrovac: i also have another question [00:26:35] mobrovac: it would be very useful if restbase would log to local host in beta (rather than logstash), is that something we can change? [00:26:53] you can do it in puppet [00:27:09] e.g. for RB we have two configs - one for prod, another for beta [00:30:18] nuria: ok, cassandra is back up on deployment-aqs01 [00:30:27] you should be able to do the deploy now [00:30:50] nuria: note that from deployment-tin you now have to pass the -f param to scap deploy, otherwise it won't complete the action [00:31:25] 10Analytics: Aqs beta should log to localhost rather thna logstash - https://phabricator.wikimedia.org/T155791#2954868 (10Nuria) [00:31:34] mobrovac: like scap deploy -f [00:31:44] mobrovac: or something else? [00:32:07] Pchelolo: looks like cassandra is up but i think we are still going to run into teh schema issue [00:32:09] *the [00:32:34] nuria: yup [00:34:07] 10Analytics: Aqs beta should log to localhost rather than logstash - https://phabricator.wikimedia.org/T155791#2954882 (10Nuria) [00:37:49] ok nuria here's what's going on [00:37:56] Pchelolo: yessir [00:38:52] to detect that the table schema was changed we're storing the previous schema in a cassandra table. So in prod you've got away with upgrading the schema and not bumping the version because you've rebuilt the cluster from scratch, so there was no previous schema stored. [00:39:22] here you have the previous schema stored, so the schema-change-detection login actually works and you get this error in beta. [00:40:03] The easiest workatround is to bump the schema version to 3 on line 35 as I've said. It will update the data model in beta and from production it will be a no-op [00:40:12] Pchelolo: ok, any suggestions on best way to fix it? I guess we can manually update version on cassandra beta? [00:41:21] Pchelolo: isn't it easier to update schema version on beta? since change is a fake. I rather do that than having a code change that is meaningless and might create confusion later on. [00:41:26] nuria: ok, give me a moment to fix it [00:41:35] Pchelolo: aham [00:42:37] I'll stop aqs there for a timebeing [00:42:40] it's logging a lot [00:44:19] Pchelolo: k, let me know when you have run the update and i will try to deploy [00:52:39] nuria: aqs is happily running on deployment-aqs01 [00:52:55] Pchelolo: did you updated schema vs on cassandra? [00:53:07] Pchelolo: redeploying [00:53:58] nuria: something like that [00:54:18] Pchelolo:ok, it is a secret but now it works, yes! [00:54:31] Pchelolo, mobrovac thanks for the prompt responses. [03:01:26] 10Analytics, 10EventBus: log-events topic emitted in EventBus - https://phabricator.wikimedia.org/T155804#2955138 (10Mattflaschen-WMF) [03:13:05] 06Analytics-Kanban, 10EventBus, 10Wikimedia-Stream, 06Services (watching), 15User-mobrovac: Bikeshed what events should be exposed in public EventStreams API - https://phabricator.wikimedia.org/T149736#2761082 (10Mattflaschen-WMF) Per our Dev Summit meeting and plan (ReviewStream = review-stream-revision... [03:16:22] 06Analytics-Kanban, 10EventBus, 10Wikimedia-Stream, 06Services (watching), 15User-mobrovac: Bikeshed what events should be exposed in public EventStreams API - https://phabricator.wikimedia.org/T149736#2955159 (10Mattflaschen-WMF) The reason we may want EventBusWikiChangeEventsNewInfra (hypothetical repl... [03:17:32] 10Analytics, 10ChangeProp, 10Edit-Review-Improvements-ReviewStream, 10EventBus, and 4 others: Set up the foundation for the ReviewStream feed - https://phabricator.wikimedia.org/T143743#2955161 (10Mattflaschen-WMF) >>! In T143743#2658575, @SBisson wrote: > In the new infrastructure, RCStream will be deprec... [03:17:50] 10Analytics, 10ChangeProp, 10Edit-Review-Improvements-ReviewStream, 10EventBus, and 4 others: Set up the foundation for the ReviewStream feed - https://phabricator.wikimedia.org/T143743#2955163 (10Mattflaschen-WMF) [03:20:23] 10Analytics, 10ChangeProp, 10Edit-Review-Improvements-ReviewStream, 10EventBus, and 4 others: Set up the foundation for the ReviewStream feed - https://phabricator.wikimedia.org/T143743#2955164 (10Mattflaschen-WMF) [05:15:01] 10Quarry: Explain command forces Quarry to keep running endlessly - https://phabricator.wikimedia.org/T155808#2955224 (10Soni) [07:36:48] morning ateam! [08:52:37] fdans: o/ [08:52:58] joal: Hello! Moritz's code review for the labsdb replica firewall changes - https://gerrit.wikimedia.org/r/#/c/332457/3/modules/role/manifests/labs/db/replica.pp [08:53:34] (so it should open 3306 on labsdb-replica for traffic coming from the analytics network) [08:53:53] I was wrong about the network rules, it should be reachable only with the firewall changes [08:53:59] (/me ignorant) [08:54:13] soooo much better than what we thought :) [08:54:40] if everything looks right for you, I believe that today we could merge and you'll be able to test whenever you prefer [09:00:31] also a-team, forgot to ask - is Hue something that we suggest to use instead of beeline (as a UI replacement for it) or not? [09:08:56] * elukey commutes to the office! [09:27:05] * elukey back [09:31:20] 10Analytics-Tech-community-metrics, 06Developer-Relations (Jan-Mar-2017): Deployment of Maniphest panel - https://phabricator.wikimedia.org/T138002#2955523 (10Lcanasdiaz) 2nd ping :O [10:14:11] * fdans has been craving ribs all morning [10:17:51] 10Analytics: Disable queries for recent data on stats.grok.se - https://phabricator.wikimedia.org/T155785#2955609 (10MusikAnimal) [10:28:10] 10Analytics: Disable queries for recent data on stats.grok.se - https://phabricator.wikimedia.org/T155785#2954564 (10MusikAnimal) Not #tool-labs-tools-pageviews :) Stats.grok.se is an external service that we don't have access to, and is (was) maintained by [[ https://en.wikipedia.org/wiki/User:Henrik | User:Hen... [11:00:32] 10Analytics, 10Wikimedia-General-or-Unknown: Disable queries for recent data on stats.grok.se - https://phabricator.wikimedia.org/T155785#2955677 (10Nemo_bis) p:05Triage>03Low a:03Henrik [11:01:51] 10Analytics, 10Wikimedia-General-or-Unknown: Disable queries for recent data on stats.grok.se - https://phabricator.wikimedia.org/T155785#2954564 (10Nemo_bis) To me, serving an internal error seems quite an explicit way to disable them. :) More useful would be to redirect (not the requests for JSON but the cha... [11:06:54] 10Analytics, 10Wikimedia-General-or-Unknown: Disable queries for recent data on stats.grok.se - https://phabricator.wikimedia.org/T155785#2955688 (10Nemo_bis) [11:21:16] elukey: Thanks mate for letting me know on labsdb-network :) [11:21:33] I still need info from chasemp (hopefully I'll manage to catch later on today) [11:22:08] elukey: Do you think we could apply the network patch from now on? It would allow me to possibly test this weekend [11:24:01] yes I do, I think that Moritz already merged it (or he'll do it today) [11:24:07] checking [11:25:09] still not merged but probably it will be done this afternoon [11:25:21] awesome :) [11:25:28] joal: do we "sponsor" hue as UI replacement for beeline? [11:25:33] elukey: Thanks mate for covering me on that part :) [11:25:35] or something like Quarry [11:25:44] elukey: no "sponsor" per say [11:26:03] elukey: for people prefering nive web UIs over CLI, hue on hive is fine [11:26:24] elukey: BUT, we are crazy geeks, we'll never sponsor web app over LCI :-P [11:27:42] ahhh okok.. because Amir1 asked access to Hue as "quarry" for analytics [11:28:11] It's a bit different, but has some comparable things [11:29:32] It would be nice to have it, hopefully Druid fill this gap [11:30:02] Amir1: Can you be more precise on what you mean by 'it' ? [11:30:33] "Quarry for hadoop" [11:31:05] a GUI that I can refer to or extract graphs, etc. [11:32:54] Amir1: I think the closest you'll have for now os hive-on-hue (it's not too bad), and that madhuvishy is working on providing an internal version of PAWS, that will allow us to have python notebooks connected to hive [11:33:19] (see Amir1 you don't have to trust me, I say silly things all the times :) [11:35:01] joal: Thanks! elukey: You know much better than me. I still need to learn from you! [11:38:45] leaving for the afternoon - later a-team [11:57:02] 10Analytics, 10ChangeProp, 10Edit-Review-Improvements-ReviewStream, 10EventBus, and 4 others: Set up the foundation for the ReviewStream feed - https://phabricator.wikimedia.org/T143743#2955780 (10SBisson) >>! In T143743#2955161, @Mattflaschen-WMF wrote: > [...] > * review-stream-revision-create - revision... [12:00:16] 10Analytics, 10Analytics-Cluster, 06Operations: Move cloudera packages to a separate archive section - https://phabricator.wikimedia.org/T155726#2955790 (10MoritzMuehlenhoff) p:05Triage>03Normal [12:42:48] 06Analytics-Kanban: Run a 1-off sqoop over the new labsdb servers - https://phabricator.wikimedia.org/T155658#2950064 (10elukey) Moritz patch (https://gerrit.wikimedia.org/r/#/c/332457/) has been abandoned after a chat with Jaime. We should not open ports on labsdb replicas but use `labsdb-analytics.eqiad.wmnet`... [12:47:34] joal: did I miss you again? [12:48:46] yesss [12:48:49] he is out :) [12:55:01] elukey: in looking at https://phabricator.wikimedia.org/T155658#2955883 have you confirmed you can access 3306 at that service url? [12:55:08] (from wherever in analytics land you need I mean) [12:57:28] chasemp: nope just talked with Jaime, still need to test [12:57:38] but maybe I can rapidly check with telnet [12:57:48] please, that's really all I was after [12:58:01] then I know a bit about how blocked you guys are as joal and I keep mising each other [12:58:26] it seems not working :( [12:58:50] elukey@analytics1030:~$ telnet labsdb-analytics.eqiad.wmnet 3306 [12:58:51] Trying 10.64.37.14... [12:59:13] (an1030 is an hadoop node) [12:59:52] 06Analytics-Kanban: Run a 1-off sqoop over the new labsdb servers - https://phabricator.wikimedia.org/T155658#2955906 (10elukey) ``` elukey@analytics1030:~$ telnet labsdb-analytics.eqiad.wmnet 3306 Trying 10.64.37.14... ``` Seems like a network ACL preventing access.. [13:16:59] thanks elukey, not sure what the deal is but at least we know [13:18:31] chasemp: just confirmed with Faidon, there might be some network whitelist work to do :) [13:19:23] makes sense [13:23:14] 10Analytics, 10Wikimedia-General-or-Unknown: Disable queries for recent data on stats.grok.se - https://phabricator.wikimedia.org/T155785#2955945 (10Effeietsanders) @Nemo_bis The error made me try several different pages, come back a few times etc before concluding it would not get fixed. It does not come acro... [15:16:16] 06Analytics-Kanban: Run a 1-off sqoop over the new labsdb servers - https://phabricator.wikimedia.org/T155658#2956060 (10MoritzMuehlenhoff) It's not blocked by ferm, so this in fact needs network ACL changes. [15:17:29] moritzm: yeah I am trying to figure out what's wrong on cr1/cr2, but apparently there seems to be no ACL over there [15:57:11] 06Analytics-Kanban, 10EventBus, 10Wikimedia-Stream, 06Services (watching), 15User-mobrovac: Bikeshed what events should be exposed in public EventStreams API - https://phabricator.wikimedia.org/T149736#2956117 (10Nuria) +1 to @Mattflaschen-WMF comment. There is also an operational argument. Seems that RC... [15:58:03] 10Analytics-Tech-community-metrics, 06Developer-Relations (Jan-Mar-2017): Deployment of Maniphest panel - https://phabricator.wikimedia.org/T138002#2956118 (10Aklapper) Sorry. I have emailed @mmodell a few days ago asking him for input here. Waiting for a reply here... :-/ [15:59:32] 06Analytics-Kanban, 10Fundraising-Backlog, 13Patch-For-Review: Productionize banner impressions druid/pivot dataset - https://phabricator.wikimedia.org/T155141#2956119 (10mforns) a:03mforns [16:04:06] 06Analytics-Kanban, 07Easy, 03Google-Code-In-2016, 13Patch-For-Review: Add monthly request stats per article title to pageview api - https://phabricator.wikimedia.org/T139934#2956129 (10Nuria) a:05Phantom42>03Nuria [16:06:31] 10Analytics, 10Wikimedia-General-or-Unknown: Disable queries for recent data on stats.grok.se - https://phabricator.wikimedia.org/T155785#2956146 (10Aklapper) > on the tool page Where is "the tool page"? [16:08:40] 10Analytics-Tech-community-metrics, 06Developer-Relations (Jan-Mar-2017): Deployment of Maniphest panel - https://phabricator.wikimedia.org/T138002#2956150 (10Lcanasdiaz) Thank you @Aklapper ! We could deploy it next with (even with a subset of the data) so you can play with it. It is saving a lot of time here... [16:16:22] 10Analytics-Dashiki, 06Analytics-Kanban, 13Patch-For-Review: Add extension and category (ala Eventlogging) for DashikiConfigs - https://phabricator.wikimedia.org/T125403#2956175 (10Milimetric) @greg I did ask whether security review was required on this extension's original patch, and added Darian, but didn... [16:33:20] 10Analytics-EventLogging, 06Analytics-Kanban: Add user_agent_map field to EventCapsule - https://phabricator.wikimedia.org/T153207#2872765 (10Milimetric) Reminder to also change the description of the userAgent field in the capsule: https://meta.wikimedia.org/wiki/Schema:EventCapsule [16:41:00] 10Analytics, 10Wikimedia-General-or-Unknown: Disable queries for recent data on stats.grok.se - https://phabricator.wikimedia.org/T155785#2956218 (10Effeietsanders) @Aklapper I was referring to http://stats.grok.se/ [16:53:49] Hey chasemp :) [16:53:53] I'm on now :) [16:57:57] joal: still no luck in unblocking the proxy :( [16:58:47] :( [16:59:02] elukey: maybe back to moritzm original patch, or will it be the same thing? [17:02:17] joal: no no Jaime -2ed it, the proxy is the only way forward [17:02:27] it is just a matter of finding the right ACL [17:02:47] and there is not firewall rule for the proxies [17:03:42] ok elukey :) [17:06:06] joal: it's going to take a bit of looking at, I'm not feeling very well today either [17:07:04] chasemp: monday would be better? [17:07:34] :) early in the week? monday my day is 6hrs of meetings usually [17:08:07] wow chasemp, I don't want to be the one adding to that :) [17:09:54] chasemp: What about Tuesday, just before Technology Management meeting? [17:10:02] chasemp: 1/2h meeting? [17:10:17] you're thinking to do a hangout and sync up? [17:10:39] chasemp: yessir (except if you prefer differently) [17:11:56] sounds delightful [17:12:11] shoot me a cal apt? [17:12:47] chasemp: talking about DBs architecture has never been named delightful by a colleague before, I take this as a good sign ;) [17:12:50] sure chasemp [17:14:48] chasemp: Just sent [17:15:06] chasemp: With an update about duration [17:15:12] joal: I'll restart working on Monday on this, it is probably how the Analytics VLAN works that blocks us [17:15:25] really elukey ? Bizarre ! [17:16:11] well from the explanation that I got and from what I can see in the router's configs we filter inbound traffic to the analytics VLAN [17:16:32] but it seems the other way around (as also Andrew remember that it should work) [17:16:41] but we'd need to wait for the network masters :) [17:16:45] my thinking when I had amoment was to go through teh ACL's there and propose a rule change on a task for input [17:16:52] or one of us to do that :) [17:17:21] milimetric: Hi ! [17:17:36] brb joal, watching the world end [17:17:46] sure milimetric, send me the link ! [17:17:53] https://www.nytimes.com/video/us/politics/100000004881180/watch-live-inauguration.html [17:18:16] chasemp: I tried to run tcpdump on both analytics1030 and the labsdbproxy, making the telnet connection from an1030 to labsdb. I don't see the SYN packet logged on the labsdb proxy host.. [17:18:34] and there is not inbound traffic rule afaics on the router for it [17:19:44] elukey: I'm sorry but I'm not even trying to help on that - The effort I need to just understand the three lines you wrote tell me I need not to take patrt :) [17:22:53] chasemp: are you familiar with networks ACL on cr1/cr2 eqiad? [17:23:52] I think I can come up with the necessary change yeah but time is one thing I'm short on (and then flu) [17:27:46] ok, joal, world's end seems to have been extended to the next 4-8 years, so I have time now [17:27:53] :D [17:28:36] Just wanted to confirm on the plan to plan something for January metrics meeting :) [17:28:44] chasemp: ahhh okok! [17:29:04] don't worry, I wanted to understand where to look because I can't find the problem :( [17:29:07] next week :) [17:29:27] joal: yes, I am going to work on the slides today, mostly trying to brainstorm the best example to show in 7 minutes [17:29:47] Awesome milimetric, please let me know if you think I can do anything [17:30:02] so the slides are due Monday, we can maybe work on them together Monday morning? [17:30:21] milimetric: hopefully by then we'll have confirmation that we can do it on labs, and we'l be able to tell about it even if not yet done :) [17:30:23] Marcel and I were going to ponder over them some more later after meetings / lunch [17:30:28] yay [17:30:44] Sounds great milimetric - monday morning ! [17:33:13] 06Analytics-Kanban: Run a 1-off sqoop over the new labsdb servers - https://phabricator.wikimedia.org/T155658#2956313 (10elukey) So far I have followed what @faidon suggested, namely: 1) Checking input filters ACLs on cr1/cr2 for the ports related to `labsdb-analytics.eqiad.wmnet` and `analytics10XX.eqiad.wmnet... [17:33:33] all right running afk! [17:33:35] byyyeeee o/ [17:34:28] Bye elukey ! [17:40:15] 06Analytics-Kanban, 13Patch-For-Review: Productionize loading of edit data into Druid (contingent on success of research spike) - https://phabricator.wikimedia.org/T141473#2956337 (10JAllemandou) done using Druid loading rules. Indexing involves full dataset, like that it is available on hadoop deepstorage if... [17:48:41] milimetric, I'm around, let me know when you're ready with lunch and stuff and you want to cave :] [17:49:36] I'm ready to cave any moment now :) http://www.macmillandictionary.com/us/dictionary/american/cave_2 [17:49:53] I'm gonna go brainstorm for a while, will ping when back [17:56:06] ok [17:56:22] xD [17:58:00] gah, that word was already taken [18:07:00] lately i've been using the latino spanish word for batcave [18:07:07] La Baticueva [18:07:29] * joal like *la baticueva*, sounds like a dance ! [18:07:57] * fdans dances la baticueva vigorously [18:08:31] :D [18:08:48] halfak: Hi firend, are you close by and having two minutes? [18:11:10] Hey joal. Am making lunch now. Will be back in 30 minutes [18:11:19] sure halfak, later :) [18:11:23] kk [18:15:31] Hi mforns :) [18:15:37] hey joal :] [18:20:30] leaving for the weekend, see you on monday a-team!! 🙋🏻‍♂️ [18:20:45] (emoji fail) [18:20:45] fdans, bye!! [18:20:48] hehehe [18:21:23] have a nice weekend! [18:23:23] Bye fdans! Have a good weekend :) [18:48:09] * nuria loves baticueva, yatusabeh [19:15:57] urandom: yt? have a (i think) fast question [19:16:33] nuria: what's up? [19:17:03] urandom: I would like to dump a few records from cassandra on prod to load them on staging so we can do proper testing [19:17:35] urandom: not a full table obviously as that might be to big for staging/beta labs/however -we-call-it [19:18:08] urandom: I saw the copy command that seems to be geared for full tables [19:18:26] urandom: is there a way to dump a few records? [19:18:36] nuria: not really, no [19:18:41] urandom: ains [19:18:49] i mean, not short of querying them [19:19:34] nuria: you already have machinery for import though, no? will that not work? [19:19:49] urandom: ahem.. me no understand [19:20:10] i mean, what you use daily, and what you used to back-fill the new cluster [19:20:30] urandom: but that is prod talking to prod [19:20:37] urandom: staging is on labs [19:21:09] so you're looking to copy to file first, and then copy the file(s) to labs? [19:21:41] the prod cluster is prod too, is what i'm wondering [19:21:48] urandom: right, i was thking (maybe old fashion). dump some of prod records to csv ->move csv to labs -> import records into labs [19:21:58] oh, ok [19:22:23] yeah, there is the copy command in cqlsh, but i don't think that will work for you [19:22:33] urandom: but maybe there is something better? labs is not reachable from analytics vlan so i cannot load directly from cluster [19:23:11] nuria: you could query and write to csv [19:23:28] you want some subset, do you know know the shape of that subset? [19:24:02] like, a subset of pages with full history? subset of pages with partital history? all pages with partial history? [19:26:09] urandom: query cluster you mean? or query cassandra? [19:26:19] either, really [19:26:35] you might get more throughput out of cassandra [19:26:39] urandom: better cassandra as cluster does loads of manipulation of data when loading [19:26:53] urandom: to "fit" cassandra structure [19:28:05] urandom: the select > out.txt would be the recommended method? [19:28:24] i think so, yeah [19:28:48] urandom: k, lemme try , thanks for the prompt response! [19:29:02] no worries, let me know if you need any help with that [19:29:37] urandom: and .. BTW, did you get data from cluster to do your load testing like you wanted? [19:29:59] no yet, i just got the new test environment up [19:30:18] nuria: between holidays, devsummit, and allhands, things have languished :)L [19:31:06] urandom: jaja, yes. ok let me know when you get back to that [19:31:18] will do. [19:43:30] urandom: me little scared of doing select that eats up cluster resources [19:43:45] urandom: does this sound ok? "select article from local_group_default_T_pageviews_per_article_flat where article='Barack_Obama' and ts='201701010000';" [19:47:26] nuria: that should be fine, article is the partition key, and ts is a clustering key, yes? [19:47:40] urandom: yes, PRIMARY KEY (("_domain", project, article, granularity), timestamp, "_tid") [19:47:42] milimetric, yt? [19:47:47] hey mforns [19:47:54] yeah, was just staring at the slides [19:48:04] cave? [19:48:05] hey, do you want to bat-cave? [19:48:07] ok [19:48:19] la baticueva cha-cha-cha [19:48:37] lol [19:48:54] nuria: so you're going to have _domain, project, and granularity there as well? [19:50:33] urandom: adding [19:51:16] you could omit timestamp too if you want, page through the results [19:51:30] or use a range for timestamp [19:51:36] urandom: wait.. now I am not sure of what is _domain [19:52:41] nuria: yeah, i'm not sure what project is given that you have _domain [19:52:56] on the restbase cluster, _domain is like en.wikipedia.org [19:55:12] nuria: try: select "_domain", project, article, granularity from local_group_default_T_pageviews_per_article_flat.data limit 1; [19:56:18] urandom: [19:56:22] https://www.irccloud.com/pastebin/4sidcsKS/ [19:56:51] urandom: ah sorry, quotes [19:57:02] urandom: [19:57:06] https://www.irccloud.com/pastebin/ow8LLB7E/ [19:58:11] oh [19:58:19] nuria: try: select "_domain", project, article, granularity from "local_group_default_T_pageviews_per_article_flat".data limit 1; [19:58:28] (quotes around the keyspace name) [19:58:30] urandom: right, right just did [19:58:45] urandom: sorry, too slow on irc but i get it, it is a "fake" domain [20:01:49] nuria: again, not sure what you're looking for with this subset of data, but you can use a range for timestamp and page through the results [20:02:25] urandom: okeis,yes, will do that. [20:49:39] Pchelolo: which are the user and pw for cassandra instances on labs? [21:19:57] urandom: one more question if you may [21:20:32] nuria: sure, what's up? [21:21:19] urandom: i dump a thousand records as txt, transform those to csv and moved the file to nuria@deployment-aqs01:~/cassandra-loading$ pwd [21:21:19] /home/nuria/cassandra-loading [21:22:02] when loading with copy command (in labs) i get an error about hexadecimal uuid [21:22:23] urandom: but that hasn't changed [21:22:31] urandom: Example: [21:23:12] https://www.irccloud.com/pastebin/To37sKEs/ [21:23:18] https://www.irccloud.com/pastebin/DPhGe6qp/ [21:23:29] wait let me make sure is matching columns right ..... ay ay [21:24:45] urandom: ya, it is [21:25:09] urandom: but it errors [21:25:12] https://www.irccloud.com/pastebin/0lmSSLxA/ [21:25:27] 06Analytics-Kanban, 10Fundraising-Backlog, 13Patch-For-Review: Productionize banner impressions druid/pivot dataset - https://phabricator.wikimedia.org/T155141#2934689 (10DStrine) Hi all, Just joining this work here. I've seen this tool in action and it looks pretty cool!! I love dev summit meetings and hac... [21:26:03] nuria: let me look at something [21:26:31] 06Analytics-Kanban, 10Fundraising-Backlog, 13Patch-For-Review: Productionize banner impressions druid/pivot dataset - https://phabricator.wikimedia.org/T155141#2956901 (10Nuria) @Dstrine: actually we are going to go ahead and productionize it ourselves, just try to see what is going on for further modificati... [21:26:33] urandom: yessir [21:27:44] nuria: i wonder if it's the quotes [21:28:08] urandom: ok, let me try 1 record w/o [21:28:38] urandom: no quotes on any field? [21:28:47] well, on the UUID at least [21:28:55] 06Analytics-Kanban, 10Fundraising-Backlog, 13Patch-For-Review: Productionize banner impressions druid/pivot dataset - https://phabricator.wikimedia.org/T155141#2956916 (10DStrine) @Nuria ok, that's cool. Thanks for the help! [21:28:58] quotes aren't needed for them in the CQL grammar [21:29:07] though i'm surprised they'd create an issue [21:31:11] yeah, cqlsh definitely doesn't like them [21:31:20] quotes around a UUID [21:31:37] single quotes either [21:32:15] urandom: file w/o quotes whatsoever also fails [21:32:17] urandom: [21:32:23] https://www.irccloud.com/pastebin/aXqCy0jz/ [21:32:43] does that fail in the same way? [21:32:56] urandom: yes [21:32:59] https://www.irccloud.com/pastebin/w93msp4l/ [21:34:39] bye team, have a nice weekend! [21:35:37] mforns: ciao [21:37:48] is that an actual UUID from the prod cluster? [21:38:34] nevermind [21:39:42] urandom: yes [21:45:20] urandom: to replicate on deployment-aqs01 go to /home/nuria/cassandra-loading files are there, will change permits [21:47:13] urandom: i triple checked columns match [22:21:20] nuria: fyi, node is still on version 4.4.6 on deployment-aqs01, should we update to node 6?