[09:49:56] (PS1) QChris: Add kraken-etl skeleton for camus based ETL [analytics/kraken] - https://gerrit.wikimedia.org/r/88945 [09:49:57] (PS1) QChris: Turn kraken-etl jar into shaded jar [analytics/kraken] - https://gerrit.wikimedia.org/r/88946 [09:56:10] qchris: hi [09:56:17] Hi average [12:41:47] (PS1) Erik Zachte: use different input/output folders [analytics/wikistats] - https://gerrit.wikimedia.org/r/88973 [12:42:02] (CR) jenkins-bot: [V: -1] use different input/output folders [analytics/wikistats] - https://gerrit.wikimedia.org/r/88973 (owner: Erik Zachte) [13:00:06] hi, know where the index.html of stats.wikimedia.org is edited? [13:01:08] nevermind, got it [13:13:12] (PS1) Dzahn: use new wikivoyage logo on stats.wikimedia.org [analytics/wikistats] - https://gerrit.wikimedia.org/r/88978 [13:13:24] (CR) jenkins-bot: [V: -1] use new wikivoyage logo on stats.wikimedia.org [analytics/wikistats] - https://gerrit.wikimedia.org/r/88978 (owner: Dzahn) [13:28:59] (PS1) Erik Zachte: upd link for Wikidata (linked to draft version) [analytics/wikistats] - https://gerrit.wikimedia.org/r/88981 [13:54:01] (CR) Ottomata: [C: 2 V: 2] Add kraken-etl skeleton for camus based ETL [analytics/kraken] - https://gerrit.wikimedia.org/r/88945 (owner: QChris) [13:54:18] (CR) Ottomata: [C: 2 V: 2] Turn kraken-etl jar into shaded jar [analytics/kraken] - https://gerrit.wikimedia.org/r/88946 (owner: QChris) [13:54:56] ottomata: You're fast on the submit button :-D Thanks. [13:55:11] heh :) [13:56:33] oo qchris [13:56:39] yup? [13:56:40] not sure if you want to build using this but [13:56:41] http://search.maven.org/#search%7Cga%7C1%7Ckafka [13:57:34] Oh. Cool. [13:57:47] I've built camus using the kafka they provide. [13:57:53] (CR) Erik Zachte: [C: 2 V: 2] use new wikivoyage logo on stats.wikimedia.org [analytics/wikistats] - https://gerrit.wikimedia.org/r/88978 (owner: Dzahn) [13:57:54] (CR) jenkins-bot: [V: -1] use new wikivoyage logo on stats.wikimedia.org [analytics/wikistats] - https://gerrit.wikimedia.org/r/88978 (owner: Dzahn) [13:58:08] Do you know how far apart they have drifted? [13:58:26] Or how strict camus is about requiring it's own kafka builds? [14:00:00] its probably all compatible, but i betcha the kafka camus has is the same as what's in maven [14:00:07] it was a beta release made at that time [14:00:18] we're building kafka for broker installs with latest from 0.8 branch right now [14:00:23] until they release stable 0.8 [14:00:56] ok. [14:03:48] hey there [14:04:12] just found out that https://integration.wikimedia.org/ci/job/analytics-wikistats/ job is failing on patch submission [14:04:43] looking at the repository, some commit have been pushed directly to analytics/wikistats , thus bypassing Gerrit / Jenkins [14:04:47] and that broke the repo :( [14:04:59] just speculation [14:06:42] thanks hashar, I'll let drdee know too [14:07:18] yep, that's why the logo change hasn't been merged though Erik just gave it +2 [14:07:19] Our code quality is no worse than anyone else in the industry [14:08:45] milimetric: might want to remove push right as well and enforce Gerrit :D [14:09:11] I don't think Erik is comfortable with that [14:09:33] and I have a long history of getting pretty screwed by gerrit / git-review as well [14:10:03] I usually try to at the very minimum git review and self merge [14:10:16] qchris, we should get your access to labs stuff fixed [14:10:16] hm [14:10:22] but sometimes things get completely messed up and instead of spending an hour figuring out why, I just git push [14:10:23] That process requires human oversight that nobody was providing [14:10:40] ottomata: agreed. [14:10:50] I already asked again some hours ago. [14:10:52] let's make sure to ask in #labs in a few hours when [people are awake [14:10:53] ok [14:15:01] milimetric: unfortunately can't have both afaict, either never push or always push but not have jenkins [14:15:11] afaik [14:16:35] that seems odd, but then again we haven't yet set up jenkins on my projects [14:17:10] milimetric: one day we will have to sort out the issue you are facing :-] [14:17:10] too many technologies to fight with, way too little time [14:17:29] milimetric: but I guess most of your projects are single person and git is merely for history not that much for pair reviewing [14:17:54] i would love to hashar, last time I tried to give into "The Gerrit" I got Really close to achieving what I wanted [14:18:09] yeah, that's the primary reason for self-review [14:18:20] that make sense [14:18:28] somehow [14:18:37] but we're hoping to start working as a team on some projects [14:18:42] will have to keep it mind to sell you the Gerrit / Test / review concept [14:18:43] hehe [14:18:55] oh dude, I'm 100% sold on the concept [14:19:04] that takes a bunch of time to switch to that kind of concept :/ [14:19:13] and you need to have some free time to invest [14:19:18] nono, I'm serious, I Love the concept [14:19:26] I just never have time to debug it with gerrit [14:20:33] we will have to pair over google hangout one day [14:20:35] milimetric: i've mostly worked out gerrit stuff these days to the point where I don't usually have problems with it [14:20:45] or if I do I just make new commits and abandon old ones [14:20:50] next time you have problems lemme know and maybe we can figure it out [14:21:00] next time you are tempted to push directly you could dump the problem into a Bugzilla instead. it's valid to even just ask for an "enhancement" [14:21:20] when we switched from svn to git/gerrit, I spent countless hours doing screen sharing with random volunteers/ staff . [14:21:25] I completely agree with mutante. [14:21:29] thanks guys, I feel really bad doing that but I'll consider it next time [14:21:32] screen sharing definitely helped. [14:21:44] milimetric: whenever you get some free time, ping me :-] [14:21:57] milimetric: we can hangout for half an hour about it and find a way to solve what ever is blocking [14:22:03] i will be VERY happy to assist. [14:22:30] ok, I promise I won't push without consulting you next hashar [14:22:31] and thank you [14:22:44] milimetric: keep pushing [14:22:56] i mean git push directly [14:23:00] milimetric: come to me whenever you have time / mood / whatever :-) I am not willing to force anything. [14:25:54] meanwhile the pending patch, you can override jenkins [14:25:57] and just merge it anyways [14:26:11] right hashar [14:30:31] hey milimetric, i'm working on this python hive partitioner again, using some insight and code from luigi [14:30:32] :-D \o/ [14:30:38] Thanks hashar [14:30:51] i'm sure there are bunches of things I could do smarter [14:31:02] would you give me a code review and give me tips? [14:31:05] hangout? [14:31:11] morning guys [14:31:45] morning [14:32:27] morning! [14:32:54] one sec ottomata, i'll be right there [14:32:56] !card 1168 [14:33:20] !card is https://mingle.corp.wikimedia.org/projects/analytics/cards/$1 [14:33:20] You are not authorized to perform this, sorry [14:33:42] lol what? [14:35:59] https://meta.wikimedia.org/wiki/Wm-bot#User_access [14:36:26] !card is https://mingle.corp.wikimedia.org/projects/analytics/cards/$1 [14:36:26] You are not authorized to perform this, sorry [14:37:01] https://meta.wikimedia.org/wiki/Wm-bot#.40trusted [14:37:12] @trusted [14:37:12] I trust: jeremyb!.*@wikimedia/jeremyb (2admin), [14:37:43] mutante: Is wm-bot2 different from wm-bot? [14:37:58] mutante: hasha-r just trusted me, didn't he? [14:38:14] @trustadd *@wikimedia/qchris trusted [14:38:14] You are not authorized to perform this, sorry [14:38:56] qchris_wmf: no, i think it's the same thing, when i just had a bot join my new channel i got wm-bot3 [14:39:12] qchris_wmf: and the trust is per channel, i think only JeremyB can add more people [14:39:20] and there is also #wmbot [14:39:45] eh,#wm-bot [14:40:14] i'd try to ask there (well, or JeremyB) [14:40:49] Thanks. [14:41:15] !card is https://mingle.corp.wikimedia.org/projects/analytics/cards/$1 [14:41:16] You are not authorized to perform this, sorry [14:41:44] !card 1168 [14:43:10] hi drdee [14:43:22] @trusted [14:43:22] I trust: jeremyb!.*@wikimedia/jeremyb (2admin), [14:44:39] @trusted [14:44:39] I trust: jeremyb!.*@wikimedia/jeremyb (2admin), [14:44:58] what's @trusted ? [14:45:01] qchris: ? [14:45:13] average: It's for wm-bot [14:45:22] average: https://meta.wikimedia.org/wiki/Wm-bot [14:46:16] @trust .*@wikimedia/qchris trusted [14:46:23] @trustadd .*@wikimedia/qchris trusted [14:46:23] Successfuly added .*@wikimedia/qchris [14:46:27] !card is https://mingle.corp.wikimedia.org/projects/analytics/cards/$1 [14:46:27] Key was added [14:46:34] T13: \o/ [14:46:35] Stupid smart phone... [14:46:37] T13: Thanks [14:46:42] Np [14:46:51] !card 1168 [14:46:51] https://mingle.corp.wikimedia.org/projects/analytics/cards/1168 [14:52:09] * jeremyb looks up [14:52:37] * jeremyb lowercases mutante :) [14:52:48] @whoami [14:52:48] You are root identified by name .*@wikimedia/jeremyb [14:53:25] jeremyb: heh, thanks [14:53:28] @trusted [14:53:28] I trust: jeremyb!.*@wikimedia/jeremyb (2admin), .*@wikimedia/qchris (2trusted), [14:54:18] jeremyb: fyi, #wikimedia-planet new channel that uses wm-bot and the RSS feature ;) [14:54:31] qchris: did you remove your name from the RFC? [14:54:36] well, at least i got some updates from the Russian one so far, but i added them all [14:54:40] mutante: i saw! [14:54:48] in #wikimedia [14:55:00] ah, yea:) [14:55:54] and all the docs about adding a new planet language still talk about svn , cough, at least i left a message askingfor help with the cleanup [14:56:18] but i'll stop talking offtopic now:) cya [14:57:03] guys check https://www.mediawiki.org/w/index.php?title=Analytics/Hypercube&action=history [14:57:08] there is quite some activity [14:57:12] really encouraging [15:13:21] mutant, still here? [15:13:43] i am trying to add an RSS feed to the analytics wiki pages to make mingle stuff more visible [15:13:47] i get the following error: [15:13:48] Extension:RSS -- Error: "https://mingle.corp.wikimedia.org/projects/analytics/feeds/8z8k6vUfniLmWc2qGe6GMQsxBujvKRKuybLd8mdTbMFHwGfLH3oxK*MU0E8zM6go.atom" is not in the whitelist of allowed feeds. The allowed feeds are as follows: http://blog.wikimedia.org/feed/ and https://git.wikimedia.org/feed/mediawiki/extensions/Translate.git. [15:13:51] mutante: ^^ [15:14:01] where can I whitelist the mingle feed? [15:17:18] drdee: this is on wikitech? [15:17:41] mediawikiwiki [15:17:46] i found the config [15:17:55] (https://noc.wikimedia.org/conf/highlight.php?file=InitialiseSettings.php) [15:18:05] is it possible to whitelist the domain but not the exact path? [15:18:34] for example can we whitelist "https://mingle.corp.wikimedia.org/projects/" [15:18:59] mutante: ^^ [15:19:55] drdee: i see this the first time :p [15:20:08] k [15:20:12] i will read the source code [15:20:14] i just wrote the very first revision of that extension [15:20:29] but it didnt have any whitelists [15:21:36] drdee: but i got something else for you if you want RSS :) [15:21:44] shoot [15:22:10] @rss+ mingle https://mingle.corp.wikimedia.org/projects/analytics/feeds/8z8k6vUfniLmWc2qGe6GMQsxBujvKRKuybLd8mdTbMFHwGfLH3oxK*MU0E8zM6go.atom [15:22:10] Permission denied [15:22:14] @rss-on [15:22:14] Permission denied [15:22:16] :) [15:22:30] https://meta.wikimedia.org/wiki/Wm-bot#.40rss.2B [15:22:34] qchris: can now do it:) [15:22:48] you'll have feed output here on channel [15:23:18] that could be neat or it could flood the channel [15:23:28] even @rss-setstyle [15:23:37] qchris, milimetric, ottoman: shall we try that? [15:23:47] enable the mingle rss feed into this channel? [15:24:02] ok... [15:24:20] you can turn it on and off ..like during meetings [15:24:29] it doesn't forget the URL [15:30:04] Sorry. Back. Reading backscroll [15:31:21] drdee, mutante: @rss-on requires admin rights. I am only trusted. [15:32:34] @rss+ mingle https://mingle.corp.wikimedia.org/projects/analytics/feeds/8z8k6vUfniLmWc2qGe6GMQsxBujvKRKuybLd8mdTbMFHwGfLH3oxK*MU0E8zM6go.atom [15:32:34] Item was inserted to feed [15:32:38] :p summons T13 [15:32:50] oh:) [15:32:51] mutante can i also have admin rights for this bot ? [15:32:59] @trusted [15:32:59] I trust: jeremyb!.*@wikimedia/jeremyb (2admin), .*@wikimedia/qchris (2trusted), [15:33:12] T13 needs to add you or jeremyb [15:33:27] T13 could you add me as admin for wm-bot2? [15:34:03] @trusted [15:34:04] I trust: jeremyb!.*@wikimedia/jeremyb (2admin), .*@wikimedia/qchris (2trusted), [15:34:34] @trustadd .*@wikimedia/qchris admin [15:34:34] Unable to add user because this user is already in a list [15:34:44] @trustdel .*@wikimedia/qchris [15:34:44] User was deleted from access list [15:34:48] @trustadd .*@wikimedia/qchris admin [15:34:48] Successfuly added .*@wikimedia/qchris [15:34:54] Thanks T13 [15:34:59] me too :) [15:35:01] :) [15:35:15] mutant and qchris can vouch for me [15:35:23] @rss-on [15:35:23] Rss feed has been enabled on channel [15:35:36] Can you take care of drdee since they are a pain to do from mobile [15:35:47] Ok. [15:35:49] Thanks T13 [15:36:11] @trustadd .*@wikimedia/drdee admin [15:36:11] Successfuly added .*@wikimedia/drdee [15:36:30] T13: thanks [15:36:31] Just use .*/mediawiki/drdee [15:36:43] Isn't he a mediawiki cloak? [15:36:55] yes [15:36:55] @trustdel .*@wikimedia/drdee admin [15:36:55] User was deleted from access list [15:37:40] @trustadd .*@mediawiki/drdee [15:37:40] Wrong number of parameters, go fix it - example @trustadd regex (admin|trusted) [15:37:44] @trustadd .*@mediawiki/drdee admin [15:37:44] Successfuly added .*@mediawiki/drdee [15:37:55] drdee: Can you try? [15:37:56] @trusted [15:37:56] I trust: jeremyb!.*@wikimedia/jeremyb (2admin), .*@wikimedia/qchris (2admin), .*@mediawiki/drdee (2admin), [15:38:00] ty [15:38:08] Thanks T13! [15:38:13] Np [16:03:20] drdee_: you should try @whoami [16:03:36] @whoami [16:03:36] You are admin identified by name .*@mediawiki/drdee [16:03:42] sweet [17:11:08] ottomata: Look at the filesizes on stat1002 in ll /a/squid/archive/mobile/mobile-sampled-100.tsv.log-201307* [17:13:28] hmm, ah ok [17:17:14] (CR) Jdlrobson: [C: 2] Story 1253: Limn graphs for mobile Quarterly goal - mobile registrants -> active editors [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/88265 (owner: JGonera) [17:17:26] (CR) Jdlrobson: [V: 2] "oh yeh no jenkins" [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/88265 (owner: JGonera) [17:29:32] while not team.satiated: [17:29:42] time.sleep(60) [17:30:15] ottomata.ping(milimetric) [17:30:21] qchris.ping(milimetric) [17:30:25] drdee.ping(milimetric) [17:30:29] average.ping(milimetric) [17:30:38] # please execute the pseudocode above :) [17:56:14] i am in the process of satiating! [18:06:03] self.ping(milimetric) [18:07:03] nice [18:07:06] 3 more :) [18:10:43] * qchris_away is satiated [18:11:02] self.ping(milimetric) [18:11:37] this is like a barrier, or a petri net transition :)) [18:12:05] drdee, you ready to hang out? [18:12:05] :-) [18:15:49] hey ottomata/drdee/average got time for the batcave? [18:16:07] ok [18:37:04] self.ping(drdee) [19:08:54] milimetric: real quick explanation of the camus thing I was talking about [19:09:02] sure [19:09:18] camus is built to load data from kafka, examine a timestamp, and then store in hdfs bucket in that timestamp [19:09:23] but, each of those tasks are modular [19:09:27] you can plug in whatever classes you want [19:09:41] the timestamp and bucketing bit is the part that it abstracts [19:09:42] so [19:10:03] we could write a MessageDecoder class that knows how to import from dumps, and set the timestamp based on the filename [19:10:24] and a RecordWriter class that knows how to write the unzipped file back into hdfs [19:10:26] gotcha [19:10:35] so why souldn't we do it now? [19:10:36] The marketing department made us put that there [19:11:06] i think the clunky part might be the MessageDecoder timestamp thing, it is built to examine every record [19:11:06] is it hard to write a MessageDecorder & RecordWriter? [19:11:15] and direct that record to the proper location [19:11:26] in our case the Message and Record is an hourly dump file [19:11:28] not a line [19:11:31] so we'd be trying to move it from the row level to the file level [19:11:32] but ithink it is possible [19:11:34] yeah [19:11:39] ok, I see [19:11:54] it'd be interesting to try [19:11:55] then let's just use python now, and see how nasty it is [19:12:03] yeah i think that's what we should do too [19:27:32] ah milimetric [19:27:33] FAILED: SemanticException [Error 10019]: Line 7:37 OR not supported in JOIN currently '16' [19:27:39] boo [19:27:39] hehe [19:27:41] trying to do [19:27:44] and ((G.DAY BETWEEN and G.HOUR BETWEEN 16 and 23) or (G.DAY = 08 and G.HOUR BETWEEN 0 and 16)) [19:27:54] to get just the 48 hours period that I have [19:28:08] ah yea [19:28:13] that's why i had to make the query a bit nastier [19:28:14] It must be a hardware problem [19:28:17] OR is not supported [19:28:33] hm [19:28:35] you can just left join twice [19:28:55] are you inner joining or outer? [19:29:01] http://codeshare.io/aITwc [19:30:03] haha [19:30:53] what's the first G.DAY supposed to be, there's a random BETWEEN there [19:31:21] *24 hour period [19:32:32] hmmi hsoul dhave more data than this [19:32:33] hm [19:33:02] ok, I fixed the query to do what you wanted but why do you care about restricting the day and hour anyway? [19:33:02] I couldn't find any examples of how that can be done anywhere else in the project [19:33:12] i don't want the borders [19:33:15] you can just say where G.YEAR > 2001 in the first version [19:33:17] oh [19:33:25] i could say not [19:33:25] hm [19:33:34] ok, without trying it, I am not sure but what I wrote should work [19:34:02] oh wait, no [19:34:07] where NOT (day = 07 and HOUR = 15) and NOT (day = 08 and hour > 16) [19:34:13] ? [19:35:24] geez [19:35:24] ha [19:36:34] :) yeah, funny, that might work [19:36:51] if not, i just fixed the double join version [19:40:34] you are a crazy awesome man [19:41:40] ottomata should https://www.mediawiki.org/wiki/Analytics/Infrastructure/Stat1 be moved to wikitech? [19:42:18] naw, deleted? [19:42:41] we rarely delete wiki pages, why? [19:42:41] That behaviour is in the original specification [19:42:51] add a note saying its old? [19:43:23] ok [19:51:22] DarTar: are you aware of this page https://www.mediawiki.org/wiki/Analytics/Key_metrics_explained_laymen%27s_terms. [19:51:54] ottomata: what about this page: https://www.mediawiki.org/wiki/Analytics/Kraken/Firehose [19:52:50] meh? [19:58:43] what's meh? [20:25:48] drdee: no, interesting, but that link points to a non existing page [20:26:23] uhhhhh [20:26:28] i copy pasted it [20:27:15] milimetric: i forget, should the count query always return 0, or should it return 2 [20:27:17] for the borders? [20:27:46] well, i saw you were using the version with the L(esser) and G(reater) combined [20:27:58] with that, you'd have to do count(*)-2 to get 0 [20:28:30] basically you have to select count(*) - NumberOfLeftJoins to get 0 [20:28:40] which version are you using? [20:34:42] http://codeshare.io/aITwc [20:34:43] did that one [20:34:47] the last in that file [20:34:50] didn't run your join [20:35:15] I realized that I was looking at the wrong topic, which is why I thought i 48 hours but only saw 24 [20:35:16] I'm still working on that as we speak [20:35:18] varnish1 has more data [20:35:36] I got bored and set up my host forwarding [20:35:44] hadoop/hive access, we can haz :D [20:35:50] wooo [20:36:04] although I have some table structure concerns already. because, well, I'm me ;p [20:36:20] so, milimetric, I would try your join query [20:36:25] but I need 3 day conditions, [20:36:32] I want all of 08 [20:36:45] Ironholds: whaaats up? [20:36:46] :) [20:36:49] you got some hive? [20:37:06] he's got hive beef it sounds like [20:37:07] :) [20:37:29] ok so ottomata perhaps you can tell me what you're trying to do [20:37:44] ok, i want to just select against 48 hours of data [20:37:49] check it for missing seqs [20:37:50] maybe you can just run the original query and we can figure out what we'd need to subtract from the result [20:37:56] i don't want any borders [20:38:09] do we know ahead of time how many borders there *should* be? [20:38:19] i'm saying borders of the camus imports [20:38:22] i don't want the first hour [20:38:25] and i don' want the last hour [20:38:40] I ran my query at the bottom of the page there [20:38:42] you don't want to check the hour at all [20:38:46] and got 3 missing seq runs from a single host [20:39:03] from amsterdam [20:39:21] ottomata: indeed have hive, which is nice :) [20:41:21] milimetric: i'm going torun your query to output the actual sequence numbers [20:42:15] yeah, i think the main problem is you can't say between day x, hour y and day x1, hour y1 [20:42:24] because day and hour are separate things [20:42:26] hm... [20:42:48] milimetric: that's my beef too [20:43:04] I mean, I hate mediawiki timestamps, but mostly because they're not POSIX compliant ;p [20:43:10] having a single time field is good times (aha) [20:43:18] well, no, these fields were never meant for querying [20:43:34] they correspond to hive partitions and should really be left alone [20:43:39] ottomata: here's what we should do [20:43:54] when we import this data, we should concat the y-m-d h together [20:44:00] and use that to query [20:44:20] Ironholds: we do have a timestamp [20:44:39] oh? [20:44:43] yeah, dt field [20:44:45] its a string [20:44:47] we could use that [20:44:51] ah, cool [20:44:54] let's [20:45:07] but, the thing about the partitions is that it limits where hive will look for the data [20:45:09] so, we can use both [20:45:11] and then leave the G.YEAR type things set to generic > 2001 or whatever just to make hive happy [20:45:21] where day BETWEEN 07 and 09 and dt BETWEEN … and ... [20:45:22] yeah, that's fine [20:45:31]