[01:08:31] Analytics-Engineering, operations: Install enchant on stat1002 and stat1003 - https://phabricator.wikimedia.org/T99030#1284064 (Dzahn) We would install packages via puppet, not manually. So this needs a puppet code change with a list of all the package names. [01:11:46] Analytics-Engineering, operations: Install enchant on stat1002 and stat1003 - https://phabricator.wikimedia.org/T99030#1284065 (Dzahn) There is no common role that is used on both stat1002 and stat1003. So i'm wondering where this should be added. Is it definitely needed on both of them? stat1003 is only... [01:46:20] Analytics-Engineering, operations, Patch-For-Review: Install enchant on stat1002 and stat1003 - https://phabricator.wikimedia.org/T99030#1284074 (Dzahn) @halfak here's the list i get on stat1002: https://gerrit.wikimedia.org/r/#/c/210846/1/modules/statistics/manifests/enchant.pp including that in the... [01:46:47] Analytics-Engineering, operations, Patch-For-Review: Install enchant on stat1002 and stat1003 - https://phabricator.wikimedia.org/T99030#1284075 (Dzahn) a:Dzahn [03:00:50] Analytics-Engineering, operations, Patch-For-Review: Install enchant on stat1002 and stat1003 - https://phabricator.wikimedia.org/T99030#1284110 (Halfak) I was planning to use them on stat1003, but stat1002 fills a similar use-case for me. It's hard for me to answer this question, but if I were to choo... [03:11:44] Analytics-Tech-community-metrics: Community Bonding evaluation for "Allowing contributors to update their own details in tech metrics" - https://phabricator.wikimedia.org/T98045#1284126 (NiharikaKohli) @Sarvesh.onlyme Are you working on this? Have you had any meetings so far? Have you requested a Phabricator... [03:12:58] Analytics-Tech-community-metrics, ECT-May-2015, Epic, Google-Summer-of-Code-2015: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#1284132 (NiharikaKohli) [03:12:59] Analytics-Tech-community-metrics: Community Bonding evaluation for "Allowing contributors to update their own details in tech metrics" - https://phabricator.wikimedia.org/T98045#1284133 (NiharikaKohli) [10:43:52] Analytics-Tech-community-metrics, ECT-May-2015, Epic, Google-Summer-of-Code-2015: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#1284635 (Sarvesh.onlyme) [10:56:33] around? [10:56:47] hehe, tried to ping milimetric [11:34:32] Analytics-Tech-community-metrics: Community Bonding evaluation for "Allowing contributors to update their own details in tech metrics" - https://phabricator.wikimedia.org/T98045#1284682 (Sarvesh.onlyme) [11:38:55] Analytics, MediaWiki-extensions-ContentTranslation: Limn language dashboard: eswiki graph is wrong/stuck - https://phabricator.wikimedia.org/T99074#1284691 (KartikMistry) NEW [11:45:10] Analytics-Tech-community-metrics: Community Bonding evaluation for "Allowing contributors to update their own details in tech metrics" - https://phabricator.wikimedia.org/T98045#1284707 (Sarvesh.onlyme) [11:54:05] (CR) Joal: [C: -1] "Some changes, let's discuss when you wish :)" (11 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/210632 (owner: Madhuvishy) [14:37:59] Analytics-Kanban, VisualEditor, Editing Department 2014/15 Q4 blockers: Schema:Edit seems to incorrectly set users as anonymous {lion} - https://phabricator.wikimedia.org/T92596#1284878 (Milimetric) Thanks Halfak, the numbers are crazy, but crazy good! :) [14:51:25] (PS4) Madhuvishy: Add generic oozie component for emailing data [analytics/refinery] - https://gerrit.wikimedia.org/r/210632 [14:53:53] (PS5) Madhuvishy: Add generic oozie component for emailing data [analytics/refinery] - https://gerrit.wikimedia.org/r/210632 [14:56:44] (PS6) Madhuvishy: Add generic oozie component for emailing data [analytics/refinery] - https://gerrit.wikimedia.org/r/210632 [14:58:33] (PS7) Madhuvishy: Add generic oozie component for emailing data [analytics/refinery] - https://gerrit.wikimedia.org/r/210632 [15:24:30] Analytics-Kanban, Need-volunteer: Top Articles ad-hoc Report - https://phabricator.wikimedia.org/T99083#1284977 (Milimetric) NEW [16:39:45] mforns: when is the showcase? [16:40:09] madhuvishy, in 1h20m [16:40:27] madhuvishy, do you want to present the validate again task? [16:41:27] mforns: I'm not sure. I've not seen how the showcase works. [16:42:39] madhuvishy, I'll guide the showcase: welcome everyone, state the agenda, give an overview on the tasks, etc. Basically follow the slides [16:43:17] madhuvishy, and then I will showcase the first task: screenshare my desktop and play with wikimetrics staging to show the feature [16:43:47] madhuvishy, I'm preparing right now some examples to be able to show the tasks [16:44:11] mforns: alright. I can do that for validate again. [16:44:17] madhuvishy, ok [16:44:24] the code is deployed now in staging [16:44:39] mforns: okay great. [16:44:59] madhuvishy, so if you want to prepare a cohort for your example [16:45:10] madhuvishy, let me know if you need help [16:46:21] milimetric: Thank you for opening my eyes on how easy it could be to pick dimensions :) [16:46:39] everything's easy when people work together [16:46:45] it's within our own minds that we get lost [16:47:00] Yeah, I know the feeling :) [16:47:14] Anyway, thanks :) [16:47:28] Analytics, Research-and-Data: Referrer data for en:Glitter for shareafact test - https://phabricator.wikimedia.org/T93270#1285200 (ggellerman) [16:51:10] ggellerman: I'm on the way to office. Will be a few minutes late to the retrospective. [16:52:03] k...thanks for letting me know [17:00:44] am I in the alternate batcave? [17:15:55] madhuvishy, yt? [17:20:09] mforns: yeah [17:20:23] madhuvishy, did you make it to the office? [17:20:29] mforns: yup :) [17:20:59] ok, let me know if you need help with setting up the example for the showcase :] [17:21:39] hellOOoOoOo [17:21:48] Hi ottomata :) [17:22:17] madhuvishy, by the way, I found an easy way to create an example for you... :] [17:22:29] mforns: oh? [17:22:57] madhuvishy, create a cohort with your mediawiki username and a project that you never visited, i.e.: ruwiki? [17:23:18] madhuvishy, it will not validate, because the user does not exist in the db [17:23:29] madhuvishy, then access ru.wikipedia.org [17:24:02] madhuvishy, and after *some* time, you can validate again and it should validate [17:24:23] mforns: hmmm, alright. i'll try that [17:24:32] madhuvishy, by accessing a new wiki with your username, you automatically create a new user for that wiki [17:25:48] mforns: aah, thanks i din't know that. [17:37:09] Hey ottomata, do we reschedule today'e meeting ? [17:38:28] i think so, sure, sounds like some researchers won't be there, i don't have an agenda [17:38:36] ok :) [17:44:05] joal: I've made all the changes to the oozie module. Thanks for the comments :) They made perfect sense. Feel free to review it again when you have time. [17:44:14] Hey Madhu [17:44:19] madhuvishy: --^ [17:44:20] :) [17:44:25] joal: also i wanted to make sure that the name of the module was okay. [17:44:38] I was trying to run the module, to double check [17:44:50] Got an error from file too big [17:45:00] joal: yeah - i got that too [17:45:03] We can only send 2048 bytes of data ... [17:45:23] I really wonder if this thing will be usefull knowing that limitiation :( [17:45:30] joal: i tried with a small one and it was fine. is that something we should work around? [17:45:40] Anyway --> the code looks great, [17:46:01] You should add andrew (ottomata) as a reviewer, and then we'll merge :) [17:46:10] Analytics-Engineering, operations, Patch-For-Review: Install enchant on stat1002 and stat1003 - https://phabricator.wikimedia.org/T99030#1285421 (Dzahn) >>! In T99030#1284110, @Halfak wrote: > Might it make sense to create a task for a role common to stat machines? It seems that this is a common proble... [17:46:26] I think the only way to overcome is to rewrite emails sending through shell [17:46:27] hellooooo! [17:46:32] Hulllo :) [17:46:33] i am still checking so many emailllls [17:46:37] hehe :) [17:46:44] joal: hmmm, alright - who sets the 2048 byte restriction? [17:47:03] oozie settings [17:47:26] joal: can we increase it? should we? [17:47:32] madhuvishy: I don't knowreally what's best, to change the settings or to change the apporahc :( [17:47:47] We should discuss that with ottomata [17:47:52] madhuvishy: --^ [17:48:11] joal: right - i dont have much background on why this module was needed - so i'm not sure. Sure! [17:48:22] may be when he's caught his breath :) [17:48:27] madhuvishy: Actually I don't have neither, so let'sd ask :) [17:48:33] Analytics-Engineering, operations, Patch-For-Review: Install enchant on stat1002 and stat1003 - https://phabricator.wikimedia.org/T99030#1285430 (Halfak) I wonder if statistics::cruncher would be a good common role between machines. To me, it sounds like an appropriate name for use-cases common to both... [17:48:58] Yeah, I know email reading is one of his prevered task ;) [17:49:47] wuuuuut are we talking about? [17:50:24] sending emails with data using oozie [17:50:28] ottomata: --^ [17:50:50] oh! [17:50:55] nuria knows much more about that [17:51:08] yeah, I know that as well ... [17:52:04] I guess best is to wait for Nuria then ? [17:52:31] maybe, i will investigate shortlyyyYy [17:54:50] Analytics-Engineering, operations, Patch-For-Review: Install enchant on stat1002 and stat1003 - https://phabricator.wikimedia.org/T99030#1285446 (Dzahn) If we would put the existing statistics::cruncher role on stat1002 now we would effectively change that server and install all the things that role ins... [18:04:45] Analytics-Engineering, operations, Patch-For-Review: Install enchant on stat1002 and stat1003 - https://phabricator.wikimedia.org/T99030#1285453 (Ottomata) Note that both role::statistics::cruncher and role::statistics::private include the module class statistics::compute. I would just ensure these pac... [18:19:54] hey mforms. can you send me a screenshot of the "re-run report" button, so I can include that in my email? [18:20:11] whoops *mforns ^^ [18:20:42] J-Mo, sure! do you mind if I do that after our retrospective? [18:20:53] not at all. [18:21:26] ok, madhuvishy, joal, when you are done with retro, can talk about whatever (oozie emails?) [18:26:51] Analytics-Cluster, operations: Kafka Broker disk usage is imbalanced - https://phabricator.wikimedia.org/T99105#1285537 (Ottomata) NEW [18:30:47] Analytics-Kanban: Quarterly tracked animals - https://phabricator.wikimedia.org/T99108#1285565 (ggellerman) NEW a:kevinator [18:40:25] mforns: we also need to finish code review of my task(after my updates based on your comments) and merge it [18:40:49] madhuvishy, right! thanks for the heads up [18:41:05] madhuvishy, going to do it right now [18:41:22] mforns: great thanks [18:41:32] madhuvishy, I've seen you left the validate again button on the left side, that's ok [18:42:08] mforns: aah did you want me to move it? the buttons had no space between them - which i fixed [18:42:20] i thought that was the problem [18:44:35] mforns: aah i think i missed the comment about moving it to the other end - sorry about that! [18:45:30] madhuvishy, you can change it if you like, but I was really nitpicky in that, so you choose [18:46:20] mforns: hmmm, i'm okay with where it is [18:46:31] madhuvishy, cool! [18:47:04] madhuvishy, I've also seen the comments in the migration, and in fact I run the queries in staging and they went fine, so yes [18:47:27] mforns: great! [18:56:39] madhuvishy, one question, there are some changes in the last patch that come from other wikimetrics changes, right? [18:57:03] mforns: yeah [18:57:22] mforns: that should be after my rebase with master [18:57:56] madhuvishy, aha, did you need some manual conflict solving when rebasing? [18:58:33] mforns: no. it went fine. i only needed to update my migration and mark yours as the downgrade [18:59:16] madhuvishy, ok, cool [18:59:51] (CR) Mforns: [C: 2 V: 2] "LGTM" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/206346 (https://phabricator.wikimedia.org/T78339) (owner: Madhuvishy) [19:00:30] madhuvishy, merged it :] [19:00:59] mforns: yay :D [19:01:22] ottomata: joal and I were talking about this patch I made - https://gerrit.wikimedia.org/r/#/c/210632/ for this task - https://phabricator.wikimedia.org/T88433. It works fine - but oozie says we can only send 2048 bytes of data. We were wondering if it's useful despite this limitation - or if there's need to work around it somehow. [19:06:32] milimetric: factor 50 when aggregating an hour [19:06:34] Soun [19:06:38] sounds great [19:06:44] hey hey [19:06:47] that's not shabby [19:06:50] But not queriable straight from impala though [19:06:57] hm [19:07:01] still to be used using hive [19:07:18] ~ 500 Mo ? hour [19:07:26] I wonder if it would still be useful to someone at the daily level [19:07:30] probably :) [19:07:58] but this seems like a good second refinement after the refined pageviews [19:08:03] or aggregated among other dimensions [19:08:10] from here we can get daily and the stuff you need, and my adhoc report [19:08:10] Completely agreed [19:08:18] :D [19:08:21] T [19:08:31] thanks a lot alot :) [19:09:09] cool, I'm really excited, I think we're on the brink of really great releases [19:09:25] About articles extraction, do we have some specs ? [19:09:30] milimetric: yup, that's cool ! [19:09:45] madhuvishy: joal, no clue about the 2048 [19:10:02] what's happening, the email that gets sent isn't longer than 2048 chars? [19:10:12] or, is it the capture-output part maybe? [19:10:20] ottomata: no, if the file is bigger - it throws an error [19:10:36] ottomata: oozie.action.max.output.data [19:11:23] oh, we just need to increase that? [19:11:38] Either we increase that, or we take a different approach [19:11:51] joal: I don't think we have a spec, let me check out some samples from the logs, see if I can think it out [19:11:53] Increasing that makes sense, but only for small enough reports [19:12:02] milimetric: Nice :) [19:12:35] milimetric: email me your findings, I'll have a code review for tomorrow evening on that hting :) [19:12:38] (CR) Mforns: [C: 2 V: 2] "LGTM" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/206343 (owner: Milimetric) [19:12:58] joal: will do [19:13:31] joal: i don't think we shoudl be emailing out large reports from this [19:13:38] this is more for operational use [19:13:44] aggreed [19:13:45] did you have a different approach in mind? [19:13:50] (CR) Ottomata: Add generic oozie component for emailing data (3 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/210632 (owner: Madhuvishy) [19:13:51] But do we even want to send reports ? [19:14:03] My view is that we want to send path, not reportts [19:14:09] ottomata: --^ [19:14:26] ? [19:14:28] send paths? [19:14:35] And in that case, oozie email action is fine without global utility [19:14:42] Yeah, path of the report [19:14:52] of the job results, you mean? [19:14:57] yes [19:15:09] aye, yeah, i don't think this email would be used for end users [19:15:14] Something like: You got a new report here ! [19:15:15] it would be more for making sure data is run and looks good [19:15:22] similar to the webrequest dump report thing [19:15:32] job is run* [19:15:44] could be (I still don't have those emails by the way) [19:16:10] Or we could send a hue link for the files ? easier to access for end users [19:16:47] My point is: the email sending function with the action passing data limitiation is not a good approach in my opinion if we want to send reports to users [19:18:10] My time, we can talk about that tomorrow in standup :) [19:18:16] Talk to you all tomorrow [19:19:11] oook [19:19:12] laters [19:19:20] byeee [19:20:19] okay gnite! [19:21:22] milimetric, ready? [19:21:34] milimetric, do you have 10 minutes today to pair-deploy wikimetrics changes? As there are 6 significant changesets being deployed together, I'd like to make sure that I don't mess up :] [19:22:59] milimetric, I see you have previous engagements, talk to you later! [19:30:27] yurik: yes [19:30:43] mforns_snack: I was going to do it after my talk with madhu [19:30:50] Hey, I'm having trouble ssh-ing into stat1003. I can ssh to bast1001 correctly, but when I try to ssh to stat1003 from there, I get a "Permission denied (publickey)" error. Any thoughts? My ssh config is at https://phabricator.wikimedia.org/P648. [19:30:50] don't worry, we'll figure it out :) [19:31:43] neilpquinn: I've got meetings until 2 your time, I can help after that (re-ping me if I forget about you) [19:31:57] but did you request access to the researchers group for stat1003? [19:32:00] from ops-access-requests? [19:32:19] yurik: I'm ready, can I have 15 minutes? [19:32:25] I mean, 15 minutes of the presentation? [19:32:34] millimetric: sounds good [19:32:45] milimetric, sure [19:32:51] you can have even more :) [19:33:31] millimetric: but yes, I was added to researchers: https://phabricator.wikimedia.org/T97746 [19:34:19] neilpquinn: I had some trouble with similar stuff before and ops recommended i try ssh-add .ssh/ [19:35:03] neilpquinn: and then try to ssh. [19:37:43] madhuvishy, millimetric: hey! that worked! thanks :) [19:38:06] neilpquinn: great! :) [19:52:24] thanks madhuvishy : ) [19:52:46] :) [20:23:37] Analytics-Engineering, operations, Patch-For-Review: Install enchant on stat1002 and stat1003 - https://phabricator.wikimedia.org/T99030#1285868 (Dzahn) @ottomata thanks, added to statistics::compute. also, used "ensure_packages" like for the other existing ones @halfak done, the packages have been ins... [20:27:43] Analytics-Engineering, operations, Patch-For-Review: Install enchant on stat1002 and stat1003 - https://phabricator.wikimedia.org/T99030#1285873 (Dzahn) Open>Resolved [21:06:23] madhuvishy: so, wanna deploy soon? [21:18:19] milimetric, let's deploy [21:20:09] mforns: you're still around! not allowed! [21:20:17] xD [21:20:20] I can do it with madhu, no? [21:20:33] I'll make sure nothing goes wrong, no worries [21:21:07] milimetric, mmm I feel like responsible for that, and would like to empty the back of my head to continue with your review tomorrow :] [21:22:00] mforns: ok, sorry, if it's an empty head operation, I support it [21:22:01] milimetric, by the way, great presentation! [21:22:03] let's batcave! [21:22:07] sure [21:22:13] madhuvishy: fyi we're deploying now [21:23:54] milimetric: i'm running an inner join query (constrained to one day with a an upper bound left side around 237k records) on the el db defined in the system .cnf from stat1003, lemme know if i need to back out, and i'll constrain to an hour and project (however imperfect that may be due to geo-related events) [21:25:15] milimetric: okay :) [21:25:31] madhuvishy: you're welcome to join us in the batcave if you wanna help [21:25:40] dr0ptp4kt: I'm doing a deployment, be with you in a bit [21:26:00] milimetric: shoot, i have to change floors and am going to loose my connection i'm sure. i'm just going to kill this query, just wanted to give you a heads up...should be back on and re-attempting a bit later [21:26:30] milimetric: ha, well ^c managed to remove me from the top of top :) [21:26:32] milimetric: ttyl [23:07:17] halfak: yuvipanda headed to your side [23:13:46] Analytics-Engineering: Analytics-Engineering availability to support VE A/B test during hackathon - https://phabricator.wikimedia.org/T99014#1286454 (Krenair) [23:13:57] Analytics-Engineering, Hackathon-Lyon-2015: Analytics-Engineering availability to support VE A/B test during hackathon - https://phabricator.wikimedia.org/T99014#1283579 (Krenair) [23:19:06] Analytics-Kanban, Need-volunteer: Top Articles ad-hoc Report - https://phabricator.wikimedia.org/T99083#1286464 (Milimetric)