[00:55:20] <wikibugs>	 Analytics, VisualEditor, Wikimedia-Site-requests, Database, VisualEditor 2014/15 Q4 blockers: Backfill rctag data for VisualEditor from 2015-05-23–2015-05-28 - https://phabricator.wikimedia.org/T101270#1394944 (Neil_P._Quinn_WMF) @Jdforrester-WMF, I just queried enwiki's `change_tag` and `tag_s...
[07:15:46] <wikibugs>	 Analytics-Tech-community-metrics, ECT-June-2015, Epic, Google-Summer-of-Code-2015: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#1395470 (NiharikaKohli)
[08:21:54] <wikibugs>	 Analytics-Tech-community-metrics, ECT-June-2015: Active changeset *authors* and changeset *reviewers* per month - https://phabricator.wikimedia.org/T97717#1395645 (Dicortazar) Viz is ready to go at http://korma.wmflabs.org/browser/scr.html  You'll see that there are two new charts. One of them contains th...
[08:22:24] <wikibugs>	 Analytics-Tech-community-metrics, ECT-June-2015: Gerrit changes reviewed per month (on scr.html) - https://phabricator.wikimedia.org/T97716#1395647 (Dicortazar) Viz is ready to go at http://korma.wmflabs.org/browser/scr.html  You'll see that there are two new charts. One of them contains the information v...
[08:41:15] <grrrit-wm>	 (PS1) Joal: Add webstatcollector projectview transformation [analytics/refinery] - https://gerrit.wikimedia.org/r/220426 (https://phabricator.wikimedia.org/T101118)
[14:06:40] <joal>	 ottomata: Hi !
[14:07:00] <joal>	 I'd like some advice for a puppet change
[14:10:09] <ottomata>	 morning
[14:10:10] <ottomata>	 k
[14:10:11] <ottomata>	 wasssup
[14:11:31] <joal>	 I am willing to modify the aggregator.pp in the statistics module to add aggregation for new page views
[14:11:47] <joal>	 And I'd like your opinion on how to approach the thing :)
[14:11:50] <joal>	 ottomata: --^
[14:13:55] <ottomata>	 does it just need a new cron?
[14:14:19] <joal>	 I think it needs a new git repo
[14:14:35] <ottomata>	 oh a new data one?
[14:14:36] <ottomata>	 hm.
[14:14:41] <joal>	 yup that's the idea
[14:14:46] <ottomata>	 maybe we shoudl separate out the setup and the data+cron part
[14:14:48] <ottomata>	 i'd say:
[14:15:03] <ottomata>	 make a new directory in there called aggregator/
[14:15:19] <ottomata>	 make a new class in that called aggregator::projectcounts
[14:15:39] <ottomata>	 and move the aggregator/data.git and the crons into that class, and make that class include statistics::aggregator
[14:15:43] <joal>	 entire new module, right ?
[14:15:45] <ottomata>	 nawww
[14:16:19] <ottomata>	 sorry, the first class should be statistics::aggregator::projectcounts
[14:16:24] <ottomata>	 i thnk anyway
[14:16:33] <ottomata>	 not sure if this really deserves its own module
[14:16:50] <ottomata>	 so
[14:16:52] <joal>	 k
[14:17:10] <ottomata>	 statistics::aggregator - setup and aggregator_code clone, etc.
[14:17:23] <ottomata>	 statistics::aggregator::projectcounts - projectcounts data.git clone and cron jobs
[14:17:24] <joal>	 yeah makes sense
[14:17:27] <ottomata>	 then your new classs can be
[14:17:29] <ottomata>	 ja you got it :)
[14:17:56] <ottomata>	 joal: i can't remember thouhg, did we decided to go with using aggregator?
[14:18:09] <joal>	 About the files, for the moment I have aggregator.pp in module/statistics/manifest
[14:18:42] <joal>	 ottomata: we did so, in order to easily replicate the existing behavior
[14:18:50] <joal>	 Let's batcave that a min if you wish
[14:37:53] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Write a pidfile only when daemonized [analytics/kafkatee] - https://gerrit.wikimedia.org/r/219378 (owner: Faidon Liambotis)
[14:42:12] <wikibugs>	 Analytics, Tool-Labs-tools-Other: Work on Metrics tools wm-metrics and MediaCollectionDB, refactoring and code quality. - https://phabricator.wikimedia.org/T100710#1396448 (JeanFred) Open>Resolved I am happy with the results in the context of the hackathon. Work will continue but this can be closed.
[14:51:28] <grrrit-wm>	 (PS1) Ottomata: Merge branch 'master' into debian [analytics/kafkatee] (debian) - https://gerrit.wikimedia.org/r/220467
[14:51:38] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Merge branch 'master' into debian [analytics/kafkatee] (debian) - https://gerrit.wikimedia.org/r/220467 (owner: Ottomata)
[15:28:17] <grrrit-wm>	 (PS1) Milimetric: Update for July Meeting [analytics/reportcard/data] - https://gerrit.wikimedia.org/r/220483
[15:28:46] <grrrit-wm>	 (CR) Milimetric: [C: 2 V: 2] Update for July Meeting [analytics/reportcard/data] - https://gerrit.wikimedia.org/r/220483 (owner: Milimetric)
[15:31:10] <joal>	 ottomata: Standuuuuup :)
[15:40:16] <wikibugs>	 Analytics-Kanban: Vet data in intermediate aggregate {wren} [8 pts] - https://phabricator.wikimedia.org/T102161#1396719 (ggellerman) a:JAllemandou
[15:40:26] <wikibugs>	 Analytics-Kanban, Patch-For-Review: Processor writes valid and invalid events to separate Kafka topics {stag} [13 points] - https://phabricator.wikimedia.org/T98781#1396720 (kevinator)
[15:41:00] <wikibugs>	 Analytics-Kanban: Gather information on all the schemas {tick} [13 pts] - https://phabricator.wikimedia.org/T102515#1396721 (kevinator)
[15:41:21] <wikibugs>	 Analytics-Kanban: Gather information on all the schemas {tick} [13 pts] - https://phabricator.wikimedia.org/T103366#1396722 (kevinator)
[16:02:43] <mforns>	 mili|away, test
[16:03:52] <milimetric>	 cool, thx
[16:04:00] <mforns>	 :]
[16:04:25] <YuviPanda>	 mforns: hey! I didn't get to recovering you guys' files earlier :( give me a link again?
[16:04:29] <YuviPanda>	 to the bug that is
[16:04:44] <mforns>	 YuviPanda, no problem, just a sec
[16:05:34] <mforns>	 YuviPanda, https://phabricator.wikimedia.org/T103530
[16:05:53] <mforns>	 thanks!
[16:09:08] <YuviPanda>	 mforns: done and updated tickets
[16:10:56] <mforns>	 YuviPanda, should /data/projects and the home folders be mounted now in wikimetrics1?
[16:11:04] <YuviPanda>	 if you run puppet...
[16:11:15] <mforns>	 YuviPanda, OK cool! thanks!
[16:36:35] <joal>	 will be away for some time then back !
[17:28:20] <milimetric>	 ottomata: so are you backfilling that data with the batch size set to 1?
[17:29:38] <ottomata>	 oof, i haven't yet, still trying to fix kafktee on oxygen :(
[17:33:09] <grrrit-wm>	 (PS2) Ottomata: Remove kafkatee.service's ExecStopPost [analytics/kafkatee] (debian) - https://gerrit.wikimedia.org/r/219377 (owner: Faidon Liambotis)
[17:34:07] <wikibugs>	 Analytics, VisualEditor, Wikimedia-Site-requests, Database: Backfill rctag data for VisualEditor from 2015-05-23–2015-05-28 - https://phabricator.wikimedia.org/T101270#1397182 (Neil_P._Quinn_WMF) Open>declined
[17:37:58] <wikibugs>	 Analytics, VisualEditor, Wikimedia-Site-requests, Database: Backfill rctag data for VisualEditor from 2015-05-23–2015-05-28 - https://phabricator.wikimedia.org/T101270#1397195 (Neil_P._Quinn_WMF) It looks like T100439 affected tags in the `recentchanges` table, but not in `change_tag` or `tag_summ...
[17:44:11] <madhuvishy>	 ottomata: I'm around, we can chat when you're free :)
[17:44:37] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Remove kafkatee.service's ExecStopPost [analytics/kafkatee] (debian) - https://gerrit.wikimedia.org/r/219377 (owner: Faidon Liambotis)
[17:46:07] <grrrit-wm>	 (PS1) Ottomata: Update version to 0.1.4-1 [analytics/kafkatee] (debian) - https://gerrit.wikimedia.org/r/220511
[17:46:21] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Update version to 0.1.4-1 [analytics/kafkatee] (debian) - https://gerrit.wikimedia.org/r/220511 (owner: Ottomata)
[17:46:32] <ottomata>	 madhuvishy: 10-15 mins?
[17:46:40] <madhuvishy>	 ottomata: sure
[17:55:03] <dr0ptp4kt>	 milimetric: the meaty stuff is embodied in https://github.com/wikimedia/apps-android-wikipedia/blob/master/wikipedia/src/main/java/org/wikipedia/page/linkpreview/PreviewFetchTask.java and other files in that directory. dbrant should be able to fill you in on any details. full rollout hasn't started - the guys are verifying it's efficacy in different forms,
[17:55:03] <dr0ptp4kt>	 with presumable rollout later. nonetheless, it may cause increases in such API requests
[17:55:26] <milimetric>	 dr0ptp4kt: thanks, I wrote an FYI to the team, thanks for bringing it up
[17:56:12] <ottomata>	 phew, ok madhuvishy
[17:56:14] <ottomata>	 batcave?
[17:59:00] <milimetric>	 ottomata: just in case you didn't see this 'cause I accidentally rebased and so hid my comments: https://gerrit.wikimedia.org/r/#/c/220290/3/server/bin/eventlogging-processor
[17:59:51] <madhuvishy>	 ottomata: yes joining in a sec
[18:28:51] <ottomata>	 milimetric:  i see it, thanks!
[18:39:46] <madhuvishy>	 joal: think we can get the app session metrics job running now?
[18:39:51] <joal>	 Sure !
[18:39:57] <madhuvishy>	 if it's too late we can do later
[18:40:08] <joal>	 It's really ok :)
[18:40:50] <joal>	 So, the way I do it: I have oozie job launching templates savedin a file I keep, and reuse them over and over :)
[18:40:53] <wikibugs>	 Analytics-Backlog, Labs, Labs-Infrastructure: Report page views for labs instances - https://phabricator.wikimedia.org/T103726#1397418 (Spage) NEW
[18:40:57] <joal>	 madhuvishy: --^
[18:41:15] <madhuvishy>	 joal: aah
[18:41:23] <joal>	 How do you want us to go for that ?
[18:41:26] <joal>	 batcave ?
[18:41:31] <madhuvishy>	 joal: sure. joining
[18:46:45] <wikibugs>	 Analytics-Backlog, Labs, Labs-Infrastructure: Report page views for labs instances - https://phabricator.wikimedia.org/T103726#1397438 (Spage)
[18:57:19] <joal>	 ottomata: Heyyyyaaaaa :)
[18:57:48] <joal>	 ottomata: Is it a quick one to give Madhu hdfs sudo rigths on stat1002 ?
[18:57:50] <ottomata>	 yo
[18:57:58] <joal>	 If not, I'll launch the job
[18:58:12] <joal>	 But it would nice if she does ;)
[18:58:12] <ottomata>	 no, gotta phab ticket + get sign off, + 3day waiting: (
[18:58:17] <joal>	 :(
[18:58:23] <ottomata>	 actually, that might be one that has to be talked about in ops meeting
[18:58:27] <joal>	 Ok, We'll start the process, and I'll launch the thing
[19:05:11] <joal>	 ottomata: Do you think you could plan on trying dynamic resource allocation for spark on the cluster ?
[19:05:20] <joal>	 It would really be awesomne :)
[19:06:22] <joal>	 ottomata: Madhu prod job fight for instantaneously available resource, while having other jobs allocating resource on the fly makes it difficult
[19:06:53] <ottomata>	 joal: yes!  i have not thought about it, but if it is a priority I can try to find some time
[19:07:01] <ottomata>	 maybe after I get through this next round of EL stuff?
[19:07:02] <joal>	 That would be great :)
[19:07:06] <joal>	 For sure !
[19:07:11] <joal>	 Not so urgent ;)
[19:22:52] <ottomata>	 milimetric: i am backfilling!
[19:35:51] <madhuvishy>	 joal: something's wrong - the job's not running - and it seems to get restarted
[19:37:11] <ottomata>	 milimetric:  fixed those
[19:37:14] <ottomata>	 comments
[19:37:51] <ottomata>	 if you (or I can) merge i will deploy this code in beta. it shouldnt' need any puppet changes
[19:38:01] <milimetric>	 (looking)
[19:38:43] <milimetric>	 done
[19:40:38] <joal>	 madhuvishy: I'll kill the coordinator and restart it with 16 executors
[19:54:08] <madhuvishy>	 joal: okay..
[19:54:10] <joal>	 ottomata: NEEEEEED :) https://databricks.com/blog/2015/06/22/understanding-your-spark-application-through-visualization.html
[19:54:30] <joal>	 madhuvishy: There a super heavy job currently ongoing
[19:54:36] <ottomata>	 OH cool!
[19:54:39] <joal>	 So we'll see if youre get launched
[20:00:21] <joal>	 madhuvishy, ottomata : weird spark stuff --> job is launched, resource allocated, but nothing happens :( no job, no stage
[20:01:24] <madhuvishy>	 joal: yeah
[20:01:26] <ottomata>	 can you load the app master page?
[20:01:27] <ottomata>	 i'm trying
[20:01:40] <joal>	 worked a while ago, not anymore for me either
[20:01:57] <madhuvishy>	 joal: ottomata hmmm :|
[20:05:20] <ottomata>	 ah now it is loading
[20:05:30] <ottomata>	 but ja nothing running
[20:05:33] <joal>	 yes, but not worling
[20:05:41] <ottomata>	 is this the first tiem we are trying a big job through oozie?
[20:06:00] <joal>	 I did spark oozie jobs for a while, working fine wuth 8 workers
[20:06:27] <ottomata>	 i mean all 30 days
[20:06:29] <ottomata>	 ?
[20:06:36] <madhuvishy>	 ottomata: joal i was running this through oozie while testing
[20:06:41] <madhuvishy>	 i dont think that's it
[20:06:49] <madhuvishy>	 something is failing
[20:08:18] <ottomata>	 oh this is session
[20:08:20] <ottomata>	 so 7 days, right?
[20:08:22] <ottomata>	 no?
[20:08:23] <ottomata>	 uhhh
[20:08:24] <ottomata>	 no
[20:08:24] <madhuvishy>	 no
[20:08:26] <ottomata>	 i get confused
[20:08:28] <ottomata>	 heh
[20:08:29] <madhuvishy>	 once in 7 days
[20:08:32] <madhuvishy>	 30 day period
[20:08:46] <madhuvishy>	 joal: ottomata I couldn't go to lunch before because meeting, so just going - but will ping when I come back. Joal it's probably late for you so if we dont figure it out we can tomorrow
[20:09:03] <joal>	 sounds good madhu :)
[20:09:06] <joal>	 thanks :)
[20:09:19] <joal>	 I'll continue for some time, and then will go to sleep ;)
[20:09:34] <madhuvishy>	 okay thanks :)
[20:10:26] <ottomata>	 weirsd
[20:10:28] <ottomata>	 weird
[20:12:15] <ottomata>	 joal: just to check, what happens if you launch this job manually, rather than via oozie
[20:12:20] <ottomata>	 same input. does it run?
[20:12:26] <joal>	 I have not tries
[20:12:33] <joal>	 I am investingating logs
[20:12:35] <ottomata>	 k
[20:12:47] <ottomata>	 milimetric: can i get a quick brain bounce?
[20:13:39] <milimetric>	 ottomata: I'm in the cave
[20:16:10] <joal>	 ottomata: something we want to do as well is drop some logs in /var/log/hadoop-yarn/apps/hdfs
[20:16:23] <joal>	 Can't even ls with such a big list !
[20:21:50] <ottomata>	 joal
[20:21:59] <ottomata>	 yarn logs -applicationId <app_id>
[20:22:13] <joal>	 yup hmm
[20:22:45] <joal>	 no logs for our case though
[20:22:49] <joal>	 ottomata: --^
[20:22:52] <joal>	 WEird
[20:24:30] <wikibugs>	 Analytics-Tech-community-metrics: Weekly report for "Allow contributors to update their own details in tech metrics directly" - https://phabricator.wikimedia.org/T101134#1397795 (Sarvesh.onlyme)
[20:24:32] <joal>	 ottomata: will kill the coordinator for now and investigate more tomorrow
[20:24:40] <joal>	 time to get to bed
[20:29:19] <ottomata>	 joal: logs don't s how up until job is done or dies
[20:29:24] <ottomata>	 joal:  ok
[20:29:35] <wikibugs>	 Analytics-Tech-community-metrics, Engineering-Community: Check whether it is true that we have lost 40% of code contributors in the past 12 months - https://phabricator.wikimedia.org/T103292#1397805 (Nemo_bis) Worth checking that recently created repositories are being included in the counts. People tend...
[20:32:47] <mforns>	 kevinator, yt?
[20:34:14] <joal>	 ottomata: coordinator killed
[20:34:27] <joal>	 ottomata: I let the app running, see if anything happens ...
[20:34:39] <joal>	 See y'all tomorrow !
[20:36:42] <ottomata>	 laters!
[20:37:13] <ottomata>	 milimetric: https://gerrit.wikimedia.org/r/#/c/220614/
[20:37:40] <mforns>	 bye joal|night
[20:38:18] <milimetric>	 ottomata: you're gonna wait for ori's review right?
[20:38:50] <ottomata>	 i'm going to email him now ja, he said he'd be gone for a bit, so that's fine.
[20:39:40] <milimetric>	 let me know, I can merge it whenever
[21:04:12] <grrrit-wm>	 (CR) Yuvipanda: [C: 2] "Oopsy on yeti. Not sure how exactly to replace - just move to another theme, I suppose?" [analytics/quarry/web] - https://gerrit.wikimedia.org/r/219422 (owner: Ricordisamoa)
[21:04:19] <grrrit-wm>	 (Merged) jenkins-bot: Load CodeMirror from cdnjs, using minified files [analytics/quarry/web] - https://gerrit.wikimedia.org/r/219422 (owner: Ricordisamoa)
[21:07:17] <wikibugs>	 Analytics, Analytics-Kanban, Readership-Web: Debug blank datafiles generated by generate.py [8 pts] {lamb] - https://phabricator.wikimedia.org/T103387#1397904 (mforns) a:mforns
[21:18:11] <wikibugs>	 Analytics, Research-and-Data: Referrer data for en:Glitter for shareafact test - https://phabricator.wikimedia.org/T93270#1397954 (ggellerman) Open>declined
[21:25:02] <grrrit-wm>	 (CR) Ricordisamoa: "Yeti should be ok as long as http://quarry.wmflabs.org/static/vendor/yeti.bootstrap.min.css does not include googleapis.com. Bootswatch an" [analytics/quarry/web] - https://gerrit.wikimedia.org/r/219422 (owner: Ricordisamoa)
[21:37:25] <wikibugs>	 Analytics, Engineering-Community, Research-and-Data, ECT-July-2015: Metrics about the use of the Wikimedia web APIs - https://phabricator.wikimedia.org/T102079#1398018 (ggellerman) @Qgil Research can help you develop your measurements if you help clarify what it is that you actually want measured....