[13:31:13] <wikibugs>	 Analytics-Wikistats: Percentage pageviews from Russia is too low in recent geographical breakdowns in Wikistats - https://phabricator.wikimedia.org/T109582#1552786 (ezachte) NEW
[14:31:16] <wikibugs>	 Analytics-EventLogging, operations, Patch-For-Review: Create a package for python-pykafka for ubuntu precise and debian sid - https://phabricator.wikimedia.org/T109567#1552896 (Ottomata) Ah!  Forgot about hafnium, and that it was precise.  Will package it for precise too.
[14:40:10] <joal|night>	 craintesHi ottomata
[14:40:17] <joal|night>	 Hi ottomata sorry :)
[14:40:20] <ottomata>	 hello!
[14:40:31] <joal|night>	 Oops, I'm looking nighty !
[14:40:47] <joal>	 How is it today ?
[14:43:08] <ottomata>	 haven't checked status yet, although dan and I were seeing sometihng fishy yesterday.....>.......mainly about duplicates coming in, but also about potentially rerunning more jobs...due to tweaks we made to my hourly loss query :(
[14:43:15] <ottomata>	 i'm reading your cassandra loader now :)
[14:44:33] <joal>	 ottomata: ok
[14:45:01] <joal>	 Every text load job since yesterday 4AM is shown red in hue --> seems worng :(
[14:45:08] <joal>	 ottomata: --^
[14:48:21] <ottomata>	 hm, that does sound wrong, will look
[14:48:24] <ottomata>	 joal: https://github.com/spotify/hdfs2cass ?
[14:48:32] <ottomata>	 seen that?
[14:48:44] <joal>	 yup
[14:48:58] <joal>	 avro based
[14:49:47] <ottomata>	 we can write avro in hadoop pretty easy though, no?
[14:49:55] <ottomata>	 we can very easily output an avro dataset from hive query
[14:50:09] <ottomata>	 and why not, eh?  pageview could be avro
[14:50:14] <ottomata>	 its all in hadoop right now
[14:50:29] <joal>	 for sure could be :)
[14:51:21] <ottomata>	 camus is all caught up. hm.
[14:51:33] <ottomata>	 i bet its all duplicates.
[14:51:34] <ottomata>	 hm.
[14:51:35] <joal>	 And there are bizarre behavior in kafka
[14:51:37] <ottomata>	 checking.
[14:51:38] <ottomata>	 oh?
[14:52:04] <ottomata>	 what?
[14:52:06] <joal>	 Like cpu iowait is higher on 1021 and 1022
[14:52:26] <joal>	 disk await max higher on 1021
[14:52:44] <joal>	 I would have expected things to level up after leader reelection
[14:53:46] <ottomata>	 ah, i also find that strange, but i think it has been like that for a while, and the difference isn't a lot.
[14:53:57] <joal>	 yeah, true
[14:54:02] <joal>	 Machiens are different ?
[14:54:58] <ottomata>	 hardare?  no.
[14:55:05] <joal>	 k
[14:55:51] <ottomata>	 yeah, joal, i think our reporting is all messed up now, especially by that cp1008.wikimedia.org host
[14:56:05] <ottomata>	 looking into that now, maybe can turn off vk there
[14:56:06] <joal>	 ok ottomata
[14:56:22] <ottomata>	 i don't see very little loss (accoriding to my new query) now.
[14:56:24] <joal>	 so wrong load jobs, but no data loss
[14:56:35] <ottomata>	 btw, this is what
[14:56:35] <ottomata>	 http://codeshare.io/aITwc
[14:56:40] <ottomata>	 i am using to check
[14:56:41] * joal breath again
[14:57:30] <joal>	 ottomata: So, about the cassandra loader, I'd choose not to go for avro because of the change weight
[14:57:51] <ottomata>	 aye, +avro is a lot
[14:58:10] <ottomata>	 but, also your job is not short :)
[14:58:17] <joal>	 ottomata: But if you prefer, we can go to avro and project that (we'll have to if we want to go for confluent anyway ;)
[14:58:24] <ottomata>	 lots of logic there about convention with fields, etc.
[14:58:33] <joal>	 ottomata: Very true
[14:58:45] <ottomata>	 wait, pageview is in parquet, right?
[14:58:50] <ottomata>	 or just text
[14:58:51] <joal>	 Type conversion, therefore schema definition
[14:58:59] <joal>	 pageview is parquet
[14:59:01] <ottomata>	 where is this xsv coming from?
[14:59:22] <joal>	 Genericity
[14:59:28] <ottomata>	 Genericity?
[15:00:18] <joal>	 Same job for any XSV data type
[15:00:31] <joal>	 Like historical pageview for instance
[15:00:38] <ottomata>	 aye
[15:00:45] <ottomata>	 btw, i think its pretty easy to read parquet as avro
[15:00:46] <ottomata>	 https://github.com/Parquet/parquet-mr/blob/master/parquet-avro/src/test/java/parquet/avro/TestReadWrite.java#L72
[15:01:32] <joal>	 Indeed, easy to convert parquet to avro it seems :)
[15:01:49] <ottomata>	 yeah, anyway, i think you could be right, just want to make sure we consider this
[15:02:16] <joal>	 I have looked at it, and choosed the shrtest path in my opinion
[15:02:23] <joal>	 But it can be changed
[15:03:51] <joal>	 CAn't read your codeshare ottomata
[15:04:00] <ottomata>	 no?
[15:04:01] <ottomata>	 why?
[15:04:12] <joal>	 application error :(
[15:08:29] <ottomata>	 hm
[15:08:31] <ottomata>	 ok gist
[15:08:47] <ottomata>	 joal: https://gist.github.com/ottomata/bbab3caa708c5b213ec0
[15:08:59] <joal>	 better thx
[15:11:45] <joal>	 ottomata: your request seems correct to me
[15:20:20] <ottomata>	 ok, joal, i'm going to rerun some loads again...and compare one more time, ok?
[15:20:30] <ottomata>	 going to rerun for these https://gist.github.com/ottomata/bbab3caa708c5b213ec0#gistcomment-1554721
[15:20:50] <ottomata>	 and then compare those percent_lost after the load finishes, and any of those we will rerun refine and depdendents.
[15:20:54] <ottomata>	 am sorry this is not yet over!!! :(
[15:21:09] <ottomata>	 i checked this yesterday: upload  2015    8   17  20  157311435   58018892    0.0 26.94413407
[15:21:10] <ottomata>	  and it was actually fine. so ja.
[15:21:59] <joal>	 ottomata: let's go for it, and also maybe for 2015-08-18 ?
[15:22:23] <joal>	 ottomata: We also need to investigate and pinpoint the issue
[15:22:44] <ottomata>	 indeed.  joal, that is all hours where percent_lost > 1% in august up til today
[15:22:45] <joal>	 Cause that's really taking a lot of resource (people _ cluster) to catch up
[15:22:47] <ottomata>	 so nothing from yesterday
[15:22:57] <joal>	 neither today
[15:23:01] <joal>	 Ok cool
[15:23:06] <joal>	 Let's go
[15:23:12] <joal>	 We'll talk about that in standup I guess
[15:23:18] <ottomata>	 indeed.  i was hoping to get through this upgrade and then do post mortem and fix all this stuff.  ESPECIALLY our 2 hour job fireing thing
[15:23:22] <ottomata>	 that is the source of a lot of these problems
[15:23:40] <ottomata>	 well, maybe not the source, but it is a dumb thing that woudl have prevented a lot of this work
[15:23:43] <ottomata>	 if we fired load off smarter
[15:23:46] <ottomata>	 than waiting for 2 hours
[15:23:56] <ottomata>	 i think we should do it by examining camus offset/timestamps
[15:24:00] <ottomata>	 for each partition
[15:24:10] <joal>	 right
[15:24:25] <joal>	 I think a first step would be to lauch load with 4 hours lag instead of 2
[15:24:32] <joal>	 Easy win
[15:24:33] <ottomata>	 joal:  if you like, maybe I should wait until we get through the full kafka upgrade/exampsion
[15:24:37] <ottomata>	 before i do mroe job rerunning stufff?
[15:24:42] <ottomata>	 that way we do it once and for all?
[15:24:53] <ottomata>	 joal, 'id be fine with that, but that would delay job output by a lot
[15:25:03] <joal>	 ottomata: feasible, but the cluster will be buuuuuusy at that moment !
[15:25:06] <ottomata>	 yeah
[15:25:14] <joal>	 hm
[15:25:27] <joal>	 Job output timing is not really as important as data quality
[15:25:29] <joal>	 I think
[15:25:45] <joal>	 So if having 2 hours more lag gives us quality, I buy that !
[15:26:35] <grrrit-wm>	 (PS2) Milimetric: Pass the buck [analytics/geowiki] - https://gerrit.wikimedia.org/r/232426 (https://phabricator.wikimedia.org/T106229)
[15:26:56] <joal>	 I actually dont mind rerunning jobs while we stop load during reinstall
[15:27:00] <joal>	 ottomata: --^
[15:29:12] <ottomata>	 i agree joal
[15:29:18] <ottomata>	 joal, aye.
[15:31:42] <joal>	 Even if it's tedious, I think trying to catch up as we go makes sense
[15:31:47] <joal>	 ottomata: --^
[15:35:18] <ottomata>	 k
[15:37:53] <madhuvishy>	 ottomata: https://gerrit.wikimedia.org/r/#/c/232408/
[16:00:17] <mutante>	 hi
[16:00:41] <mutante>	 stat1003 tries to git pull geowiki-scripts, but that fails
[16:00:50] <mutante>	 since an hour
[16:01:02] <mutante>	 Error: /usr/bin/git  pull --quiet returned 1 instead of one of [0]
[16:01:11] <mutante>	 Error: /Stage[main]/Geowiki/Git::Clone[geowiki-scripts]/Exec[git_pull_geowiki-scripts]
[16:01:19] <mutante>	 any ideas if there was a change to that an hour ago?>
[16:04:50] <wikibugs>	 Analytics-Engineering: stat1003 - git pull geowiki scripts fails - https://phabricator.wikimedia.org/T109594#1553329 (Dzahn) NEW
[16:05:15] <wikibugs>	 Analytics-Engineering: stat1003 - git pull geowiki scripts fails - https://phabricator.wikimedia.org/T109594#1553336 (Dzahn) Exec[git_pull_geowiki-scripts]/returns:** You are not currently on a branch**
[16:27:50] <madhuvishy>	 milimetric: were you working on geowiki-scripts? ^
[16:28:18] <milimetric>	 yes :) talking to them in -ops now
[16:28:22] <milimetric>	 thx
[16:28:31] <wikibugs>	 Analytics-Cluster, operations, ops-eqiad, Patch-For-Review: rack new hadoop worker nodes - https://phabricator.wikimedia.org/T104463#1553505 (Cmjohnson) All are racked  and @ottomata has already added as worker nodes.  Unfortunately, we had 2 more servers that were came to us not working.    Dell t...
[16:29:48] <ottomata>	 woot, +6 hadoop nodes this morning!
[16:30:04] <madhuvishy>	 ottomata: :D nice!
[16:32:13] <wikibugs>	 Analytics-Engineering: stat1003 - git pull geowiki scripts fails - https://phabricator.wikimedia.org/T109594#1553516 (Milimetric) This is my fault, sorry for the noise.  I'm messing with the repository there and I have to keep it in this bad state until I can be sure I fixed the original problem.  Let me know...
[16:38:28] <joal>	 ottomata: COOOOOL ! :)
[16:38:37] <joal>	 ottomata: faster backfill ;)
[16:38:48] <joal>	 ottomata: need me for something ?
[16:39:33] <ottomata>	 naw, i'm about to start rerun load jobs
[16:39:36] <ottomata>	 i'm not going to rerun misc
[16:39:38] <ottomata>	 or bits
[16:39:43] <ottomata>	 and i'm not going to rerurn jobs from 08-03
[16:39:50] <ottomata>	 i think we are pretty sure those are lost
[16:39:57] <joal>	 ottomata: right
[16:40:38] <joal>	 Cool, I'll make a pause until this evening interview then, it's already been a long day
[16:40:59] <ottomata>	 yeah
[16:40:59] <ottomata>	 k
[16:41:10] <ottomata>	 i'm going to start these and then get lunch and run home, then will do some reinstalls
[16:41:13] <joal>	 I'll still be online though, so in case, just ping
[16:41:32] <joal>	 Enjoy your lunch :)
[16:42:37] <wikibugs>	 Analytics-Engineering: stat1003 - git pull geowiki scripts fails - https://phabricator.wikimedia.org/T109594#1553617 (Dzahn) I ACKed it in Icinga  so monitoring won't report it anymore until it works again. The ideal way would be to schedule a downtime in Icinga (and disable puppet) but you'd have to ask ops...
[16:42:52] <ottomata>	 k laters
[16:42:59] <wikibugs>	 Analytics-Engineering: stat1003 - git pull geowiki scripts fails - https://phabricator.wikimedia.org/T109594#1553618 (Dzahn) p:Triage>Low
[17:19:03] <dr0ptp4kt>	 bearND: able to make the scrum of scrums today? hoping we can cover the link previews and content service stuff a little bit over video there
[17:22:25] <bearND>	 dr0ptp4kt: I can join and be happy to answer any questions.
[17:24:32] <dr0ptp4kt>	 bearND: thx ^ ottomata, bearND is coming to scrum of scrums. you'll be there, right?
[17:25:17] <ottomata>	 oh , wasn't planning on it, i was just called into a meeting with kevin about something
[17:25:18] <ottomata>	 shoudl I?
[17:26:18] <dr0ptp4kt>	 ottomata: actually, is milimetric going to be at scrum of scrums? i hear you if you need to be at another meeting.
[17:46:25] <grrrit-wm>	 (PS1) Ottomata: Update camus property files with names of new brokers [analytics/refinery] - https://gerrit.wikimedia.org/r/232535 (https://phabricator.wikimedia.org/T106581)
[17:46:53] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Update camus property files with names of new brokers [analytics/refinery] - https://gerrit.wikimedia.org/r/232535 (https://phabricator.wikimedia.org/T106581) (owner: Ottomata)
[17:57:09] <joal>	 madhuvishy: Heya
[17:57:13] <joal>	 You still with Paul ?
[18:19:32] <wikibugs>	 Analytics-Kanban, Patch-For-Review: Foundation-only Geowiki stopped updating - https://phabricator.wikimedia.org/T106229#1554067 (Milimetric) FYI the script currently in code review works, it's been running for about 5 hours and it looks to be about 1/3 of the way through updating the private data.  I'd ex...
[18:20:13] <ottomata>	 !log does this log work?
[18:20:25] <ottomata>	 guess not!
[18:21:39] <ottomata>	 FYI I am starting reinstall of analytics1022 now
[18:50:57] <grrrit-wm>	 (CR) Milimetric: [C: -1] "I need to add some notes directly into this patch so this is easier in the unfortunate case that this happens again. I'll call them READM" [analytics/geowiki] - https://gerrit.wikimedia.org/r/232426 (https://phabricator.wikimedia.org/T106229) (owner: Milimetric)
[18:51:25] <joal>	 Hey mforns
[18:51:30] <joal>	 batcave for debrief ?
[18:51:35] <mforns>	 joal, sure!
[18:51:37] <joal>	 in 5 mins ?
[18:51:57] <mforns>	 joal, ok!
[18:53:34] <joal>	 ready mforns :)
[18:53:38] <mforns>	 yep
[18:53:41] <mforns>	 omw
[18:57:19] <grrrit-wm>	 (CR) Milimetric: Pass the buck (1 comment) [analytics/geowiki] - https://gerrit.wikimedia.org/r/232426 (https://phabricator.wikimedia.org/T106229) (owner: Milimetric)
[18:57:34] <madhuvishy>	 milimetric: should I add access method to the top endpoint? atm it only takes project and timestamp
[18:57:40] <madhuvishy>	 joal: ^
[18:57:52] <milimetric>	 madhuvishy: I'd think so, for completeness
[18:58:09] <milimetric>	 it would be nice to know which articles are trending for folks on mobile only
[18:58:24] <madhuvishy>	 milimetric: yeah, okay makes sense
[18:58:48] <milimetric>	 if there are systematic differences between top on mobile and top on desktop, that could inform some interesting decisions for readership
[19:03:52] <milimetric>	 joal / ottomata: what's up with this thing: log4j:WARN No such property [maxBackupIndex] in org.apache.log4j.DailyRollingFileAppender.
[19:04:11] <joal>	 milimetric: from where ?
[19:04:19] <milimetric>	 when i run hive
[19:04:41] <milimetric>	 I'm just issuing a simple select distinct with tablesample on wmf.webrequest
[19:04:55] <joal>	 hmm
[19:05:09] <ottomata>	 probably from some log4j misconfiguration
[19:05:19] <joal>	 ottomata: new machines ?
[19:05:32] <joal>	 I received an email from aswin as well, I'll forward it to you
[19:05:50] <joal>	 It seems to have started today (why I think machine additions)
[19:05:59] <ottomata>	 ?
[19:06:07] <ottomata>	 milimetric:  is it causing you problems?
[19:06:22] <milimetric>	 no, I was just making sure it's not causing other problems
[19:06:40] <milimetric>	 like, if it's misconfigured and it can never roll the logs, is it just growing and growing?
[19:07:29] <ottomata>	 pretty sure hive cli isn't writing logs
[19:08:21] <joal>	 ottomata: just forwarded you ashwin email
[19:09:43] <ottomata>	 thanks joal, that is a change i made today.
[19:09:44] <ottomata>	 looking
[19:09:52] <joal>	 thx !
[19:13:19] <joal>	 ottomata: chocolate day ! https://twitter.com/jaykreps/status/634069079524012032
[19:14:12] <ottomata>	 ha
[19:15:42] <leila>	 joal: do you want Ashwin's email to Analytics public list?
[19:16:28] <joal>	 Hey leila, the one about hashing I guess
[19:16:35] <leila>	 the one about Hadoop
[19:17:16] <joal>	 Good idea leila, if somebody else has an issue they might bounce as well
[19:17:23] <ottomata>	 the one you sent me?
[19:17:36] <ottomata>	 just replied, and CCed ashwin
[19:17:37] <leila>	 ooki, will tell him, joal
[19:17:39] <ottomata>	 should be fixed shortly
[19:17:53] <joal>	 leila: given ottomata feedback, no need
[19:19:09] <joal>	 ottomata: analytics1022 is dead, welcome kafka1022 I guess :)
[19:19:13] <ottomata>	 :)
[19:19:22] <ottomata>	 yeah, waiting for that stupid PTR record to change again
[19:19:27] <joal>	 :)
[19:19:30] <ottomata>	 next time I will change that ahead of time
[19:19:32] <leila>	 ah okay, joal.
[19:19:39] <joal>	 But thx leila for asking :)
[19:19:47] <joal>	 And sorry for the missfit :)
[19:20:00] <joal>	 Guys, I'm off for today !
[19:20:01] <leila>	 np. :-)
[19:20:07] <leila>	 have a good night joal
[19:20:10] <joal>	 Thx !
[19:20:32] <joal>	 ottomata: if you need me to rerun some stuff, please email, like that I start in he morning (less traffic)
[19:21:48] <ottomata>	 joal|night:  i emailed you a few mins ago
[19:21:52] <ottomata>	 oh, i didn't
[19:21:54] <ottomata>	 i'm still refining
[19:21:57] <ottomata>	 ther are only 2 partitions more to do
[19:22:16] <ottomata>	 oh, and they are done
[19:22:16] <joal|night>	 yeah I was wondering how it came unoticed :)
[19:22:17] <ottomata>	 emailling now
[19:22:21] <joal|night>	 Ok
[19:22:32] <ottomata>	 k, goodnight!
[19:26:58] <mforns>	 ottomata, hi! did the candidate write a task?
[19:27:17] <ottomata>	 yes
[19:27:19] <ottomata>	 you want?
[19:27:24] <mforns>	 yes please! :]
[19:28:06] <ottomata>	 sent
[19:28:58] <mforns>	 ottomata, thanks!
[20:22:45] <wikibugs>	 Analytics-Kanban: Legal request, data point 1 - https://phabricator.wikimedia.org/T109626#1554591 (ggellerman) NEW
[20:25:15] <wikibugs>	 Analytics-Kanban, Research-and-Data: Analyze referrer traffic to determine report format {hawk} [8 pts] - https://phabricator.wikimedia.org/T108886#1554616 (Milimetric) I picked an hour I'm not going to disclose on purpose, and I did (notice the 0.01% sampling)      select referer, count(*)       from wmf....
[20:41:51] <grrrit-wm>	 (PS3) Milimetric: Pass the buck [analytics/geowiki] - https://gerrit.wikimedia.org/r/232426 (https://phabricator.wikimedia.org/T106229)
[20:53:32] <wikibugs>	 Analytics-Kanban, Research-and-Data: Analyze referrer traffic to determine report format {hawk} [8 pts] - https://phabricator.wikimedia.org/T108886#1554764 (kevinator) @milimetric can you run using a whole day instead of an hour?  I'm surprised to see Japan so high and I'm left wondering if this was done a...
[20:57:22] <wikibugs>	 Analytics-Kanban, Research-and-Data: Legal request, data point 1 - https://phabricator.wikimedia.org/T109626#1554779 (DarTar)
[21:22:19] <bearloga>	 ottomata: what's the best way to kill a hive job?
[21:31:40] <grrrit-wm>	 (PS1) Ottomata: [WIP] Generate hourly aggregate statistics about webrequest sequence stats [analytics/refinery] - https://gerrit.wikimedia.org/r/232644
[21:31:56] <ottomata>	 bearloga: https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hive#Killing_a_running_query
[21:36:41] <bearloga>	 ottomata: thank you very much!
[21:40:52] <grrrit-wm>	 (CR) Ottomata: "I haven't run this exact version of the query, or tested this at all, hence WIP. Whatcha think?" [analytics/refinery] - https://gerrit.wikimedia.org/r/232644 (owner: Ottomata)
[21:59:25] <grrrit-wm>	 (CR) Mforns: Refactor queries into reportupdater and dashiki (1 comment) [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/232645 (https://phabricator.wikimedia.org/T107504) (owner: Mforns)
[22:05:26] <grrrit-wm>	 (CR) Mforns: "I don't know what this flake8 error is, it seems it crashed. In my machine flake8 passes OK. :/" [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/232645 (https://phabricator.wikimedia.org/T107504) (owner: Mforns)
[22:06:51] <grrrit-wm>	 (CR) Milimetric: "yeah, that's a system error, not your code. I'd wait until tomorrow and just upload a new patchset" [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/232645 (https://phabricator.wikimedia.org/T107504) (owner: Mforns)
[22:07:29] <mforns>	 milimetric, thx
[22:18:32] <wikibugs>	 Analytics, Labs, Labs-Infrastructure, Labs-Sprint-108, Patch-For-Review: Set up cron job on labstore to rsync data from stat* boxes into labs. - https://phabricator.wikimedia.org/T107576#1555114 (yuvipanda) /me pokes @akosiaris again
[22:19:18] <wikibugs>	 Analytics-Backlog: Delete obsolete schemas {tick} - https://phabricator.wikimedia.org/T108857#1555124 (mforns)
[22:32:51] <grrrit-wm>	 (CR) Milimetric: [C: 1] [WIP] Generate hourly aggregate statistics about webrequest sequence stats (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/232644 (owner: Ottomata)
[22:39:55] <kevinator>	 Ironholds , milimetric i'm signing off and headed to the dentist... I won't be answering emails for a bit ;-)
[22:40:28] <Ironholds>	 kevinator, I'm going to go watch The Wire and eat raspberry ice cream with marshmallows in it
[22:40:40] <Ironholds>	 which probably means I'll be heading to the dentist tomorrow ;p
[22:40:55] <Ironholds>	 have fun! Or, as much fun as you can have! And let's work on this problem in the morrow when we're fresh and rested :)
[23:32:41] <wikibugs>	 Analytics-Kanban, Analytics-Wikistats: {lama} Wikistats 2.0 - https://phabricator.wikimedia.org/T107175#1555494 (kevinator) Here's another use case I came across recently:  As a legal team member, I want traffic reports by country so I can provide evidence of traffic in a particular country.  ( http://sta...