[00:53:36] <wikibugs>	 Analytics-Wikimetrics, MediaWiki-Vagrant: Vagrant Setup alembic config errors - https://phabricator.wikimedia.org/T99631#1295554 (bd808)
[00:55:09] <wikibugs>	 Analytics-Wikimetrics, MediaWiki-Vagrant: Vagrant Setup alembic config errors - https://phabricator.wikimedia.org/T99631#1295505 (bd808) The interesting part of the Puppet trace is: ``` ==> default: Notice: /Stage[main]/Wikimetrics::Database/Exec[alembic_upgrade_head]/returns: Traceback (most recent call...
[03:04:54] <wikibugs>	 Analytics-Cluster, operations: Build Kafka 0.8.1.1 package for Jessie  and upgrade Brokers to Jessie. - https://phabricator.wikimedia.org/T98161#1295682 (Gage) I followed this: https://git.wikimedia.org/blob/operations%2Fpuppet.git/2cdd08f9686b040816bd0dd8e63e712f4b084a7a/modules%2Fpackage_builder%2FREADM...
[03:05:15] <wikibugs>	 Analytics-Cluster, operations: Build Kafka 0.8.1.1 package for Jessie  and upgrade Brokers to Jessie. - https://phabricator.wikimedia.org/T98161#1295683 (Gage)
[13:06:16] <joal>	 Good morning milimetric :)
[13:06:22] <milimetric>	 morning
[13:06:39] <milimetric>	 I don't see aaron around
[13:06:42] <joal>	 Aaron not being here, do we cancel the meeting or do you want us to talk ?
[13:06:44] <milimetric>	 so I figure the meeting's off
[13:06:52] <joal>	 he ent an email :)
[13:06:56] <milimetric>	 oh, sorry, didn't read
[13:07:00] <joal>	 np :)
[13:07:02] <milimetric>	 nah, let's cancel, I could use the time
[13:07:10] <joal>	 Cool
[13:07:15] <joal>	 Talk later then :)
[13:07:42] <milimetric>	 joal: if you want to talk, of course I have the time, btw
[13:07:53] <joal>	 So far so good :)
[14:29:52] <ottomata>	 moooornin
[14:29:54] <ottomata>	 joal: you around?
[14:30:00] <joal>	 Heya !
[14:30:03] <joal>	 I am :)
[14:30:07] <ottomata>	 helloOooO
[14:30:13] <joal>	 Was waiting for you to ping, not to be invasive :)
[14:30:18] <ottomata>	 hehe :)
[14:30:32] <joal>	 How are you doing ?
[14:30:39] <ottomata>	 thanks, slow morning for me.  went to cafe, forgot my bike lock, had to run back home to get it
[14:30:41] <ottomata>	 now settled in at cafe!
[14:30:52] <ottomata>	 am doing gooood
[14:30:57] <ottomata>	 how you doiiiin?
[14:31:08] <ottomata>	 how's baby joal?
[14:31:16] <joal>	 Lino is great :)
[14:31:37] <joal>	 Smiling every day more, and shouting when hungry, as his father  :D
[14:31:40] <ottomata>	 :)
[14:32:00] <joal>	 And he lets us sleep not too bad, which is really cool
[14:32:24] <joal>	 I'd advise anybody who wants a really incredible adventure to make a child :D
[14:32:29] <ottomata>	 that is great!  you should bring baby lino to mexico! :)
[14:32:56] <joal>	 I am sure he'd love, but my concentration would be completely gone :)
[14:33:00] <ottomata>	 ha
[14:33:10] <ottomata>	 maybe nuria can bring her little one too!
[14:33:16] <ottomata>	 SOOOooOOo um jajjajajaj
[14:33:17] <joal>	 Next year, depending on the place maybe
[14:33:17] <ottomata>	 this guy:
[14:33:19] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/209642/
[14:33:48] <ottomata>	 do you think my worry is legitimate?
[14:33:52] <ottomata>	 the comment?
[14:33:57] <joal>	 Just read it
[14:34:11] <joal>	 So let's math a bit
[14:34:25] <joal>	 2G / container by default
[14:34:35] <ottomata>	 see comment in file on line 211
[14:34:36] <joal>	 64g per new machine (right ?)
[14:34:44] <ottomata>	 ya
[14:34:45] <joal>	 - 12G for system ?
[14:35:12] <joal>	 Isn't there some setting to reserve memory space for the systme ?
[14:35:21] <joal>	 I recall something like that
[14:35:24] <ottomata>	         elsif $::memorysize_mb <= 73728 {
[14:35:24] <ottomata>	             $reserve_memory_mb = 8192
[14:35:24] <ottomata>	         }
[14:35:34] <joal>	 k
[14:35:34] <ottomata>	         $available_memory_mb = $::memorysize_mb - $reserve_memory_mb
[14:35:41] <ottomata>	         # Since I have chosen a static $memory_per_container of 2048 across all
[14:35:41] <ottomata>	         # node sizes, we should just choose to give NodeManagers
[14:35:41] <ottomata>	         # $available_memory_mb to work with.
[14:35:41] <ottomata>	         # This will give nodes with 48G of memory about 21 containers, and
[14:35:41] <ottomata>	         # nodes with 64G memory about 28 containers.
[14:36:31] <ottomata>	 this is probably relevant too: https://phabricator.wikimedia.org/T90640
[14:36:33] <joal>	 28 containers but 24 (virtual) procs, right
[14:37:28] <ottomata>	 yarn_nodemanager_resource_cpu_vcores     - Default: max($::processorcount - 1, 1)
[14:37:49] <joal>	 I think it could cause trouble
[14:39:17] <joal>	 trouble = slow down jobs for bad reasons (waiting for an allocated container with no CPU to finish while the rest of ocntainers have finished)
[14:40:02] <joal>	 It really depends on the jobs though --> IO bottlenecked jobs, no problems
[14:40:04] <ottomata>	 well, except that mostly yarn schedules based on memory
[14:40:09] <joal>	 CPU bounded jobs, problems
[14:40:12] <ottomata>	 bu ya
[14:40:13] <ottomata>	 but ya
[14:40:31] <ottomata>	 cpu load would likely increase a lot, hm.  but wait, what is it now?
[14:40:34] <ottomata>	 let's see
[14:40:47] <joal>	 That why you could have trouble: some containers scheduled, but no real CPU resource available
[14:41:07] <joal>	 can you share you url for acluster load ?
[14:41:16] <ottomata>	 ah, right now
[14:41:26] <ottomata>	 it is about 21 containers per node no matter what
[14:41:52] <ottomata>	 joal: this  has more than just worker nodes in it, but:
[14:41:53] <ottomata>	 http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&m=cpu_report&s=by+name&c=Analytics%2520cluster%2520eqiad&tab=m&vn=&hide-hf=false
[14:42:21] <ottomata>	 psshhhhh
[14:42:24] <ottomata>	 1017 is out of cluster
[14:42:33] <ottomata>	 why no balacner! :)
[14:42:35] <ottomata>	 checking
[14:42:59] <joal>	 k
[14:43:02] <ottomata>	 wha it hasn't run since the 10th
[14:43:08] <ottomata>	 or, hasn't logged since then
[14:43:10] <ottomata>	 checking cron logs
[14:43:12] <joal>	 wow
[14:43:27] <joal>	 No unhealthy node why you weren't here though
[14:43:39] <ottomata>	 ok phew, so not that long
[14:43:42] <joal>	 This is my way of checking if balancver runs
[14:43:44] <ottomata>	 mabye its only been like that for a bit
[14:44:00] <ottomata>	 I should make an alert for unhealthy nodes
[14:44:01] <ottomata>	 that would be easy
[14:44:07] <joal>	 yup
[14:44:12] <joal>	 Would like that as well
[14:44:13] <ottomata>	 i do see it attempted to run today at least
[14:46:00] <ottomata>	 hmmhm, i think its been running
[14:46:25] <ottomata>	 hm i think its just not logging well
[14:47:13] <joal>	 ok
[14:47:20] <joal>	 weird :(
[14:47:57] <ottomata>	 yeah ah its output is mostly on stderr, fixing that.
[14:48:13] <ottomata>	 Nneewway
[14:48:47] <ottomata>	 so ja
[14:48:55] <ottomata>	 right now, i think all nodes get about 21 containers
[14:49:20] <ottomata>	 hm, wonder if it says that somewhere
[14:49:25] <ottomata>	 total # containers in cluster
[14:49:59] <ottomata>	 whoa, joal, did you deploy pageviews_hourly job?
[14:49:59] <joal>	 this number is due to memory calculation, correct ?
[14:50:05] <ottomata>	 in production?
[14:50:07] <ottomata>	 yes
[14:50:28] <ottomata>	 https://github.com/wikimedia/operations-puppet/blob/production/manifests/role/analytics/hadoop.pp#L187
[14:50:33] <joal>	 ottomata: used a oozie job in my folder, but ran it in production queue to try to get some resources
[14:50:51] <ottomata>	 ah cool
[14:50:56] <ottomata>	 just saw it and was like whoooaaoo COOOL
[14:50:57] <ottomata>	 :)
[14:51:02] <ottomata>	 ja, so
[14:51:06] <joal>	 This is another topic I'd like to discuss: spark fight for resources
[14:51:08] <ottomata>	 the previous settings i was using
[14:51:08] <ottomata>	 did
[14:51:13] <ottomata>	         # number of containers = min (2*CORES, 1.8*DISKS, (Total available RAM) / MIN_CONTAINER_SIZE)
[14:51:17] <ottomata>	 its a min
[14:51:22] <ottomata>	 and since all nodes have the same # of disks
[14:51:24] <ottomata>	 (12)
[14:51:38] <ottomata>	 floor(1.8*12) = 21.6
[14:51:40] <ottomata>	 sorry
[14:51:40] <ottomata>	 21
[14:51:49] <joal>	 ok
[14:52:06] <joal>	 That is good for IO
[14:53:03] <joal>	 But assuming that non-IO bounded containers will be evenly distributed, we can probably add some containers in there, based on RAM
[14:53:22] <ottomata>	 hm
[14:53:24] <joal>	 leading to --> your modification
[14:53:29] <joal>	 right ?
[14:53:35] <ottomata>	 right, but the worry is allocating more than cores now
[14:53:43] <joal>	 yep
[14:53:51] <joal>	 That's not a very good idea
[14:53:52] <ottomata>	 cpu load looks hihg on those newer nodes now
[14:53:56] <ottomata>	 but, that's because it is a %
[14:54:05] <ottomata>	 and hyperthreading is off, so it looks like they have 12 cores
[14:54:11] <ottomata>	 assuming they each get 21 containers
[14:54:20] <ottomata>	 21/12 > 100%
[14:54:24] <ottomata>	 (that's if YARN maxed them out)
[14:54:37] <joal>	 I think we should setup hyperthreading, don't you think ?
[14:55:02] <ottomata>	 i've googled this before, doing it again, don't remmeber which
[14:55:06] <ottomata>	 is should def be consistent
[14:55:12] <joal>	 yeah
[14:55:51] <ottomata>	 OO
[14:55:53] <ottomata>	 this is our hw
[14:55:54] <ottomata>	 http://www.slideshare.net/lhrc_mikeyp/ai10-optimizing-poweredgehadoopvfnl
[14:56:45] <ottomata>	 http://image.slidesharecdn.com/ai10optimizingpoweredgehadoopvfnl-130608190555-phpapp01/95/optimizing-dell-poweredge-configurations-for-hadoop-15-638.jpg?cb=1370718739• Assume 1.5 Hadoop Tasks per physical core– Turn Hyperthreading on– This allows headroom for other processes
[14:56:45] <ottomata>	 ?
[14:56:49] <ottomata>	 sure then let's turn it on. ;)
[14:56:52] <ottomata>	 :)
[14:58:18] <ottomata>	 ok. joal.  let's do this:  I merge this puppet change as is, but I will reboot each nodemanager and turn on hyperthreading and fix that ticket as we go
[14:58:21] <joal>	 And then, if they say 1.5 container per core, 21 seems even a bit high, no ?
[14:58:28] <ottomata>	 12*1.5
[14:58:30] <ottomata>	 oops
[14:58:31] <ottomata>	 ha
[14:58:32] <ottomata>	 yes
[14:58:46] <joal>	 Let's keep it this way and see, ok ?
[14:59:03] <ottomata>	 keep it what way?
[14:59:16] <joal>	 when you say fix that ticket, you're talking about hte jdk version one ?
[14:59:27] <ottomata>	 things:
[14:59:37] <ottomata>	 jdk upgrade, that's for security.  that will be fine
[14:59:43] <joal>	 k
[14:59:48] <ottomata>	 i meant to do that when we did the cluster upgrade
[14:59:50] <ottomata>	 but forgot
[14:59:54] <ottomata>	 that's easy
[14:59:58] <joal>	 :)
[15:00:00] <ottomata>	 hyperthreading, i should just turn it on
[15:00:04] <joal>	 ok
[15:00:17] <ottomata>	 made that ticket a while ago but didn't want to deal with it (have to reboot all nodes into bios, etc. really tedious)
[15:00:23] <joal>	 About memory, I like the idea of having predefined size
[15:00:38] <ottomata>	 so, you think we should try my change?
[15:00:56] <joal>	 I think we should, but I want to confirm
[15:01:04] <ottomata>	 confirm...that things don't explode?
[15:01:37] <ottomata>	 so. actually, something we are saying isn't exactly right.
[15:01:40] <ottomata>	 mostly it is
[15:01:44] <ottomata>	 so, we are tweaking mainly 2 settings
[15:01:52] <ottomata>	 1.  amount of RAM available to NMs
[15:01:57] <ottomata>	 and
[15:02:11] <ottomata>	 the default job RAM size
[15:02:26] <joal>	 but not the max number of containers :)
[15:02:26] <ottomata>	 the number of containers we are talking about comes from that
[15:02:54] <joal>	 let's batcave for a minute
[15:02:57] <ottomata>	 so, for example, we know that impala will likely allocate containers that are smaller than the default job size
[15:03:00] <ottomata>	 ok.
[15:04:29] <joal>	 CAn't hear you :(
[15:05:33] <joal>	 You there?
[15:06:06] <wikibugs>	 Analytics-Dashiki: Dashboard Directory research: Look at Hay's directory - https://phabricator.wikimedia.org/T99675#1296343 (Milimetric) NEW
[15:11:45] <wikibugs>	 Analytics-Kanban, Analytics-Visualization: Integrate Dygraphs into Vital Signs {crow} [13 pts] - https://phabricator.wikimedia.org/T96339#1296357 (Milimetric) p:Normal>Triage
[15:12:11] <wikibugs>	 Analytics-Kanban, Quarry: it would be useful to run the same Quarry query conveniently in several database - https://phabricator.wikimedia.org/T95582#1296360 (Milimetric) a:Milimetric>None
[15:12:40] <wikibugs>	 Analytics-Cluster, Analytics-Kanban: Report pageviews to the annual report - https://phabricator.wikimedia.org/T95573#1296361 (Milimetric) a:Milimetric>None
[15:13:32] <wikibugs>	 Analytics-Kanban: Fix analysis that categorizes users (anonymous is bad) {lion} - https://phabricator.wikimedia.org/T92300#1296369 (Milimetric) Open>Invalid This turned out to be untrue, the users are tagged correctly as far as we can see, it's just that the rates of anonymous usage of Visual Editor was...
[15:14:45] <wikibugs>	 Analytics-Dashiki, Analytics-Kanban: Add time range selection to Limn dashboards (or new Dashiki dashboards) {frog} - https://phabricator.wikimedia.org/T87603#1296381 (Milimetric) p:Low>Normal
[15:14:58] <wikibugs>	 Analytics-Dashiki, Analytics-Kanban: Add time range selection to Limn dashboards (or new Dashiki dashboards) {frog} - https://phabricator.wikimedia.org/T87603#994953 (Milimetric) p:Low>Normal
[15:41:20] <wikibugs>	 Analytics-EventLogging, Analytics-Kanban: Backfill EL missing data for 2015-05-06 - https://phabricator.wikimedia.org/T98729#1296460 (mforns) Open>Resolved
[15:41:36] <wikibugs>	 Analytics-EventLogging, Analytics-Kanban: ContentTranslationError event logging table is not receiving new events - https://phabricator.wikimedia.org/T98842#1296463 (mforns)
[15:41:38] <wikibugs>	 Analytics-EventLogging, Analytics-Kanban, Patch-For-Review: Troubleshoot EL performance problems on 2015-05-06 - https://phabricator.wikimedia.org/T98588#1296462 (mforns) Open>Resolved
[15:48:08] <wikibugs>	 Analytics-Cluster, Analytics-Kanban: Report pageviews to the annual report - https://phabricator.wikimedia.org/T95573#1296474 (Heather) What does up for grabs mean? :)
[15:51:55] <wikibugs>	 Analytics-Wikistats: Wikistats list of Wikipedias incomplete - https://phabricator.wikimedia.org/T99677#1296482 (Mdennis-WMF) NEW
[16:02:38] <wikibugs>	 Analytics-Wikistats: Wikistats list of Wikipedias incomplete - https://phabricator.wikimedia.org/T99677#1296522 (Mdennis-WMF)
[16:19:07] <wikibugs>	 Analytics-EventLogging, Analytics-Kanban: Code to write a new Camus consumer and store the data in two Hive tables [21 points] {oryx} - https://phabricator.wikimedia.org/T98784#1296582 (Milimetric)
[16:19:17] <wikibugs>	 Analytics-EventLogging, Analytics-Kanban, Patch-For-Review: Configuration for Python & Kafka in Beta labs [8 points] {oryx} - https://phabricator.wikimedia.org/T98780#1296583 (Milimetric)
[16:19:25] <wikibugs>	 Analytics-EventLogging, Analytics-Kanban, Patch-For-Review: Forwarder modification to produce to multiple outuputs, 1 to zmq and 1 to Kafka [8 points] {oryx} - https://phabricator.wikimedia.org/T98779#1296587 (Milimetric)
[16:24:58] <joal>	 ottomata: let me know when you are ready for a chat
[16:26:44] <ottomata>	 k, getting lunch...
[16:26:51] <joal>	 Enjoy :)
[17:04:23] <ottomata>	 joal: HIyyy, i haven't started the nodes yet, but we can chat
[17:04:27] <ottomata>	 can we just chat here?
[17:04:33] <joal>	 sure :)
[17:05:11] <joal>	 So, let's start with ELhn
[17:05:37] <joal>	 The comment you made on the coed review about key
[17:05:44] <joal>	 What do you think we should do there
[17:05:47] <joal>	 ?
[17:07:01] <ottomata>	 hn?
[17:07:03] <ottomata>	 commennnnnt
[17:07:05] <ottomata>	 looking
[17:10:46] <ottomata>	 joal: i forget, what does the event look like if it is raw?
[17:10:50] <ottomata>	 i think i am a bit out of hte loop here
[17:11:00] <joal>	 :)
[17:11:07] <joal>	 it depends where it comes from
[17:11:24] <ottomata>	 why did you need to add raw for all these writers?
[17:11:43] <joal>	 well, only two :)
[17:12:12] <ottomata>	 5, no?
[17:12:14] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/210701/8/server/eventlogging/handlers.py
[17:12:26] <ottomata>	 raw was added as a parameter for 5 of them
[17:12:39] <ottomata>	 sorry
[17:12:40] <ottomata>	 no 4
[17:13:10] <joal>	 forgot I had added them to other writers
[17:13:13] <joal>	 let me explain
[17:13:49] <joal>	 before the change, the forwarder was treating raw event and using a zmq publisher without the writer abstractio
[17:14:24] <joal>	 having the writer abstraction, we need to tell those writers if the events we write are raw or not
[17:14:46] <joal>	 And I added the raw param to stdout and log to ensure compatibility with th
[17:14:59] <joal>	 with the forwarde
[17:15:04] <ottomata>	 hmm, ok, i think i get it
[17:15:07] <joal>	 makes sense ?
[17:15:29] <joal>	 The writer abstraction used always to decode json
[17:15:37] <ottomata>	 why does zmq writer get json.dumps with check_circular, but not kafka
[17:15:40] <joal>	 now we need to send raw as well
[17:15:40] <ottomata>	 oh...that might have been me.
[17:15:46] <ottomata>	 since I wrote the kafka one
[17:15:47] <ottomata>	 hm
[17:16:04] <joal>	 don't know, didn't get into that level f detail for json :)
[17:16:38] <ottomata>	 hm, yea weird, not sure why that is set at all, ori	 must know
[17:16:40] <joal>	 Now about the key thing, my idea was to be able to pass a key as parameter for raw events (not defined in the event itself)
[17:16:58] <joal>	 But I am not sure if it is usefull
[17:17:29] <ottomata>	 hmm, i see, but that also makes it consistent with other writers
[17:17:33] <ottomata>	 i think that is a good idea
[17:17:53] <ottomata>	 but, yeah
[17:18:04] <ottomata>	 you should probably not reset the key to the schema_rev is key was provided
[17:18:22] <joal>	 Agreed !
[17:18:30] <joal>	 Will change that and resubmit
[17:18:45] <joal>	 We good on that, or other comments ?
[17:18:51] <ottomata>	 ok cool.  do we need to use a string key at all?  can we do Nil key?  not sure.
[17:19:02] <ottomata>	 pretty sure kafka can use null keys
[17:19:06] <ottomata>	 not sure bout python kafka
[17:19:14] <joal>	 yeah, that's the thing
[17:19:27] <joal>	 we got errors when not encoding keys as urf8
[17:19:30] <joal>	 so weird
[17:19:40] <ottomata>	 hm, ok, well i guess it doesn't really matter
[17:19:56] <ottomata>	 can you add some documentation?
[17:20:02] <ottomata>	 that specifies the behavoir?
[17:20:04] <ottomata>	 right now it just says
[17:20:07] <ottomata>	  """Write events to Kafka, keyed by SCID."""
[17:20:22] <joal>	 yup, will do !
[17:20:27] <ottomata>	 make it say that if events are not raw and key is not set, the key will be schema_rev
[17:20:29] <ottomata>	 cool
[17:20:33] <ottomata>	 aside from that +1
[17:20:33] <ottomata>	 :)
[17:20:41] <joal>	 cool, thx :)
[17:21:48] <wikibugs>	 Analytics-Wikimetrics, MediaWiki-Vagrant: Vagrant Setup alembic config errors - https://phabricator.wikimedia.org/T99631#1296778 (Milimetric) I took a look this morning and this worked out of the box for me.  The error pasted above basically means the configuration file didn't get written to /etc/wikimet...
[17:21:49] <joal>	 Let me do that and then get back to you about spark resources :)
[17:22:43] * milimetric goin to grab lunch
[17:23:09] <ottomata>	 k
[17:26:29] <joal>	 ottomata: let me know if you want help monitoring stuff while you apply the change
[17:26:50] <ottomata>	 joal: i think mostly keeping an aye out to make sure the jobs all run
[17:27:02] <joal>	 k, will do
[17:28:11] <joal>	 will get diner first :)
[17:28:15] <joal>	 back in a bit
[17:29:24] <ottomata>	 mmk
[17:32:56] <wikibugs>	 Analytics-Cluster: Audit hyperthreading on analytics nodes. - https://phabricator.wikimedia.org/T90640#1296812 (Ottomata) I will be starting on nodes 1028 - 1041 today.
[17:59:47] <ottomata>	 walking home, back shortly
[18:12:07] <mforns>	 milimetric, do you have 10 minutes to talk about dashiki build?
[18:14:16] <milimetric>	 mforns: yes, just finished eating, to the batcave!
[18:14:28] <mforns>	 milimetric, ok