[01:25:06] <wikibugs>	 Analytics-Backlog, Labs, Wikimedia-Apache-configuration, operations, and 3 others: https://wikitech.wikimedia.org/beacon/statsv 404 Not Found - https://phabricator.wikimedia.org/T104359#1600598 (Krenair) Open>Resolved
[08:32:50] <wikibugs>	 Analytics, operations, Graphite: Graphite `reqstats.` hierarchy is filled with apparently unused metrics for each of our wiki domains - https://phabricator.wikimedia.org/T111318#1601266 (hashar) NEW
[08:45:39] <wikibugs>	 Analytics, operations, Graphite: Graphite `reqstats.` hierarchy is filled with apparently unused metrics for each of our wiki domains - https://phabricator.wikimedia.org/T111318#1601287 (hashar)
[08:45:42] <wikibugs>	 Analytics-Kanban, operations, Monitoring, Patch-For-Review: Overhaul reqstats - https://phabricator.wikimedia.org/T83580#1601288 (hashar)
[08:46:39] <wikibugs>	 Analytics, operations, Graphite: Graphite `reqstats.` hierarchy is filled with apparently unused metrics for each of our wiki domains - https://phabricator.wikimedia.org/T111318#1601266 (hashar) Per @fgiunchedi , `reqstats.` is being overhauled: {T83580}.
[13:23:15] <icinga-wm>	 PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 26.67% of data above the critical threshold [30.0]
[13:25:24] <icinga-wm>	 RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 25.00% above the threshold [20.0]
[14:17:16] <nuria>	 holaaaaa
[14:18:51] <mforns>	 hey nuria!
[14:22:16] <milimetric>	 hey, you're up early
[14:22:58] <milimetric>	 5am the cat came and woke me up.  I think I'm gonna get a dog :)
[15:09:33] <joal>	 Hi nuria
[15:09:39] <nuria>	 holaaa joal
[15:09:52] <joal>	 Quick question about the ticket update
[15:10:12] <nuria>	 joal: yes, please correct as needed
[15:10:38] <joal>	 You say the check implies change to pageview definition itself
[15:10:47] <joal>	 And I don't get why
[15:11:03] <nuria>	 joal: ah, poor wording on my part
[15:11:09] <joal>	 So before being an idiot in the ticket, I prefer to be one on IRC :)
[15:12:27] <nuria>	 joal;
[15:12:30] <nuria>	 corrected
[15:13:00] <nuria>	 joal: "This implies changes to the code that tags pageviews but not to the pageview definition itself."
[15:13:00] <joal>	 hm, I still don't get it :)
[15:13:11] <joal>	 batcave for a minute ?
[15:13:33] <nuria>	 joal:k
[15:24:53] <wikibugs>	 Analytics-Cluster, operations, ops-eqiad, Patch-For-Review: rack new hadoop worker nodes - https://phabricator.wikimedia.org/T104463#1602601 (Ottomata) I just installed and puppetized these nodes.  Thanks!
[15:25:07] <wikibugs>	 Analytics-Cluster, Analytics-Kanban: {mule} Hadoop Cluster Expansion - https://phabricator.wikimedia.org/T99952#1602603 (Ottomata) All new Hadoop workers have been racked and installed, woot!
[15:31:39] <madhuvishy>	 ottomata: Standup :)
[15:31:57] <ottomata>	 !
[16:12:31] <wikibugs>	 Analytics-Backlog, Analytics-Cluster: Audit kernel version on analytics worker nodes - https://phabricator.wikimedia.org/T109834#1602736 (Ottomata) Currently doing:    apt-get install linux-headers-3.13.0-62 linux-headers-3.13.0-62-generic linux-image-3.13.0-62-generic linux-image-extra-3.13.0-62-generic...
[16:31:50] <nuria>	 ottomata: coming to retro?
[16:32:01] <nuria>	 ottomata: i wasn't sure if you attend now
[16:42:20] <wikibugs>	 Analytics-Kanban: Puppetize dashiki dashboard deployments - https://phabricator.wikimedia.org/T110351#1602952 (Milimetric) a:Milimetric
[16:43:00] <wikibugs>	 Analytics-Kanban, Reading-Admin, Patch-For-Review, Wikipedia-Android-App: Update definition of page view and implementation for mobile apps {hawk} [8 pts] - https://phabricator.wikimedia.org/T109383#1602958 (Milimetric) a:Milimetric>Nuria
[16:43:13] <icinga-wm>	 PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 26.67% of data above the critical threshold [30.0]
[16:45:22] <icinga-wm>	 RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 25.00% above the threshold [20.0]
[16:48:29] <wikibugs>	 Analytics-Backlog: Resurrect logbot so we can leave a breadcrumb trails for incident reports, teammates, etc. - https://phabricator.wikimedia.org/T111393#1603009 (Milimetric) NEW
[17:11:12] <wikibugs>	 Analytics-Backlog, Analytics-EventLogging, Privacy: Opt-out from logging some of the default EventLogging fields - https://phabricator.wikimedia.org/T108757#1603196 (Nuria) ClientIp is always encrypted and takes no space so I do not think is an issue.
[17:13:29] <wikibugs>	 Analytics-Backlog, Analytics-EventLogging, Privacy: Opt-out from logging some of the default EventLogging fields - https://phabricator.wikimedia.org/T108757#1603202 (Milimetric) p:Triage>Normal
[17:14:37] <wikibugs>	 Analytics-Backlog, Research consulting, Research-and-Data: Workshop to teach analysts, etc about Quarry, Hive, Wikimetrics and EL {flea} - https://phabricator.wikimedia.org/T105544#1603206 (Milimetric)
[17:17:53] <wikibugs>	 Analytics-Backlog, Analytics-Cluster: Create Kafka deployment checklist on wikitech  - https://phabricator.wikimedia.org/T111408#1603220 (ggellerman) NEW
[17:18:33] <icinga-wm>	 PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 53.33% of data above the critical threshold [30.0]
[17:21:39] <wikibugs>	 Analytics-Backlog, Analytics-Cluster: Spike replacing Camus with Gobblin - https://phabricator.wikimedia.org/T111409#1603236 (ggellerman) NEW
[17:22:33] <icinga-wm>	 RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 25.00% above the threshold [20.0]
[17:26:47] <grrrit-wm>	 (CR) Bmansurov: "Dan, sorry this has been dragging on for too long. I've been pretty busy with other stuff, but tomorrow I'm planning on wrapping this up." [analytics/dashiki] - https://gerrit.wikimedia.org/r/231424 (https://phabricator.wikimedia.org/T104261) (owner: Milimetric)
[17:29:11] <wikibugs>	 Analytics-Backlog, Analytics-EventLogging: Send EventLogging validation logs to Logstash {oryx?} - https://phabricator.wikimedia.org/T111412#1603272 (mforns) NEW
[17:39:34] <wikibugs>	 Analytics-Backlog, Analytics-EventLogging: Send EventLogging validation logs to Logstash {oryx?} - https://phabricator.wikimedia.org/T111412#1603378 (Ottomata) With {stag} (and currently but not official yet), all event processing errors go to a special topic in Kafka 'eventlogging_EventError'.  EventErro...
[17:58:35] <milimetric>	 ottomata: so the EL dashboard shows 3 clear holes: http://grafana.wikimedia.org/#/dashboard/db/eventlogging
[17:58:46] <milimetric>	 and the logs and db are fine during those times
[17:58:49] <milimetric>	 no loss or anything
[17:59:13] <milimetric>	 so something must be up with how stats are being reported or received, heard anything about it from ops?
[18:00:05] <milimetric>	 heh... so now grafana has errors loading the data altogetehr
[18:01:00] <ottomata>	 milimetric:  yes, extra strange is that evnetloging zmq data doesn't have the holes
[18:01:29] <ottomata>	 yeah dunno
[18:01:56] <madhuvishy>	 milimetric: oooh congrats :D
[18:02:18] <milimetric>	 ottomata: the zmq has the holes, they just show up a little different
[18:02:32] <milimetric>	 for the hole yesterday, only the raw rate shows up, the other two disappear
[18:02:49] <milimetric>	 for the first hole today, at 13:00, all the metrics are not on the graph if you zoom in there
[18:02:54] <milimetric>	 congragz?
[18:03:02] <milimetric>	 oh, metrics meeting, lemme tune in
[18:03:12] <madhuvishy>	 milimetric: yeah, they just announced you finished 3 years :D
[18:04:39] <milimetric>	 ooh, I had forgotten, yay :)
[18:52:26] <joal>	 ottomata: problem with camus earlier ?
[18:52:50] <joal>	 looking at the kafka chart, 2 hours ago
[19:01:53] <ottomata>	 yeah joal i saw that too, i looked at camus logs, and everything seems fine
[19:02:06] <ottomata>	 oh, i did do a rolling restart of nodemanagers (2 hours ago though?  no.)
[19:02:07] <ottomata>	 hm
[19:02:51] <joal>	 weird
[19:43:52] <nuria>	 madhuvishy: tried every trick but the fact that pykafka has a tests package inside it's what tripping the test runner and doesn't look like there is an easy way to not make the test running go into that directory, i will try a bit more
[19:48:19] <ottomata>	 nuria:  whatcha doin?
[19:50:56] <nuria>	 ottomata: getting tests on EL master to run
[19:51:10] <nuria>	 ottomata: with ahem a very ..crude workarround
[19:51:23] <nuria>	 ottomata: cause now they error (nothing wrong with tests)
[19:51:45] <nuria>	 ottomata: rather the pykafka egg has a test directory that trips the runner
[19:52:11] <nuria>	 ottomata: makes sense?
[19:53:37] <madhuvishy>	 nuria: wanna talk now ish?
[19:53:48] <madhuvishy>	 just had too much good chinese food :)
[19:54:04] <nuria>	 madhuvishy: let me submit the crude workarround, to see if you can think of anything better
[19:54:08] <nuria>	 madhuvishy: one sec
[19:54:11] <madhuvishy>	 nuria: alright
[19:56:58] <ottomata>	 hm yea
[20:04:50] <wikibugs>	 Analytics-Backlog, Analytics-EventLogging: eventlogging tests in mainline error - https://phabricator.wikimedia.org/T111438#1603865 (Krenair)
[20:06:46] <wikibugs>	 Analytics-Backlog, Analytics-EventLogging, Analytics-Kanban: eventlogging tests in mainline error - https://phabricator.wikimedia.org/T111438#1603883 (Nuria)
[20:07:15] <joal>	 hey milimetric
[20:07:22] <joal>	 You have a minute ?
[20:07:27] <milimetric>	 sure joal
[20:07:47] <joal>	 I'd like to sync on when we are on PV API
[20:07:51] <nuria>	 madhuvishy: ok, change is in, ...very high tech as you see
[20:07:55] <joal>	 where we are sorry
[20:08:02] <nuria>	 madhuvishy: all yours
[20:08:08] <milimetric>	 joal: no movement at all yet
[20:08:14] <milimetric>	 nothing on any of the tickets we opened
[20:08:14] <madhuvishy>	 nuria: the gerrit patch is not linked on phab
[20:08:28] <milimetric>	 joal: if you wanna try to ping those guys in the european morning tomorrow, I'd appreciate it
[20:08:47] <milimetric>	 Faidon, Alex, or Filipo I guess
[20:09:21] <joal>	 I can do that :)
[20:09:34] <joal>	 more precisely: I will do that
[20:09:49] <milimetric>	 thx much
[20:10:08] <nuria>	 madhuvishy: right, i wonder why? let me add it by hand
[20:10:25] <joal>	 I'm looking for the master ticket milimetric
[20:10:30] <wikibugs>	 Analytics-Backlog, Analytics-EventLogging, Analytics-Kanban: eventlogging tests in mainline error - https://phabricator.wikimedia.org/T111438#1603915 (Nuria) https://gerrit.wikimedia.org/r/#/c/235849/
[20:10:32] <madhuvishy>	 nuria: ya i see it linked in the commit message - this has happened a couple times now
[20:10:37] <joal>	 backlog dashboard ?
[20:10:43] <milimetric>	 looking
[20:10:49] <joal>	 Found it
[20:10:54] <madhuvishy>	 nuria: may be it needs to be the line above change id?
[20:10:54] <joal>	 kanban, paused 1st
[20:11:08] <madhuvishy>	 dont know
[20:11:11] <nuria>	 madhuvishy: let me correct that
[20:11:15] <milimetric>	 joal: https://phabricator.wikimedia.org/T107056
[20:11:23] <milimetric>	 right
[20:11:24] <joal>	 thanx milimetric
[20:11:41] <joal>	 So there is only material provisionning so far, right ?
[20:11:46] <joal>	 milimetric: --^
[20:11:50] <milimetric>	 in terms of blocking tasks?
[20:11:54] <joal>	 yup
[20:11:56] <milimetric>	 yes
[20:12:11] <milimetric>	 well, that's kind of all that's left really for puppetization
[20:12:13] <joal>	 plus the puppet stuff in the ticket itself
[20:12:18] <joal>	 right
[20:12:27] <milimetric>	 and once they approve a certain setup we can know how to change the REST endpoints
[20:12:34] <madhuvishy>	 nuria: yeah now it appeared
[20:12:40] <milimetric>	 and then that'll tell us how to change the load job, if at all
[20:12:55] <joal>	 right
[20:13:05] <nuria>	 madhuvishy: world is coming to an end !!! i forgot the format of commit messages.... good that qchris is not watching
[20:13:10] <milimetric>	 so now we're just waiting on ops, and have to ping :)
[20:13:15] <madhuvishy>	 nuria: lol why does it say uploaded by alex monk
[20:13:16] <milimetric>	 I'll ping too if they don't get back to us soon
[20:13:28] <joal>	 milimetric: I'll still build an oozie bundle with hive cubes
[20:13:37] <qchris>	 nuria: :-)
[20:13:39] <joal>	 To have a bse ready to be cahnged
[20:13:56] <milimetric>	 joal: agreed
[20:14:01] <joal>	 coo
[20:14:03] <joal>	 cool
[20:14:30] <milimetric>	 he's always watching nuria, otherwise the world would come to an end
[20:14:37] <joal>	 ottomata: quick sync on hardware for PV API please?
[20:14:38] <nuria>	 madhu: ok, ready if you want to talk about your job or anything else
[20:14:49] <madhuvishy>	 nuria: yeah, batcave?
[20:15:33] <joal>	 ottomata: I'm gonna try to ping Faidon or Alex or Filipo tomorrow european morning
[20:16:07] <milimetric>	 joal: when you ping them, the main thing we need is a review on the puppetization
[20:16:10] <joal>	 ottomata: I'd like to confirm how many spare we have for Cassandra/Restbase
[20:16:14] <milimetric>	 the rest we can mostly take care of
[20:16:23] <nuria>	 madhu: give me 2 mins
[20:16:24] <milimetric>	 like the hardware, I think otto can help us with
[20:16:31] <madhuvishy>	 sure
[20:16:33] <joal>	 ottomata: Faidon agreed on having 3 better than two
[20:16:42] <joal>	 right milimetric
[20:17:03] <joal>	 milimetric: As you see, I'm trying to grab information ;)
[20:17:58] <joal>	 Also milimetric, we'll wait for better knowledge of core restbase integration before correcting the top end point, right ?
[20:18:37] <milimetric>	 joal: hm, I'm a little skeptical.  I'm more leaning towards going with mostly what we have now
[20:18:52] <milimetric>	 and then later shifting towards what gabriel was suggesting
[20:19:05] <joal>	 so yearly; monthly, daily are only for last year, last month and last day, right
[20:19:14] <milimetric>	 oh, that
[20:19:16] <joal>	 oh, global setting
[20:19:16] <milimetric>	 we can talk about that with the team
[20:19:26] <milimetric>	 yes, global domain I was talking about
[20:19:28] <joal>	 milimetric: missujnderstanding :)
[20:19:31] <ottomata>	 joal:  yes here
[20:19:44] <milimetric>	 but the /top endpoint I don't have strong opinions about.  Maybe let's talk at standup tomorrow
[20:19:45] <joal>	 So milimetric, fully agreed
[20:19:53] <joal>	 Let's start with what we have
[20:19:58] <joal>	 Then maybe change
[20:20:32] <ottomata>	 joal: we have 4 older dells left
[20:20:33] <joal>	 But we still need the 'top' endpoint corrected
[20:20:46] <ottomata>	 if 3 go to cassandra(?) and 1 as an extra stat box
[20:20:49] <ottomata>	 i think that will be fine
[20:20:55] <joal>	 ottomata: awesome :)
[20:21:32] <joal>	 ottomata: I prefer to know a bit before talking with the ops ;)
[20:21:41] <joal>	 ottomata: thanks !
[20:21:50] <madhuvishy>	 ottomata: question - can we change the /etc/spark/conf/log4j.properties to append logs to a file or something
[20:21:55] <madhuvishy>	 joal too ^
[20:22:06] <madhuvishy>	 it's now configured to append to console
[20:22:17] <madhuvishy>	 which means when we run through oozie, they are lost
[20:24:39] <joal>	 madhuvishy: http://stackoverflow.com/questions/28454080/how-to-log-using-log4j-to-local-file-system-inside-a-spark-application-that-runs
[20:25:37] <joal>	 madhuvishy: You can pass a very ugly parameter to the job for a specific conf file (on hdfs I think, for multi-node purpose)
[20:25:42] <ottomata>	 madhuvishy: sure
[20:25:59] <madhuvishy>	 joal: yeah, i think we should change it though
[20:26:09] <joal>	 madhuvishy: why not :)
[20:26:32] <joal>	 milimetric: let's discuss the 'top' end-point tomorrow in standup :)
[20:26:38] <madhuvishy>	 ottomata: is this stuff in a repo? I can submit a patch if so
[20:26:50] <madhuvishy>	 or is it like tell ottomata and it will happen :)
[20:27:38] <milimetric>	 joal: sure, I'll try to remember if you don't
[20:27:43] <joal>	 :)
[20:27:58] * joal is a specialist of forgetting to talk about stuff :)
[20:31:32] <ottomata>	 madhuvishy: its in puppet
[20:31:37] <ottomata>	 in the cdh module
[20:31:40] * madhuvishy looks
[20:31:45] <ottomata>	 its just a file there
[20:31:52] <ottomata>	 could be templatized, mabye.
[20:32:10] <madhuvishy>	 ottomata: it's a submodule?
[20:32:18] <ottomata>	 modules/cdh/files/spark/log4j.properties
[20:32:19] <ottomata>	 yup
[20:33:34] <ottomata>	 madhuvishy: i'm heading out, any other last qs?
[20:33:34] <ottomata>	 :)
[20:33:50] <madhuvishy>	 ottomata: no.. I'm looking at it, thanks :)
[20:35:47] <ottomata>	 mk laterrssrsrsr
[20:40:02] <joal>	 madhuvishy: I have a lead !
[20:40:09] <madhuvishy>	 joal: oh?
[20:40:46] <joal>	 In hue, the failed workflow tells you an error code: JA018
[20:40:59] <joal>	 Then in oozie doc:  JA018 is output directory exists error in workflow map-reduce action
[20:41:08] <madhuvishy>	 huh
[20:41:29] <joal>	 I think my original guess (yesterday) was correct
[20:41:34] <joal>	 I explain more in detail
[20:41:42] <madhuvishy>	 sure. batcave?
[20:41:46] <joal>	 sure !
[20:41:49] <joal>	 omw
[20:56:23] <grrrit-wm>	 (CR) Mforns: [C: 1 V: 2] "LGTM!" (2 comments) [analytics/wikihadoop] - https://gerrit.wikimedia.org/r/233937 (https://phabricator.wikimedia.org/T108684) (owner: Joal)
[21:02:11] <wikibugs>	 Analytics-Backlog, Analytics-EventLogging: Send EventLogging validation logs to Logstash {oryx?} - https://phabricator.wikimedia.org/T111412#1604211 (mforns) This is awesome! \o/ Probably there's a tool to easily move those to Logstash, Will look for it. Thx
[21:11:59] <leila>	 just thought to say hi milimetric, now that I'm around for the rest of the day. ;-)
[21:16:15] <wikibugs>	 Analytics-Backlog, Team-Practices-This-Week: Get regular traffic reports on TPG pages - https://phabricator.wikimedia.org/T99815#1604284 (JAufrecht) Since automation is not cheap, let's do one more manual inspection of data and then decide whether or not there's anything in the data worth the investment of...
[21:25:19] <wikibugs>	 Analytics-Backlog, Team-Practices-This-Week: Get regular traffic reports on TPG pages - https://phabricator.wikimedia.org/T99815#1604341 (JAufrecht) a:JAufrecht>kevinator
[21:31:23] <joal>	 madhuvishy: I'll try to investigate more tomorrow
[21:31:40] <joal>	 madhuvishy: Let me know if you find anythin by email :)
[21:31:43] <madhuvishy>	 joal: no problem, i'll try these things and leave an update for you
[21:31:45] <madhuvishy>	 :)
[21:31:51] <joal>	 Have a good end of day !
[21:32:03] <nuria>	 ciao joal, madhu let me know what you find out
[21:32:11] <joal>	 Bye a-team, see you tomorrow !
[21:32:22] <mforns>	 bye joal see you!
[21:35:01] <madhuvishy>	 nuria: sure
[21:35:13] <madhuvishy>	 good night joal!
[21:49:08] <wikibugs>	 Analytics-Kanban, Reading-Admin, Patch-For-Review, Wikipedia-Android-App: Update definition of page view and implementation for mobile apps {hawk} [8 pts] - https://phabricator.wikimedia.org/T109383#1604466 (Nuria) Apps need to tag on X-Analytics header whether a request is a pageview or a preview (r...
[21:53:37] <nuria>	 milimetric: then, for the mobile stuff i will just go ahead and do code changes?
[21:54:13] <nuria>	 milimetric: did we agree that if we have preview=1 in the x-analytics header it will not be counted as a pageview?
[22:06:14] <nuria>	 madhu: about to logoff for a bit, did you learned something?
[22:06:55] <madhuvishy>	 nuria: nope, i dont see the logs show up with those options
[22:07:02] <madhuvishy>	 i'll keep trying
[22:11:26] <nuria>	 madhuvishy: ya, it might work in a simper setup than ours
[22:11:34] <madhuvishy>	 nuria: hmmm
[22:11:48] <madhuvishy>	 wonder if the logs go to the actual machine which executes it
[22:11:52] <madhuvishy>	 not sure
[22:12:09] <madhuvishy>	 spark.executor.extraJavaOptions and spark.driver.extraJavaOptions
[22:30:32] <wikibugs>	 Analytics-Backlog, Analytics-Dashiki, Editing-Analysis, Research-and-Data, VisualEditor: Start generating a visual editor adoption metric - https://phabricator.wikimedia.org/T109158#1604667 (Halfak) I think I need a better notion of what is meant/desired by "adoption" before I can comment effect...
[22:40:13] <wikibugs>	 Analytics-Backlog, MediaWiki-API, Research-and-Data: log user agent in api.log - https://phabricator.wikimedia.org/T108618#1604694 (ggellerman)
[22:44:51] <wikibugs>	 Analytics-Backlog, Research consulting, Research-and-Data: Workshop to teach analysts, etc about Quarry, Hive, Wikimetrics and EL {flea} - https://phabricator.wikimedia.org/T105544#1604702 (Halfak) I'm in.  Let me know when you want to run the event and I'll help out.
[23:35:53] <wikibugs>	 Analytics-Backlog, Analytics-Dashiki, Editing-Analysis, VisualEditor, Patch-For-Review: Improve the edit analysis dashboard {lion} - https://phabricator.wikimedia.org/T104261#1604805 (Neil_P._Quinn_WMF) p:High>Normal
[23:50:06] <wikibugs>	 Analytics-Backlog, Analytics-Dashiki, Editing-Analysis, Research-and-Data, VisualEditor: Start generating a visual editor adoption metric - https://phabricator.wikimedia.org/T109158#1604878 (Neil_P._Quinn_WMF) p:High>Normal
[23:55:21] <grrrit-wm>	 (PS7) Madhuvishy: Report RESTBase traffic metrics to Graphite [analytics/refinery/source] - https://gerrit.wikimedia.org/r/234453 (https://phabricator.wikimedia.org/T109547)