[00:32:29] <nuria>	 madhuvishy: ok, after looking into this for 3 hours i know where 25% of our nocookie traffic comes from
[00:32:41] <nuria>	 madhuvishy: that leaves... ahem.. 75%
[00:33:06] <nuria>	 madhuvishy: i think 4% are -for sure - users and 20% are -for sure- bots
[00:33:15] <nuria>	 madhuvishy: will review my numbers
[00:36:43] <nuria>	 madhuvishy: so to our last access numbers we need to add at least 1%
[00:51:21] <ebernhardson>	 is there any issue with me temporarily using ~120G on the analytics cluster? ganglia makes it look like its pretty minimal compared to available space, but thought i should ask
[00:52:30] <ebernhardson>	 basically creating a weeks fill of a reduced pageview_hourly table (only project, page_title, view_count and page_id fields) so i have something with page_id values to work with for testing generation of cirrus scoring information from it
[00:52:35] <ebernhardson>	 (in /user/ebernhardson)
[04:47:46] <icinga-wm>	 PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 26.67% of data above the critical threshold [30.0]
[04:51:38] <icinga-wm>	 RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 25.00% above the threshold [20.0]
[09:52:22] <grrrit-wm>	 (PS1) Addshore: Ignore values with _s in them [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255512
[09:52:38] <grrrit-wm>	 (CR) Addshore: [C: 2 V: 2] Ignore values with _s in them [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255512 (owner: Addshore)
[10:02:44] <grrrit-wm>	 (CR) Joal: [C: 2 V: 2] "Merging :)" [analytics/refinery] - https://gerrit.wikimedia.org/r/236224 (owner: Joal)
[10:28:44] <wikibugs>	 Analytics-Backlog, Analytics-EventLogging, Analytics-Kanban, Patch-For-Review: More solid Eventlogging alarms for raw/validated {oryx} [8 pts] - https://phabricator.wikimedia.org/T116035#1833425 (JAllemandou) a:JAllemandou>Ottomata
[10:29:40] <wikibugs>	 Analytics-Backlog, Analytics-EventLogging, Analytics-Kanban, Patch-For-Review: More solid Eventlogging alarms for raw/validated {oryx} [8 pts] - https://phabricator.wikimedia.org/T116035#1738860 (JAllemandou) Code looks ok, have been merged, then a ton of icinga check errors were raised. Andrew take...
[11:31:31] <grrrit-wm>	 (Draft4) Addshore: Add rc script [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/255518
[11:46:20] <mforns>	 hi joal?
[12:08:41] <joal>	 Hey mforns :)
[12:09:19] <joal>	 will need to go and give lunch to Lino soon
[12:09:35] <joal>	 I have reviewed / re-organised the etherpad a bit
[12:10:06] <joal>	 I think now we should look for the link between distinct IPs per ciry and per UAs
[12:10:11] <joal>	 mforns: --^
[12:12:45] <mforns>	 joal, hey
[12:12:47] <mforns>	 :]
[12:12:56] <mforns>	 I saw you organized it
[12:13:31] <mforns>	 I was reading a bit on k-anonymity and l-diversity and t-closeness
[12:13:50] <joal>	 k
[12:13:55] <mforns>	 and thinking of how we were going to approach the sanitizing
[12:14:55] <mforns>	 I have some ideas if you want to discuss in the batcave, but first go have lunch, I'll try to get some relation between #ips and #pageviews
[12:15:05] <joal>	 great mforns
[12:15:17] <joal>	 I'll ping you when the little one goes back to bed :)
[12:15:23] <mforns>	 cool joal
[13:49:44] <joal>	 hey mforns
[13:49:49] <joal>	 I'm back :)
[13:49:50] <mforns>	 hey joal :]
[13:49:52] <mforns>	 cool
[13:49:59] <mforns>	 batcave?
[13:50:05] <joal>	 sure, omw
[13:56:39] <icinga-wm>	 PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 26.67% of data above the critical threshold [30.0]
[13:58:46] <icinga-wm>	 RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 25.00% above the threshold [20.0]
[16:19:45] <joal>	 ottomata !!!! Helloooooo :)
[16:19:57] <joal>	 Sorry to jump on you on arrival :)
[16:20:55] <joal>	 hue is dead I think ottomata :(
[16:23:34] <ottomata>	 HMm ok...
[16:23:46] <joal>	 ottomata: sorry :(
[16:25:01] <ottomata>	 !log restarting hue on analytics1027
[16:25:15] <joal>	 ottomata: Do I have the right to do that ?
[16:26:39] <ottomata>	 hm, don't think so
[16:26:42] <ottomata>	 but, i dont' htink it helped
[16:26:47] <joal>	 qrfn
[16:26:51] <ottomata>	 you can look at logs in /var/log/hue there though
[16:26:56] <ottomata>	 [26/Nov/2015 16:25:20 ] supervisor   ERROR    Process /usr/lib/hue/build/env/bin/hue runcpserver exited abnormally. Restarting it.
[16:27:02] <ottomata>	 not sure why yet though..
[16:27:02] <joal>	 Oh ok
[16:28:11] <ottomata>	 hm ok its ok now
[16:28:19] <ottomata>	 it just didn't restart properly because one of the processses didn't shut down
[16:28:24] <ottomata>	 so there was a port conflect
[16:28:30] <ottomata>	 not sure why it broke though
[16:28:38] <joal>	 ok, so you actually killed then start instead of restart
[16:29:43] <ottomata>	 yeah
[16:29:57] <joal>	 Ok, thanks a lot ottomata :)
[16:30:01] <ottomata>	 joal:  btw, afaik your check graphite stuff is totally fine
[16:30:19] <ottomata>	 its out on all kafka_drerr checks for varnishkafka now
[16:30:24] <joal>	 yeah, I mean, I expected so, reviewed multiple times, tested etc
[16:30:24] <ottomata>	 i'm going to just enable it for the eventlogging one
[16:30:26] <joal>	 But dpeloy ?
[16:30:36] <ottomata>	 and then we can make it happen for all on monday
[16:30:43] <joal>	 ok ottomata
[16:30:44] <ottomata>	 just want to keep iteasy to revert until it is not holiday :)
[16:30:52] <joal>	 Supposedly no deploy in the next 7 weeks ?
[16:31:03] <joal>	 ottomata: --^ will be ok ?
[16:31:04] <ottomata>	 no, not true, that is for mediawiki
[16:31:09] <joal>	 ah ok :)
[16:31:11] <ottomata>	 and, it is not all next 7 weeks
[16:31:17] <ottomata>	 just these two
[16:31:17] <joal>	 rught :)
[16:31:22] <ottomata>	 and then the few around xmas and the all hands
[16:31:47] <joal>	 k makes sense
[16:32:39] <joal>	 ottomata: if you apply the change to EL, would be awesome if you could change the metric and thresholds as well
[16:34:00] <ottomata>	 ja am doing that bit, just taking exactly what you had for it and using that
[16:34:05] <ottomata>	 this right?
[16:34:05] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/254846/3/modules/eventlogging/manifests/monitoring/graphite.pp
[16:35:05] <ottomata>	 i also have a vested interest in this thing working, btw
[16:35:10] <ottomata>	 am tired of getting texted by it :p
[16:35:10] <ottomata>	 :)
[16:35:19] <ottomata>	 joal https://gerrit.wikimedia.org/r/#/c/255550/
[16:35:57] <joal>	 awesomeyeah, same for me re3viewing emails :)
[17:01:09] <joal>	 ottomata: quick standup?
[17:01:27] <joal>	 Saying hello ottomata ;)
[17:02:05] <ottomata>	 okj!
[17:11:19] <icinga-wm>	 PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 26.67% of data above the critical threshold [30.0]
[17:13:19] <icinga-wm>	 RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 25.00% above the threshold [20.0]
[17:25:32] <ottomata>	 ok joal, mforns.  eventlogging_difference_raw_validated now using --until 5min
[17:25:43] <joal>	 great ottomata : Thanks !
[17:33:17] <icinga-wm>	 PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [30.0]
[17:35:17] <icinga-wm>	 RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 20.00% above the threshold [20.0]
[17:53:17] <icinga-wm>	 PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [30.0]
[18:03:37] <icinga-wm>	 RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 20.00% above the threshold [20.0]
[18:09:46] <icinga-wm>	 PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 26.67% of data above the critical threshold [30.0]
[18:09:55] <joal>	 ottomata: you here?
[18:11:29] <ottomata>	 ja
[18:11:32] <ottomata>	 dawww
[18:11:35] <joal>	 yup
[18:12:48] <ottomata>	 why the heck does it even think that
[18:13:37] <icinga-wm>	 RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 20.00% above the threshold [20.0]
[18:14:10] <ottomata>	 it thinks that 3 minutes in the 15 minute window had over 30 not validated events?
[18:14:14] <ottomata>	 is that right?
[18:14:47] <ottomata>	 i didn't get texted for these though...?
[18:14:57] <ottomata>	 ohh
[18:14:58] <joal>	 no, it thinks that 3 points in the last 15 minutes (withj 5 minutes lag) are not equal
[18:15:01] <ottomata>	 but that' because i have no signal at home
[18:15:01] <ottomata>	 haha
[18:15:12] <ottomata>	 not equal?
[18:15:23] <ottomata>	 no the threshold for alerting critical is 30
[18:15:24] <ottomata>	 right?
[18:17:10] <ottomata>	 k btw, if i remove the absolute from the graph, it dips below 0 at times
[18:17:17] <ottomata>	 i assume just because of the same reason this alert is funky
[18:17:30] <joal>	 yeah, it's not absolute the issue, it's lag that should be bigger
[18:17:45] <ottomata>	 because the numbers are reported at slightly different times, so the difference is weird
[18:17:48] <ottomata>	 yeah, maybe larger lag should help
[18:17:51] <ottomata>	 or maybe we should just ditch this alert
[18:17:53] <ottomata>	 altogether
[18:18:03] <ottomata>	 and just rely on EventError or somerthing...
[18:18:05] <ottomata>	 not as good, but
[18:18:06] <ottomata>	 hm
[18:18:06] <joal>	 absolute is good --> since we diff values that can erratically be bigger or smaller, we want the absolute
[18:18:29] <joal>	 But, looking with marcel, the lag having diff is bigger than 5 minutes: actually more about 8
[18:18:36] <ottomata>	 i mean i guess so, in that case the movingAverage is better, because the abosluted negatives are realy strange to see
[18:18:49] <ottomata>	 ok i guess so, submit that thar patch and I will merge it:)
[18:18:51] <ottomata>	 make it 10 :)
[18:18:58] <joal>	 Awesome
[18:19:00] <joal>	 Thanks
[18:19:03] <ottomata>	 worth a try
[18:19:13] * joal goes to pupeet
[18:22:31] <joal>	 ottomata: https://gerrit.wikimedia.org/r/255556
[18:23:00] <ottomata>	 joal # At least 3 of the (20 - 5) = 15 readings
[18:23:05] <ottomata>	 is that now 25 - 10
[18:23:05] <ottomata>	 ?
[18:23:15] <joal>	 It is ottomata :)
[18:23:54] * joal amends !
[18:24:16] <joal>	 ottomata: done
[18:30:52] <joal>	 thanks for merging ottomata
[18:31:05] <joal>	 I'm off for now, but will look occasionally to the chan
[18:31:13] <joal>	 Have a good end of day a-team :)
[18:31:23] <mforns>	 bye joal :]
[18:34:15] <ottomata>	 laters!
[20:47:22] <mforns>	 a-team see you tomorrow!
[22:04:04] <wikibugs>	 Analytics-Backlog, Wikimedia-Developer-Summit-2016: Developer summit session:  Pageview API  overview - https://phabricator.wikimedia.org/T112956#1834472 (Symac) I don't know where is the best place to report this but when I had an average time of 200ms for each request last week, I am mainly today at 1s,...
[22:33:40] <wikibugs>	 Analytics-Backlog, Wikimedia-Developer-Summit-2016: Developer summit session:  Pageview API  overview - https://phabricator.wikimedia.org/T112956#1834516 (JAllemandou) >>! In T112956#1834472, @Symac wrote: > I don't know where is the best place to report this but when I had an average time of 200ms for ea...