[04:24:07] <wikibugs>	 Analytics, MediaWiki-Vagrant: confluent-kafka (dependency of eventlogging) fails to build in mediawiki-vagrant - https://phabricator.wikimedia.org/T140447#2465102 (dpatrick)
[04:29:33] <wikibugs>	 Analytics, MediaWiki-Vagrant, Patch-For-Review: confluent-kafka (dependency of eventlogging) fails to build in mediawiki-vagrant - https://phabricator.wikimedia.org/T140447#2465119 (bd808) Open>Resolved a:dpatrick
[08:44:11] <wikibugs>	 Analytics-Kanban, Operations, Traffic, Patch-For-Review: Verify why varnishkafka stats and webrequest logs count differs - https://phabricator.wikimedia.org/T136314#2465314 (elukey) Ran again the query, no empty dt fields for the past hours too. The issue seems solved! We'll might need to tune a...
[08:44:19] <wikibugs>	 Analytics-Kanban, Operations, Traffic: Verify why varnishkafka stats and webrequest logs count differs - https://phabricator.wikimedia.org/T136314#2465315 (elukey)
[10:53:36] <grrrit-wm>	 (PS1) Addshore: Count global users with beta features enabled [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/299125 (https://phabricator.wikimedia.org/T140229)
[11:03:06] * elukey lunch!
[11:09:42] <mforns>	 buon appetito
[11:41:17] <elukey>	 lol
[11:41:19] <elukey>	 :)
[11:41:28] <elukey>	 grazie in ritardo mforns!
[11:46:27] <mforns>	 :]
[11:51:53] <grrrit-wm>	 (CR) WMDE-leszek: Count global users with beta features enabled (3 comments) [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/299125 (https://phabricator.wikimedia.org/T140229) (owner: Addshore)
[12:00:18] <milimetric>	 morning
[12:01:03] <milimetric>	 mforns: reporting for duty
[12:01:40] <mforns>	 milimetric, hey :]
[12:02:20] <mforns>	 what should we do?
[12:02:28] <mforns>	 https://hangouts.google.com/hangouts/_/wikimedia.org/eds-batcave ?
[12:02:30] <milimetric>	 I don't know, what's there to do?
[12:02:35] <mforns>	 hehe
[12:02:38] <milimetric>	 :P
[12:03:01] <milimetric>	 ok, sure, let's do ehr-batcave though - edit history reconstruction
[12:03:04] <milimetric>	 wait what's eds?
[12:03:17] <mforns>	 hehe, edit data sprint
[12:03:21] <mforns>	 whatever
[12:03:37] <milimetric>	 https://hangouts.google.com/hangouts/_/wikimedia.org/ehr-batcave
[12:50:49] <grrrit-wm>	 (CR) WMDE-leszek: Count global users with beta features enabled (1 comment) [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/299125 (https://phabricator.wikimedia.org/T140229) (owner: Addshore)
[12:55:10] <grrrit-wm>	 (CR) WMDE-leszek: [C: -1] Count global users with beta features enabled (1 comment) [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/299125 (https://phabricator.wikimedia.org/T140229) (owner: Addshore)
[13:07:54] <grrrit-wm>	 (PS2) Addshore: Count global users with beta features enabled [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/299125 (https://phabricator.wikimedia.org/T140229)
[13:12:06] <elukey>	 mobrovac: o/ time for a scap3 question?
[13:15:10] <grrrit-wm>	 (PS3) Addshore: Count global users with beta features enabled [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/299125 (https://phabricator.wikimedia.org/T140229)
[13:25:12] <ottomata>	 hmmm elukey you have had more eyes on GC metrics than I have
[13:25:26] <ottomata>	 i'm looking at the your jmxtrans patch, trying to understand how the other GC things work
[13:25:43] <ottomata>	 why do I reuse resultAlias a few times for different metrics?
[13:26:06] <elukey>	 ottomata: o/
[13:26:15] <elukey>	 that one is my only doubt
[13:26:35] <elukey>	 I supposed that since only a couple of them can be active at the same time maybe it is fine to repeat?
[13:26:41] <ottomata>	 i was about to advise to use name=* and typeNames => ['name']
[13:26:42] <ottomata>	 aye
[13:26:47] <ottomata>	 i was gonna say that
[13:26:51] <ottomata>	 but there are already a few there?
[13:27:22] <elukey>	 no idea, I followed what I found in there.. is it that important the resultAlias?
[13:27:54] <ottomata>	 ja it changes the metric name sent
[13:27:58] <ottomata>	 no?
[13:28:51] <elukey>	 ah snap
[13:28:55] <elukey>	 didn't know it
[13:30:14] <ottomata>	 elukey:  i have not looked at this in  soooo long!
[13:32:23] <elukey>	 ottomata: whenever you have 5 minutes I'd need to brainstorm about scap and refinery
[13:32:39] <elukey>	 I think I got it but there are some unclear points
[13:32:43] <ottomata>	 ok cool
[13:32:48] <ottomata>	 gimme a few, am looking at jmxtrans stuff
[13:32:54] <ottomata>	 will leave a comment there
[13:33:02] <ottomata>	 i think we can clean up some crap i did too :p
[13:36:23] <ottomata>	 elukey:  i can brainstorm via IRC now, or batcave in like 30 mins :)
[13:37:00] <elukey>	 sure! I can write something and then you'll answer when you have time
[13:37:04] <elukey>	 ok?
[13:38:01] <ottomata>	 ja!
[13:38:19] <ottomata>	 i have time now, i'm just ina  room with some other people atm so can't jump in batcave
[13:40:19] <elukey>	 sooo I've read https://doc.wikimedia.org/mw-tools-scap/scap3/ssh-access.html and I merged it with https://wikitech.wikimedia.org/wiki/Scap3/Migration_Guide
[13:40:56] <elukey>	 so I'd say that we could create a pair of pub/priv ssh keys for deploy-analytics
[13:41:10] <elukey>	 upload the priv to the private repo under the proper folder
[13:41:19] <elukey>	 add the config in scap server.yaml
[13:41:45] <elukey>	 and then in scap::target configure the appropriate user etc..
[13:42:18] <elukey>	 but (maybe I am wrong) I didn't see any mention to how to add the ssh pub key to the target host
[13:42:28] <elukey>	 in this case analytics1027
[13:42:34] <elukey>	 to allow tin to connect
[13:42:42] <elukey>	 (and I didn't see any config for eventlogging)
[13:42:45] <elukey>	 am I crazy?
[13:43:51] <ottomata>	 hmmm sounds about right
[13:43:54] <ottomata>	 adding pub key.. hmmm
[13:44:03] <ottomata>	 i thought scap::target does that, looking
[13:44:22] <ottomata>	 ja elukey scap::target does it
[13:44:25] <ottomata>	 via $manage_user
[13:44:50] <ottomata>	 i think that might not be used for eventlogging
[13:44:58] <ottomata>	 ja
[13:45:01] <ottomata>	 for eventlogging it is in
[13:45:06] <ottomata>	 eventlogging::deployment::target
[13:45:10] <ottomata>	 since the eventlogging user already existed before scap
[13:45:17] <ottomata>	 and was used for non scap things
[13:45:37] <elukey>	 checking!
[13:46:01] <ottomata>	 elukey:  there is also the matter of re-arming the keyholder with the new key
[13:46:07] <elukey>	 yep yep
[13:46:15] <ottomata>	 https://wikitech.wikimedia.org/wiki/Keyholder
[13:46:16] <elukey>	 but it should be a one time thing right?
[13:46:17] <ottomata>	 aye cool
[13:46:19] <ottomata>	 just so you know
[13:46:19] <ottomata>	 yup!
[13:46:22] <elukey>	 super
[14:06:30] <wikibugs>	 Analytics-Kanban, Operations, Traffic: Verify why varnishkafka stats and webrequest logs count differs - https://phabricator.wikimedia.org/T136314#2465959 (Ottomata) NICE WORK!
[14:07:43] <elukey>	 \o/
[14:54:14] <elukey>	 ottomata: about the ssh pub key - content => secret("keyholder/${key_name_safe}.pub"),
[14:54:31] <elukey>	 do I need to add it somewhere, or will I just need to add it once manually to the keyholder?
[14:55:13] <ottomata>	 hmm, that i'm not sure ummm
[14:55:38] <ottomata>	 maybe secret module in private?
[14:55:57] <ottomata>	 yup
[14:55:58] <ottomata>	 elukey:
[14:56:03] <ottomata>	 private/modules/secret/secrets
[14:56:10] <ottomata>	  /keyholder
[14:56:11] <ottomata>	 i thikn
[14:56:15] <ottomata>	 yup
[14:57:32] <wikibugs>	 Analytics-Cluster, Operations, Packaging: libcglib3-java replaces libcglib-java in Jessie - https://phabricator.wikimedia.org/T137791#2466093 (fgiunchedi) p:Triage>Low
[14:57:44] <wikibugs>	 Analytics-Cluster, Analytics-Kanban, EventBus, Operations, and 2 others: Better monitoring for Zookeeper - https://phabricator.wikimedia.org/T137302#2466094 (Ottomata) Woot, preliminary dash here:  https://grafana.wikimedia.org/dashboard/db/zookeeper
[14:58:35] <elukey>	 ottomata: ah snap I put the private under /private/modules/secret/secrets/ssh/tin
[14:58:38] <elukey>	 grrr
[14:58:44] <elukey>	 I am going to move it and add the pub
[15:00:55] <nuria_>	 ottomata: no change in log sizes, i guess we can restart namenode but will try -again- to find more info about this
[15:01:16] <ottomata>	 hah yeah
[15:01:17] <ottomata>	 growl
[15:01:27] <ottomata>	 well, i mean, we did restart RMs near end of day yesterday
[15:01:29] <ottomata>	 so hasn't been 24 hours
[15:01:33] <ottomata>	 but ja, you are probably right
[15:05:58] <mobrovac>	 elukey: sorry, was out for a bit
[15:06:02] <mobrovac>	 elukey: what's up?
[15:06:41] <elukey>	 mobrovac: already resolved thanks :)
[15:06:47] <elukey>	 ottomata: keys added!
[15:06:59] <elukey>	 ETOOMUCHTRIBALKNOWLEDGE
[15:07:39] <nuria_>	 ottomata: also i think time is relative to the time the application took to run, so it might not apply retroactively
[15:09:06] <grrrit-wm>	 (CR) WMDE-leszek: [C: 1] Count global users with beta features enabled [analytics/wmde/scripts] - https://gerrit.wikimedia.org/r/299125 (https://phabricator.wikimedia.org/T140229) (owner: Addshore)
[15:09:39] <wikibugs>	 Analytics, Analytics-Cluster, Analytics-Kanban, Deployment-Systems, and 3 others: Deploy analytics-refinery with scap3 - https://phabricator.wikimedia.org/T129151#2466150 (elukey) Just generated and added the pub/priv key to the private repo.
[15:13:12] <ottomata>	 hmm, interesting possibly nuria_ doesn't seem quite right but could be.
[15:13:36] <nuria_>	 ottomata: i am going to spend an hour looking for docs see if i find something
[15:13:53] <ottomata>	 k
[15:14:10] <ottomata>	 nuria_:  you could probably read code for it and figure out how it does it and see if it looks at really logs too
[15:14:30] <nuria_>	 ottomata: ya, looked at deletion service in source forge
[15:14:36] <wikibugs>	 Analytics-Cluster, Analytics-Kanban, EventBus, Operations, and 2 others: Better monitoring for Zookeeper - https://phabricator.wikimedia.org/T137302#2466180 (mobrovac) Nice!
[15:14:46] <nuria_>	 ottomata: but i could not see anything related to thta setting
[15:14:48] <nuria_>	 *that
[15:26:08] <nuria_>	 ottomata: ok, the thing that deletes these logs lives inside JobHistoryServer
[15:27:35] <ottomata>	 nuria_:  HMM, you know...which we don't explicitly run...
[15:27:42] <ottomata>	 this is something i have been a little confused about
[15:28:04] <nuria_>	 no wonder
[15:28:06] <ottomata>	 oh wait
[15:28:09] <ottomata>	 YES we do
[15:28:15] <ottomata>	 man i am stale on this stuff
[15:28:17] <ottomata>	 its called hadoop-mapreduce-historyserver
[15:28:19] <ottomata>	 and we run it
[15:28:21] <ottomata>	 of course we do.
[15:28:38] <ottomata>	 this makes more sense, i had some notion that it had moved inside of yarn rm or something
[15:28:55] <nuria_>	 ok, let me look at the code but that would be the service taht needs a restart (i will document this once we .. ahem... know what is going on)
[15:28:58] <ottomata>	 nuria_:  this can just be restarted, will do so now.
[15:29:04] <nuria_>	 ottomata: k
[15:29:05] <ottomata>	 ok, want me to wait?
[15:29:28] <ottomata>	 nuria_: ?
[15:29:38] <nuria_>	 ottomata: no no, please restart
[15:29:43] <ottomata>	 k
[15:29:45] <ottomata>	 !log restarting hadoop-mapreduce-historyserver to apply yarn log aggreation retention settings
[15:30:22] <ottomata>	 nuria_:  OH
[15:30:25] <ottomata>	 will you look at that...
[15:30:26] <ottomata>	 Deleting aggregated logs in hdfs://analytics-hadoop/var/log/hadoop-yarn/apps/addshore/logs/application_1441303822549_25187
[15:30:27] <ottomata>	 haha
[15:30:33] <nuria_>	 ottomata: ta-ta-channnnn
[15:30:34] <ottomata>	 lots of those in logs
[15:30:42] <ottomata>	 nice nuria_ sorry for that confusion, i shoulda known that
[15:30:42] <elukey>	 ottomata: does it look good? https://puppet-compiler.wmflabs.org/3357/
[15:31:38] <ottomata>	  {"name"=>"java.lang:name=*  "typeNames"=>["name"] "result_alias"=>"GarbageCollector"
[15:31:40] <ottomata>	 looks good to me!
[15:31:40] <elukey>	 urandom: o/ I'd need you for a couple of questions
[15:31:56] <elukey>	 about life, univers and all that
[15:31:57] <ottomata>	 elukey:  you can test in labs if you want, but this won't hurt anything to just apply in prod and fix if it isn't quite right
[15:32:04] <ottomata>	 up to you
[15:32:15] <ottomata>	 might be faster easier to just do in prod
[15:32:51] <elukey>	 mmmmmm
[15:33:03] <elukey>	 I'll probably grab some time to test it on Monday
[15:33:11] <elukey>	 it seems not a good idea on Friday
[15:33:23] <elukey>	 especially on a EU evening :P
[15:33:27] <wikibugs>	 Analytics-Cluster, Analytics-Kanban, EventBus, Operations, and 2 others: Better monitoring for Zookeeper - https://phabricator.wikimedia.org/T137302#2466240 (Ottomata) Alerts too!  https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=conf1001&service=Zookeeper+Alive+Client+Connecti...
[15:33:53] <ottomata>	 elukey:  yeah, but we don't have alerts or antyhing based on it
[15:33:55] <ottomata>	 and it is just monitoring
[15:33:57] <ottomata>	 but jaaa
[15:34:01] <ottomata>	 i guess itis your evening
[15:34:04] <urandom>	 elukey: when the time comes, choose the one on the left.
[15:34:04] <ottomata>	 and i don't want to babysit it :p
[15:34:12] <urandom>	 elukey: trust me.
[15:34:50] <nuria_>	 ottomata: now documented here: https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hadoop/Administration#Logs
[15:36:08] <elukey>	 urandom: :D
[15:36:48] <elukey>	 urandom: so I was reviewing https://wikitech.wikimedia.org/wiki/User:Elukey/Ops/AQS_Settings today and one thing that I wondered (popped up earlier also discussing with Joal) is the space requirement for each cassandra instance
[15:37:09] <elukey>	 we need ~6TB for each cassandra instance to have 3 years of retention
[15:37:40] <elukey>	 this is because we think that one year of data will be ~2TB on each instance..
[15:37:48] <elukey>	 but this is true for the _actual_ cluster
[15:37:51] <elukey>	 not the new one
[15:37:57] <elukey>	 so I am wondering.. going from 3 to 6
[15:38:05] <elukey>	 (and changing compaction)
[15:38:23] <elukey>	 should change also the space consumption right? (I suppose reducing it)
[15:38:41] <urandom>	 2T compressed, yes?
[15:38:44] <elukey>	 because from my calculations with 2 disks failure we are sort of in a very bad position
[15:39:58] <urandom>	 elukey: i might be confused, what do you mean going from 3 to 6?
[15:40:13] <urandom>	 TB?
[15:40:44] <elukey>	 nope instances sorry
[15:40:44] <urandom>	 no, that's not right...
[15:40:46] <elukey>	 or nodes
[15:41:03] <nuria_>	 ottomata: ok, i see 3TB of logs got deleted
[15:41:22] <urandom>	 elukey: oh, i see
[15:41:41] <elukey>	 so now I am confused about the compression
[15:41:48] <elukey>	 I know that the sstables file are compressed
[15:41:56] <ottomata>	 nuria_:  its still going too!
[15:41:58] <ottomata>	 if you want to watch
[15:42:01] <ottomata>	 on analytics1001
[15:42:06] <ottomata>	 tail -f /var/log/hadoop-mapreduce/mapred-mapred-historyserver-analytics1001.out
[15:42:09] <elukey>	 but nodetool status reports 2.2TB meanwhile df -h 2.6TB
[15:42:17] <nuria_>	 ottomata: k, good, will check in a bit.
[15:42:19] <elukey>	 but I guess it is fine
[15:42:34] <urandom>	 elukey: which node?
[15:42:35] <elukey>	 urandom: I am trying to understand if raid10 would be out of the question or not
[15:42:38] <elukey>	 aqs1001
[15:42:43] <ottomata>	 ja good for documenting nuria, pretty weird that it is a yarn-site setting for a 'mapreduce' service.  i bet that is some legacy thing
[15:43:54] <elukey>	 nuria_: https://grafana.wikimedia.org/dashboard/db/analytics-hadoop?panelId=25&fullscreen :)
[15:44:19] <urandom>	 elukey: so w/ aqs100[1-3], you figured 2T per host of utilization, per each year of data, is that right?
[15:44:58] <elukey>	 urandom: this is what I got, but I could be wrong
[15:45:06] <urandom>	 where in aqs100[1-3], host and instance can be used interchangeably
[15:45:16] <elukey>	 but calculations fit, since we have raid0 arrays of ~6TB
[15:45:30] <elukey>	 urandom: yeah
[15:46:10] <urandom>	 so yeah, i would expect that w/ aqs100[4-6], that would work out to roughly 1T per instance
[15:46:38] <urandom>	 the same quantity of data
[15:47:44] <urandom>	 also, nodetool status, that value "load" is supposed to be the total of SSTable files on disk
[15:47:49] <urandom>	 so "compressed"
[15:47:57] <elukey>	 okok.. now that I recall during some tests joal was puzzled about the size of the sstables on aqs100[456] after some load job..
[15:48:04] <elukey>	 okok
[15:48:05] <urandom>	 why it reports differently than df, i have no clue
[15:48:12] <elukey>	 ok so I am not crazy
[15:49:21] <urandom>	 anyway, i'm may have lost track of your concerns playing catch up here, could you summarize them again?
[15:49:29] <elukey>	 yes sure
[15:49:48] <elukey>	 so we chose to have a 4 disks raid0 array for each cassandra instance
[15:50:13] <urandom>	 yup
[15:50:22] <elukey>	 and if I made my assumptions correctly in https://wikitech.wikimedia.org/wiki/User:Elukey/Ops/AQS_Settings the new cluster will tollerate only 2 disks failures
[15:50:45] <urandom>	 by disks, you mean raid arrays?
[15:50:47] <elukey>	 even worse, each time that a disk fail a RAID0 array will need to get rebuilt
[15:51:06] <urandom>	 and i guess you mean simultaneous failures?
[15:51:13] <elukey>	 or in a rapid sequence
[15:51:24] <elukey>	 you know, very bad luck for analytics week or something similar :)
[15:51:42] <elukey>	 ops asked to us to motivate really well why we put raid0 in prod
[15:51:51] <elukey>	 since disk failures should not become emergencies
[15:52:05] <elukey>	 I mean, a couple
[15:52:24] <elukey>	 with RAID10 we'd have more resiliency
[15:52:40] <elukey>	 but it was discarded long time ago because we needed space
[15:52:52] <urandom>	 elukey: you know, i keep saying this (a lot), but we seem to get drives of the same model from the same source
[15:53:06] <urandom>	 i have to assume they are from the same lot
[15:53:27] <urandom>	 and so having disks fail in rapid succession is actually not that surprising
[15:53:40] <urandom>	 it would not surprise me at all
[15:54:04] <urandom>	 this has happened to me more than once
[15:54:11] <urandom>	 we should really be mixing things up a bit
[15:54:22] <elukey>	 yeah...
[15:54:33] <elukey>	 so from https://grafana.wikimedia.org/dashboard/db/aqs-cassandra-system and IIRC, joal loaded one month of data
[15:54:40] <elukey>	 that is ~200GB
[15:54:55] <urandom>	 per instance?
[15:54:59] <elukey>	 yes
[15:55:06] <elukey>	 so assuming a linear increase
[15:55:14] <urandom>	 that's a lot more than 1T
[15:55:21] <elukey>	 ah snap
[15:55:41] <elukey>	 do you mean for a year?
[15:55:46] <urandom>	 yeah
[15:55:51] <urandom>	 that's 2400G
[15:56:45] <elukey>	 may I ask what is the calculation that you made?
[15:56:53] <urandom>	 200 * 12
[15:57:09] <elukey>	 sure that is not 1.2 TB as my brain was suggesting
[15:57:14] <elukey>	 sigh friday brainfarts
[15:57:33] <urandom>	 it's friday morning for me and i am already struggling, so no worries :)
[15:57:47] <elukey>	 could this be due to the levelled compaction?
[15:58:00] <urandom>	 i don't think so
[15:58:01] <elukey>	 or it is completely not relevent?
[15:58:03] <elukey>	 okok
[15:58:31] <urandom>	 i mean, i guess that compaction could effect compression efficiency
[15:58:42] <urandom>	 but i wouldn't expect by a lot
[15:58:48] * urandom is thinking out loud
[15:59:20] <elukey>	 but this means that doubling the instances does not reduce sstable size?
[15:59:26] <elukey>	 it is a bit weird
[15:59:48] <urandom>	 not sure what you mean
[15:59:59] <urandom>	 the largest SSTable will be smaller when you have more instances
[16:00:07] <urandom>	 but the total amount of data remains unchanged
[16:00:39] <elukey>	 yeah but the keyspace should be split among more instances, so I'd expect less data per node in the sstables.. no?
[16:01:01] <nuria_>	 elukey: right now we have 2 TB per node with half a year
[16:01:26] <urandom>	 that would fit the match
[16:01:30] <urandom>	 s/match/math/
[16:01:54] <nuria_>	 elukey: standdup?
[16:01:55] <elukey>	 nuria_: we have HALF a year?
[16:02:12] <urandom>	 4T per year per instance @ 3 instances, would be 2T per year per instance @ 6 instances
[16:02:13] * elukey cries in a corner
[16:02:34] <elukey>	 yes now it makes sense
[16:02:44] <elukey>	 I was convinced about a full year stored
[16:02:52] <elukey>	 sorry urandom, now I feel better
[16:03:53] <urandom>	 elukey: also, remember you need working space for range moves and compaction, so you don't want to run full utilized
[16:04:02] <urandom>	 s/full/fully/
[16:05:21] <nuria_>	 elukey: NO SORRY, 1 year
[16:06:06] * elukey cries again
[16:06:29] <nuria_>	 nuria is WRONG!!!
[16:07:38] <nuria_>	 urandom, elukey : pageviews from july 2015: https://analytics.wikimedia.org/dashboards/vital-signs/#projects=eswiki,itwiki,enwiki,jawiki,dewiki,ruwiki,frwiki/metrics=Pageviews
[16:10:00] <wikibugs>	 Analytics-Kanban, Operations, Traffic: Verify why varnishkafka stats and webrequest logs count differs - https://phabricator.wikimedia.org/T136314#2466298 (Nuria) Open>Resolved
[16:28:43] <nuria_>	 urandom: but while our total data size is 2TB per year and that doesn't change, now we have 6 cassandra instances when before we had 3.
[16:29:31] <nuria_>	 urandom: all things being equal we would expect 1TB of data per cassandra instance if we have teh same replication , correct? cc elukey (just trying to understand)
[16:30:57] <elukey>	 something is off, I'll review it with joal on Monday
[16:31:08] <elukey>	 hopefully he has just loaded 2 months of data
[16:31:14] <elukey>	 and calculations will be ok :)
[16:33:24] <urandom>	 nuria_: you guys have 3x replication, yes?
[16:33:34] <nuria_>	 urandom: yes
[16:37:51] <urandom>	 nuria_: i dunno, i think i might be pretty confused at this point
[16:38:14] <urandom>	 grafana shows 200MB per instance for what is reported to be 1 month
[16:38:24] <urandom>	 for the new nodes
[16:38:44] <urandom>	 which is more than 2T for a year at 6 instances
[16:38:56] <urandom>	 what would be more than 4T under the current production cluster
[16:39:08] <urandom>	 does that not fit?
[16:41:59] <urandom>	 to restate: that would be 2T per-instance per-year in a 6 instance configuration, or 4T per-instance per-year under a 3 instance configuration
[16:45:56] <elukey>	 yeah but we have 2TB for a year of data per instance (as it seems) in a 3 instance config
[16:46:40] <elukey>	 anyhow, let's resync on Monday with joal, I am probably missing something
[16:46:51] <urandom>	 elukey: ok, i thought nuria_ was saying that was actually 6 months
[16:46:52] <elukey>	 don't want to waste everybody's time :)
[16:47:09] <elukey>	 yeah but then she corrected and said the we have one year
[16:47:28] <urandom>	 auh, ok, i missed that
[16:47:33] <urandom>	 whew!
[16:47:35] <urandom>	 :)L
[16:47:38] <urandom>	 :)
[17:00:55] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations (Jul-Sep-2016): Create basic/high-level Kibana (dashboard) documentation - https://phabricator.wikimedia.org/T132323#2466390 (Aklapper)
[17:01:49] <wikibugs>	 Analytics-Tech-community-metrics, JavaScript: Syntax error, unrecognized expression on Korma profiles - https://phabricator.wikimedia.org/T126325#2466391 (Aklapper) p:Low>Lowest
[17:02:29] <wikibugs>	 Analytics-Tech-community-metrics: Deployment of Gerrit (basic) panel - https://phabricator.wikimedia.org/T137999#2466393 (Aklapper) p:Triage>Normal
[17:04:02] <elukey>	 going afk team! byyyeeeeee
[17:48:21] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations (Jul-Sep-2016): Identify Wikimedia's most important/used info panels in korma.wmflabs.org - https://phabricator.wikimedia.org/T132421#2466540 (Aklapper) p:High>Normal
[17:48:52] <milimetric>	 hey mforns sorry finishing up phone call so I'll be a little longer
[17:49:01] <mforns>	 milimetric, np
[17:49:19] <wikibugs>	 Analytics-Tech-community-metrics, Phabricator: Decide on wanted metrics for Maniphest in kibana - https://phabricator.wikimedia.org/T28#2466546 (Aklapper)
[17:50:29] <wikibugs>	 Analytics-Tech-community-metrics: Maniphest support to be added to GrimorieLab - https://phabricator.wikimedia.org/T138003#2466551 (Aklapper)
[17:50:31] <wikibugs>	 Analytics-Tech-community-metrics, Phabricator: Decide on wanted metrics for Maniphest in kibana - https://phabricator.wikimedia.org/T28#2466550 (Aklapper)
[18:02:43] <milimetric>	 sorry mforns, ok ready, to the ehr-batcave!
[18:02:54] <mforns>	 milimetric, :] there
[20:08:55] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations (Jul-Sep-2016): Create basic/high-level Kibana (dashboard) documentation - https://phabricator.wikimedia.org/T132323#2467108 (Aklapper) Well, what I currently know and understand is on https://www.mediawiki.org/wiki/Community_metrics#dashboard.bitergi...
[22:21:56] <urandom>	 nuria_: congrats!
[22:58:07] <neilpquinn>	 Does anyone know if pageviews are accurate for special pages? The numbers for Special:RecentChanges on enwiki show a big drop at the end of May, which seems odd.
[22:58:09] <neilpquinn>	 https://tools.wmflabs.org/pageviews/?project=en.wikipedia.org&platform=all-access&agent=user&range=latest-60&pages=Special:RecentChanges
[23:01:39] <neilpquinn>	 It could just a bot losing interesting or something like that, though.