[14:11:36] <volans>	 godog, herron: while looking for something completely unrelated I've found something odd regarding the ES logging on the logstash hosts
[14:11:52] <herron>	 volans: hey, what did you find?
[14:12:08] <volans>	 it seems that we rotate and compress the production-logstash-eqiad.log but there are only ~2 lines written there, while the current logging is going to
[14:12:11] <volans>	 production-logstash-eqiad_jvm_gc.pid65614.log.1.current
[14:12:19] <volans>	 and we somehow rotate those, but not compress too
[14:12:28] <volans>	 see /var/log/elasticsearch on logstash1010 for example
[14:13:06] <herron>	 ok having a look
[14:14:30] <volans>	 thanks, let me know if you need me to open a task
[14:15:02] <godog>	 volans: thanks!
[14:15:04] <herron>	 so the current jvm garbage collection log is going into the jvm_gc log, and afaict the other log is just low noise
[14:17:23] <volans>	 but why it's logging to a .1?
[14:22:02] <herron>	 as I understand it java rotates through the gc logs based on the values for NumberOfGCLogFiles and GCLogFileSize settings
[14:22:16] <herron>	 ES is running with -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=20M
[14:22:30] <herron>	 so writes to .0 until that’s 20M, then .1, and so on
[14:26:22] <volans>	 but that conflicts with logrotate settings
[14:26:31] <volans>	  /var/log/elasticsearch/*.log {
[14:26:52] <volans>	 wait, .0 is the first?
[14:27:21] <volans>	 ahh ok
[14:27:27] <herron>	 yeah, not sure it would ever hit since the files are named .N or .N.current
[14:27:37] <volans>	 weird naming
[14:27:51] <volans>	 and depends on the PID so we have not 2 .current
[14:27:54] <volans>	 but only one is real
[14:27:58] <volans>	 blergh
[14:28:47] <herron>	 yeah, per our -Xloggc:/var/log/elasticsearch/production-logstash-eqiad_jvm_gc.%p.log
[14:28:59] <herron>	 %p being process number
[14:29:23] <herron>	 which I think is default for our es puppetization?  I don’t think the logging cluster customizes that at least
[15:05:49] <volans>	 ack