[06:45:57] <elukey>	 herron, shdubsh o/
[06:46:53] <elukey>	 I was checking an icinga alarm about logstash1011's GC run:
[06:47:02] <elukey>	 https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=production-logstash-eqiad&var-instance=logstash1011&panelId=37&from=1586409239266&to=1586414779787
[06:47:40] <elukey>	 the alarm was originally created (IIUC) for the Search ES cluster, but probably it is used also for ES-ELK
[06:47:53] <elukey>	 (not sure what it is the correct name forgive me :D)
[06:48:30] <elukey>	 anyway, with the Search team we are investigating the same problem, namely sometimes the old gen heap fills up and the number of old gen's GC runs rise and rise
[06:48:43] <elukey>	 those are expensive GC runs
[06:49:33] <elukey>	 it seems as if the JVM wouldn't be able to free old gen space anymore, ending up in a sort of trashing. Usually it goes away restarting, but it might be related to a certain workload
[06:49:47] <elukey>	 just leaving a note in case you have ideas :)
[06:49:53] <elukey>	 also Cc: godog --^
[07:07:17] <godog>	 elukey: sweet! thanks for the note/insight, haven't looked at the alert yet myself
[07:10:54] <elukey>	 np! I'll follow up with the Search team, in case something comes up I'll alert you guys too
[07:11:24] <godog>	 sounds good
[09:56:49] <_joe_>	 cdanis, bblack, rlazarus: I did some verifications and both setcookie() in php and apache in front set all cookies as capitalized
[13:26:30] <arturo>	 cdanis, rlazarus can you take a look at this? https://gerrit.wikimedia.org/r/c/operations/puppet/+/587730
[13:43:27] <cdanis>	 _joe_: so you think we can just turn on envoy's translation?
[13:43:39] <_joe_>	 cdanis: yes
[13:50:30] <_joe_>	 I'm going to merge the change now
[14:06:43] <rlazarus>	 _joe_: okay, sgtm
[14:07:04] <rlazarus>	 are we planning to do that indefinitely, or temporarily?
[14:07:17] <rlazarus>	 arturo: ack, will look shortly :)
[14:07:18] <_joe_>	 rlazarus: not sure :D
[14:07:30] <_joe_>	 for now, I wanted to do it quickly so we unbreak old clients
[14:07:31] <rlazarus>	 haha indefinitely then, got it
[14:11:14] <_joe_>	 and in a marvelous turn of events, somehow I did something wrong in my patch it seems
[14:31:13] <rlazarus>	 arturo: LGTM
[14:43:30] <bblack>	 I donno if it was already pasted here, but there's a new Google SRE book:         req->client_identity = NULL;
[14:43:33] <bblack>	         req->storage = NULL;
[14:43:36] <bblack>	 heh wrong paste :)
[14:43:45] <bblack>	 https://landing.google.com/sre/books/
[14:46:29] <bblack>	 _joe_: all the varnish-set ones use Set-Cookie as well
[14:46:39] <_joe_>	 bblack: exactly
[14:47:15] <bblack>	 so we could fix that one at the varnish layer in isolation from other headers, if we think this is probably the only critical case
[14:47:58] <bblack>	 at least, I think we can
[14:49:08] <bblack>	 [unless varnish is too smart for its own good, and "set resp.http.Set-Cookie = ..." over the top of an existing one does a case-insensitive match for the existing one, and then doesn't bother rewriting the case to match that specified in the VCL... in whch case we might have to drop to C to do it, making it a little more complicated]
[14:50:15] <bblack>	 we could also just fix it up from Lua on the ats-be side when its first received, might be better anyways
[14:50:53] <volans>	 bblack: yeah I pasted the other day in another chan, but good to re-paste it ;)
[15:02:07] <vgutierrez>	 case (in)sensitive issues in 2020... amazing world
[15:15:50] <arturo>	 rlazarus: thanks! may I ask you to merge it? I'm not familiar with the system it is changing
[15:17:32] <rlazarus>	 arturo: sure thing
[15:17:37] <rlazarus>	 cdanis: may I ask you to merge it? I'm not familiar with the system it is changing
[15:17:38] <rlazarus>	 :D
[15:17:49] <cdanis>	 haha
[15:18:04] <cdanis>	 arturo: if you address bd808's comment, I'll get coffee and then merge it :)
[15:18:11] <rlazarus>	 (or, I'm happy to, with some pointers if there's anything to do besides puppet-merge)
[15:18:32] <arturo>	 ack :-)
[15:19:07] <cdanis>	 there isn't anything to do aside from puppet-merge; you *could* run the VTC tests if you wanted (some brief notes at https://wikitech.wikimedia.org/wiki/Varnish#Configuration assuming you have vagrant/virtualbox installed)
[15:20:28] <rlazarus>	 what, installed locally? I don't but I could, I wouldn't mind setting up to know how to do this
[15:21:03] <rlazarus>	 tell you what though, I have a meeting in 10 so this isn't the time -- if you don't mind, cdanis why don't I leave you the merge, and maybe I'll ask you more questions later when nobody's waiting for it
[15:21:04] <cdanis>	 yeah, there's no automation that runs them, they're just manually-invoked
[15:21:08] <cdanis>	 +1
[15:21:17] <rlazarus>	 thank you!
[15:24:12] <arturo>	 cdanis: https://gerrit.wikimedia.org/r/c/operations/puppet/+/587730 should be ready now
[15:31:03] <chaomodus>	 o/
[15:34:01] <cdanis>	 arturo: cool.  out of curiosity what's the expected URL formats for toolforge.org?
[15:34:29] <arturo>	 cdanis: https://wikitech.wikimedia.org/wiki/News/Toolforge.org
[15:34:37] <cdanis>	 ah ofc it's all on wikitech :)
[15:34:40] <arturo>	 cdanis: https://wikitech.wikimedia.org/wiki/News/Toolforge.org#New_URLs_for_webservices
[15:34:45] <arturo>	 that particular section
[15:34:58] <arturo>	 let me know if that makes sense
[15:35:08] <cdanis>	 yeah, it does
[15:35:11] <arturo>	 basically, <tool>.toolforge.org
[15:35:16] <cdanis>	 I ask because I'm adding some VTC tests
[15:35:20] <cdanis>	 -hdr "Referer: https://cooltool.toolforge.org/"
[15:35:24] <cdanis>	 -hdr "Referer: https://some-vps.wmcloud.org/"
[15:36:48] <arturo>	 in the second case, we may have <whatever>.subdomain.wmcloud.org
[16:13:51] <arturo>	 thanks cdanis !!
[16:14:07] <cdanis>	 👍