[07:54:32] <elukey>	 moar gpus https://phabricator.wikimedia.org/T246472#5951271 :)
[13:27:49] <elukey>	 memcached 1.6.0 released :) https://github.com/memcached/memcached/wiki/ReleaseNotes160
[13:28:48] <elukey>	 the extstore part is really interesting for the future
[13:28:51] <cdanis>	 some pretty nice changes there
[13:29:11] <elukey>	 I'd be happy to get to 1.5.x first though :D
[13:30:03] <cdanis>	 the "Atomic Stampeding Herd Handling" that's part of the meta protocol is very interesting
[13:41:11] <elukey>	 yes the meta protocol is very nice
[13:43:21] <cdanis>	 yeah really there's like half a dozen things in the meta protocol that seem interesting from a WANCache perspective
[13:44:17] <_joe_>	 not sure the memcached php extension supports them
[13:45:16] <_joe_>	 akosiaris: ok to go with https://gerrit.wikimedia.org/r/#/c/operations/deployment-charts/+/578318/ ?
[13:45:52] <cdanis>	 _joe_: do the comments need updating?
[13:46:04] <_joe_>	 cdanis: I am not sure :P
[13:46:25] <_joe_>	 because we don't have new benchmarks, just production results
[13:46:55] <akosiaris>	 I think it's fine for now, we anyway want to update the chart itself and have the comments in there
[13:48:12] <akosiaris>	 _joe_: let's see how mediawiki and envoy are going to behave when you deploy that
[13:48:22] <_joe_>	 yep
[13:48:40] <_joe_>	 akosiaris: go on an appserver and tail /var/log/envoy/sessionstore.log
[13:48:46] <_joe_>	 see if you see errors
[13:53:52] <_joe_>	 akosiaris: 3 errors
[13:53:59] <_joe_>	 given we're at what, 4k rps?
[13:54:06] <akosiaris>	 6
[13:54:12] <akosiaris>	 but yeah, it's nothing
[13:54:46] <_joe_>	 so 4k means about 300 rps per server
[13:54:59] <_joe_>	 I'm impressed
[13:55:11] <_joe_>	 big props to kubernetes
[13:55:41] <akosiaris>	 latencies increased a bit
[13:56:09] <akosiaris>	 but overall pretty nice
[13:57:18] <akosiaris>	 I am not even sure it's actually related
[13:57:23] <akosiaris>	 it seems like a 404 spike
[13:57:29] <akosiaris>	 not really something related to the deploy
[13:57:39] <akosiaris>	 got a meeting in 3 btw
[15:14:37] <_joe_>	 akosiaris: any idea of a page where I find something rendered via mathoid?
[15:15:33] <cdanis>	 _joe_: https://en.wikipedia.org/wiki/Canny_edge_detector
[15:18:49] <akosiaris>	 _joe_: yup, https://wikimedia.org/api/rest_v1/media/math/render/svg/4a36d7f727beeaff58352d671bb41a3aca9f44d6
[15:19:01] <akosiaris>	 as cdanis pointed out. But that's probably cached by rb
[15:19:12] <_joe_>	 yeah I generated a new one
[15:19:36] <_joe_>	 akosiaris: wanna do eventgate too?
[15:19:44] <_joe_>	 or do we want to wait for otto tomorrow?
[15:19:58] <akosiaris>	 sure, it looks like it's going well anyway
[15:20:07] <akosiaris>	 I am looking at mathoid graphs, no changes
[15:20:32] <akosiaris>	 actually...
[15:20:43] <akosiaris>	 so scratch that, no changes
[15:20:59] <akosiaris>	 maybe some pattern will become discernible later
[15:21:04] <akosiaris>	 go ahead with eventgate
[15:21:14] <akosiaris>	 TLS ? or no TLS ?
[15:21:44] <akosiaris>	 _joe_: maybe get a +1 from ottomata?
[15:21:56] <_joe_>	 he's off today
[15:21:59] <_joe_>	 that's with TLS
[15:22:06] <akosiaris>	 and yet online?
[15:22:15] <_joe_>	 I don't think so
[15:22:16] <akosiaris>	 weird...
[15:22:23] <akosiaris>	 no I mean he has no bouncer
[15:22:26] <_joe_>	 oh right
[15:22:29] <_joe_>	 ottomata: you around??
[15:22:38] <_joe_>	 it should be a US holiday today
[15:25:55] <ottomata>	 hello!
[15:26:01] <ottomata>	 i'm kinda around but also not!
[15:26:03] <ottomata>	 whatcha doing?
[15:28:05] <ottomata>	 akosiaris: ^
[15:30:50] <_joe_>	 ottomata: switching eventgate analytics to use https from mediawiki
[15:31:08] <_joe_>	 and we wanted your imprimatur
[15:31:36] <_joe_>	 ottomata: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/576008/
[15:32:14] <ottomata>	 OHhhohohoi
[15:32:24] <ottomata>	 sounds scary but yall know what you are doing
[15:32:26] <ottomata>	 eventgate is ready for it
[15:32:51] <ottomata>	 6004 goes to eventgate-analytics 4592 ?
[15:33:45] <ottomata>	 _joe_:  i won't be watching it but other things are already using the TLS ports, so MW should be able to as well
[15:33:48] <ottomata>	 if yall want to I say: proceed!
[15:35:34] <_joe_>	 ottomata: yeah {socket_address: {address: eventgate-analytics.discovery.wmnet, port_value: 4592}
[15:35:44] <ottomata>	 pfct
[15:36:27] <_joe_>	 I'm gonna deal with any eventual fallout
[15:36:34] <_joe_>	 but can you +1 the change?
[15:36:43] <_joe_>	 btw what's the adequate timeout for eventgate?
[15:37:24] <_joe_>	 atm we have a 1 second timeout but we need to tune it
[15:39:35] <ottomata>	 _joe_:  we have this now
[15:39:35] <ottomata>	 https://github.com/wikimedia/operations-mediawiki-config/blob/ce4d7a0f1bcd170a0d06955af14f528bc727d458/wmf-config/CommonSettings.php#L3806-L3816
[15:39:45] <ottomata>	 actually the hasty one should be way faster than that
[15:39:47] <ottomata>	 the other could be longer
[15:39:52] <ottomata>	 ok, i gotta run tho, proceed good luck!
[15:40:19] <_joe_>	 I hope nothing lasts that long from mediawiki
[16:04:44] <_joe_>	 akosiaris: so, we're authorized
[16:04:47] <_joe_>	 I raised the timeout
[16:04:50] <_joe_>	 let's go on?
[16:07:02] <akosiaris>	 yup
[16:07:05] <akosiaris>	 +1
[16:09:30] <akosiaris>	 _joe_: I 'll do my sessionstore unreachable tests tomorrow btw
[16:09:38] <_joe_>	 ack
[16:14:35] <_joe_>	 akosiaris: releasing now
[17:05:46] <ottomata>	 o/  _joe_  how goes?  :)
[17:06:04] <_joe_>	 ottomata: we rolled back because the cpu on eventgate got throttled
[17:06:11] <_joe_>	 tls negotiation needs more cpu
[17:06:20] <ottomata>	 ah1
[17:06:21] <_joe_>	 that's what happened last time too
[17:06:23] <ottomata>	 intereesting
[17:06:33] <_joe_>	 only this time envoy in the middle saved mediawiki from crumbling
[17:06:40] <ottomata>	 it was the k8s cpy throttling last time?
[17:06:40] <ottomata>	 huh
[17:06:42] <ottomata>	 cpu*
[17:06:48] <_joe_>	 I'm pretty sure it was
[17:06:54] <ottomata>	 didn't know that!  thought it was just https badness
[17:06:55] <ottomata>	 ok
[17:07:02] <_joe_>	 me too
[17:07:06] <_joe_>	 but now it seems clear
[17:07:13] <_joe_>	 so, we'll roll forward tomorrow
[17:07:18] <ottomata>	 ya makes sense
[17:07:25] <ottomata>	 ok cool, yall are going to up the CPU limits?
[17:07:26] <_joe_>	 both analytics and main I think
[17:07:38] <_joe_>	 ottomata: yeah but I'm getting afk now, and alex too
[17:07:42] <ottomata>	 ok
[17:07:43] <ottomata>	 cool
[17:07:55] <ottomata>	 we should up tthose limits for all charts prob
[17:07:59] <ottomata>	 eg charts
[17:08:02] <_joe_>	 yes
[17:08:02] <ottomata>	 eg...helmfiles*
[17:32:32] <jbond42>	 godog: ok to merge your change
[17:32:41] <godog>	 jbond42: yup, go ahead!
[17:33:13] <jbond42>	 cheers, merging