[12:42:42] <inflatador>	 <o/
[13:37:13] <pfischer>	 o/
[13:39:55] <pfischer>	 Regarding today’s retro: David’s out shall we postpone? Guessing from the google docs, the last one (September 25th) has been skipped, too. 
[14:02:34] <inflatador>	 I'm OK w/postponing
[14:04:29] <ebernhardson>	 \o
[14:04:35] <ebernhardson>	 ya we can postpone
[14:09:24] <inflatador>	 heads-up that we're restarting pybal to turn off the plaintext WDQS endpoints for internal-main and internal-scholarly ( ref https://phabricator.wikimedia.org/P83716 ) 
[14:10:01] <inflatador>	 No impact is expected, but I'm watching QPS
[15:03:44] <inflatador>	 I'm just spitballing, but does anyone have any theories when internal-scholarly traffic might've dropped to zero starting at 1:30? ref https://grafana.wikimedia.org/goto/uq2n0SeNR?orgId=1 . We merged a patch to change the services to HTTPS a few hours before https://gerrit.wikimedia.org/r/c/operations/puppet/+/1187772 
[15:03:53] <inflatador>	 1:30 UTC that is
[15:05:55] <inflatador>	 cc ryankemper . I'm looking at T337013 to see if we can figure out what the internal clients might be
[15:05:56] <stashbot>	 T337013: [Epic] Splitting the graph in WDQS - https://phabricator.wikimedia.org/T337013
[16:08:58] <inflatador>	 incident report (of sorts) at https://etherpad.wikimedia.org/p/wdqs-internal-plaintext ...note that we don't actually know if this is user-impacting yet, it could also be a metrics collection problem
[16:09:05] <inflatador>	 workout/errand, back in ~40
[17:51:32] <inflatador>	 Closing the loop on ^^, the traffic recovered about 90m ago. Not yet sure why, more details in the etherpad
[17:57:36] <cdanis>	 inflatador: 16:10-16:30 was the first scap deployment since these patches
[17:57:52] <cdanis>	 and some of the settings you're changing in the service catalog affect how the k8s envoy config is generated
[17:58:02] <cdanis>	 so I suspect that your internal client is mediawiki
[17:58:14] <cdanis>	 (of qdqs-internal-scholarly)
[17:58:40] <inflatador>	 cdanis interesting. Scap deployment of mediawiki, that is?
[17:59:02] <cdanis>	 yes
[17:59:16] <cdanis>	 13:28	<esanders@deploy2002>	esanders: Backport for [[gerrit:1194588|Revert "Invalidate Flow cache on enwiktionary"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
[17:59:44] <cdanis>	 that was probably the first scap deploy since https://gerrit.wikimedia.org/r/c/operations/puppet/+/1187772 was merged
[18:00:11] <cdanis>	 but when you're setting encryption=>false that way, I suspect what happens is, you're telling the envoys to connect to port 443 but using http
[18:01:00] <inflatador>	 that makes sense. I stopped digging since things went back to normal, but when we did the graph split we did reach out to our clients, T374021 is an example. I should probably put these clients in Wikitech
[18:01:09] <stashbot>	 T374021: Make WikibaseQualityConstraints use split-graph query service - https://phabricator.wikimedia.org/T374021
[18:01:14] <inflatador>	 Good find though, I'll update the etherpad
[18:01:16] <cdanis>	 or figure out some other observability ideas yeah :)
[18:01:37] <inflatador>	 That seems quite reasonable as well ;)
[18:02:39] <cdanis>	 sigh puppetboard displays local time
[18:02:57] <cdanis>	 inflatador: https://puppetboard.wikimedia.org/report/deploy1003.eqiad.wmnet/bc92018ced14ab8505e44bf83b5a2c692d8c6eb5
[18:03:31] <cdanis>	 here you can see Puppet editing the helmfile default values.yaml that then influences the `mesh.configuration` chart module and via that the Envoy configmap
[18:04:15] <inflatador>	 hey, that's why I set my computer to UTC ;P I think I may've actually picked up that trick from you
[18:09:00] <inflatador>	 I'm poking around the envoy telemetry dashboard as well...maybe some hints there
[18:09:47] * ebernhardson gets to re-experience the joys of conda...how lovely!
[18:10:46] <ebernhardson>	 installing the discolytics package to the default notebook env causes conda-pack to be unhappy...would think the obvious choice is install discolytics as a conda package, but conda packaging is a whole separate thing from python packaging :P
[18:11:08] <inflatador>	 Hah, Conda provoked a rant from one of the Texas Linux Fest exhibitors when I mentioned it
[18:14:16] <ebernhardson>	 i guess the alternate is to document how to pull re-use the conda env we package for airflow jobs...but that seems overly complicated each time we use it
[18:18:58] <inflatador>	 lunch/errand, back in ~1h
[18:35:45] <pfischer>	 ebernhardson: are you around
[18:35:49] <pfischer>	 ?
[18:35:54] <ebernhardson>	 pfischer: oh yea, sec
[19:45:58] <inflatador>	 sorry, been back awhile