[06:02:54] <wikibugs>	 10Traffic, 10Operations: Move cache text cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231627 (10Vgutierrez)
[06:03:33] <wikibugs>	 10Traffic, 10Operations: Move cache text cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231627 (10Vgutierrez) p:05Triage→03Normal
[06:19:43] <wikibugs>	 10Traffic, 10Operations: Cannot download STL files due to network error - https://phabricator.wikimedia.org/T231422 (10Vgutierrez) I can reproduce the issue, using HTTP/2 and HTTP/1.1 on eqsin: `willikins:~ vgutierrez$ curl https://upload.wikimedia.org/wikipedia/commons/d/de/65-msr-sandaled-foot-5.stl -o /dev/...
[06:29:31] <wikibugs>	 10Traffic, 10Operations: Perform HTTPS redirect without crossing domain boundaries for non canonical domains - https://phabricator.wikimedia.org/T231513 (10Vgutierrez) 05Open→03Resolved
[08:00:30] <wikibugs>	 10Traffic, 10Operations: Cannot download STL files due to network error - https://phabricator.wikimedia.org/T231422 (10ema) p:05Triage→03Normal a:03ema
[08:12:27] <wikibugs>	 10Traffic, 10Analytics, 10Operations: varnishkafka statsv and webrequest crashed on cp1081 - https://phabricator.wikimedia.org/T231331 (10Nuria) 05Open→03Resolved
[08:12:32] <wikibugs>	 10Traffic, 10Analytics, 10Operations: varnishkafka statsv and webrequest crashed on cp1081 - https://phabricator.wikimedia.org/T231331 (10Nuria) Agreed, closing.
[13:48:52] <mutante>	 one more service with TLS, planet has switched to envoy and https:// discovery name for backends too
[13:49:42] <bblack>	 thanks!
[13:52:43] <mutante>	 i learned from ema's docs how to generate the certs so i can follow that for others too.  there is only one minor drawback.. actually it was active/active before and now codfw is commented out. that is because then i could use a CNAME to create planet.discovery.wmnet and point it to planet1001.  currently if you want geo DNS discovery and be active/active you also have to be behind LVS which is 
[13:52:49] <mutante>	 not the case and overkill for this case
[13:53:58] <mutante>	 so in one way that is a small step back.. but the other advantages make up for it
[13:54:55] <wikibugs>	 10Traffic, 10Operations, 10serviceops, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10Dzahn)
[13:55:06] <bblack>	 the discovery dns stuff can be used indepdently of LVS, AFAIK
[13:55:13] <bblack>	 it's just not the common pattern
[13:55:44] <mutante>	 oh! then i should try to figure that out later
[13:56:42] <bblack>	 oh now that I look, I see what you mean
[13:57:15] <bblack>	 for most purposes, the discovery DNS stuff and LVS are completely separate things at runtime, and fundmentally there's no reason the former should require the latter
[13:57:18] <bblack>	 however:
[13:58:30] <bblack>	 https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/authdns/templates/discovery-geo-resources.erb
[13:58:43] <bblack>	 ^ in there, the way the geo resources are templated draws directly from LVS configuration
[13:59:05] <bblack>	 which is what ties them together in practice, for CM
[14:00:28] <bblack>	 https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/hieradata/common/discovery.yaml
[14:00:56] <bblack>	 ^ but you could potentially add new subkeys to this (that various manifests and templates understand) to allow it to declare non-LVS services by their eqiad and codfw IP addresses directly or whatever
[14:01:12] <bblack>	 it might be rather ugly and complex work to add that support, though
[14:19:57] <mutante>	 bblack: thanks! yea that's kind of the conclusion we came to when i was walking to e.ma. but maybe in the future we can get there
[16:07:04] <wikibugs>	 10Traffic, 10MobileFrontend, 10Operations, 10Mobile: https://en.wikipedia.org/wiki/Heteromyidae shows the mobile version on desktop - https://phabricator.wikimedia.org/T231620 (10AntiCompositeNumber)
[16:21:23] <wikibugs>	 10Traffic, 10MobileFrontend, 10Operations, 10Mobile: https://en.wikipedia.org/wiki/Heteromyidae shows the mobile version on desktop - https://phabricator.wikimedia.org/T231620 (10CDanis) Seems likely this is the same as {T231504} ?
[18:15:41] <wikibugs>	 10Traffic, 10Operations, 10ops-eqiad: cp1085 - IPMI not working - https://phabricator.wikimedia.org/T231525 (10wiki_willy) a:03Cmjohnson
[18:19:25] <wikibugs>	 10netops, 10Operations, 10ops-eqiad: (Need By: Sept 30) upgrade msw1-eqiad from EX4200 to EX4300 - https://phabricator.wikimedia.org/T225121 (10wiki_willy) a:05Papaul→03Cmjohnson
[18:47:19] <wikibugs>	 10Traffic, 10Operations, 10Performance-Team (Radar): Some load.php requests failing due to "ERR_SPDY_PROTOCOL_ERROR 200" - https://phabricator.wikimedia.org/T220022 (10Krinkle)
[18:47:28] <wikibugs>	 10Traffic, 10Wikimedia-Apache-configuration, 10Operations, 10Patch-For-Review, and 2 others: Visual Editor gets stuck opening article (net::ERR_SPDY_PROTOCOL_ERROR 200/Loading failed for the <script> with source ...) - https://phabricator.wikimedia.org/T213214 (10Krinkle)
[18:48:29] <wikibugs>	 10Traffic, 10Operations, 10Performance-Team (Radar): Some load.php requests failing due to "ERR_SPDY_PROTOCOL_ERROR 200" - https://phabricator.wikimedia.org/T220022 (10matmarex) I proposed a possible fix for this on the task you merged: https://gerrit.wikimedia.org/r/521575
[18:48:58] <wikibugs>	 10Traffic, 10Operations, 10Performance-Team (Radar): Some load.php requests failing due to "ERR_SPDY_PROTOCOL_ERROR 200" - https://phabricator.wikimedia.org/T220022 (10Krinkle)
[18:49:47] <wikibugs>	 10Traffic, 10Operations, 10Performance-Team (Radar): Some load.php requests failing due to "ERR_SPDY_PROTOCOL_ERROR 200" - https://phabricator.wikimedia.org/T220022 (10Krinkle) Seen again today when using Flow.  `Failed to load resource: net::ERR_SPDY_PROTOCOL_ERROR` `name=URL,lines=2 https://www.mediawiki.o...
[18:51:13] <wikibugs>	 10Traffic, 10Operations, 10Performance-Team (Radar): Some load.php requests failing due to "ERR_SPDY_PROTOCOL_ERROR 200" - https://phabricator.wikimedia.org/T220022 (10Krinkle)