[15:47:01] <brouberol>	 How does one investigate what looks to be envoy errors when accessing an http service hosted on k8s? I'm *sometimes* seeing "upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection failure, transport failure reason: delayed connect error: 111" when accessing https://airflow-test-k8s.wikimedia.org.
[15:47:02] <brouberol>	 Thanks
[15:54:53] <brouberol>	 well, s/sometimes/most of the time now/
[15:59:07] <elukey>	 brouberol: did you inspect the sidecar's logs to see if anything pops up?
[15:59:38] <elukey>	 another thing that I'd try is to use nsenter -t $PID -n $command on the host where the sidecar container runs
[16:00:06] <elukey>	 it should be easier to reproduce the error, and/or to see if there is a problem
[16:01:25] <brouberol>	 when I see the error in the UI, no logs are generated in the envoy tls proxy sidecar 
[16:01:41] <brouberol>	 so the error might come from the ingress gateway?
[16:11:43] <brouberol>	 I was indeed able to capture a log in istio-ingressgateway-4xr9n: https://phabricator.wikimedia.org/P70595
[16:15:44] <brouberol>	 the error codes UF,URX seem to indicate an upstream failure. Inspecting the airflow-production-tls-service endpoints gave me the answer: the service was resolving to pods created by airflow as well 🤦‍♂️ 
[16:16:20] <brouberol>	 the service selector is much too broad
[16:17:43] <brouberol>	 aand fixed
[16:40:27] <elukey>	 brouberol: ahhh sorry I thought it was an error accessing airflow-test, my bad
[16:40:46] <brouberol>	 no worries, you were a perfect 🦆
[16:40:47] <elukey>	 from a pod namely
[16:41:13] <elukey>	 super :)
[16:41:53] <brouberol>	 and speaking of: we currently don't have any way to specify an extra set of k/v label selectors for the mesh service. In the case of airflow, we have 4 permanent pods and many transient ones, and we'd need the tls-service to resolve to a single pod (the one labeled `component: webserver`). I could do that by specifying an extra label selector on
[16:41:53] <brouberol>	 top of the `chart: x, routed_via: y`. Would it make sense to y'all to add that to the vendored modules? 
[16:43:00] <brouberol>	 and that extra set of label selectors could be scaffolded to {} by default
[16:45:10] <brouberol>	 (I'm not sure if I'm very clear)