[00:49:30] 10Traffic, 10Operations: Proxy-connection HTTP response header being sent to some users in some cases causing HTTP/2 protocol errors - https://phabricator.wikimedia.org/T238509 (10Krenair) [01:02:34] 10Traffic, 10Operations: Proxy-connection HTTP response header being sent to some users in some cases causing HTTP/2 protocol errors - https://phabricator.wikimedia.org/T238509 (10Krenair) [01:10:29] Regarding Krenair's T238509, see also: https://serverfault.com/questions/937253/https-doesnt-work-with-safari [01:10:30] T238509: Proxy-connection HTTP response header being sent to some users in some cases causing HTTP/2 protocol errors - https://phabricator.wikimedia.org/T238509 [03:25:11] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712: Pages whose title ends with semicolon (;) are intermittently inaccessible - https://phabricator.wikimedia.org/T238285 (10Vgutierrez) >>! In T238285#5666584, @Urbanecm wrote: > @Vgutierrez I guess what you quoted wouldn't be valid, b... [03:51:02] 10Traffic, 10Operations, 10Patch-For-Review: Move cache text cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231627 (10Vgutierrez) [04:14:12] 10Traffic, 10Operations: Enable HSTS for non canonical domains using the ncredir service - https://phabricator.wikimedia.org/T231514 (10Vgutierrez) 05Open→03Resolved [04:15:53] 10Traffic, 10Operations: Submit non-canonical domains to the HSTS preload list - https://phabricator.wikimedia.org/T238513 (10Vgutierrez) [04:16:47] 10Traffic, 10Operations, 10Patch-For-Review: ats-tls shows spikes on H/2 recv settings bad param errors - https://phabricator.wikimedia.org/T238307 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez Solved with 8.0.5-1wm11 [04:17:57] 10Traffic, 10Operations: Submit non-canonical domains to the HSTS preload list - https://phabricator.wikimedia.org/T238513 (10Vgutierrez) p:05Triage→03Normal [06:25:51] 10Traffic, 10Operations, 10Patch-For-Review: Move cache text cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231627 (10Vgutierrez) [06:42:40] 10Traffic, 10Operations, 10Patch-For-Review: Move cache text cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231627 (10Vgutierrez) [07:08:57] 10Traffic, 10Operations, 10Patch-For-Review: Move cache text cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231627 (10Vgutierrez) [07:31:18] 10Traffic, 10Operations: Disable TLSv1/TLSv1.1 on sites without caching layer - https://phabricator.wikimedia.org/T238518 (10Vgutierrez) [08:00:03] 10Traffic, 10Operations: Disable TLSv1/TLSv1.1 on sites without caching layer - https://phabricator.wikimedia.org/T238518 (10Vgutierrez) [08:00:17] 10Traffic, 10Operations: Disable TLSv1/TLSv1.1 on sites without caching layer - https://phabricator.wikimedia.org/T238518 (10Vgutierrez) p:05Triage→03Normal [08:36:19] 10Traffic, 10Operations, 10Patch-For-Review: Disable TLSv1/TLSv1.1 on sites without caching layer - https://phabricator.wikimedia.org/T238518 (10Vgutierrez) [08:57:08] 10Traffic, 10Operations, 10Patch-For-Review: Disable TLSv1/TLSv1.1 on sites without caching layer - https://phabricator.wikimedia.org/T238518 (10MoritzMuehlenhoff) [11:37:10] 10Traffic, 10Operations: cergen fails signing CSR - https://phabricator.wikimedia.org/T231423 (10DannyS712) [batch] remove patch for review tag from resolved tasks [11:37:48] 10Traffic, 10Operations: Investigate segfaults on ats-tls running on cp5001 - https://phabricator.wikimedia.org/T232298 (10DannyS712) [batch] remove patch for review tag from resolved tasks [14:38:14] 10Traffic, 10Operations, 10Performance-Team: 200ms / 50% response start regression starting around 2019-11-11 - https://phabricator.wikimedia.org/T238494 (10ema) There's been a decrease in local backend hitrate on ats-be compared to varnish-be. While on 2019-11-11 (before reimages to ats) the local hitrate... [15:14:35] bblack: happy day! Could you please check if https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/551531/ makes sense? [15:40:09] 10Traffic, 10Operations, 10Performance-Team: 200ms / 50% response start regression starting around 2019-11-11 - https://phabricator.wikimedia.org/T238494 (10Gilles) a:03ema [15:40:50] 10Traffic, 10Operations, 10Performance-Team: 200ms / 50% response start regression starting around 2019-11-11 - https://phabricator.wikimedia.org/T238494 (10Gilles) Assigning this to you, as it sounds like a very likely root cause to the sizeable performance regression [15:50:43] 10Traffic, 10Operations, 10Performance-Team: 200ms / 50% response start regression starting around 2019-11-11 - https://phabricator.wikimedia.org/T238494 (10BBlack) @Gilles This could also be related to TLS certificate changes that were happening around the same dates, and could be inflating the bytes transf... [16:51:48] 10Traffic, 10Operations: Proxy-connection HTTP response header being sent to some users in some cases causing HTTP/2 protocol errors - https://phabricator.wikimedia.org/T238509 (10crusnov) p:05Triage→03Normal [16:51:56] 10Traffic, 10MediaWiki-extensions-Translate, 10Operations, 10Wikidata, 10User-DannyS712: Bug: 502 error when marking page for translation - https://phabricator.wikimedia.org/T237319 (10crusnov) p:05Triage→03Normal [16:58:52] 10Traffic, 10Operations: Proxy-connection HTTP response header being sent to some users in some cases causing HTTP/2 protocol errors - https://phabricator.wikimedia.org/T238509 (10Vgutierrez) @ema this could be caused by ats-be? ats-tls is setting would set `Proxy-Connection: close` but we are actually prevent... [17:34:12] 10Traffic, 10Operations: Proxy-connection HTTP response header being sent to some users in some cases causing HTTP/2 protocol errors - https://phabricator.wikimedia.org/T238509 (10Vgutierrez) From what I'm seeing on https://github.com/apache/trafficserver/blob/8.0.x/proxy/http/HttpTransact.cc#L6859-L6882 I don... [17:42:06] 10Traffic, 10Operations: Proxy-connection HTTP response header being sent to some users in some cases causing HTTP/2 protocol errors - https://phabricator.wikimedia.org/T238509 (10Vgutierrez) as a PoC, you can force ATS to send `proxy-connection: keep-alive` using HTTP/1.0 like this: ` $ curl --http1.0 -H 'Pro... [18:10:38] 10Traffic, 10Operations: Proxy-connection HTTP response header being sent to some users in some cases causing HTTP/2 protocol errors - https://phabricator.wikimedia.org/T238509 (10Vgutierrez) so we have been able to reproduce the issue, apparently this happens when nginx is being used as the TLS termination l... [18:50:42] 10Traffic, 10MediaWiki-extensions-Translate, 10Operations, 10Wikidata, 10User-DannyS712: Bug: 502 error when marking page for translation - https://phabricator.wikimedia.org/T237319 (10Vgutierrez) For `2019-11-14 18:33:15 GMT`, ats-be in cp4030 is reporting `20191114.18h33m15s CONNECT: could not connect... [19:31:27] 10Traffic, 10Operations, 10Inuka-Team (Kanban), 10Patch-For-Review, 10Performance-Team (Radar): Code for InukaPageView instrumentation - https://phabricator.wikimedia.org/T238029 (10Krinkle) >>! @Krinkle wrote on Gerrit (Patch Set 13): > This seems to vary server-response for anon users by aspects (user-... [20:49:45] 10Traffic, 10Operations: Proxy-connection HTTP response header being sent to some users in some cases causing HTTP/2 protocol errors - https://phabricator.wikimedia.org/T238509 (10Krenair) a:03Vgutierrez Thanks Valentin.