[00:14:30] <wikibugs>	 10Domains, 10Traffic: Redirecting incoming queries to non-existent subpages - https://phabricator.wikimedia.org/T212914 (10Dzahn)
[00:15:54] <wikibugs>	 10Domains, 10Traffic, 10Operations: Redirecting incoming queries to non-existent subpages - https://phabricator.wikimedia.org/T212914 (10Dzahn) @Thomas_Shafee Is it an option to stop using GoDaddy? It seems pretty bad to me that they are adding "random strings", maybe another registrar would be the best solu...
[00:17:22] <wikibugs>	 10Domains, 10Traffic, 10Operations: Redirecting incoming queries to non-existent subpages - https://phabricator.wikimedia.org/T212914 (10Dzahn) https://www.godaddy.com/community/Managing-Domains/subdomain-forwarding-is-adding-extra-random-characters-to-the/td-p/61774
[02:08:55] <XioNoX>	 https://he.net/3d-map/
[13:30:06] <wikibugs>	 10Domains, 10Traffic, 10Operations: Redirecting incoming queries to non-existent subpages - https://phabricator.wikimedia.org/T212914 (10ema) p:05Triage→03Normal
[14:34:49] <wikibugs>	 10Traffic, 10Operations: HTTP/2 requests fail with too-long URLs - https://phabricator.wikimedia.org/T209590 (10ema) P7916, which worked for @Huji, does not work for me. The path in P7916 (/wiki/Main_Page?_=xxxx...) is 4698 characters long.  In my tests, the following works (path length 4683):  ` $ curl --http...
[14:38:59] <ema>	 XioNoX: cool! :)
[14:39:05] <vgutierrez>	 ema: interesting issue
[14:39:10] <ema>	 vgutierrez: right?
[14:39:41] <vgutierrez>	 URL length is so long that increases the number of roundtrips?
[14:40:26] <vgutierrez>	 which curl version are you using?
[14:40:35] <vgutierrez>	 I don't get the same output as you
[14:40:59] <ema>	 I've tried 7.62.0, same version as the submitter
[14:41:09] <ema>	 and also 7.52.1, with different output
[14:42:36] <vgutierrez>	 so if I get rid of the grep part of the cmd, i get the EOF
[14:42:41] <vgutierrez>	 * TLSv1.2 (IN), TLS alert, Client hello (1):
[14:42:41] <vgutierrez>	 * Unexpected EOF
[14:42:54] <vgutierrez>	 here --> curl 7.54.0 (x86_64-apple-darwin18.0) libcurl/7.54.0 LibreSSL/2.6.4 zlib/1.2.11 nghttp2/1.24.1
[14:43:23] <ema>	 looks similar to the output you'd get from a cache node (curl 7.52.1, see bottom of T209590#4855055)
[14:43:24] <stashbot>	 T209590: HTTP/2 requests fail with too-long URLs - https://phabricator.wikimedia.org/T209590
[14:43:52] <ema>	 * TLSv1.2 (IN), TLS alert, Client hello (1):
[14:43:52] <ema>	 { [2 bytes data]
[14:43:52] <ema>	 * Unexpected EOF
[14:45:40] <vgutierrez>	 yep
[14:46:13] <vgutierrez>	 nginx is the one speaking h2 right?
[14:46:21] <ema>	 correct
[14:46:23] <vgutierrez>	 and varnish talks the good old http1.1
[14:47:29] <ema>	 also correct, nginx<->varnish is http1.1
[14:48:16] <vgutierrez>	 yup, if I hit http:// with the long query param (4666 chars) I get the proper 302 response by varnish
[14:48:21] <vgutierrez>	 even with h2
[14:49:57] <ema>	 it's not really h2, varnish does not support it
[14:52:48] <ema>	 as in, it does say this:
[14:52:49] <ema>	 -   Debug          "H2 upgrade attempt"
[14:53:10] <vgutierrez>	 yeah.. I was trying to test if the issue was just nginx http2 parsing
[14:53:34] <vgutierrez>	 damn.. my curl is too old to support TLS master key dumping
[14:54:03] <ema>	 but then really keeps on using HTTP/1.1
[14:54:04] <ema>	 -   ReqProtocol    HTTP/1.1
[14:54:10] <ema>	 -   RespProtocol   HTTP/1.1
[14:54:59] <ema>	 ah there's a feature flag to enable it
[14:55:07] <ema>	 I'll give it a try on cp1008
[14:56:56] <ema>	 the flag is "-p feature=+http2", FTR
[14:58:58] <ema>	 yeah, it works, I get HTTP/2 301
[14:58:59] * vgutierrez compiling curl 7.63... like in the old times
[14:59:25] <ema>	 not that the varnish h2 thing matters at all with the problem :)
[15:00:54] <ema>	 vgutierrez: who knows what's faster, building curl or installing a debian sid VM 
[15:01:08] <vgutierrez>	 hahaha, already done
[15:01:17] <vgutierrez>	 but probably fetching a container with curl would be faster yeah
[15:02:03] <ema>	 vgutierrez: keep in mind that if you want to mess with things in isolation you can use cp1008
[15:07:15] <vgutierrez>	 so it seems an http2 issue
[15:07:34] <vgutierrez>	 let met see if I can get a text output using tshark instead of wireshark
[15:07:52] <ema>	 vgutierrez: I got something under cp1008:~ema
[15:08:04] <ema>	 443-too-long.dump and 443-not-too-long.dump
[15:08:22] * vgutierrez checking
[15:08:42] <ema>	 which (obviously!) are tcpdumps of a unsuccessful and successful repro respectively :)
[15:11:10] <ema>	 heh, the other way around. 443-too-long.dump is a successful repro (the URL is too long) 
[15:11:52] <ema>	 I see in the first trace that the server sent two TLSv1.2 Encrypted Alerts
[15:12:13] <ema>	 there's one Encrypted Alert only being sent in the second trace
[15:15:25] <vgutierrez>	 so... on the crash
[15:15:32] <vgutierrez>	 https://www.irccloud.com/pastebin/FRP6LaEj/
[15:16:22] <vgutierrez>	 VS the working one
[15:16:27] <vgutierrez>	 https://www.irccloud.com/pastebin/DObntAxe/
[15:19:19] <vgutierrez>	 the HTTP2 GOAWAY packet contains the following error
[15:19:32] <vgutierrez>	 Error: ENHANCE_YOUR_CALM (11)
[15:21:30] <vgutierrez>	 so are we returning a 420 now? :)
[15:32:12] <vgutierrez>	 ema: that errors seems to be generated by nginx
[15:32:39] <ema>	 interesting
[15:32:56] <vgutierrez>	 the 4666 request doesn't generate any output in varnishlog
[15:33:11] <vgutierrez>	 (checking against our lovely pink unicorn)
[15:33:32] <vgutierrez>	 could we increase logging in nginx @ cp1008?
[15:33:41] <ema>	 of course!
[15:33:53] <vgutierrez>	 maybe a proper error log could give us some insights
[15:35:50] <ema>	 so it seems that nginx returns NGX_HTTP_V2_ENHANCE_YOUR_CALM in two cases:
[15:36:04] <ema>	     if (len > h2c->state.header_limit) {
[15:36:04] <ema>	         ngx_log_error(NGX_LOG_INFO, h2c->connection->log, 0,
[15:36:04] <ema>	                       "client exceeded http2_max_header_size limit");
[15:36:04] <ema>	         return ngx_http_v2_connection_error(h2c, NGX_HTTP_V2_ENHANCE_YOUR_CALM);
[15:36:07] <ema>	     }
[15:36:16] <ema>	 and
[15:36:19] <ema>	     if ((size_t) len > h2scf->max_field_size) {
[15:36:19] <ema>	         ngx_log_error(NGX_LOG_INFO, h2c->connection->log, 0,
[15:36:19] <ema>	                       "client exceeded http2_max_field_size limit");
[15:36:19] <ema>	         return ngx_http_v2_connection_error(h2c, NGX_HTTP_V2_ENHANCE_YOUR_CALM);
[15:36:22] <ema>	     }
[15:37:32] <vgutierrez>	 so.. the header "path" on the failing request has a length of 4684 bytes
[15:39:10] <vgutierrez>	 those limits are handled by http2_max_field_size and http2_max_header_size
[15:40:10] <ema>	 unfortunately I've got to go afk now, but it looks like we're on the right track :)
[15:40:14] <ema>	 keep me posted
[15:40:15] <ema>	 o/
[15:40:20] <vgutierrez>	 ack, see you later
[15:46:01] <vgutierrez>	 got it, http2_max_field_size is the culprit here
[16:02:31] <wikibugs>	 10Traffic, 10Operations: HTTP/2 requests fail with too-long URLs - https://phabricator.wikimedia.org/T209590 (10Vgutierrez) Ok, so after capturing a crashing request, we can see the following stuff going on at HTTP2 level:  `willikins:~ vgutierrez$ tshark -r ~/crash.pcap.pcapng -o ssl.keylog_file:/tmp/crash.ke...
[16:15:01] <wikibugs>	 10Traffic, 10Operations: HTTP/2 requests fail with too-long URLs - https://phabricator.wikimedia.org/T209590 (10Huji) Thanks for working on this. I found that 420 message hilarious! I resisted making a 420 pun, so instead, I offer you this meme to thank you for all the hard work :)  {F27790968}
[16:26:40] <wikibugs>	 10Traffic, 10Operations: HTTP/2 requests fail with too-long URLs - https://phabricator.wikimedia.org/T209590 (10Anomie) Note it's ok if the server implements some limits. The problem here is that the client isn't getting the expected 4xx response (e.g. 414 or 431) when the limit is hit, all it sees is a droppe...
[20:46:39] <wikibugs>	 10Traffic, 10Commons, 10MediaWiki-File-management, 10Multimedia, and 11 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214 (10Krinkle)
[20:47:15] <wikibugs>	 10Traffic, 10Commons, 10MediaWiki-File-management, 10Multimedia, and 11 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214 (10Krinkle)
[20:47:48] <wikibugs>	 10Traffic, 10Commons, 10MediaWiki-File-management, 10Multimedia, and 11 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214 (10Krinkle)
[21:01:03] <wikibugs>	 10Traffic, 10Operations: HTTP/2 requests fail with too-long URLs - https://phabricator.wikimedia.org/T209590 (10ema) >>! In T209590#4855448, @Anomie wrote: > Note it's ok if the server implements some limits. The problem here is that the client isn't getting the expected 4xx response (e.g. 414 or 431) when the...