[04:13:08] 10Traffic, 10Operations: Move cache text cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231627 (10Vgutierrez) [04:34:22] 10Traffic, 10Operations, 10Patch-For-Review: Move cache text cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231627 (10Vgutierrez) [05:20:12] 10Traffic, 10Operations, 10Patch-For-Review: Move cache text cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231627 (10Vgutierrez) [06:37:49] 10Traffic, 10Core Platform Team, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712: Pages whose title ends with semicolon (;) are intermittently inaccessible - https://phabricator.wikimedia.org/T238285 (10eranroz) [08:38:02] 10Traffic, 10Operations, 10Patch-For-Review: Move cache text cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231627 (10Vgutierrez) [08:54:57] 10Traffic, 10Operations, 10Patch-For-Review: Move cache text cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231627 (10Vgutierrez) [09:14:11] 10Traffic, 10Operations: Move cache text cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231627 (10Vgutierrez) [09:36:09] 10Traffic, 10Operations, 10Patch-For-Review: Move cache text cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231627 (10Vgutierrez) [09:54:55] 10Traffic, 10Operations, 10Patch-For-Review: Replace Varnish backends with ATS on cache text nodes - https://phabricator.wikimedia.org/T227432 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts: ` ['cp3062.esams.wmnet'] ` The log can be found in `/var/log/wm... [10:24:15] 10Traffic, 10Core Platform Team, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712: Pages whose title ends with semicolon (;) are intermittently inaccessible - https://phabricator.wikimedia.org/T238285 (10ema) >>! In T238285#5665013, @Ladsgroup wrote: > I think this has to do something with... [10:29:12] 10Traffic, 10Operations, 10Patch-For-Review: Replace Varnish backends with ATS on cache text nodes - https://phabricator.wikimedia.org/T227432 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp3062.esams.wmnet'] ` and were **ALL** successful. [11:00:21] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712: Pages whose title ends with semicolon (;) are intermittently inaccessible - https://phabricator.wikimedia.org/T238285 (10mobrovac) [11:02:29] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712: Pages whose title ends with semicolon (;) are intermittently inaccessible - https://phabricator.wikimedia.org/T238285 (10ema) p:05Triage→03Normal [11:11:28] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712: Pages whose title ends with semicolon (;) are intermittently inaccessible - https://phabricator.wikimedia.org/T238285 (10Vgutierrez) hmm it looks like ATS URL parsing is at fault here. ATS is using a semi colon as a separator betwee... [11:23:26] not an expert, but shouldn't ; be url-encoded based on https://tools.ietf.org/html/rfc3986#page-12 ? [11:25:01] it's actually a deeply complex issue, how one interprets that stuff [11:25:07] although it also says "If a reserved character is found in a URI component and no delimiting role is known for that character, then it must be interpreted as representing the data octet corresponding to that character's encoding in US-ASCII." [11:25:30] so it should be url encoded, but if it is not, it shout work anyway [11:25:41] that is my interpretation [11:25:45] the more important distinction isn't so much about whether it's in the reserved set, but about why ATS considers it essential to structural parsing of URIs [11:26:43] I see, e.g. to not break some clients assumptsion, for example? [11:26:59] I'm not sure yet, I'm trying to dig up a good reason [11:27:35] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712: Pages whose title ends with semicolon (;) are intermittently inaccessible - https://phabricator.wikimedia.org/T238285 (10Vgutierrez) BTW, Checking RFC 3986, I'm not sure that `https://ban.wikipedia.org/wiki/Mal:;` is a valid URL whe... [11:27:36] I am a huge fan of that kind of work you do, it is not easy [11:27:54] but from the rfc3986 sort of viewpoint, the semicolon isn't very special. it's a sub-delim, not a gen-delim, and even gen-delims can be used as long as they're not used in a confusing place within the URI [11:28:12] yeah [11:28:17] I'd say that the end of the path is a confusing place :) [11:29:06] the ATS parser I don't think cares whether it was at the end [11:29:39] it's puzzling to me, and I'm tempted to think ATS is just wrong in its handling of the semicolon here, but I assume there's a reason, so digging on standards to see if I can infer what it is/was [11:30:25] generally speaking, once you've dealt with the breakdown of "scheme://host:port/" at the front and you're down to the path and beyond [11:30:34] hmmm actually it cares [11:30:44] if the path is /wiki/Mal:;x, the semicolon doesn't get stripped [11:30:46] almost everything is legit unencoded data except the ? and # to start query and fragment [11:31:05] at least that's my usual loose understanding [11:31:33] and I'm struggling to figure out why ; would be any different than, say, : or / in that regard [11:31:38] that would be my understanding too, based on that rfc [11:32:02] at the application layer you can have meanings for any/all of these things, which is what makes all this so complex [11:32:22] but even the forward slashes within the path are just app-specific bullshit, and the unencoded / can mean whatever the app wants it to mean [11:33:21] but clearly there must be some other sidecar standard or historical standard here where the semicolon, in the path part, was as important as the question-mark, or ATS wouldn't have this in its parser (which as a proxycache, obviously should take the broadest view, not app-specific interpretation) [11:36:02] so from stack overflow apparently 1999 W3C recommendations recommended to use ";" as the param separator instead of "&" [11:36:05] (although from a pragmatist point of view: we've revproxied mediawiki prod traffic through all kinds of implementations before and not hit this case, so I think ATS is in the minority viewpoint here, and this might really just be a bug in ATS (well, design flaw from misunderstanding something)) [11:36:25] vgutierrez: but that would only apply *within* the query part of the URI, which starts with the ? [11:36:37] right? [11:36:40] 2014 edition says that only "&" must be used [11:37:30] nothing interesting at: https://issues.apache.org/jira/browse/TS-1592?jql=project%20%3D%20TS%20AND%20text%20~%20semicolon [11:37:33] https://github.com/apache/trafficserver/blob/master/proxy/hdrs/URL.cc#L1344 [11:37:53] ^ here you can see that structurally, the way they spell out their local variables here... [11:38:07] it sounds like they're declaring some other component that sits between "path" and "query" called "param" [11:38:12] err "params" [11:38:27] yep [11:39:11] maybe this is URI-vs-URL stuff? (maybe ;params exists in the broader URI spec but not in the more-specific HTTP URL schema?) [11:39:29] but ATS is trying to be broad enough to work for unknown non-HTTP cases of URI here? [11:39:44] this is how ATS parses "https://ban.wikipedia.org/wiki/Mal:;?vgutierrez=1" [11:39:55] https://www.irccloud.com/pastebin/Y8nldz1W/ [11:40:45] that's the very first parse [11:40:49] before remapping [11:41:14] and you can see that Path is already missing the semi-colon and the "params" part of the url is empty [11:41:24] and query gets "vgutierrez=1" [11:42:26] what if you put some data between ;? [11:43:01] in any case, https://tools.ietf.org/html/rfc3986#section-3.3 (about Path) says explicitly in the first paragraph: [11:43:05] The path is terminated by the first question mark ("?") or number sign ("#") character, or by the end of the URI. [11:43:15] hmmm interesting [11:43:25] if I write "https://ban.wikipedia.org/wiki/Mal:;" on Chrome, it works for me [11:43:36] I guess that Chrome is encoding that semicolon for me [11:43:41] probably [11:44:03] UAs do differ in how aggressively they encode things [11:44:42] (in some cases this causes other kinds of pathology too, since major UAs different in the auto-encoding of a character can cause redirect loops if we attempt to redirect all requests to the canonical encoding) [11:47:31] nah.. is giving a 400 or that's what the inspector says... and I end seeing a page saying "murda sane nenten manut" [11:48:36] so further down in the Path section, there's stuff about individual segments [11:49:13] the RFC does talk about the path /abc/xyz/foo as being parseable in /-delimited segments, and all the special rules there, but it's really also (confusingly) digging into applayer details that can vary [11:49:44] but it does talk about how (again, at the applayer discretion) it might be common to use ; to set per-segment parameters [11:50:06] For example, the semicolon (";") and equals ("=") reserved characters are often used to delimit parameters and parameter values applicable to that segment. The comma (",") reserved character is often used for similar purposes. For example, one URI producer might use a segment such as "name;v=1.1" to indicate a reference to version 1.1 of [11:50:11] "name", whereasanother might use a segment such as "name,1.1" uto indicate the same. Parameter types may be defined by scheme-specific semantics, but in most cases the syntax of a parameter is specific to the implementation of the URI's dereferencing algorithm. [11:50:56] but in that context, you'd have to parse things like "/foo;a=b/bar;a=c/baz;a=z/..." all as part of Path still, and those parameter lists being separate for each path segment. [11:52:53] oh god, and then they actually use the semicolon in a reference example: [11:52:56] https://tools.ietf.org/html/rfc3986#section-5.4 [11:53:25] but none of the examples below that (of how to interpret a relative URI, relative to that example) contradict the non-specialness of the semicolon, I think. [11:53:37] but they must have put that there in the example for some clarifying/historical reason... [11:53:51] yup [11:53:59] I've pinged upstream via their slack channel [11:54:14] but I gotta leave now or my gf is going to kill me [11:54:19] (7pm here already ) [11:54:20] :) [11:54:44] "d;p" ";x" "g;x" and such in the example set there, all seem to be treated opaquely [11:54:50] (if confusingly) [11:54:59] vgutierrez: ok :) [11:59:26] RFC 1630 (which is very early on) describes a possible use in ftp:// URIs [11:59:44] as in ftp://asdf/xyz.zip;type=IMAGE [12:00:13] or maybe it's just type=I [12:00:19] either way there was IMAGE and ASCII, etc... [12:10:47] and here's another non-http case: [12:10:54] https://tools.ietf.org/html/rfc1738#section-3.11 [12:11:12] prospero:// URLs have a distinct ;-delimted part after the path [12:14:42] https://tools.ietf.org/html/rfc2396 is outdated (RFC 3986 referenced a bunch earlier obsoletes it), but it specifically mentions in https://tools.ietf.org/html/rfc2396#section-1.6 [12:14:55] This document uses two conventions to describe and define the syntax [12:14:55] for URI. The first, called the layout form, is a general description [12:14:58] of the order of components and component separators, as in [12:15:00] /;? [12:15:06] so there it is [12:15:28] 10Traffic, 10Varnish, 10Education-Program-Dashboard, 10Operations, 10Programs-and-Events-Dashboard-Sprint 2: Cache education dashboard pages - https://phabricator.wikimedia.org/T120509 (10ema) @awight: is there anything to do here or can we close the task? [12:15:45] there was no "gen-delim" vs "sub-delim" split back then, and semicolon was the first character of the generic "reserved" set as a delimiter [12:16:19] still, all non-ancient, HTTP-specific standards don't spell out this semicolon thing [12:22:54] 10Traffic, 10Operations, 10Wikimedia-Incident: upload.wikimedia.org returns HTTP status code 503 for truncated urls, not 404 - https://phabricator.wikimedia.org/T106517 (10ema) I cannot reproduce with URLs such as https://upload.wikimedia.org/wikipedia/commons/thumb/6/6b/Kitagawa_Utamaro_-_Toji_san_bijin_%28... [12:29:48] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712: Pages whose title ends with semicolon (;) are intermittently inaccessible - https://phabricator.wikimedia.org/T238285 (10Urbanecm) @Vgutierrez I guess what you quoted wouldn't be valid, but https://ban.wikipedia.org/wiki/Mal:%3B sho... [12:47:06] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712: Pages whose title ends with semicolon (;) are intermittently inaccessible - https://phabricator.wikimedia.org/T238285 (10BBlack) There's some confusion on historical standards interpretation here, I think. There are some ancient st... [13:03:09] 10Traffic, 10Operations, 10Patch-For-Review: Create a second text-lb IP address for test purposes - https://phabricator.wikimedia.org/T237492 (10faidon) p:05Normal→03High It looks like there are proposed patches for this, so perhaps we're not too far off? This ties to an exploration we're doing with a ve... [14:15:06] 10Traffic, 10Operations: Create a second text-lb IP address for test purposes - https://phabricator.wikimedia.org/T237492 (10BBlack) Should work now! [14:28:50] 10Traffic, 10Operations, 10Patch-For-Review: Replace Varnish backends with ATS on cache text nodes - https://phabricator.wikimedia.org/T227432 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts: ` ['cp3064.esams.wmnet'] ` The log can be found in `/var/log/wm... [14:43:00] 10Traffic, 10Operations, 10SRE-tools, 10Goal, and 3 others: Automate generation of Management DNS records from Netbox - https://phabricator.wikimedia.org/T233183 (10Volans) >>! In T233183#5665281, @BBlack wrote: > Seems sane! The only thing I'm a little iffy about iis from the "SHA1 written to etcd" onwar... [14:58:02] 10Traffic, 10Operations, 10Patch-For-Review: Replace Varnish backends with ATS on cache text nodes - https://phabricator.wikimedia.org/T227432 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp3064.esams.wmnet'] ` Of which those **FAILED**: ` ['cp3064.esams.wmnet'] ` [15:17:01] 10Traffic, 10Operations, 10Patch-For-Review: Replace Varnish backends with ATS on cache text nodes - https://phabricator.wikimedia.org/T227432 (10ema) [15:17:37] there's no trace of varnish-be left in our PoPs :) [15:17:48] codfw and eqiad are next to be converted [15:19:18] \o/ [15:19:27] my arms have been waiting since my false alarm yesterday [15:19:33] haha I know! [19:42:53] 10Traffic, 10Operations, 10SRE-tools, 10Goal, and 3 others: Automate generation of Management DNS records from Netbox - https://phabricator.wikimedia.org/T233183 (10Volans) A simplified version could be to use a cookbook to couple stuff: - have a script on the netbox hosts to generate the snippets from th...