[10:16:32] <gilles>	 https://blogs.dropbox.com/tech/2019/12/evaluating-bbrv2-on-the-dropbox-edge-network/
[10:34:37] <wikibugs>	 10netops, 10Operations, 10cloud-services-team (Kanban): Return traffic to eqiad WMCS triggering FNM - https://phabricator.wikimedia.org/T240789 (10akosiaris) I guess parsing past alerts can help answer this question? If the number of exceptions that need to be defined is small enough it might not be worth it...
[10:41:55] <wikibugs>	 10netops, 10Operations, 10cloud-services-team (Kanban): Return traffic to eqiad WMCS triggering FNM - https://phabricator.wikimedia.org/T240789 (10ayounsi) Good point! Looking at past alerts only that one was a false positive.  2) would allow us to have stricter thresholds, but I agree that it's outside the...
[13:11:36] <wikibugs>	 10Traffic, 10Operations, 10User-jbond: Setup a new PKI software as an alternative to the puppet CA for managing services certificates - https://phabricator.wikimedia.org/T194031 (10jbond) a:05Volans→03jbond
[13:20:21] <wikibugs>	 10netops, 10Operations, 10cloud-services-team (Kanban): Return traffic to eqiad WMCS triggering FNM - https://phabricator.wikimedia.org/T240789 (10CDanis) +1 to doing #1 and revisiting if it becomes a problem again.
[13:38:52] <wikibugs>	 10netops, 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): WMCS: cleanup network allocations - https://phabricator.wikimedia.org/T240670 (10aborrero) Patches are merged. I can do the Netbox cleanup (delete all those objects) by myself if you confirm @ayounsi
[14:25:42] <wikibugs>	 10Traffic, 10Operations: /sec-warning page: please add a helpful XML comment explaining why it's being delivered. - https://phabricator.wikimedia.org/T240794 (10DavidBrooks) Suggest: near the top of the /sec-warning file, add an HTML comment:  <!-- If this content has been delivered in response to a MediaWiki...
[14:27:19] <wikibugs>	 10Traffic, 10Operations: /sec-warning page: please add an HTML comment that is more easily visible to API and transport-level inspection/debugging - https://phabricator.wikimedia.org/T240794 (10DavidBrooks)
[15:11:03] <wikibugs>	 10Traffic, 10Operations: /sec-warning page: please add an HTML comment that is more easily visible to API and transport-level inspection/debugging - https://phabricator.wikimedia.org/T240794 (10BBlack) Or a patch to template this in.  The problem is it's implemented from a standard template for the top 30-40 l...
[15:59:56] <wikibugs>	 10Traffic, 10Operations: HTTPS/Browser Recommendations page on Wikitech is outdated - https://phabricator.wikimedia.org/T240813 (10BBlack) The wording issues here are actually a bit tricky.  We've done several TLS standards upgrades over time, and there are still a few to go:  Done sometime in the past: * HTTP...
[16:08:05] <wikibugs>	 10netops, 10Operations, 10cloud-services-team (Kanban): WMCS: cleanup network allocations - https://phabricator.wikimedia.org/T240670 (10ayounsi) For sure! Ping me if you have any issue.
[16:32:21] <XioNoX>	 "The 130+ ms RTT band represents cross-Pacific traffic and hence very likely a GSLB failure to properly route users to the closest PoP. We’ll talk about how we are utilizing RUM data to avoid that in the following blog post."
[16:32:53] <cdanis>	 is that from a Google post?
[16:33:21] <cdanis>	 oh, DRopbox
[16:34:06] <bblack>	 maybe their following post will outline all the same RUM stuff we've been talking about here for years and never made any progress on :)
[16:34:45] <bblack>	 (TL;DR - 1/1K client-side JS pinging all our edges and feeding us RUM used to build a better dataset for our dns geographic routing than maxmind)
[16:36:49] <XioNoX>	 I'm wondering if BBR2 will have much differences for us as we have smaller pages
[16:37:24] <wikibugs>	 10netops, 10Operations, 10cloud-services-team (Kanban): WMCS: cleanup network allocations - https://phabricator.wikimedia.org/T240670 (10aborrero) 05Open→03Resolved Done, I deleted all the objects referred in the task description from Netbox.
[16:37:42] <bblack>	 and better text compression
[16:37:55] <bblack>	 (hello to our missing support of brotli)
[16:38:09] <bblack>	 and TLSv1.3 killing 1xRTT off of most of our connection startups
[16:38:27] <bblack>	 there's a lot of good perf stuff to look forward to, on some timeline that's hard to see at times
[16:39:30] <cdanis>	 silly question, what TLS version do we use internally (eg between ATS and appservers)?
[16:40:10] <bblack>	 1.2 of course
[16:40:43] <bblack>	 but that shouldn't be a *huge* deal because those connections are heavily reused and amortize the startup costs...
[16:41:02] <bblack>	 (except they don't, because appservers.svc nginx closes connections on ats-be way more often than it should....)
[16:42:34] <bblack>	 (because we don't want to store-and-forward buffer the response flow at the apache/nginx level , because then if php slow-streams a large output it would add huge repsonse latency vs streaming-through all the layers...)
[16:43:14] <bblack>	 (but then once you've decided not to store-and-forward in any of these proxy layers, if php throws an exception halfway through the output of a specific page and doesn't finish it (or right after headers or whatever), the connection becomes invalid and has to be torn down by one side or the other, possibly after a timeout)
[16:44:01] <bblack>	 there are some deeply-troubling architectural things buried in here when you start staring at this stuff...
[16:46:03] <bblack>	 cdanis: but back to your point: we need the appservers and the ats boxes to both be on debian buster, in practice, to start using 1.3 there
[16:46:36] <bblack>	 it would also be nice to get icinga at least on buster sooner rather than later, because it can't monitor TLSv1.3 over the network right now
[16:46:45] <bblack>	 and I've already deployed a TLSv1.3-only service that it can't monitor :P
[16:47:05] <cdanis>	 heh heh heh
[16:47:15] <cdanis>	 great :D
[17:15:17] <bblack>	 cdanis: I had another random thought this morning as I was making coffee, re: pursuing request-ids of some kind we can trace through the stack in a header...
[17:15:34] <cdanis>	 bblack: I'm in a meeting -- but please, go on! will just reply async
[17:15:56] <bblack>	 cdanis: we should also (separately, in some sense, although it could perhaps share some implementation details and/or the header) tack on connection-tracking through the layers as well
[17:17:12] <bblack>	 cdanis: like, it might be interesting to know that request#ffabc631... that we're tracing through the stack, used connectionids 123, 456, 789, and abc at various layers (the shared/reused connections at ths ats-tls->v-fe, v-fe->ats-be, ats-be->appservers.svc nginx, that -> apache, ...
[17:17:32] <bblack>	 so that it's possible to track things that affect connections, like the stuff I was rambling about above
[17:18:21] <bblack>	 insights like "oh this request had anomalous behavior yet nothing seems wrong with the request, but let's look at the connection it was flowing over between v-fe->ats-be.  Oh there was a different request right before it that broke the state of the connection"
[17:18:23] <cdanis>	 oh interesting! I like it
[17:19:21] <bblack>	 because I think a lot of our hard-to-trace problems do tend to come down to leakage of brokeness from one request to the next in some shared connection at some layer
[17:19:50] <cdanis>	 that's the reason for resetting of a connection on error, right?
[17:20:13] <bblack>	 in theory, we do that in places where we can
[17:20:35] <bblack>	 sometimes the two ends of a connection don't even have a shared view of the protocol state, so it gets tricky
[17:20:56] <cdanis>	 right
[17:22:05] <bblack>	 e.g. you can imagine a scenario (which I think we've seen live variants of) where varnish-be is talking to appserver-apache, and varnish-be things that request#345 of that connection did a full req->resp cycle and the connection is idle and waiting to be re-used for connection#346, and isn't trying to read the supposedly-idle connection's data at present.
[17:22:25] <bblack>	 but apache thought it wasn't done with the response and has already sent another 8932 bytes of html into some buffer somewhere
[17:22:52] <bblack>	 and then varnish-be sends a new request down this idle connection, and the response is garbage (the trailing HTML of #345, not the response headers of #346)
[17:23:13] <bblack>	 and then it closes when it sees the garbage, and errors out request#346, and we don't even know how to find the faulty req in #345
[17:23:54] <bblack>	 s/things/thinks/ a few lines up, and s/connection#346/request#346/ too
[17:24:53] <bblack>	 but if we had the conn-tracking through the layers with the req-tracking...
[17:25:24] <bblack>	 you might say "there are all these 503s on random URIs, there's no pattern", oh but wait, the pattern is that every connection they flowed over at layer X shared a request for faulty URI Y shortly before it"
[17:28:17] <bblack>	 luckily connections already have unique identifiers within reasonable time bounds thanks to sockets
[17:29:00] <bblack>	 a timestamp and an ip:port<->ip:port, and maybe a tag to help humans not have to parse the IP just to know this was the entry created by v-fe or ats-tls or nginx-appserver or whatever
[17:29:44] <wikibugs>	 10Traffic, 10Operations: /sec-warning page: please add an HTML comment that is more easily visible to API and transport-level inspection/debugging - https://phabricator.wikimedia.org/T240794 (10DavidBrooks) Ah, that makes it more complex. Again, I guess there's no evidence that anyone else is encountering the...
[21:23:13] <wikibugs>	 10Domains, 10Traffic: nameserver change for wikimedia.sk - https://phabricator.wikimedia.org/T241084 (10RobH) 05Open→03Stalled p:05Triage→03Normal
[21:26:52] <wikibugs>	 10Domains, 10Traffic, 10Operations: nameserver change for wikimedia.sk - https://phabricator.wikimedia.org/T241084 (10RobH) a:03Luky001
[21:29:11] <wikibugs>	 10Domains, 10Traffic, 10Operations: nameserver change for wikimedia.sk - https://phabricator.wikimedia.org/T241084 (10RobH)