[07:51:09] 10Traffic, 06Operations: Age header reset to 0 after 24 hours on varnish frontends - https://phabricator.wikimedia.org/T141373#2514310 (10ema) I've left varnishncsa running for six hours on a cache_upload machine, and the maximum value for Age found there was 601758 (1 week). Same procedure on a cache_text sy... [11:53:19] 10netops, 10Datasets-General-or-Unknown, 06Operations: dumps.wikimedia.org seems to have poor throughput towards some destinations - https://phabricator.wikimedia.org/T120425#2515046 (10Nemo_bis) This is a test of download from upload.wm.o on 217.30.184.184 (lakka.kapsi.fi). Just dumping here ``` federico@l... [11:56:49] 10netops, 10Datasets-General-or-Unknown, 06Operations: dumps.wikimedia.org seems to have poor throughput towards some destinations - https://phabricator.wikimedia.org/T120425#2515066 (10mark) So you're currently seeing 4+ MB/s, or ~40+ Mbps from eqiad, downloading from busy servers with many many connections... [14:13:23] 10Traffic, 06Operations: Support TLS chacha20-poly1305 AEAD ciphers - https://phabricator.wikimedia.org/T131908#2515382 (10MoritzMuehlenhoff) Some summarised comments in mostly random order :-) First of all, I don't think moving to BoringSSL/LibreSSL is doable (or even desirable). BoringSSL is essentially an... [14:47:08] 10Traffic, 06Operations: Support TLS chacha20-poly1305 AEAD ciphers - https://phabricator.wikimedia.org/T131908#2515438 (10BBlack) >>! In T131908#2515382, @MoritzMuehlenhoff wrote: > Some summarised comments in mostly random order :-) Thanks for the feedback! I know it's a complex annoying topic :) > First... [14:48:18] moritzm: I'm still holding my ground, but I don't mean to deter you from arguing back forcefully either if you still think it's bad idea :) [14:58:48] paravoid: you might take an interest in this debate linked above, too. but fair warning, it's a complex topic with very long posts. [14:59:20] oh I've read it :) [15:02:11] :) [15:05:06] I'll comment, sure :) [15:06:15] do you know what happens if you configure ciphers in nginx that the libssl version doesn't know about? [15:06:41] does it just ignore them or does it error out? [15:06:56] I'd guess the former but I don't remember every trying that [15:08:40] it ignores them [15:08:50] 10netops, 06Discovery, 06Operations, 03Discovery-Search-Sprint: deploy elasticsearch/plugins to relforge1001-1002 servers - https://phabricator.wikimedia.org/T141085#2515472 (10Gehel) Some notes of our discussion with @chasemp: [[ https://github.com/wikimedia/operations-puppet/blob/production/hieradata/co... [15:08:52] I've already put chapoly ciphers in our deployed lists that are being ignored :) [15:08:56] oh, nice [15:09:38] the only real incompatibility between 1.0.2+cf and 1.1.0+patches on that front is ordering [15:09:39] so even if a new libssl version comes out with a security fix, that conflicts with the cloudflare patch (an unlikely possibility judging from your comment) [15:09:53] yes, ssl_ciphersuite remains unchanged [15:09:54] we can just deploy the fixed version without chacha temporarily [15:09:59] right [15:10:06] just taking the perf hit [15:10:21] but re: moving from 1.0.2+cf to 1.1.0+equalpref, our ciphersuite ordering of those chapoly options has to change on the transition [15:10:50] the cloudflare pref hack needs the chapoly options at the top. 1.1.0+equalpref wants them ordered correctly using [A|B] syntax. [15:11:21] what's in our deployed list today just puts them in there beneath aes128-gcm, which means probably almost nothing will negotiate it, but it's technically there as an available option. [15:11:21] oh, that's cool [15:11:51] * ostriches spots a wild Platonides! [15:12:05] of course, if upstream isn't going to accept the equalpref patch (ever) anyways, I could dump that for a simpler approach like cloudflare's and keep ssl_ciphersuite compatible across both versions, too [15:12:18] Hi ostriches xD [15:13:17] backtracking from all the depth on this, because I doubt I ever gave a rationale well at the beginning: [15:14:14] for applicable Android mobile devices and older Intel PCs with Chrome (in both cases, lacking AES-NI and/or AVX), switching them to chapoly cuts crypto CPU usage for the client by ~50-66% depending on CPU. [15:14:26] so less battery waste, faster page load, blah blah [15:14:54] all this applies to modern Opera too, because Opera's now essentially based on Chrome. [15:16:52] (that and it's our only backup strong cipher if AES-GCM develops attacks/issues down the road, but that alone only argues for waiting on 1.1's official and un-prioritized support. this stuff is about making it work today for real clients) [15:18:19] is there an openssl bug about that equalpref patch? [15:18:49] Platonides: https://github.com/openssl/openssl/issues/541 [15:19:33] thanks [15:19:40] I was looking at rt.openssl [15:23:25] bblack: yeah, I'm aware of the chacha benefits (and excited to see you work on it!) [15:26:41] bblack: I'm totally fine with proceeding with 1.0.2+cloudflare for now, I guess I'm just a lot of more optimistic about openssl 1.1. I think we'll rather see less bugs in 1.1 compared to 1.0.2, e.g. CVE-2016-2177 was fixed by code refactoring in master for 1.1. once 1.1.0 is out we can work on it in parallel by introducing a src:openssl1.1 and see where to move from there [15:27:35] let me know if I can help with anything for openssl 1.0.2/cloudflare [15:37:49] moritzm: ok, thanks :) [15:38:15] I'll probably stab at deploying the package tomorrow or thursday [15:38:34] if nothing else changes by then, or someone else doesn't make a compelling argument against it [15:43:06] 10netops, 10Datasets-General-or-Unknown, 06Operations: dumps.wikimedia.org seems to have poor throughput towards some destinations - https://phabricator.wikimedia.org/T120425#2515591 (10Nemo_bis) Yes, but of course Helsinki is a very happy place. Also, upload.wm.o is much faster than dumps.wm.o, which AFAICT... [15:51:31] 10netops, 10Datasets-General-or-Unknown, 06Operations: dumps.wikimedia.org seems to have poor throughput towards some destinations - https://phabricator.wikimedia.org/T120425#2515609 (10BBlack) upload is very different than dumps. It's part of our traffic-optimize cache cluster termination, and dumps isn't.... [16:24:36] 10netops, 10Datasets-General-or-Unknown, 06Operations: dumps.wikimedia.org seems to have poor throughput towards some destinations - https://phabricator.wikimedia.org/T120425#2515751 (10BBlack) Oh, also if you're in Helsinki, hopefully you're hitting the upload.wm.o terminator in esams, which is close to you... [16:59:53] 10Traffic, 10Varnish, 06Operations, 13Patch-For-Review: Convert upload cluster to Varnish 4 - https://phabricator.wikimedia.org/T131502#2515835 (10ema) >>! In T131502#2512853, @ema wrote: > 1) User1 requests range 1-10 of a 1GB object. We want to make sure that varnish fetches the > whole thing, and s... [17:03:34] ema: yeah that's interesting :) [17:04:17] ema: I suspect the test results you have so far work out basically the same way on our varnish3, which is why we have VCL workarounds for the extreme cases [17:06:06] bblack: right. Also 3) is not true (if it had been for the final 10 bytes of the object, it should be considered a miss) [17:06:09] (frontends of course we have "special" range stuff, but even in the backends we already do ignore_busy for >=33MB mark on range requests, and then also pass-on-miss for the same... either the first part is redundant with the pass, or maybe v3 considers that case a hit but still stalls?) [17:06:35] bblack: not sure about v3, v4 certainly does consider it a hit and stalls [17:07:15] ema: so basically your observation is that V4 without special upload-vcl hacks acts in the most-dumb way it can while still basically-supporting Range, right? Which is to say: it ignores the Range part for cache/backend purposes, but once the whole object is fetched or found it returns the correct sub-range. [17:08:13] bblack: correct. Perhaps it's not literally the most-dumb way because at least it does not wait for the whole object to be returned on a low-range miss :) [17:08:27] well... kinda [17:08:37] it only does that for the first client though, not the others that stall on it [17:09:01] yup. It could only do worse by stalling on the first client as well [17:09:16] I would assume user1 1-10/1GB is satisified immediately, but user2 1-10/1GB stalls on the whole load for user1 [17:09:36] oh, let's try [17:09:48] it's probably true, given your other results [17:10:06] it's true [17:10:35] another interesting question (but not really necessary to answer, tbh) is what happens with storage in these cases [17:11:06] assume: we get 10 requests for bytes 1-10/1GB on this one object, and the requests are spaced by a few milliseconds. [17:12:01] I assume we effectively allocate 10GB of room in the cache, and then assuming they all finish in perfect sequence like they started, we commit the first copy of the object, then the next commits and destructs the first, etc, etc... [17:12:17] overallocation -> thrash -> eventually just 1GB [17:12:53] but there's probably a point in that sequence where we're effectively allocating (evicting) objsize * concurrency [17:13:18] whatever concurrency level we can reach before the first one completes a full load and starts satisfying hits, anyways [17:13:59] ignoring that problem, which undoubtedly also exists today and is probably 10 times worse under persistence and v3 anyways.... [17:15:31] our frontends basically pass on all Range requests [17:16:12] our existing backend VCL has magic to ensure high-range requests don't stall too long, with a tunable set at 32MB, which doesn't seem unreasonable. which triggers what I'm talking about above (otherwise they would stall instead of thrashing) [17:16:54] the only other issue I see about all of this we should look into: [17:17:08] our current VCL also passes on the Range header to the backend (including through varnish tiers) [17:17:18] whereas stock varnish strips it for the backend request [17:17:48] does varnish handle (well? at all?) a ranged response? and how does that change everything above? [17:18:02] (for that matter, does swift do ranges or just ignores them?) [17:18:49] because all of the above is predicated on the idea that the backend fetch is going to get the whole object [17:19:11] I'm not even sure how all of this behaves in V3, much less V4 :( [17:24:07] lots of interesting questions :) [17:24:52] bblack: so our current VCL sends Range to the appserver right? [17:25:39] but then I guess that in my test apache should respond with a 206, not the whole object [17:26:09] right [17:26:21] I'd test swift itself (make some test reqs to prod with curl?) [17:26:39] it may make a big difference in this whether swift does 206 or 200 [17:26:59] I'll certainly also test swift [17:27:14] in the case of apache, though, we know that it should respond with a 206 [17:27:45] but then why is the second range request for the same range stalling? [17:28:01] were you using our upload vcl? [17:28:06] yep [17:28:26] I donno then [17:28:36] let's dig further [17:28:55] the underlying question is: if we have range in bereq and varnish gets a 206 response, what does it even do with the content? [17:29:20] does it cache it somehow (as a separate ranged object? as a partially-satisified object?) does it toss it out after giving it to the client? [17:30:11] it would seem like it's a bad idea to pass range between cache backend layers if it kills caching of anything [17:30:26] maybe *that's* the missing magic our 3.x-plus had that v4 doesn't have. [17:31:22] if that turns out to be the case, we might want to slightly-alter our VCL on the backends more like: [17:32:17] oh, varnish probably strips Range before hitting the appserver [17:32:39] because apache responds with the full object (200) in case of range requests coming from varnish [17:32:39] yeah but we intentionally hack around that in VCL [17:33:25] (we copy req.http.Range over to bereq.http.Range through an intermediary, manually) [17:33:47] 10netops, 06Operations, 10ops-eqiad: cr2-eqiad temperature alerts ("system warm") - https://phabricator.wikimedia.org/T141898#2516015 (10faidon) [17:33:55] yeah I remember that (the ugly hack!) [17:38:59] mmh I've started sniffing a bit and there is no trace of Range being sent to apache [17:40:24] perhaps the hack doesn't work with v4 [17:42:37] check our upload cluster backend reqs, maybe it never worked :) [17:43:26] the Range stuff still falls into a category of "legacy VCL hacks that were there already when I started looking deeply, so I haven't touched them much since because nothing's obviously-broken and we have other things to do" [17:43:36] so really a lot of this I have no idea at all what's really happening [17:44:38] 10netops, 06Discovery, 06Operations, 03Discovery-Search-Sprint: deploy elasticsearch/plugins to relforge1001-1002 servers - https://phabricator.wikimedia.org/T141085#2516056 (10Gehel) 05Open>03Resolved Correcting network categorization solved the plugin deployment issue. Closing this task. [17:45:47] bblack: Range certainly gets sent from the FE to the BE [17:58:11] but it really doesn't look like we're sending it to swift [17:58:27] 0 results with: [17:58:28] varnishlog -b -m TxHeader:"Range" [17:58:49] that would solve the mistery! [18:01:16] OT: we also get lots of lowercase headers, because of h2 I guess [18:12:30] bblack: so yes, on cache_upload frontends we simply convert Range requests into pass and copy req.http.Range into bereq.http.Range [18:13:11] on backends, we only add the Range header to requests "over a certain threshold" [18:52:41] really? [18:53:06] oh you're right, I missed that looking at the VCL earlier [18:53:23] ok, so, we *would* send it to swift, if it was over the threshold [18:53:39] (the same threshold that causes full-pass behavior, basically) [18:54:10] and in the frontend we also pass and send Range onwards [18:54:41] so, high-range requests get the Range header passed through all layers and (pass) throughout, and I can only assume Swift answers range with 206 in that case. [18:55:09] low-range requests: frontend acts as above. first backend converts it to a full fetch with 200 response (no bereq.http.Range). [18:55:20] from there onwards to the applayer it would keep being a full request. [18:56:08] that's all on misses of course. range requests can obviously hit, too. [18:56:40] (and for high-range hits, apparently we still do hash_ignore_busy.... to avoid the stalled-hit thing on concurrency) [21:22:11] 10Traffic, 06Operations: TLS stats regression related to Chrome/41 on Windows - https://phabricator.wikimedia.org/T141786#2516829 (10BBlack) [21:36:39] 10Traffic, 06Operations: TLS stats regression related to Chrome/41 on Windows - https://phabricator.wikimedia.org/T141786#2516864 (10BBlack) The history around the Windows updates involved is confusing (it seems they released a bad patch, then stopped offering it as an update, then started offering a replaceme... [21:47:35] 10Traffic, 06Operations: TLS stats regression related to Chrome/41 on Windows - https://phabricator.wikimedia.org/T141786#2516938 (10BBlack) I take that back, I've found a way to reason about the latter part. There are supposed to be commas on the end when counting bytes. Counting them like that, the 1024-bo... [22:01:10] 10Traffic, 06Operations: TLS stats regression related to Chrome/41 on Windows - https://phabricator.wikimedia.org/T141786#2517011 (10BBlack) `KB3163018` is a better search term. That's the big cumulative update to Win10 that included these things (among others). Some links related to it that are relevant: h...