[00:46:31] 10Traffic, 10Operations, 10ops-ulsfo: setup bast4001/WMF7218 - https://phabricator.wikimedia.org/T179050#3711643 (10RobH) [00:46:40] 10Traffic, 10Operations, 10ops-ulsfo: decommission/replace bast4001.wikimedia.org - https://phabricator.wikimedia.org/T178592#3711660 (10RobH) [00:46:43] 10Traffic, 10Operations, 10ops-ulsfo: setup bast4001/WMF7218 - https://phabricator.wikimedia.org/T179050#3711643 (10RobH) [00:49:04] 10Traffic, 10Operations, 10ops-ulsfo: setup bast4001/WMF7218 - https://phabricator.wikimedia.org/T179050#3711661 (10RobH) [09:01:38] 10Traffic, 10Operations, 10Patch-For-Review: Allow setting varnish connection timeouts in puppet - https://phabricator.wikimedia.org/T159429#3711925 (10ema) 05Open>03Resolved a:03ema All varnish runtime parameters can now be specified with the `profile::cache::base::be_runtime_params` hiera setting. [11:27:11] bblack: so, with multiple varnishes if the user-facing one disables streaming CL is set. If instead streaming is disabled by a later varnish (in our case let's say a varnish-be) and not by the one hit by the user, then CL is not set [11:27:19] https://phabricator.wikimedia.org/P6190 [11:28:01] I'm missing a "then" or a comma up there :) [11:28:20] with multiple varnishes, if the user-facing one disables streaming, then CL is set [11:28:24] better ^ [11:29:55] also, exp patch updated: https://gerrit.wikimedia.org/r/#/c/386192/ [12:16:06] 10HTTPS, 10Traffic, 10Operations, 10Parsoid, 10VisualEditor: Parsoid, VisualEditor not working with SSL / HTTPS - https://phabricator.wikimedia.org/T178778#3712186 (10Deskana) >>! In T178778#3707527, @PlanetKrypton wrote: > @Deskana I had plugin temporarily disabled so people didn't try to use it. Try no... [12:22:12] ema: ok so that kinda sucks, but maybe we can work with it. We could still potentially disable streaming on lack of CL at the backendmost, and then record the CL in an alternate header to pass around for storage/caching decisions, or something [12:23:01] although it will only be set on the outbound resp in that case [12:23:51] the open question then is whether disabling stream + doing gzip emits a CL [12:37:30] funny [12:37:57] I'm not managing to read CL in VCL in the test pasted above [12:38:32] it's as if CL was set at the very end (even after vcl_deliver) [12:38:57] but you do get on the beresp side in the next varnish up? [12:41:02] apparently not [12:41:05] so, we could have some generic code pretty early in backend_response that says "if (CL) { copy CL to X-CL }" to preserve length info through layers once we've ever seen it, and adjust CL-sensitive code to look for that header. [12:41:28] and then if streaming is off in the backend-most, the next one up should get to set the header that way? [12:42:18] bblack: https://phabricator.wikimedia.org/P6191 [12:42:25] the very last expect fails [12:43:30] * ema is perplexed [12:43:56] yeah me too, I thought we had seen at least single-layer behavior before with !stream->generate-CL [12:51:04] as I'm staring at related things... [12:51:25] I don't think we've properly taken into account the 4.x -> 5.0 -> 5.1 changes in hit-for-pass -related things yet [12:52:16] 5.0 killed the traditional hit-for-pass, replacing it with "hit-for-miss", and 5.1 then re-introduced a different form of "hit-for-pass" with different VCL conventions than the old one [12:54:11] https://varnish-cache.org/docs/5.1/whats-new/changes-5.0.html#hit-for-pass-is-now-actually-hit-for-miss [12:54:19] https://varnish-cache.org/docs/5.1/whats-new/changes-5.1.html#hit-for-pass-has-returned [12:54:52] it's confusing because the old code from 4.x is still syntactically correct, but by 5.1 it's definitely not doing what we think it's doing anymore, and there's new syntax to do what we used to want to do. [12:55:19] my best TL;DR summary so far (which might be wrong!): [12:56:48] 1) The old way to create hit-for-pass (beresp.uncacheable=true + beresp.ttl=X) now creates a hit-for-miss object. hit-for-miss objects avoid coalescing like hit-for-pass (basically imply req.hash_ignore_busy when hitting these objects), but every fetch through them is considered a miss rather than a pass, and thus is potentially cacheable for future clients as a real cache entry. [12:57:27] 2) If you want the old hit-for-pass behavior, in 5.1+, you need to now instead "return (pass(300s))" from vcl_backend_response to create hit-for-pass objects. [12:58:11] ah! the VTC test was wrong! CL *is* set if the backendmost varnish disables streaming https://phabricator.wikimedia.org/P6192 [12:58:35] the previous test was not going through v2 heh [12:58:55] 3) There's a bunch of meandering wording in the updates about how this affects conditional requests that I don't fully comprehend. Maybe hit-for-miss was better than hit-for-pass in some cases, but hfp is better for conditionals, or something.... [13:01:57] it kinda sounds to me like maybe people were invoking hfp behavior on transient errors, and hfm is better for those cases because it will recover to a cacheable object faster? but the new hfp is what we really want for non-error hfp cases. [13:02:16] but I don't yet understand all the 304/IFN/IMS considerations of hfp-vs-hfm [13:09:50] test updated: https://phabricator.wikimedia.org/P6193 (a) disabling streaming at the backend-most varnish is enough to access CL on the next varnish (b) no chance to read CL on the backend-most varnish itself [13:10:27] nice [13:12:04] oh wait, so it did transit on its own, too, right? [13:13:07] yes your test seems to confirm that [13:13:13] right, v1 in test case /3 has streaming enabled, v2 has streaming disabled. v1 gets CL. [13:13:15] v2 is !stream, v1 is streaming, and client gets CL [13:13:22] :) [13:13:40] and yeah, v1 *and* client get CL, right [13:13:55] is the response from v1 to the client actually streamed? [13:14:01] (and is it chunked?) [13:14:18] and the other question: now that v2 is hooked up right, can v2 see the outbound CL at its own deliver-time even? [13:14:25] (not that it helps much there) [13:14:44] no, v2 doesn't seem to be able to see the outbound CL [13:14:58] (see the assertion on beresp.http.X-v2-CL) [13:15:21] I've tried to set it in vcl_deliver instead to no avail [13:17:17] hmmm [13:17:47] so, the next most important question is probably about do_gzip [13:18:06] Transfer-Encoding is unset on the client [13:18:09] if server s1 sends some text, and varnish v2 does do_gzip = true [13:18:30] if server s1 sends some text, and varnish v2 does do_gzip = true + do_stream = false [13:18:49] I bet v1 and client don't get a CL [13:19:41] probably also matters that the client is ok with gzip in accept-encoding [13:23:00] it would be really handy if we could get CL information at all layers for storage-binning in the backends [13:23:29] I'd settle for getting CL (possibly smuggled in an alternate header) at the frontends for exp(-size/c) on 100% of reqs, but it would need to work for the gzip case too. [13:26:34] (trying) [13:27:22] the thing I keep having a hard time separating in my head when I think about this, is that streaming+chunked are independent [13:28:12] bblack: you win the bet. With do_gzip=true in v2, c1 gets no CL [13:28:22] you can have, from the perspective of a given varnishd: chunked-or-not on the client side, streaming or not through the core, and chunked-or-not on the backend side [13:28:50] it's possible to non-stream chunked->chunked. it's possible to stream notchunked->notchunked, etc... (and probably every other combination) [13:29:48] does it change anything if you re-order the statements do_gzip=true+do_stream=false in v2? [13:29:51] bblack: but! X-v1-CL is set [13:30:04] (to 31, the compressed length) [13:31:25] so v1 got a CL, but didn't forward it to c1 [13:31:30] right [13:31:32] was c1 accepting gzip? [13:32:06] let's see [13:32:27] (if not, then v1 had to un-gzip it first, which might change things too) [13:34:39] so the request performed by c1 has no header whatsoever [13:34:55] trying with AE:gzip [13:35:33] ah! [13:35:39] with AE:gzip c1 gets CL:31 [13:38:21] ok [13:38:52] so that's fine. our varnishes will always AE:gzip to each other when it matters, basically. [13:39:07] if the client doesn't, it gets an auto-ungzipped responses in some cases that lacks CL and is chunked+streamed out [13:39:36] but all varnishes except for the backend-most one will have CL [13:39:53] right [13:40:20] so we've established then, that it's universally true that if our backend-most varnish sets do_stream=false when it doesn't find a CL header, all further varnishes will observe a CL header, right? [13:41:15] yes, including do_gzip cases [13:41:21] I do wonder what's actually happening from v2's perspective in the do_gzip=true+do_stream=false case that it still emits a (compressed) CL [13:41:52] I guess the only answer that makes sense, is that it's sucking up the whole backend response, gzipping it for storage/forwarding, then responding with a CL [13:42:02] (in other words, the gzip isn't really streamed on the output side, at least) [13:42:55] so [13:43:29] do_stream=false at backend-most only, when CL is not provided by the applayer, seems like a small cost to get consistent CL at the FEs [13:44:02] we'll just have to live with the fact that we can't know CL in time to pick a storage bin at the backendmost [13:44:41] maybe if we're lucky we can eventually get rid of storage binning anyways [14:07:14] 10Traffic, 10Operations, 10ops-ulsfo: cp4024 kernel errors - https://phabricator.wikimedia.org/T174891#3712359 (10BBlack) I left this down because @RobH was due on-site a short while later. He observed no SEL entry while on-site. Powercycled back to the OS this morning with traffic depooled. No useful obs... [14:07:35] 10Traffic, 10Operations, 10ops-ulsfo: cp4024 kernel errors - https://phabricator.wikimedia.org/T174891#3712361 (10BBlack) a:05BBlack>03RobH [14:07:52] bblack: something like this? https://gerrit.wikimedia.org/r/386616 [14:08:58] jenkins got confused again [14:08:59] Line 8: Expected 'Change-Id:' to be in footer [14:09:27] ema: also, kill the similar code in templates/misc-backend.inc.vcl.erb ? [14:09:37] oh, right [14:11:35] it's because of the commit msg hook :). [14:12:04] if you do Something: (notice :) then the commit hook will put it under the change id [14:16:31] paladox: it doesn't get confused by Bug: BANANA though, only Ref: POTATO [14:17:19] because bananas taste better [14:17:30] fair enough [14:17:36] yeh [14:17:44] it's because the commit msg has Bug: [14:17:46] hard coded [14:18:20] ema see https://github.com/GerritCodeReview/gerrit/blob/43d985a2a15a7d59d42e19ffd60d41c0de6c3e59/gerrit-server/src/main/resources/com/google/gerrit/server/tools/root/hooks/commit-msg#L22 [14:22:39] paladox: ah nice one. Somehow I thought we were able to add Ref without jenkins getting offended, but perhaps I'm wrong [14:23:12] well no, I'm pretty sure I did use it in the past? meh [14:25:01] ema https://gerrit-review.googlesource.com/#/c/gerrit/+/136577/ :) [14:25:14] it wont be in 2.13 as 2.13 is EOL. [14:26:16] paladox: cool! [14:29:42] :) [14:30:25] bblack: how does streaming work without TE:chunked? [14:31:49] streaming in the varnish do_stream sense I think just means "start sending bytes to the client side as soon as we get bytes from the backend side". Even if both connections are non-chunked, one can stream-through the content as it arrives. [14:32:18] (as opposed to buffering it all up till the end, then starting to send it to the client) [14:33:02] right, but we can't see that just looking at req/resp headers [14:33:14] see as in check if it's happening at the vtc level [14:40:45] bblack: any further comments on https://gerrit.wikimedia.org/r/#/c/386192/? Should we give it a try? :) [14:49:44] ema: yes :) [14:53:25] bblack: mmh we should perhaps allow to set rate/base in hiera actually [14:53:55] and yes, I forgot misc! [14:55:12] ema: yeah I think rate/base in hiera is where we go in the long run. up to you if it's worth it at the moment, since we're probably a long ways from coming up with new numbers [15:00:30] bblack: right, and maybe one sunny day we'll be able to set vcl_config in hiera [15:01:18] or not have a vcl_config, I donno [15:01:24] that abstraction has issues :P [15:20:57] 10Traffic, 10netops, 10Operations, 10Pybal, 10Patch-For-Review: Frequent RST returned by appservers to LVS hosts - https://phabricator.wikimedia.org/T163674#3712530 (10elukey) Just found this nice comment in a mozilla repo (not sure if this is the core of firefox or some old thing): https://github.com/m... [16:11:30] first search result for hit-for-miss isn't helpful [16:11:32] https://www.youtube.com/watch?v=X6yXm88fCa4 [16:12:08] (nor acceptable in any way) [16:13:21] ema: +1 on latest PS, with the caveat that we still don't have an answer in there for coalescing issues [16:13:29] but still, we can test the rest on canary [16:14:10] the v4 answer is probably "always hash_ignore_busy on frontends". the v5.1 answer is probably "hit-for-miss". maybe we stall turning this on until cache_upload==v5.1 [16:15:58] or, alternatively, we find a way to convince ourselves that with a plain forced-miss (uncacheable+ttl=0), coalescing is not an issue because blah blah about how the statistics work out in the edge cases [16:17:11] the edge case being: can URL x of size y be just-popular-enough that coalesced-forced-misses cause stalling/perf issues we care about, but not quite popular enough to be likely to pass the probability filter and become a hit. [16:18:02] my suspicion at present is that such cases would exist for sufficiently-popular + sufficiently-large files [16:22:12] so our old/current code, we had the 4-hit-wonder-filter first, and then we did hfp on objects with CL>=256KB [16:22:41] right [16:23:05] looking at how probabilities scale with the calculated-c filter, for fe_mem_gb=121G (the common case other than the new ulsfo/eqsin caches) [16:23:54] CL=32KB -> adm_prob = 46%, 64KB->21%, 128KB->4.5%, 256KB->0.2% [16:25:55] for fe_mem_gb=209G (the new ulsfo/eqsin): 32KB->62%, 64KB->39%, 128KB->15%, 256KB->2.3% [16:26:14] (and 512KB->0.05%) [16:26:57] it's pretty easy to imagine a popular multi-megabyte file, which has nearly-zero chance of frontend admission [16:27:14] but has 10 or 100 clients coming in parallel and would suffer stalls without some anti-coalesce solution [16:27:50] don't we have the same problem now with the CL>=256K cutoff? [16:28:05] ah but hfp [16:29:05] even with the larger 209GB frontend caches, for example, a 4MB file has adm_prob ~= 5.793263527e-27 [16:29:17] (it will basically never get cached, and if it does maybe hell just froze over) [16:29:55] and the reason n-hit-wonder doesn't suffer this problem is because of the small N [16:30:08] at most we'll have N clients stall on it (probably) before it becomes a hit [16:30:38] I guess more than that if they all hit truly in parallel and asynchronous without previous fetches happening in the recent past [16:30:45] but it will still resolve itself fairly quickly [16:32:13] yeah it does sound like hit-for-miss would help here [16:33:00] with a ttl that sufficiently larger than expected transfer times, basically, to be sure we've killed parallel stalling reasonably [16:33:12] e.g. ttl=15m should be plenty [16:34:03] or, we do what would work on both v4 and v5.1: unconditionally hash_ignore_busy in the frontend [16:34:27] (which will probably increase fe->be connection parallelism in edge cases, but ensure we never stall while doing these complex decisions in fes) [16:34:28] 10Traffic, 10Operations, 10ops-ulsfo, 10Patch-For-Review: setup bast4001/WMF7218 - https://phabricator.wikimedia.org/T179050#3712808 (10RobH) [16:35:19] one of the core jobs of the frontend is to shield the backend from mass requests [16:36:13] if we have a sudden influx of users coming all at once asking for previously-cold URL x, there might be 1K users per frontend all hitting in parallel, but the FE ensures it's 1 req-per-fe to the next backend layer, and they in turn coalesce to 1 req-per-dc towards a common core DC's backends, etc [16:36:39] if we unconditionally set hash_ignore_busy in frontends, we lose a little slice of that protection. [16:37:24] when those first 1K users hit exactly in parallel on that cold URL, they'll all cause requests to the backend layer. but as soon as the backend layer answers once with a cacheable object, the FE will satisify them all (and all future requests within TTL) [16:37:40] significant drop in failed fetches since this morning 9:03 UTC [16:37:44] is that ema's timeouts change? [16:37:45] https://grafana-admin.wikimedia.org/dashboard/db/varnish-failed-fetches?orgId=1&from=now-7d&to=now&var-datasource=esams%20prometheus%2Fops&var-cache_type=text&var-server=All [16:38:22] mark: indeed [16:38:33] 08:58 ema: cache_text/upload varnish-be: set rutime parameter timeout_idle to 120s [16:38:44] nice [16:39:54] hit-for-miss would solve the problem more-elegantly than a blanket hash_ignore_busy [16:40:56] but that means we merge this patch now to play with on canary, but don't turn on the parameter for cache_upload until we upgrade it to v5, which is blocked on "sort out the hfp/hfm mess for upload+v5 in general", so it might be a little bit. [16:42:05] we can test the exp policy on canary and misc (using hfm as they're v5!) [16:42:14] right [16:42:22] and yeah, proceed with sorting out the hfp/hfm mess, upgrade upload, profit [16:42:32] I think to turn it into hfm on v5.1, we just need to set a non-zero ttl [16:42:48] we're non-profit [16:42:59] (which we would've thought would've made hfps like it did on v4, but now we've read about the fun incompatible changes!) [16:47:27] re: failed fetches + timeout_idle: ... [16:47:50] most likely those weren't user-visible, as the frontend retry-503-once would cover them with silent retries [16:48:07] yes [16:48:44] those failed fetches were the race condition where the "client" (varnish-fe or a remote varnish-be) started sending a new request on an idle persistent connection, while at the exact same moment the backend was closing it for reaching the (very short) timeout_idle. [16:48:53] the race is much less likely with a longer timeout_idle [16:49:09] (but still possible) [16:49:41] this also all has some interplay with our gethdr_extrachance patch [16:50:01] varnish would normally retry such a case internally and invisibly [16:50:27] (it doesn't count in stats as a retry, VCL gets no control, and if the race-like failure seems to keep happening, it will retry infinitely and eventually the whole transaction times out) [16:50:45] i see [16:50:47] we patched around that stupidity and basically told it to never retry such cases, to avoid the insane "infinite unreported internal retries" [16:51:12] because we observed that when varnish-be was talking to a broken restbase at one point, I think (or parsoid? one of the two) [16:52:12] our patch allows those retries to be configurable, but we currently configure them as zero (let it fail and let the frontend retry-503-once catch it) [16:56:45] reuters: Twitter Inc (TWTR.N) said on Thursday it may become profitable for the first time next quarter after slashing expenses over the past year and ramping up deals to sell its data to other companies, which could help to break its reliance on advertising for revenue. [16:57:03] ... [16:57:37] I don't know what to hate more: that they've been around for ~10 years as a for-profit company without making a profit, yet valued so high. Or that their answer is to sell out their users harder :P [17:05:55] https://blog.benjojo.co.uk/post/north-korea-dprk-bgp-geoip-fruad [17:18:12] 10HTTPS, 10Traffic, 10Operations, 10Parsoid, 10VisualEditor: Parsoid, VisualEditor not working with SSL / HTTPS - https://phabricator.wikimedia.org/T178778#3702250 (10Arlolra) What does your Parsoid config.yaml look like? [17:19:19] 10Traffic, 10netops, 10Operations, 10Pybal, 10Patch-For-Review: Frequent RST returned by appservers to LVS hosts - https://phabricator.wikimedia.org/T163674#3713056 (10BBlack) It looks like that commentary is still valid for even the most-recent Firefox builds. So, we may find that even modern Firefox d... [17:22:16] 10netops, 10Operations, 10ops-eqiad: Setup eqsin RIPE Atlas anchor - https://phabricator.wikimedia.org/T179042#3713069 (10faidon) p:05Triage>03Normal [17:23:12] 10netops, 10Operations, 10ops-eqiad: Setup eqsin RIPE Atlas anchor - https://phabricator.wikimedia.org/T179042#3711364 (10faidon) [17:24:58] 10netops, 10Operations, 10ops-eqiad: Setup eqsin RIPE Atlas anchor - https://phabricator.wikimedia.org/T179042#3713103 (10faidon) Image has been downloaded to the install* servers. [18:10:48] 10Traffic, 10Operations, 10ops-ulsfo: cp4024 kernel errors - https://phabricator.wikimedia.org/T174891#3713348 (10BBlack) Powering off for now, less confusing for other software-level maintenance. [18:40:14] that's interesting reading, thanks [18:43:14] I never new apnic->chinanet owned 1.3.3.7 :) [19:01:32] 10Traffic, 10Operations, 10ops-ulsfo: setup bast4001/WMF7218 - https://phabricator.wikimedia.org/T179050#3713481 (10RobH) [19:21:44] 10Traffic, 10Operations, 10ops-ulsfo: cp4024 kernel errors - https://phabricator.wikimedia.org/T174891#3713527 (10RobH) I'm going to disable its switch port and power it back into the diagnostics (also asked @bblack via irc if there was a result from him running hte output for diagnostics) [22:08:01] 10Traffic, 10Operations, 10ops-ulsfo: cp4024 kernel errors - https://phabricator.wikimedia.org/T174891#3713945 (10RobH) Ran full test today, no errors. ``` /------------------------------------------------------------------------------\ | Build : 4239.44 ePSA Pre-boot System Assessment ServiceTag...