[00:21:16] 10Traffic, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar): Define turn-up process and scope for eqsin service to regional countries - https://phabricator.wikimedia.org/T189252#4099198 (10Krinkle) [01:45:50] 10netops, 10Operations: Config discrepencies on network devices - https://phabricator.wikimedia.org/T189588#4099392 (10ayounsi) [08:55:47] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Multimedia, and 4 others: Thumbor incorrectly normalizes .jpe and .jpeg into .jpg for Swift thumbnail storage - https://phabricator.wikimedia.org/T191028#4099963 (10Gilles) [09:44:25] 10Wikimedia-Apache-configuration, 10Operations: Review/Merge/Deploy advisors.wikimedia.org apache vhost - https://phabricator.wikimedia.org/T190143#4100035 (10Reedy) 05Open>03Resolved a:03Dzahn [12:42:09] 10Traffic, 10Operations, 10media-storage, 10User-fgiunchedi: Swift invalid range requests causing 501s - https://phabricator.wikimedia.org/T183902#4100418 (10fgiunchedi) a:03fgiunchedi [12:56:45] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Upgrade pybal-test instances to stretch - https://phabricator.wikimedia.org/T190993#4100439 (10Vgutierrez) [13:06:17] so eqsin has been handling more traffic than ulsfo for the past few days, and more to come! :) [13:06:40] yeah [13:07:09] unfortunately we're now nearing the end of the low-hanging-fruit phase, and what we have left are the Complicated-with-a-capital-C cases. [13:11:25] oh [13:11:26] (which are mostly AU and CN, and maybe NZ) [13:11:51] maybe a better way to state that is: AU and maybe-NZ, and CN and maybe-KP [13:12:09] and of course the blocked-by-zero cases are what they are, but we can expect some movement by sometime in June on the remaining ones. [13:13:12] yeah I was also thinking that AU and NZ have access patterns which are probably more similar to EU/US than to JP and other countries culturally in "the east" [13:13:26] all 4 of them hypothetically should be closer to eqsin if networks and peerings and whatnot were perfect [13:14:16] yeah no doubt AU/NZ's cache-access patterns are a closer match for ulsfo than eqsin [13:14:25] but latency still wins if we can get it [13:15:11] it's gonna be interesting to see how hitrate changes when pointing them to eqsin though [13:15:11] the commonality in the eqsin access patterns is pretty phenomenal so far [13:15:26] ~2% higher than we see elsewhere [13:15:30] yep [13:15:37] commonality? [13:16:12] cache-access patterns seem more uniform than elsewhere, hence the higher hitrate [13:16:15] TL;DR - we see different hitrates at different DCs (independent of pass-traffic that has more to do with logged-in/editing rates) [13:16:23] ah :) [13:16:30] and we suppose that is due to how diverse or common the set of hot URLs hitting it is [13:16:40] (e.g. cultural/language commonality) [13:16:43] yes [13:16:56] eqsin is so far beating all previous DCs [13:16:59] good, lets return that broken server ;) [13:17:28] we've seen stable patterns 24h+ with ~98.3% hitrate for hit/(hit+miss), most others top out closer to 96.2% or so [13:18:58] mmh I'm looking at our current vcl and it seems to me that we're ignoring grace altogether [13:19:10] it's not up to VCL to ignore grace [13:19:17] (I don't think) [13:19:35] well the builtin vcl does this in vcl_hit: [13:19:40] # if (obj.ttl + obj.grace > 0s) { [13:19:48] # // Object is in grace, deliver it [13:19:48] # // Automatically triggers a background fetch [13:19:48] # return (deliver); [13:19:48] # } [13:20:12] yeah but if you take that out, return (deliver) is still what happens [13:20:51] the point is, if the object is in grace and gets hit, it still acts like a hit and goes through vcl_hit and such, while triggering a background fetch, and that's all internal-to-varnishd, not VCL. vcl_hit just allows some optional control over the process. [13:21:11] oh right, our vcl_hit ends with return(deliver) [13:21:30] (you could turn it into a miss if you didn't like the grace, as is the case when you set object-side grace high as a maximum, then trim it under backend->healthy conditions) [13:22:45] although that does raise some interesting questions I suppose, about objects that are past grace but the expiry thread hasn't gotten around to evicting them.... [13:24:56] (this is probably another one of those issues that sits near the heart of the 1799-like issues. varnishd does seem to more-or-less assume that expiry happens on time, roughly) [13:26:07] so for example, does varnish still trigger a background fetch in case of hits on a object past-grace which has not been evicted by the expiry thread yet? [13:26:15] (are those actually considered hits?) [13:26:28] well I hope it doesn't consider them pure hits, that would be insane [13:27:04] the keep factor has to come into play, too, though, so you'd think this would be handled ok [13:27:30] if keep is defined and we're in the keep window, it would be normal for ttl+grace to be < 0 but expiry to not have yet happened. [13:27:40] right [13:27:59] (in which case a hit isn't possible, so surely there's already logic guarding those cases going through vcl_hit. they're just "optimized misses") [13:29:34] if that weren't true or keep didn't exist, we might have to wonder whether we at least need a: "if(obj.grace + obj.ttl <= 0s) { return (miss); }" to avoid serving grace-hits on fully-expired objects that haven't been reaped. but then if we actually did so, we'd almost certainly trigger #1799 again. [13:30:06] that's what the builtin vcl_hit does [13:30:43] (returning miss if grace+ttl <=0) [13:30:56] yeah but it seems redundant in the way it's structured even if you did want that [13:31:02] I think it's more an example of the kind of control you can have [13:31:38] anyways, while I tend to believe my "if keep exists and works right, this can't be a problem" argument, so far I'm failing to find the evidence that this isn't a problem in the source code :) [13:32:32] surely it's not "vcl_hit -> return (miss)" code that makes keep work sanely at all? [13:32:45] (as opposed to keep acting like an ultra-extended grace with no code in vcl_hit?) [13:32:58] I wouldn't be so sure :) [13:35:07] HSH_Lookup? [13:36:37] I'm trying to find where it even triggers the bgfetch at [13:36:48] (but from the fetching request's pov) [13:37:21] I don't see "keep" being used in HSH_Lookup [13:37:32] no, but it's indirectly part of some other calculations called from there [13:37:35] I think [13:37:55] EXP_Ttl and/or hash->lookup ? [13:41:08] mmh [13:41:55] yeah nothing in the hash->lookup() calls cares about TTL [13:42:01] and EXP_Ttl doesn't look at grace or keep [13:42:53] but from that pov, it kinda looks like HSH_Lookup would never return objects that are off in grace or keep (with actual TTL < 0), so clearly I'm missing something [13:44:53] so, one hint at where some things are: [13:45:17] cache_req_fsm.c: cnt_lookup has: [13:45:45] VCL_hit_method(req->vcl, wrk, req, NULL, NULL); [13:45:45] switch (wrk->handling) { [13:45:45] case VCL_RET_DELIVER: [13:45:45] if (busy != NULL) { [13:45:45] AZ(oc->flags & OC_F_PASS); [13:45:47] CHECK_OBJ_NOTNULL(busy->boc, BOC_MAGIC); [13:45:50] VBF_Fetch(wrk, req, busy, oc, VBF_BACKGROUND); [13:45:53] } else { [13:45:55] (void)VRB_Ignore(req);// XXX: handle err [13:45:58] } [13:46:00] so VCL_hit_method is our vcl_hit [13:46:03] and VCL_RET_DELIVER is when it returns (deliver) [13:46:28] somehow "busy" seems to be triggering a background fetch there? [13:48:38] yeah but how do we end up in _hit to begin with if lookup only considers ttl? [13:50:14] I'm not sure yet, I think it's more complicated than that, though [13:50:34] cnt_lookup is what calls HSH_Lookup [13:50:47] and the 3rd argument is how HSH_Lookup sets "busy", which then causes that bgfetch [13:50:57] assignments to *bocp in HSH_Lookup basically [13:51:10] which there are a few of, and they don't all seem naturally like cases for bgfetch heh [13:55:10] interestingly, I think all miss-from-hit are background fetches [13:55:22] which isn't what I would've expected... [13:55:58] (in other words, this also applies to a hit-for-miss object. a true miss is a normal fetch, and a hit on a hit-for-miss object is a "bgfetch") [13:56:09] but I guess, it's a bgfetch that the request stalls on the results of [13:56:28] ... ? [14:00:39] what does it even mean for a hfm hit to be a bgfetch? [14:00:56] I have no idea [14:01:04] bgfetch -> deliver (possibly) stale cached object while fetching a fresh copy [14:01:28] hfm -> disable coalescing and hit the origin server w/o waitinglist [14:01:36] all I know is, VBF_BACKGROUND seems to be a bgfetch (whatever that means), and once you've hit vcl_hit that's the only kind of VBF_Fetch that can happen [14:01:53] whereas vcl_miss / vcl_pass can do other types of VBF_Fetch [14:02:56] going back to a different angle on untangling this, though [14:03:18] even indirectly through macros, when I look for all the places the "keep" value is used in the code [14:03:29] there's lots of places that log it, and there's lots of places that set it for various reasons [14:03:50] but the only decisions made based on "keep" are the ones about when the expiry thread reaps it. nothing else seems to make a keep-based decision. [14:04:28] yeah, except for choosing whether an object is shortlived [14:05:02] so, at least in the keep case (assume everything else is sane), there appears to be no distinction between an object that's past ttl+grace but still in keep, and one that's past keep but the expiry thread didn't get around to expunging it... [14:05:26] right [14:05:35] the same pattern may well apply to grace calculations (ugh) [14:06:40] so there's no difference between an object still in grace and an object past ttl+grace+keep but not expunged yet? [14:08:15] I don't know yet, all the uses of "grace" are harder to track through [14:09:58] I take that back, they're not so hard to see heh [14:10:42] as crazy as it sounds to me, I think maybe you're onto something [14:11:11] apparently ttl+grace+keep sets when the expiry thread wants to reap an object (but it could potentially fail to do so in a timely fashion) [14:11:51] during does look at just raw TTL for some reason, but it's not clear why yet. It may just be sorting through through vary-slots of an object.... [14:12:20] but in general, I suspect if an object exists (expiry thread hasn't yet killed it), and it succeeds at lookup, one way or another it ends up in vcl_hit [14:12:36] and then yes, it's up to vcl_hit to decide anything about the distinctions between ttl+grace+keep... [14:13:03] and it's the builtin vcl that brings a glimpse of sanity back onto the picture [14:13:28] but I still feel like I don't fully understand how two magical things happen regardless: [14:14:46] 1) We'd return (deliver) whether it was in normal TTL or out-of-TTL but in-grace, but somehow varnishd still needs to internally decide to trigger a bgfetch in the grace case (I'm guessing this might be the ttl check above? just looking at whether we did a hit->deliver on an expired primary-TTL at all) [14:15:58] 2) With the builtin VCL for sanity, an object past grace but out in keep time, we "return (miss)" in vcl_hit. How does it realize that it isn't a pure miss and it has an expired object to attempt conditional fetch against? [14:16:54] either way, even not fully understanding the code for those cases [14:17:28] it seems the most likely truth there is that if vcl_hit just does "return (deliver)", all objects are logically in only one of two possible states [14:17:42] within their natural TTL, and thus normal real hits [14:18:10] or outside their natural TTL, in which case they're treated like grace-hits with bgfetch. (they could be well beyond grace into keep time, or even beyond keep time if expiry thread is slow) [14:19:15] but "fixing" this involves returning (miss) from vcl_hit in some conditions, and apparently doing that in any condition is going to trigger the still-unresolved-upstream 1799 stuff [14:20:22] or the other way to fix it, is to get rid of "keep" entirely in our VCL so that objects reap at the end of grace, and of course assume expiry thread doesn't normally fall behind enough to matter. [14:21:05] but probably aprt of our storage/expiry-scaling magic since varnish4 has been the use of keep to prevent the expiry of many objects until after the varnishd gets restarted anyways [14:21:23] without keep, we'd probably ahve a much higher object eviction rate for the expiry thread in the average case [14:22:08] (then again maybe that's a good thing. maybe if the expiry thread did more work to get rid of stale objects based on their timestamps, other threads fetching for clients wouldn't have to nuke to make sudden room so often) [14:26:24] it's worth trying anyways. maybe the keep-expiry-scaling thing works differently for different clusters. [14:26:45] I imagine on text, eliminating keep might help, but on upload it might hurt [14:27:29] (because text doesn't naturally use up all storage on live objects, so expiring things faster avoids nukes. but upload has more objects than storage can fit, so even with faster natural expiries it's still going to nuke all the time, but now the expiry thread is randomly-busier too) [17:38:50] 10netops, 10Operations: Config discrepencies on network devices - https://phabricator.wikimedia.org/T189588#4101468 (10ayounsi) [18:36:25] 10netops, 10Operations, 10ops-eqiad: Rack/cable/configure asw2-a-eqiad switch stack - https://phabricator.wikimedia.org/T187960#4101662 (10Cmjohnson) Not sure where to put this but I removed to switch ports today - ge-2/0/0 { - description db1001; - disable; - } - ge-2/0/10 { - descr... [18:40:05] 10netops, 10Operations, 10ops-eqiad: Rack/cable/configure asw2-a-eqiad switch stack - https://phabricator.wikimedia.org/T187960#4101681 (10Cmjohnson) Also removed this - ge-2/0/15 { - description db1016; - disable; - } [19:24:42] so [19:24:54] this passes: https://phabricator.wikimedia.org/P6935 [19:25:31] if you comment out return(deliver) on line 15, hence using the builtin vcl, the test fails [19:27:26] so essentially keep behaves like grace if you return(deliver) in vcl_hit [19:45:45] the difference being that return(deliver) results in a bgfetch, while return(fetch) in a fetch [19:51:19] what a mess [19:51:37] it does not really behave like grace [19:51:57] it behaves like grace only for the first client request resulting in a bgfetch [19:59:11] so to be clear, this passes: https://phabricator.wikimedia.org/P6936 [20:00:43] if you instead set keep to 5s and grace to 0s in vcl_backend_response, it fails [20:34:22] (note that in both cases the bgfetch fails because the origin server is not responding anymore, it only responds to one request) [21:42:59] 10Traffic, 10Analytics, 10New-Readers, 10Operations, and 2 others: Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#4102458 (10Nuria) [21:52:48] 10Traffic, 10Analytics, 10New-Readers, 10Operations, and 2 others: Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#4102475 (10Nuria) >If we can get updates from them, can we repair the date/update things on our side retroactively? I Not likely as we would need the original re... [22:26:00] 10Traffic, 10Analytics, 10New-Readers, 10Operations, and 2 others: Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#4102622 (10atgo) >>! In T187014#4102475, @Nuria wrote: >>If we can get updates from them, can we repair the date/update things on our side retroactively? I > Ano... [22:59:42] 10Traffic, 10Analytics, 10New-Readers, 10Operations, and 2 others: Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#4102802 (10Nuria) @atgo I have checked data for january and march for Opera and i just see us receiving IP addresses of Opera's Proxy endpoints instead of Ip adr... [23:56:13] 10netops, 10Operations: Config discrepencies on network devices - https://phabricator.wikimedia.org/T189588#4102889 (10ayounsi)