[00:03:43] 10Traffic, 10Varnish, 06Discovery, 06Operations, 06Maps (Tilerator): Tilerator should purge Varnish cache - https://phabricator.wikimedia.org/T109776#1558879 (10Pnorman) I'd rather see `max-age` significantly reduced and `stale-while-revalidate` set to the current `max-age` value. This avoids the need to... [11:06:40] 10netops, 06Operations: Audit and cleanup border-in ACL on core routers - https://phabricator.wikimedia.org/T160055#3087244 (10mark) [12:44:47] ema, bblack: https://phabricator.wikimedia.org/T154934#3087501 ok to also upgrade cp1008 next? [12:58:01] ema: We still have some wrong tiles on maps after T159631. If I read the docs correctly, I should be able to invalidate with https://phabricator.wikimedia.org/P5035 [12:58:01] T159631: Tasmania is covered with water at z10+ - https://phabricator.wikimedia.org/T159631 [12:58:23] ema: could you have a look and confirm that this will probably not break things in horrible ways? [13:02:30] gehel: qq about line 3 - would it be better to do esams and ulsfo separated as you are doing for eqiad and codfw in line 1/2? [13:02:40] (not sure if needed, trying to review :) [13:03:09] our doc propose to do both at the same time : https://wikitech.wikimedia.org/wiki/Varnish#Execute_a_ban_on_a_cluster [13:03:22] elukey: thanks for looking! [13:03:30] well you have -b 1 [13:03:38] doesn't matter sorry :) [13:04:09] no problem! you're taking the time to review, I'm not going to complain! [13:04:58] gehel: line 3 - 'req.url ' \" ? [13:05:43] typo, fixed. Thanks! Good eyes! [13:06:32] lgtm! Not sure about the regex though, better wait a bit for master ema :) [13:07:26] maybee it could be good to add req.http.host if possible [13:08:06] even if /osm-intl is probably not belonging to anything else :) [13:08:13] for the regex, I'm trying to match tiles in Tasmania, like https://maps.wikimedia.org/osm-intl/10/928/642.png (osm-intl/{zoom}/{y}/{x}.png) [13:08:24] ah ok nice [13:08:52] the cache-maps cluster only serves maps, so we should probably be good [13:09:05] true :) [13:09:11] and actually, I should probably ban not only png but all formats... [13:09:48] the regex seems sane [13:10:12] maybe you could target a single Varnish maps backend first [13:10:16] say in ulsfo [13:10:21] double check that everything is good [13:10:24] and then proceed [13:10:35] * gehel does not like touching varnish... it looks scary [13:10:45] elukey: sounds like a good plan... [13:11:11] worst that can happen is that you'll execute locally sudo -i depool [13:11:34] but it would be good to see metrics for one host about the effect of the purge [13:12:31] we still did a full cache wipe not too long ago and even with a cold cache we were able to not have issues. We have a bit more traffic now, but not all that much [13:12:52] and at zoom level 10, I'm only invalidating a few 100's tiles [13:14:49] oh yes yes I am throwing out all the paranoid things that I can think of [13:15:09] anyhow, if it is not urgent ema will be probably back in a few [13:15:26] good! That's the goal of asking for a review! And I'm making sure that all the checkboxes are checked... [13:15:46] it isn't that urgent [13:33:57] moritzm: ok to upgrade cp1008 [13:36:26] k [13:37:47] gehel: seems reasonable, have you tried that out on a single host first? [13:38:09] ema: not yet [13:38:12] oh I see that good old elukey suggested that already :) [13:39:47] ema: other question, we seem to have a TTL limit of 1 day on maps cache. That's to total limit? It is not backend 1 day + frontend 1 day ? [13:40:39] I'm asking because we still have some wrong tiles in cache that should have been expired already tuesday evening... [13:43:20] ok, I'll start this ban, we'll see... [13:45:22] damn, invalid backslash sequence... [13:45:26] * gehel needs new glasses [13:50:07] ok, found it [13:59:03] ok, there is something wrong (probably with my regex) [13:59:21] gehel: things not banned? [13:59:22] https://maps.wikimedia.org/osm-intl/10/929/644.png still return "Age: 74730" [14:00:13] I did test on cp4020 and I did see X-Cache changing from a hit to a miss for cp4020... [14:00:41] gehel: so you tried manually on one host, it worked, then you did on the whole maps but nothing? [14:01:10] I would not say nothing, but at least the URL above does not seem to have been banned [14:02:57] gehel: on cp4020 I only see the ban on the frontend [14:03:23] journalctl -u varnish --since=today|grep ban [14:03:32] journalctl -u varnish-frontend --since=today|grep ban [14:03:35] my bad, mixed up the front and back... [14:03:55] starting again... [14:04:00] * gehel is learning today... [14:05:51] echo ban.list | varnishadm -n frontend [14:05:56] to see the current bans ^ [14:06:39] see? The master explained :) [14:08:19] ok, this time it seems to have worked. Thanks for the help! [14:08:46] Now I need to understand why we were still generating wrong tiles 24h ago (or why they were cached for longer than 24h) [14:09:37] gehel: do you have an example we can look at? [14:10:06] at lower zoom level probably [14:10:35] https://maps.wikimedia.org/osm-intl/11/1859/1285.png [14:10:59] Age: 75397 [14:11:52] the tile currently generated by kartotherian has land instead of water [14:13:09] kartotherian *should* have been rendering that tile correctly since ~1am UTC on March 7 [14:13:29] or at least since 7am UTC March 7 [14:14:36] kartotherian codfw is still rendering the tiles wrong (with water), but we should not have any traffic served from there, right? [14:17:58] kartotherian is not setting any cache-control headers, except ETag (which is not consistent between the nodes, we should most probably drop it) [14:18:39] <_joe_> lol [14:18:54] <_joe_> we don't set headers, but when we set them, they're wrong :P [14:18:54] and a last modified at epoch(0), mostly useless (and wrong) as well [14:19:25] where would be the fun if everything just followed the expected path? [14:19:30] :/ [14:19:48] <_joe_> gehel: well the application needs to be fixed [14:19:57] <_joe_> pretty simple! [14:20:25] agreed, but that still does not explain why my wrong tiles are still in cache... [14:41:33] <_joe_> oh sure, you weren't expecting me to make a constructive contribution, did you? [14:50:41] _joe_: you had me worried for a minute :) [17:02:52] 10netops, 06Operations, 10ops-codfw: codfw: oresrdb2002 switch port configuration - https://phabricator.wikimedia.org/T160087#3088288 (10Papaul) [17:15:31] bblack: Hi! I'd need some help on cache:misc if you have time [17:15:43] otherwise I'll follow up with ema tomorrow :) [17:16:28] elukey: depends what kind of help? [17:17:03] if this is the "can we source hash clients for wdqs" thing that I skimmed the other day, the answer is no, we're probably not going to support that [17:17:17] (at the cache layer) [17:17:45] (at this point in time in the broad sense. meaning maybe we will a yea ror two from now when we've significantly changed lots of things, but who knows!) [17:18:08] right now, it's just not a reasonable thing, it causes too many edge cases for too many other efforts [17:21:51] bblack: no idea about wdqs but it should be simpler :) [17:22:04] ok! [17:22:19] so I removed the piwik backend probes with https://gerrit.wikimedia.org/r/#/c/342007/ [17:22:34] from cache misc but I still see them on bohrium, and tcpdumps confirms.. [17:22:43] (from hosts in which puppet already ran) [17:23:02] hmmm [17:23:13] so it might be needed a restart of the backends? I hoped that it was only a VCL restart like when we added them.. [17:23:17] well let's dissect this a bit... [17:23:36] first, they should only be coming from a small handful of cache backends. like, 8 of them? 4 in eqiad 4 in codfw? [17:23:46] I don't think we probe applayer from the edge pops right? [17:24:01] but I could be wrong [17:24:32] ah ok, we do probe them from the edge sites too (pointlessly), I can see that now [17:24:36] this was what I thought as well, but then I saw probes from uslfo! [17:24:40] still, 16 hosts doing what should be a lightweight probe query is the problem? [17:25:00] e587660c-1f1a-4720-83a7-031466db51a5.be_bohrium_eqiad_wmnet probe Healthy (no probe) [17:25:09] ^ that's cp4001 in ulsfo, not currently probing [17:25:39] but really, if the probes are failing requests are failing too. it seems a little backwards to say "stop looking at failure rates, because it's failing and we want to use the failing thing anyways" [17:26:31] my theory is that it exacerbate the issue, but I could be wrong, and since the probe was originally meant to give us more info about the only backend I thought to try [17:27:05] well there's lots of things to look at here [17:27:19] but maybe the first point is: is the probe request URI actually lightweight? (it should be) [17:27:36] (and if so, it shouldn't be the cause or exacerbater) [17:27:36] re: cp4001 [17:27:37] 17:27:05.062795 IP cp4001.ulsfo.wmnet.8475 > bohrium.eqiad.wmnet.http: Flags [P.], seq 1:74, ack 1, win 58, options [nop,nop,TS val 672769348 ecr 3010942158], length 73: HTTP: GET /piwik.php HTTP/1.1 [17:27:54] are you sure that's not a real request? [17:28:19] well I see the IPs in the bohrium apache logs, I can check for cp4001 to be sure [17:28:50] "the IPs?" [17:29:20] I get that that tcpdump output means cp4001 requested /piwik.php, but was it a probe, or was it forwarded user traffic? [17:29:50] all cache_misc globally (checked with salt) at least claim in varnishadm to know of know probe for bohrium, as expected. [17:30:00] if an old VCL was stuck still probing, we'd see it there [17:30:54] the IPs of the cp misc hosts in the apache logs as requesters [17:31:02] and I can still see them [17:31:24] they are the only ones using "http://bohrium.eqiad.wmnet/piwik.php" and not the piwik domain [17:32:07] hmmm [17:32:13] ok [17:32:22] (also, duh, cp4001 should never route requests directly anyways) [17:32:43] I think it only probes the backend, could it be possible? [17:32:59] yeah the probe exists just because the backends are defined there pointlessly [17:33:09] but you deconfigured the probe, and varnish claims it reloaded VCL and is now running no probe... [17:34:28] so, clearly, varnish must have an issue where it keeps probing afterwards [17:34:55] bblack: would it be possible to try a backend restart in misc to see if it goes away? [17:35:09] well we can, but I still really question whether we should be removing probes here [17:35:23] it was meant to be an improvement, and an example we were going to use elsewhere too to get better health state [17:35:49] why is the probe URL causing heavy load? it should be something light just to confirm basic connectivity? [17:35:56] and the req rate can't be that high from 16 varnishes can it? [17:36:39] <_joe_> bblack: it's piwik, it's a kind of magic(TM) [17:36:44] it just seems strange to me that overload from probes is what's taking a service down [17:37:01] and removing the probes somehow makes life better for real clients? [17:37:04] I think it is a bit more subtle, but I could be wrong (this is why I want to test) [17:37:32] there are no 50X on bohrium, and after inspecting failed probes via varnishlog it seems that they timeout [17:37:50] e587660c-1f1a-4720-83a7-031466db51a5.be_bohrium_eqiad_wmnet probe Healthy (no probe) [17:37:54] err wrong paste [17:37:58] https://www.varnish-cache.org/trac/ticket/1061 [17:38:30] elukey: yeah, but why are probes timing out? that's still a valid "failure" from any perspective upstream of bohrium [17:39:53] bblack: didn't find a good explanation from the logs itself (apache on bohrium and varnishlog) but I suspect that during high load the bohrium apache threads gets to the point in which they queue probes for more than necessary, causing Varnish to timeout.. [17:40:10] and blocking traffic up to the point in which bohrium passes the probes [17:40:23] I saw a lot of them coming from all the hosts (16) at constant rate [17:40:35] ok you have to break that down a bit for me, since I have no idea what half of this does [17:40:44] "during high load" = during high real-client load? [17:41:25] in any case, I'm looking at the VCL staleness thing [17:41:37] it has cold VCLs from boot time listed, and 1x warm/busy and the active one with no probe [17:41:44] sure I am going to try to explain :) [17:42:07] I consider high load when apache workers are growing up too much - https://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Miscellaneous%20eqiad&h=bohrium.eqiad.wmnet&r=hour&z=default&jr=&js=&st=1489081251&v=250&m=ap_busy_workers&vl=threads&ti=Busy%20Threads&z=large [17:42:15] available auto/busy 8 root:12c25f88-4e7b-4b3a-98ed-52a629200347 [17:42:17] it is a ganeti host with 8 virtual cores [17:42:18] active auto/warm 34 e587660c-1f1a-4720-83a7-031466db51a5 [17:43:12] and the set up is apache with prefork and mod_php.. [17:43:31] I have scheduled tasks for next quarter to refactor puppet/apache/etc.. [17:48:46] <_joe_> elukey: it's LOLPHP then [17:49:22] the service definitely needs some wikilove :) [17:49:35] <_joe_> you'll get plenty for me [17:49:39] <_joe_> *from [17:50:17] Did I miss the tags by any chance? :D [17:50:47] well anyways, I think the most-probable scenario for the lingering probes is that "busy" old VCL [17:51:10] backend.list is only showing the no-probe stuff for the "current" VCL, but the previous VCL is still busy too, and it holds the old probes [17:51:31] because there's some lingering long-term client connections there that haven't closed, for some cache_misc service [17:52:47] I tried discarding the busy one, but it still shows "discarded/busy". did the cp4001 probe stop? [17:57:47] checking [17:58:27] seems to still hit bohrium [17:58:52] cp4001 does, right? [17:58:58] in which case, discarding busy does nothing [17:58:58] yep yep [17:59:29] I can re-check tomorrow morning if things have changed or not [17:59:36] I mean, it is not really super urgent :) [18:00:00] let me see how many there are that are like that [18:01:55] yeah apparently the auto/busy thing is quite common in cache_misc, probably because we have some apps that use very persistent conns [18:02:33] cp1051 is particularly bad: [18:02:35] root@cp1051:~# varnishadm vcl.list [18:02:35] available auto/cold 0 boot [18:02:35] available auto/cold 0 0953b788-8f2b-4714-954b-c3496fc98ec7 [18:02:35] available auto/cold 0 root:e3f812cd-dc6b-4eb3-824f-95ef6ef53d81 [18:02:38] available auto/busy 4 root:1c270f75-90cf-4329-b1ec-8d8261a4795a [18:02:41] available auto/busy 2 e24e881f-99a7-41ad-aa8d-cfb0c0b9b073 [18:02:44] available auto/busy 8 038e7cde-017c-46e4-ae7e-4ca719d985bc [18:02:46] available auto/cold 0 root:90ea43e9-9a14-4098-bd63-5bf2d7e02ce1 [18:02:49] available auto/busy 30 root:5017bd1d-e2e4-4a45-9dfa-65cb1a0e4f87 [18:02:52] active auto/warm 180 befd352f-e914-442b-992c-5ebcdb5c6ed8 [18:02:55] it has 4x old VCLs still busy with old conns [18:03:07] I'll try restarting cp4001 [18:04:41] elukey: cp4001 probes gone now? [18:04:45] checking [18:05:11] can't see them anymore :) [18:06:36] ok [18:06:48] I kicked off a salt to slowly restart all the others with busy threads [18:06:58] I think it will take something on the order of ~10-15 mins [18:07:34] thanks a lot, it is perfect [18:07:56] so I am going to report back about the result of my experiment [18:08:53] (brb in a bit, commuting) [18:12:18] 10netops, 06Operations, 10ops-codfw: codfw: oresrdb2002 switch port configuration - https://phabricator.wikimedia.org/T160087#3088549 (10RobH) 05Open>03Resolved switch port updated and committed, resolving task ``` robh@asw-c-codfw# show | compare [edit interfaces interface-range vlan-private1-c-codfw... [18:20:01] elukey: they're all done [18:26:31] cp1008 upgraded to 4.9.13 [18:35:51] bblack: confirmed, probes gone! thanks! [18:45:33] the whole thing still seems off to me, fwiw, but that's neither here nor there [18:45:48] probes shouldn't be more traffic/load than users, or the reason users are failing [18:59:04] bblack: you are completely right, in fact this is an experiment to see if there is any variation in the piwik 503s. The backend is definitely not working fine and needs to be refactored and improved, it currently have a lot of limitations (one above all, apache preforking model with mod_php) [19:05:56] bblack, ema : my understanding is that we have a cap on TTL at 1 day for cache-maps (https://github.com/wikimedia/puppet/blob/production/modules/role/manifests/cache/maps.pp#L48) is that correct ? [19:06:57] I see tiles with wrong content on maps, which should have been fixed much earlier than 24h ago (https://maps.wikimedia.org/osm-intl/11/1860/1298.png) [19:07:56] I can't find an explanation... [19:10:44] gehel: technically, the applayer sets the TTL [19:10:51] varnish caps it, but only per-layer, not globally [19:11:24] ok, so with 3 layers of varnish, we might get 72h of cache? [19:12:05] it doesn't really work like that usually, but that's the idea for overall cap. it varies by location and situation how many layers there are [19:12:19] for ulsfo and maps I think it would be 4 layers presently [19:12:28] and no, the app layer is not setting any caching header at this point, which needs to be fixed [19:12:45] heh [19:13:08] Ok, then that explains what I'm seeing. And the fix is obviously to expose proper caching headers... [19:13:21] gehel: can you get a dump of headers of your bad request? [19:13:31] sure [19:13:38] because everyone can fetch that url differently, but the headers of your actual bad result are telling [19:13:48] (it may be a different image for me) [19:14:33] which headers are you interested in? [19:15:07] https://phabricator.wikimedia.org/P5038 [19:15:12] X-Cache and Age mostly [19:15:27] < Age: 21365 [19:15:27] < X-Cache: cp1047 miss, cp3003 miss, cp3003 hit/71 [19:15:41] so it was fetched from maps applayer less than half a day ago [19:15:54] sounds like ttl caps are not the issue [19:16:24] (~6h ago) [19:16:39] the part that I dont understand is that the maps issue was fixed early Tuesday morning UTC [19:16:48] are you sure? :) [19:17:12] try requesting directly from all of the karto frontends [19:17:14] no I'm not, that why I'm wondering soo much [19:17:39] I did, as far as I can see, that same tile is looking good on all the eqiad nodes [19:17:40] (varnishes requests would roundrobin them all, could be just 1/N still giving old data?) [19:17:57] of course, I don't have a time machine to go see how it looked 6 hours ago [19:18:03] right [19:18:14] but I think earlier today you had someone ban it and it still came back bad? [19:19:13] I banned the a specific zoom level (10) where there was quite a few bad tiles. They did no re-apear [19:19:53] the tile linked above is not part of that ban. As this is a fairly minor issue, I wanted to dig a bit more first.. [19:21:27] the issue is still present on codfw, but I see no reason for any traffic to end up on the codfw maps servers... [19:22:25] right, the applayer route is to eqiad [19:22:46] how do I tell good and bad apart? [19:23:05] for https://maps.wikimedia.org/osm-intl/11/1860/1298.png [19:23:52] if on the left you see a road in the middle of water (blue), it is bad, if you see a road on land (grey / yellow) it is good [19:27:12] I get different results from all 3 presently (the cached tile on varnish, codfw karto, and eqiad carto) [19:28:30] so when you banned zoom level 10, that was for this same reason right? that the bad ones should've expired by now but hadn't? [19:28:57] yep, exactly [19:29:06] did you ever figure out why? [19:29:12] (and did it work?) [19:29:36] it worked (or at least I dont see any bad tiles at zoom level 10) [19:29:57] ok [19:30:03] but going back to the time of th eban in backscroll [19:30:04] 13:59 < gehel> ok, there is something wrong (probably with my regex) [19:30:04] 13:59 < elukey> gehel: things not banned? [19:30:05] 13:59 < gehel> https://maps.wikimedia.org/osm-intl/10/929/644.png still return "Age: 74730" [19:30:21] that was because bad ban execution or whatever, which was fixed after [19:30:28] but I don't know why we still had tiles at that time. Each bad tile that I checked against the applayer was rendered correctly at the applayer [19:30:39] but still, at the time you were saying "should've been fixed >24h ago", yet the bad cache object had <1d Age header [19:31:18] Age is cache-transitive, and since maps isn't providing an initial age it's invented at the backend-most varnish [19:31:22] the issue quoted was me purging only the frontend. Which I corrected afterward [19:31:32] so it does accureately represent "this object was fetched from kartotherian X seconds ago" [19:32:10] my point is even though the ban wasn't right, the evidence at the time was that the bad object was <24h old, which seems to not jive with this all being fixed for good at karto level >1d ago [19:32:34] exactly... [19:33:47] the most probable is that things were not fixed at kartotherian level when I thing they were, but I can't find any evidence of that. Which makes me wonder if we have another transient problem at the app level that hides every time I'm looking at it [19:34:19] but I dont find direct evidence of that, except those bad tiles in varnish... [19:34:35] I'd say the age values are evidence [19:37:04] yes, but it does not correlate with any other observation... [19:37:42] :) [19:38:01] I went over all of tasmania this morning at zoom level 11 on each of the server, I did not see any bad tile... [19:38:02] it doesn't have to if varnish reqs are more common than your test reqs and its intermittent [19:38:27] yeah, all point to intermitent ... I don't like that... [19:38:49] * gehel will try to find a way to dig more into this... [19:38:50] or, it could point to basic http issues [19:38:58] you mentioned earlier than etags were handled wrong? [19:39:19] yes, etags are not consistent between nodes [19:39:28] also: [19:39:30] < Last-Modified: Thu, 01 Jan 1970 00:00:00 GMT [19:39:37] ^ all tiles seem to return that [19:39:39] yes, also noted :( [19:39:50] that could be the entirety of the problem here [19:40:06] can you strip that at the front edge of karto, since it's completely wrong information? (the LM header) [19:40:24] or change its IMS behavior? [19:40:48] T108435 [19:40:48] T108435: Add proper expiry headers to kartotherian's responses - https://phabricator.wikimedia.org/T108435 [19:41:17] curl -vH 'Host: maps.wikimedia.org' -H 'If-Modified-Since: Thu, 01 Jan 1970 00:00:00 GMT' http://maps1004.eqiad.wmnet:6533/osm-intl/11/1860/1298.png [19:41:24] < HTTP/1.1 304 Not Modified [19:41:38] so this is a pretty valid explanation of what's happening: [19:41:39] I need to understand how those node services manage headers and fix them [19:41:54] doesn't it have an apache or something in front, or is it direct-to-nodejs? [19:42:04] direct to node [19:42:07] anyways, on to explanation: [19:42:28] 1) maps1004 sends "bad" tile to varnish, with Last-Modified: Thu, 01 Jan 1970 00:00:00 GMT [19:42:48] 2) maps1004 data fixed [19:43:24] 3) bad tile expires from varnish cache, varnish refreshes with "If-Modified-Since: Thu, 01 Jan 1970 00:00:00 GMT", but karto says "304 not modified" [19:43:43] 4) Thus, varnish refreshes the age on the existing object (which gives us our low age) and keeps seving bad content [19:44:25] when things expire on TTL in varnish, the object is (usually, optimization) still in storage under a longer "keep" TTL, and then used for IMS conditions like these, which when the object didn't actually change saves the BW of re-transfering an object we already have the contents of [19:44:27] Ow, that sound very plausible! I don't know why, but I expected varnish to not do conditional get [19:44:38] varnish is all about conditional get, it's a big win [19:44:48] makes a lot of sense! [19:45:03] if your server lies about the fact that the object hasn't changed since 1970 forever, even though it's constnantly changing, that will screw things up [19:45:28] Time for dinner here... have to run. [19:45:43] Thanks a lot for all the thinking! Enlightning as always! [19:47:25] ideally LM should actually be correct (when tile was updated), and then we get correct behavior and good efficiency [19:47:41] but lacking the ability to do that, a quick hack would be to just suppress the output LM header altogether [20:25:03] Hey all, I have what may be a dumb MW/Varnish question... [20:25:42] I may have been doing something wrong for years now. I have four load-balanced servers for my wikis, each running Varnish on 80 with Apache as its backend on 8080. [20:26:31] In my LocalSettings.php for my wikis, however, I have the IP addresses of all four web servers so that purges from one would go to all of the servers. Is that right or should each server only send purges to its own Varnish? [20:29:23] justinl_: assuming it's the same content on all four servers (as opposed to, say, a sharded setup where each of the 4 services separate wikis), then the purges should be going to all, yes. [20:31:01] the idea being that whlie the edit of /wiki/Foo may have happened on server1, over the past [cache TTL] /wiki/Foo may have been read by different users on all 4x of your varnish frontends, and therefore needs purging from them all to update everywhere. [20:33:55] bblack: Ok, thanks. It is the same content, shared amongst the servers via NFS. This thought just struck me last night as I'm considering architectural requirements for migrating my environment to AWS. [20:34:54] If I move to a couple of dedicated Varnish servers in front of possibly load-balanced EC2 web servers, for sure I'd want all of the Varnish servers in the EC2 serers' $wgSquidServers lists. [20:35:14] justinl_: yes! :) [20:35:47] That's another catch, though, is that if I were to have ELB between Varnish and web servers, the web servers would still need to know about the Varnish IPs for the purges. [20:40:12] yeah, possibly [20:40:29] you could set up something to advertise them to a registry or something like that, though [20:41:30] That's the catch with trying to figure out autoscaling of Apache and maybe Varnish for a MediaWiki environment is automated management of the IPs. I use SaltStack for configuration management, so I'd probably have to set up some reactors to update LocalSettings.php when hosts are dynamically added or removed. [20:41:30] AWS probably even has a KV-store you could use for such things as a service [20:42:08] Maybe Elasticache? I'll be using it to replace Memcached, unless Aurora is fast enough to obviate the need for such a cache. [20:42:19] yeah that's one possibility [20:42:27] or something etcd-like [20:42:38] Salt has Salt Mine though, so that could work if there's not an ideal AWS service. [20:47:41] bblack: hi! do you have a minute? wanted to talk about https://phabricator.wikimedia.org/T159574 [22:17:41] 10Traffic, 06Operations, 06Performance-Team, 13Patch-For-Review: Segment Navigation Timing data by continent - https://phabricator.wikimedia.org/T128709#3089607 (10Krinkle) [22:37:40] SMalyshev: the short answer is we can't do client affinity for you in varnish [22:49:46] bblack: what about IP affinity in LVS? also won't work? [22:50:59] or in fact any sort of affinity that ensures requests from one host within short time go to one server? [22:56:34] SMalyshev: LVS can do affinity for the direct clients of LVS, but wdqs's clients are varnish caches. varnish can do affinity for clients->backends when it has direct knowledge of the backend cluster nodes instead of through an abstraction like LVS, but we currently use that abstraction. [22:57:43] we made a decision a while back that at least for the present, we're not supporting client affinity for applayer services. it simplifies a bunch of things architecturally and virtually no service actually needed it. It gives us the LVS abstraction beneath varnish as a constant. [22:58:56] I really do suspect that will be re-visited at some later date, once we're past some other future architectural changes (and varnish is largely gone and internal connections are almost always TLS), but those are far-off plans [23:00:26] (at the time we made that call, I think rcstream and maybe... was it logstash? were the only ones making use of client affinity - 2x misc services, and none of our major services. We moved those to using a single node for traffic or other such solution) [23:04:01] bblack: so basically recommended solution for now is single-node? [23:06:47] if you can't fix the fundamentals some other way, yeah, use a single node for traffic (with the option to keep others hot and ready for manual failover) [23:07:53] usually with this issue, the fundamental problem is per-user state, in which case the better answer is keep using multiple nodes and track necessary per-user state in the applayer itself instead of hoping the balancers do it for you. [23:08:19] but in your case, I think this is more about consistent ordering for multiple queries that iterate a dataset via index [23:11:34] the other option is you build your own client affinity proxy layer on top of wdqs, inside the same nodes [23:12:53] conceptually the idea would be that all your nodes have apache or nginx at the front (as what varnish->LVS routes requests into randomly), and those apache/nginx instances hash on the received X-Client-IP header to choose a consistent applayer backend (which might be the same the request arrived at apache/nginx on, or one of the others) [23:13:27] it's what we would do in LVS, if LVS had visibility into the HTTP headers to make that decision [23:14:53] but still, you're solving the "inconsistent index" problem there by making it consistent for a given client IP, which is not the same as solving it generically [23:15:19] when two separate clients query a given index position in the data, they'll still see different results (with any client affinity solution here), which might eventually matter to some use-case