[10:56:30] <wikibugs>	 10Traffic, 10Wikimedia-Apache-configuration, 10Operations, 10VisualEditor: Visual Editor gets stuck opening article (net::ERR_SPDY_PROTOCOL_ERROR 200/Loading failed for the <script> with source ...) - https://phabricator.wikimedia.org/T213214 (10Schnark)
[16:46:30] <elukey>	 hi! I have a basic question about hue.wikimedia.org and static file caching. I forced httpd to serve them, and I realized that there is no cache-control policy set. So I wanted to add add Cache-control: private to the response, but I am wondering what is the best practice with Varnish
[16:50:03] <bblack>	 elukey: it depends on what you want to happen, really
[16:51:14] <bblack>	 elukey: currently hue doesn't have any special caching rules at the per-backend layer, so by default varnish will basically follow RFC-standard interpretation of related headers, and if there's no information to go on at all for a given object it will default to a 1-day cache TTL, I believe.
[16:52:08] <bblack>	 elukey: we can set "caching: pass" for all of hue to avoid all possible caching, and that could also be set for a specific subpath regex
[16:52:17] <elukey>	 bblack: so I want to see, as initial step, if my browser could stop sending get requests for the same content over and over (via chrome tools I keep seeing 200s for static content, not even an attempt to see if the have been modified)
[16:52:51] <elukey>	 in theory eventually I'd like Varnish to cache those if possible
[16:53:06] <bblack>	 what behavior do you actually want, and are you still just talking about some new static files in a certain subpath, or all of hue?
[16:53:49] <elukey>	 nono only /static/ basically.. I'd like to have them cached by the browser if possible, there are a ton of them and the page takes ages to load on average
[16:54:22] <bblack>	 so, CC:private tells shared caches not to cache it, but lets the browser cache it
[16:54:38] <bblack>	 but, you still need to tell the browser something about lifetime and how to refresh, etc
[16:55:01] <bblack>	 but if they're truly just shared static images or something, you should really let varnish cache them, too.
[16:55:12] <elukey>	 if possible then I'd love to
[16:55:53] <elukey>	 is it simply doable using cache-control: public or is there some VCL to change? (super ignorant question sorry)
[16:56:33] <bblack>	 so, as a good primer, look in https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/http-caching
[16:56:50] <bblack>	 and specifically at the chart under the heading: "Defining optimal Cache-Control policy"
[16:57:40] <bblack>	 the biggest problem isn't making it cacheable, it's deciding how long it can live in a cache and/or how often it's revalidated, and how to revalidate
[16:58:55] <bblack>	 so let's say for instance, these static files could be updated very frequently, and you really can't tolerate any delay in new content appearing to users.
[16:59:17] <bblack>	 then you might want to say that the browser must revalidate every time, but it doesn't have to transfer the data if it actually hasn't changed
[16:59:42] <elukey>	 with the Etag right?
[17:00:46] <bblack>	 by setting "CC: no-store, private, max-age=12345" and sending an ETag which actually changes if the content changes
[17:01:15] <elukey>	 ahhh okok
[17:01:39] <bblack>	 this doesn't allow shared (e.g. Varnish) caching, does allow the browser to keep a copy of what it has last seen for up to 12345 seconds, but the browser will re-check with the server every time (and get a 200 with new data, or a 304 Not Modified if it hasn't changed)
[17:02:00] <bblack>	 so you still pay the latency penalty of checking the server, but not the transfers if nothing actually changed
[17:02:10] <elukey>	 makes sense yes
[17:02:26] <elukey>	 I was mostly trying to figure out why the Etag alone wasn't enough
[17:02:33] <elukey>	 (in the response I mean)
[17:02:33] <bblack>	 or if you're ok with some staleness, you can avoid the round-trip as well and just let the browser keep reusing for a while
[17:02:53] <bblack>	 in which case you might as well let Varnish cache it too, and do something like:
[17:03:16] <bblack>	 CC: public, max-age=3600, s-maxage=7200
[17:03:40] <bblack>	 which says the browser has to refetch from varnish once an hour, and varnish can keep a copy for up to 2 hours
[17:04:01] <bblack>	 (and you can still use etag or LM to avoid transfer if nothing changes, really)
[17:04:20] <elukey>	 this seems a very good compromise
[17:05:36] <bblack>	 anyways, in whatever case, anytime varnish or browsers revalidate (query the server, but have a possibly-outdated copy already to compare against), there are two different mechanisms for revalidation, to ask the source "Hey I've got a copy that looks like X, is that ok or do you need to send me an updated one", to which the server can respond 200+content or "304 not modified" (use your cached one
[17:05:42] <bblack>	 )
[17:05:55] <bblack>	 there's the Etag/INM and the LM/IMS method
[17:06:06] <bblack>	 Etag/INM is based on a hash of the contents
[17:06:25] <bblack>	 server sends "ETag: <some hash string>" with the content anytime it sends the content
[17:06:55] <bblack>	 when client revalidates, it sends "If-None-Match: <hash string it has from last time>", and if the server's ETag hasn't changed you get a 304, otherwise you get 200 + new contents + new hash.
[17:07:13] <bblack>	 LM/IMS is similar but based on timestamps
[17:08:07] <bblack>	 server sends "Last-Modified 2019-01-01T12:34:56" (but I probably have the format wrong or whatever) with the content every time, based on its conception of when the content last changed (perhaps for a simple static webserver, filesystem mtime).
[17:08:25] <bblack>	 and clients sends "If-Modified-Since: <stamp I last saw>" to ask the server if there's a newer copy
[17:10:18] <elukey>	 just to understand - this behavior of revalidation is triggered only if 1) an appropriate CC header is returned by the server 2) the Etag header is added to the response
[17:11:19] <elukey>	 afaics the Etag is currently added by httpd by default, so in theory I only need an appropriate CC to leverage browser/varnish caching
[17:11:31] <elukey>	 that would help a lot in having a decent hue user experience
[17:11:32] <bblack>	 well, I would phrase it more like: if you care about cacheability (positively or negatively), you should set a CC header and define the behavior you want, and in addition to that, Etag and/or LM headers are a good idea for cacheable content, because they can optimize away re-transfers of the same content over and over if nothing changed.
[17:12:00] <bblack>	 if there's no CC header, I'm not 100% sure what the default behavior of the UA might be.  It might still try to revalidate if ETag/LM are present.
[17:12:10] <bblack>	 and that might vary by UA, I don't know
[17:12:23] <bblack>	 for Varnish, if there's no CC to go by, our config says go ahead and cache for a day by default
[17:13:05] <elukey>	 ack, thanks a lot for the detaled explanation!
[17:13:07] <bblack>	 building on top of all of this, there are other rules in play at the Varnish level, which might bite you in this case:
[17:14:07] <bblack>	 the most-critical one that may well apply here, is Varnish shouldn't cache objects from logged-in sessions, which are generally signalled by the use of Cookies
[17:15:06] <bblack>	 because there are many, many diverse/random services behind varnish, and we don't have deeply detailed applayer integration with many of them, it has to be conservative
[17:15:42] <bblack>	 so the rules are basically: if it sees the client request has an Authorization header (e.g. HTTP Basic Auth in use for logins), it won't cache anything.
[17:16:08] <bblack>	 and then also: if it sees any Cookie other than specially-whitelisted cookies that are known to not represent a login session, it also won't cache anything.
[17:16:45] <bblack>	 (the whitelist being some well-known analytics-related cookies from public broad use, our own analytics/geoip cookies, etc)
[17:17:37] <bblack>	 since hue is a service with logins, it probably appropriately trips the Cookie check and doesn't cache anything in Varnish.
[17:19:47] <elukey>	 yeah I can see the cookie header in the request of a static content
[17:22:05] <bblack>	 so, for high volume cases where we really care about caching, we would integrate tricky stuff at the VCL layer to work around that problem, but for various meta/internal tooling we generally don't bother.
[17:22:09] <bblack>	 win 36
[17:27:59] <elukey>	 makes sense yes
[20:10:22] <wikibugs>	 10Traffic, 10Operations, 10Parsoid, 10RESTBase, and 5 others: Consider stashing data-parsoid for VE - https://phabricator.wikimedia.org/T215956 (10mobrovac) A possible alternative is to differentiate VE's requests based on the UA header: VE [sets both `Api-User-Agent` and `User-Agent`](https://github.com/w...
[20:34:07] <wikibugs>	 10Traffic, 10Operations, 10Parsoid, 10RESTBase, and 5 others: Consider stashing data-parsoid for VE - https://phabricator.wikimedia.org/T215956 (10Pchelolo) I don't think complicating Varnish logic worths here, I don't see how UA is better than a query parameter regarding DDOs. We can mitigate the DDOS pos...
[20:42:49] <wikibugs>	 10Traffic, 10Operations, 10Parsoid, 10RESTBase, and 5 others: Consider stashing data-parsoid for VE - https://phabricator.wikimedia.org/T215956 (10mobrovac) >>! In T215956#4958249, @Pchelolo wrote: > I don't think complicating Varnish logic worths here, I don't see how UA is better than a query parameter r...
[21:05:21] <wikibugs>	 10Traffic, 10Maps, 10Operations, 10Reading-Infrastructure-Team-Backlog (Kanban): Decide on Cache-Control headers for map tiles - https://phabricator.wikimedia.org/T186732 (10Mholloway) @Jhernandez Looks like this one has a change waiting to roll out when the Stretch migration is complete, so per our discus...
[21:34:13] <wikibugs>	 10Traffic, 10Operations, 10Parsoid, 10RESTBase, and 5 others: Consider stashing data-parsoid for VE - https://phabricator.wikimedia.org/T215956 (10BBlack) Correct me if I'm wrong, but I would think all VE traffic would already be uncacheable at the Varnish level anyways, since it happens in the context of...