[05:16:21] <wikibugs>	 10Traffic, 10Operations: ATS-BE Lua mitigations for cacheable responses w/ Set-Cookie seemingly not working - https://phabricator.wikimedia.org/T264378 (10Tgr) Thanks for the thorough investigation!  So, should we try to prevent MediaWiki from emitting cookies in 304 responses, or it doesn't really matter? IIU...
[11:10:06] <wikibugs>	 10Traffic, 10Operations: Package and deploy varnish 6.0.7 - https://phabricator.wikimedia.org/T268736 (10ema)
[11:10:15] <wikibugs>	 10Traffic, 10Operations: Package and deploy varnish 6.0.7 - https://phabricator.wikimedia.org/T268736 (10ema) p:05Triage→03Medium
[11:30:38] <wikibugs>	 10Traffic, 10Operations: Package and deploy varnish 6.0.7 - https://phabricator.wikimedia.org/T268736 (10ema) I've tried building 6.0.7 on my workstation to double-check the changes between 6.0.6 and 6.0.7 with debdiff. When running the tests, `./bin/varnishtest/tests/m00035.vtc` fails with a segmentation faul...
[12:15:35] <wikibugs>	 10Traffic, 10Operations: ATS-BE Lua mitigations for cacheable responses w/ Set-Cookie seemingly not working - https://phabricator.wikimedia.org/T264378 (10ema) >>! In T264378#6647455, @Tgr wrote: > So, should we try to prevent MediaWiki from emitting cookies in 304 responses, or it doesn't really matter? IIUC...
[12:32:43] <wikibugs>	 10Traffic, 10Operations, 10Performance-Team, 10Wikimedia-Incident: 15% response start regression as of 2019-11-11 (Varnish->ATS) - https://phabricator.wikimedia.org/T238494 (10Gilles) 05Open→03Resolved Thank you for the detailed update!
[14:01:59] <wikibugs>	 10Traffic, 10Operations: Package and deploy varnish 6.0.7 - https://phabricator.wikimedia.org/T268736 (10ema) >>! In T268736#6648179, @ema wrote: > I've tried building 6.0.7 on my workstation to double-check the changes between 6.0.6 and 6.0.7 with debdiff. When running the tests, `./bin/varnishtest/tests/m000...
[14:09:36] <wikibugs>	 10Traffic, 10Operations: ATS-BE Lua mitigations for cacheable responses w/ Set-Cookie seemingly not working - https://phabricator.wikimedia.org/T264378 (10BBlack) I think especially if you start considering how `Vary: Cookie` works in all the above (both for MW on the related 200 and 304 outputs, and in the ca...
[14:11:04] <wikibugs>	 10Traffic, 10Operations: ATS-BE Lua mitigations for cacheable responses w/ Set-Cookie seemingly not working - https://phabricator.wikimedia.org/T264378 (10BBlack) (and to throw another dimension into the matrix of possibilities above - also whether the client is sending a session cookie to Vary on in either or...
[14:31:25] <ema>	 looking at the 6.0.6 / 6.0.7 debdiff:
[14:31:25] <ema>	 +.SS HTTP response status
[14:31:31] <ema>	 +With VCL code it is possible to use status codes in the form XXYZZ where the
[14:31:33] <ema>	 +overall value is lower than 65536 and the Y digit is between 1 and 9 included.
[14:31:36] <ema>	 +Only the YZZ part is sent to the client.
[14:35:59] <ema>	 whole story on https://github.com/varnishcache/varnish-cache/blob/6.0/doc/sphinx/reference/vcl_var.rst#http-response-status -- but basically this is to pass stuff around with synth/status
[14:36:25] <bblack>	 to avoid the stuff we do today where we set and match a reason string?
[14:36:44] <ema>	 that's my understanding!
[14:44:40] <bblack>	 while reading through some of the .7 links, I thought the discussion on CE+304 was interesting in a couple ways:
[14:44:43] <bblack>	 https://github.com/varnishcache/varnish-cache/issues/3169
[14:44:58] <bblack>	 because it ends up quoting that same header list being discussed in the set-cookie ticket
[14:45:29] <bblack>	 and because there's this tension in the discussion where the developers seem to want to assume that it's on the administrators to ensure all backends return logically-consistent responses
[14:45:54] <bblack>	 (which, from a varnish dev pov, would be a very nice property to be able to rely on, but isn't actually achievable for a complex large site in the real world)
[14:46:14] <ema>	 and it isn't something varnish does either, as mentioned in the ticket :)
[14:46:39] <ema>	 so in a multi-varnish setup you end up with varnish blaming the origin, which is varnish too 
[14:47:54] <bblack>	 the same sort of logical consequences may apply to other cases like Vary, too
[14:48:42] <bblack>	 what happens when a URI with live traffic undergoes a code change that causes it to begin emitting Vary for the first time, or to stop emitting a Vary it used to consistently emit, and there's no reasonable way for the administrator to ensure a point-in-time transition for all backends?
[14:49:17] <bblack>	 [for a given URI]
[14:49:57] <bblack>	 there's a clear way to handle that if you assume the inconsistency is unavoidable, but if you just assume the backend would never be so inconsistent, it can lead to Bugs
[15:45:53] <wikibugs>	 10Traffic, 10Operations, 10Technical-blog-posts: 2nd part of blog post series: the evolution of Wikimedia's Content Delivery Network - https://phabricator.wikimedia.org/T266857 (10ema) >>! In T266857#6645203, @srodlund wrote: > I decided on this image of a road: https://commons.wikimedia.org/wiki/File:On_the...
[15:49:23] <wikibugs>	 10Traffic, 10Maps, 10Operations, 10Product-Infrastructure-Team-Backlog, and 2 others: Javascript errors: Unable to add datalayers to map - https://phabricator.wikimedia.org/T267296 (10MSantos)
[15:55:28] <wikibugs>	 10Traffic, 10Maps, 10Operations, 10Product-Infrastructure-Team-Backlog, and 2 others: Javascript errors: Unable to add datalayers to map - https://phabricator.wikimedia.org/T267296 (10MSantos) >>! In T267296#6608020, @RolandUnger wrote: > It is not the first time that the map server is not working properly...
[19:37:41] <wikibugs>	 10netops, 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10mforns) @ayounsi New fields are in Druid (starting 2020-11-24T03:00:00) :] I've checked that all looks OK, but please do che...
[19:45:49] <wikibugs>	 10Traffic, 10Operations, 10Technical-blog-posts: 2nd part of blog post series: the evolution of Wikimedia's Content Delivery Network - https://phabricator.wikimedia.org/T266857 (10srodlund) Hi @ema I have published this: https://techblog.wikimedia.org/2020/11/25/wikimedias-cdn-the-road-to-ats/  Can you look...
[20:05:09] <wikibugs>	 10netops, 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10mforns) I guess this can be removed now, although it could stay if you wish! https://github.com/wikimedia/puppet/blob/produc...
[20:18:16] <wikibugs>	 10Wikimedia-Apache-configuration, 10Android-app-Bugs, 10Fundraising-Backlog, 10Operations, and 4 others: Deal with donatewiki Thank You page launching in apps - https://phabricator.wikimedia.org/T259312 (10CDanis) Sure -- SRE will get that deployed, but we're going to wait until Monday Nov 30th, in the int...
[20:49:11] <wikibugs>	 10Traffic, 10Maps, 10Operations, 10Product-Infrastructure-Team-Backlog, and 2 others: Javascript errors: Unable to add datalayers to map - https://phabricator.wikimedia.org/T267296 (10TheDJ) This proposed change seems reasonable, but if I’m not mistaken, that codepath is shared between the static map gener...