[11:02:58] 10Traffic, 10Analytics, 10Datasets-General-or-Unknown, 6Operations, 13Patch-For-Review: http://dumps.wikimedia.org should redirect to https:// - https://phabricator.wikimedia.org/T128587#2140256 (10ArielGlenn) I remember there was a discussion and I don't remember why it was wontfixed then. But we can s... [11:24:22] 10Traffic, 10Analytics, 10Datasets-General-or-Unknown, 6Operations, 13Patch-For-Review: http://dumps.wikimedia.org should redirect to https:// - https://phabricator.wikimedia.org/T128587#2079753 (10hashar) Some past tasks: * {T83675} * Declined: {T60292} The later had: > https://download.wikimedia.org/m... [12:22:21] 10Traffic, 6Operations, 10Reading-Web, 6Wikipedia-iOS-App-Product-Backlog, 13Patch-For-Review: Setup up Site Association file for Universal Link Support - https://phabricator.wikimedia.org/T111829#2140453 (10jcrespo) 5Resolved>3Open I am seeing half a million requests per minute to /.well-known/apple... [14:42:31] 7HTTPS, 10Traffic, 6Labs, 6Operations, and 2 others: Migrate tools.wmflabs.org to https only (and set HSTS) - https://phabricator.wikimedia.org/T102367#2140811 (10Andrew) [14:42:50] 7HTTPS, 10Traffic, 6Labs, 6Operations, and 2 others: Migrate tools.wmflabs.org to https only (and set HSTS) - https://phabricator.wikimedia.org/T102367#2140814 (10yuvipanda) [14:46:37] ema: I'm going to turn on the do_stream thing now I think. the upcoming SWAT doesn't look relatedly-risky. [14:47:04] bblack: alright [14:47:14] the main potential for any negative fallout that I see, is we might trip up bugs related to do_gzip. but then again the last few times I thought do_gzip was at fault, it really wasn't. [14:48:01] assuming nobody screams that something became broken and we have to revert, though, I imagine we'll see some nice perf gains, especially for e.g. POST traffic for edits and APIs and such, as large responses to them no longer have to fully buffer at 2-3 separate varnishd layers. [14:48:25] (and logged-in users' hit-for-pass objects for GETs in general) [14:48:41] nice [14:50:27] of course now that I merged the gerrit change, I realize I should probably protect against users setting that header, too [14:52:14] heh [14:52:31] (if only VCL supported setting variables) [14:54:22] I put them together in a single puppet-merge anyways [15:22:37] bblack: thanks for the long explanation in https://gerrit.wikimedia.org/r/#/c/276475, really useful [15:37:14] what the [15:37:16] Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate declaration: File[/usr/share/varnish/tests] is already declared in file /etc/puppet/modules/varnish/manifests/instance.pp:101; cannot redeclare at /etc/puppet/modules/varnish/manifests/instance.pp:101 on node puppet-ema.puppet.eqiad.wmflabs [15:38:16] probablyu from the 2x instances [15:38:26] ah! [15:38:36] varnish::instance gets invoked twice, so anything shared has to be in some include/require sort of thing [15:40:59] bblack: thanks [15:43:24] hi! https://phabricator.wikimedia.org/T130449 calls for a homepage for upload.wikimedia.org which I believe it used to be statically served by swift in eqiad (noticing only now since codfw switch) is the redirect something sensible to do at the varnish level? I considered doing it our custom rewrite middleware but sooner or later it'll go away [15:44:54] godog: yeah... if it were any other service I'd say "define that at the application layer", which is what ms-fe.svc.eqiad does today [15:45:08] but in this case, we know swift is minimal and not the best place to put things either [15:45:26] (and probably subject to change with thumbnailing plans, etc) [15:46:30] makes sense, thanks bblack ! I'll work on a patch to redirect somewhere, likely commons [15:47:35] godog: probably in templates/varnish/upload-frontend.inc.vcl.erb in sub cluster_fe_recv [15:48:06] and you'll have to put the other half of the redirect in sub cluster_fe_err_synth [15:48:38] e.g. see the example with status code 666 in text-frontend [15:49:27] hah, thanks for the pointer! yeah that helps [15:49:44] "hic sunt leones" [15:50:10] heh [15:50:35] that 666 example's whole thing with X-F-P and http:// is outdated/wrong too heh [15:51:08] but for upload purposes, just hardcode an https://foo/bar for the Location [15:56:00] indeed, I suppose 301 not 302 in this case? [16:12:53] yeah 301 [16:18:12] 10Traffic, 6Discovery, 10Kartotherian, 10Maps, and 2 others: Set up proper edge Varnish caching for maps cluster - https://phabricator.wikimedia.org/T109162#2141169 (10BBlack) [16:19:52] 10Traffic, 6Operations, 7Design, 13Patch-For-Review: Do something better than an "Unauthorized" error page at https://upload.wikimedia.org/ - https://phabricator.wikimedia.org/T130449#2136256 (10fgiunchedi) yeah what @Bawolff said! please see related patch to serve 301s out of varnish and redirect to commo... [16:21:20] while we have a little breathing room from the codfw-switchover delay, this week I want to make some progress on the esams cache cluster changes here: https://phabricator.wikimedia.org/T128813 [16:22:02] err wait, the esams ticket is https://phabricator.wikimedia.org/T125485 [16:22:27] which will indirectly unblock https://phabricator.wikimedia.org/T128813 , which is indirectly blocking some WDQS work [16:24:22] 10Traffic, 6Operations, 7Design, 13Patch-For-Review: Do something better than an "Unauthorized" error page at https://upload.wikimedia.org/ - https://phabricator.wikimedia.org/T130449#2141217 (10Krenair) @fgiunchedi: Can you please point to where the message I saw comes from? [16:26:05] 10Traffic, 6Operations, 10Reading-Web, 6Wikipedia-iOS-App-Product-Backlog, 13Patch-For-Review: Setup up Site Association file for Universal Link Support - https://phabricator.wikimedia.org/T111829#2141234 (10Krenair) Should really be a separate ticket. We could probably just add some extra apache rule to... [16:27:22] 10Traffic, 6Operations, 7Design, 13Patch-For-Review: Do something better than an "Unauthorized" error page at https://upload.wikimedia.org/ - https://phabricator.wikimedia.org/T130449#2141247 (10fgiunchedi) @krenair sure, that comes from `root` container in swift eqiad, our `rewrite.py` middleware is servi... [16:30:09] 10Traffic, 6Operations, 7Design, 13Patch-For-Review: Do something better than an "Unauthorized" error page at https://upload.wikimedia.org/ - https://phabricator.wikimedia.org/T130449#2141254 (10Krenair) Okay, but where is the HTML itself (within the puppet repo somewhere, I would hope)? It should get remo... [16:40:18] 10Traffic, 6Operations, 7Design, 13Patch-For-Review: Do something better than an "Unauthorized" error page at https://upload.wikimedia.org/ - https://phabricator.wikimedia.org/T130449#2141266 (10Krenair) Also that code references robots.txt which is also broken in the same way And requests for favicon.ico... [16:53:19] 10Traffic, 6Operations, 7Design, 13Patch-For-Review: Do something better than an "Unauthorized" error page at https://upload.wikimedia.org/ - https://phabricator.wikimedia.org/T130449#2141340 (10fgiunchedi) @Krenair in swift eqiad ATM ``` root@ms-fe1001:~# swift list root crossdomain.xml favicon.ico index... [16:56:21] 10Traffic, 6Operations, 7Design, 13Patch-For-Review: Do something better than an "Unauthorized" error page at https://upload.wikimedia.org/ - https://phabricator.wikimedia.org/T130449#2141350 (10Krenair) Agreed. [16:56:44] as always, devil is in the details... ^ [16:57:18] 10Traffic, 6Operations, 7Design, 13Patch-For-Review: Do something better than an "Unauthorized" error page at https://upload.wikimedia.org/ - https://phabricator.wikimedia.org/T130449#2141353 (10Krenair) 5Open>3Resolved a:3fgiunchedi [17:43:19] 10Traffic, 6Operations, 10Reading-Web, 6Wikipedia-iOS-App-Product-Backlog, 13Patch-For-Review: Setup up Site Association file for Universal Link Support - https://phabricator.wikimedia.org/T111829#2141565 (10jcrespo) 5Open>3Resolved I've just confirmed it on https://developer.apple.com/library/ios/do... [17:51:06] 10Traffic, 6Mobile-Apps, 6Operations: Millions of request per minute to /.well-known/apple-app-site-association producing 404s - https://phabricator.wikimedia.org/T130647#2141611 (10jcrespo) [18:17:14] speaking of the devil, I got lost into stupid details for a while in order to properly mock our backends and in the end came up with this https://gerrit.wikimedia.org/r/#/c/278948/1/modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb,unified [18:17:24] which should be simple enough [18:17:28] (for now) [18:19:17] * elukey stares the VCL syntax [18:19:30] * elukey cries a bit [18:19:34] ema: ok :) [19:40:57] directors.vcl.tpl.erb is gonna look worse than ever :) [20:10:36] :) [20:31:00] bblack: to avoid having almost one conditional per line it's probably better to just have two blocks [20:31:03] https://gerrit.wikimedia.org/r/#/c/269466/23/modules/varnish/templates/vcl/directors.vcl.tpl.erb,unified [20:33:33] ema: I donno, that duplicates a lot of complexity [20:34:39] couldn't we just conditional the the inner bit between {{range/if}} and {{end}}{{end}} ? [20:35:06] or is that way uglier? [20:35:30] mmh no, in v4 directors are defined with eg: new directorname = vslp.vslp(); [20:35:46] oh, right [20:35:50] ok [20:36:03] anyways, hopefully we get to strip away all the conditionals in a few months :) [20:36:11] :) [20:36:37] I mean there is no need to duplicate the beginning of the loop actually [20:37:11] but yeah there is no way to make that look pretty [20:50:46] bblack: my plan for tomorrow is to merge https://gerrit.wikimedia.org/r/#/c/278948/ (unless you have objections), rebase the v4 porting patch, see if I broke directors.tpl and then I guess I can start testing v4 thoroughly on my labs instance [20:51:18] ema: ok [20:51:43] ema: I think the v4 porting patch, so long as we can compiler-check that it doesn't change much for v3... we can always iterate a bit after the initial patch as well [20:52:19] so if it looks ready tomorrow, we may as well go ahead and try to merge it, and get out of rebase hell (from that point forward, all new patches need to take v4 into account) [20:52:30] that would be awesome [20:52:39] I'm out for a little while, will check in later :) [20:53:10] and I'm off for today, so see you tomorrow! :) [20:53:16] bye!