[00:07:09] 10Traffic, 10DNS, 06Operations, 10Phabricator, 13Patch-For-Review: Redirect phabricator.mediawiki.org to phabricator.wikimedia.org - https://phabricator.wikimedia.org/T137252#2370304 (10Dzahn) I don't like it very much, especially not in wikipedia.org. Where would we draw the line here before we add it t... [04:24:20] 10Traffic, 06Operations, 06Project-Admins: Create #HTTP2 tag - https://phabricator.wikimedia.org/T134960#2370538 (10Peachey88) 05Open>03declined a:05Danny_B>03None >>! In T134960#2338185, @Aklapper wrote: > Proposing to decline as per last two comments. Done [08:25:00] 10Traffic, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium): Move gallium to an internal host? - https://phabricator.wikimedia.org/T133150#2370859 (10hashar) [08:25:55] 10Traffic, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium): Move gallium to an internal host? - https://phabricator.wikimedia.org/T133150#2223557 (10hashar) [08:26:20] 10Traffic, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium): Move gallium to an internal host? - https://phabricator.wikimedia.org/T133150#2223557 (10hashar) [09:49:25] ema: hello, if you are around, do you know whether wmflabs has a nging SSL termination + equivalent of misc varnish ? [09:49:37] havent found anything in puppet, so I would assume there is not :) [09:57:59] hashar: not sure, it looks like beta.wmflabs.org is referenced in varnish vcl though [09:58:16] https://github.com/wikimedia/operations-puppet/blob/production/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb#L33 [10:07:39] ema: yeah that reproduce the text/upload varnish from prod [10:07:56] albeit without Nginx TLS/SSL termination since we lack certificats there [10:08:04] I will ask labs folks this afternoon :) [10:09:02] hashar: right, and without http2 :) [10:10:22] false ^ [10:20:15] uh we're not restarting automatically varnishxcache and friends on upgrades to varnishlog.py [10:37:56] ema: I am not sure how the varnish upgrades happen on beta cluster to be honest :( [10:38:01] maybe that is via unattended ugprade [10:45:36] in other news: no cronspam from cp* hosts since Tuesday :) [10:47:16] \O/ [13:31:32] ema: the X-Cache patch is not-quite-right yet I think. but the problem runs deeper than that patch. what exactly are the vcl_backend_error cases where we don't want to use the synthetic error page? because really I should probably be setting "err" for any entry to that function, but then the errorpage should probably be used for all entry into there too [13:32:47] in any case, I'm not feeling great today, I likely won't be online much. [13:32:56] text/call if there's something crazy [13:42:33] https://blog.cloudflare.com/optimizing-tls-over-tcp-to-reduce-latency/ [13:48:37] oh nice, they patchewd for dyn record size, we need that! [13:48:51] (we did tune the fixed record size way back, but dynamic is far cooler) [13:49:25] 10Traffic, 06Analytics-Kanban, 06Operations, 13Patch-For-Review: Verify why varnishkafka stats and webrequest logs count differs - https://phabricator.wikimedia.org/T136314#2371668 (10JAllemandou) There still are warnings from webrequest sequence checks. **TL;DR** Luca's code solved the ordering issue, bu... [13:49:31] * bblack back to bed [13:49:33] uhh [13:49:46] uhh? [13:49:56] something is weird with my irssi [13:50:11] ok [13:50:56] will patch up nginx monday-ish :) [13:51:36] hehe [13:51:54] 10Traffic, 06Analytics-Kanban, 06Operations, 13Patch-For-Review: Verify why varnishkafka stats and webrequest logs count differs - https://phabricator.wikimedia.org/T136314#2371687 (10JAllemandou) Proposed solution: If @elukey confirms that missing timestamps come from error in Varnish (timeout, connection... [14:48:36] bblack: get better soon! [15:15:03] 10Traffic, 06Operations, 13Patch-For-Review: Scripts depending on varnishlog.py maxing out CPU usage on cache_misc - https://phabricator.wikimedia.org/T137114#2371928 (10ema) 05Open>03Resolved a:03ema Grouping transactions by requests solved the problem, confirmed on cp1051 and cp1061. Closing. [16:16:00] cloudflare also enabled server push for their clients [16:16:37] I don't know if they patched nginx and if they plan on contributing it back upstream [23:02:36] -#define NGX_SSL_MAX_SESSION_SIZE 4096 [23:02:37] +#define NGX_SSL_MAX_SESSION_SIZE 16384 [23:03:08] ^ something else slipped into the TLS dynamic record size patch, not really commented/documented. but it's related to the sid-cache stuff I've been looking at the past week or two. [23:03:56] there's an if-confition in nginx code where if the size of the session data is > NGX_SSL_MAX_SESSION_SIZE, it just doesn't cache the session at all. they may have needed to bump it for dyn record size info, but only by a very small amount. [23:04:21] I wonder if they figured out that with some nginx configs + browsers + ciphers (etc), some sessions were going over that size and limiting sid-caching... [23:04:56] bblack, off the top of your head, can you think of any reason why we set cookies for upload.wm.o? [23:05:31] not really, but maybe I wouldn't know if some of our client-side JS expect certain cookies to come with images? I have no idea. [23:06:35] VCL seems to set GeoIP, WMF-Last-Access, and CP cookies [23:06:44] yeah, non are necessary [23:06:47] *none [23:07:34] anyways, don't worry about it :) I can investigate more thoroughly. Just wanted to check if there was something obvious I wasn't thinking of. Enjoy your weekend [23:07:39] we do webrequest -> hadoop for upload.wm.o traffic, don't they use WMF-Last-Access via that? or do they not care for images (I guess in the case they weren't loaded in the context of a page load on our sites)? [23:07:53] yeah [23:07:56] GeoIP I guess not. CP, we're using that for HTTP/2 and tls-ciphers graphing including upload. [23:09:05] (oh and sid reuse, that's in CP too) [23:09:06] tls ciphers are from X-Connection-Properties [23:09:19] oh, right [23:09:24] so just H2 is coming from CP? [23:09:24] ditto sid reuse [23:10:09] it definitely is when it comes to metrics we capture in js and send back [23:10:28] hmm right, even H2 is in XCP [23:10:54] we don't need the CP cookie except for any JS usage then [23:12:33] JS can't read it anyway [23:12:57] XHR disallows reading Set-Cookie [23:15:40] yeah, H2 is XCP [23:17:23] analytics pays attention to requests with no cookies at all, so it could throw them off [23:17:25] i'll check with them [23:31:13] 10Traffic, 10Analytics, 06Operations: Make upload.wikimedia.org cookieless - https://phabricator.wikimedia.org/T137609#2373208 (10ori)