[06:22:02] <_joe_> gwicke: you got wifi at your place of stay! [07:37:54] 07HTTPS, 10Traffic, 06Operations, 10Wiki-Loves-Monuments: configure https for www.wikilovesmonuments.org - https://phabricator.wikimedia.org/T118388#2398200 (10SindyM3) Done :D [10:53:31] <_joe_> ema: around? [10:57:22] <_joe_> anyways, for whenever either you or bblack are around: we're going to install a url shortener under w.wiki; the way it's implemented in labs is this [10:57:25] <_joe_> https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/mediawiki/files/apache/beta/sites/main.conf;2b8b99d28069bcd98ff06dd11081c8d079f54c2f$22 [10:57:43] <_joe_> which means that we do a ProxyPass to the public endpoint on "meta" [10:57:56] <_joe_> in prod that would mean make the appserver ask the resource to varnish [10:58:07] <_joe_> and I was wondering what you guys think about it [10:58:37] <_joe_> because making apache behave like I'd like it to (request the resource locally) isn't going to be easy [10:59:39] <_joe_> we're going to lunch now though [10:59:43] <_joe_> bbl :) [11:53:42] _joe_: I don't think the beta approach is necessary here [11:54:35] worst case we could do it in varnish (rewrite the URL from w.wiki to meta/whatever), but really why is it hard to do this in prod apache config as a rewrite rule on the way in? [12:14:24] 07HTTPS, 10Traffic, 06Operations, 10Wiki-Loves-Monuments: configure https for www.wikilovesmonuments.org - https://phabricator.wikimedia.org/T118388#2398786 (10JeanFred) 05Open>03Resolved Thanks all for this! :) [12:16:39] found an interesting Varnish test triggering a VSL timeout https://github.com/varnishcache/varnish-cache/blob/4.1/bin/varnishtest/tests/r01762.vtc#L29 [12:33:23] <_joe_> bblack: because rewrite rules can't cross vhost borders [12:33:29] <_joe_> without using proxy [12:33:36] <_joe_> and they don't want a redirect [12:33:47] <_joe_> but I think we found a better solution over lunch [12:35:35] _joe_: why does it need to be across a vhost border? it's the same cluster, right? [12:35:58] can just add w.wiki to the list of hosts for meta's vhost, and put the rewrite rule in there [12:50:03] <_joe_> bblack: d'oh right [12:51:08] <_joe_> bblack: well, I am not sure mediawiki would comply though [12:51:34] <_joe_> as I think it gets the db to fetch from the HTTP host header [13:08:07] you can rewrite the host header, right? [13:08:19] _joe_: ^ [13:08:34] I assumed one could, anyways [13:08:56] this would all be simpler if MW itself had a URL-routing layer to replace most of our apache complexity and some of our varnish complexity :) [13:09:23] <_joe_> yes [13:09:47] <_joe_> and no, it's not trivial to rewrite the host header at the right time in apache [13:10:44] well in any case, if apache can't easily do this cleanly [13:10:46] <_joe_> I mean you should use SetEnvif [13:10:52] we can always go stick yet another URL rewrite in varnish [13:10:57] <_joe_> yes [13:11:10] <_joe_> I wanted to avoid polluting varnish with this shit [13:11:18] <_joe_> apache is beyond saving [13:11:33] the upside is it's relatively painless in varnish, probably 4 easy to understand lines including bracing/formatting [13:12:41] something like: if (req.http.host == "w.wiki") { set req.http.host = "meta.wikimedia.org"; set req.url = "/...." + req.url; } [13:18:07] <_joe_> yes [13:18:16] <_joe_> I'll do that in varnish [13:18:21] <_joe_> apache is maybe doable [13:18:39] <_joe_> but it's an unholy mess and needs use of the dreaded headers module [13:19:40] <_joe_> so ok, varnish it is [13:19:48] <_joe_> I'll bake up a patch shortly [13:24:56] _joe_: there's two basic places we can do it that kinda make sense (I don't know if we're purging urlshortner stuff, and if so whether we'd purge it based on the metawiki URL or w.wiki) [13:26:32] _joe_: either at the bottom of text-frontend's cluster_recv_pre_purge (so the whole cache stack sees it as rewritten, including purges rewritten to metawiki), or in text-backend misspass_mangle (the direct case that actuall has code), which means it flows through caching as w.wiki and must be purged as w.wiki, and gets transformed at the last second on fetch from MW (so more-analogous to if we co [13:26:38] uld rewrite it in apache) [13:27:40] I tend to think misspass_mangle is cleaner, but depending on how/if purged we may need/want it to be up in cluster_recv_pre_purge [13:28:50] <_joe_> we won't purge it [13:28:51] <_joe_> no [13:28:58] <_joe_> ok [13:30:03] another non-obvious note: if doing misspass-mangle, the conditional can/should be on req.http.host, but the "set" lines inside should alter bereq.* not req.* [13:30:15] (as with the other examples there) [13:31:32] <_joe_> ues [13:31:34] <_joe_> *yes [13:32:09] bblack: https://phabricator.wikimedia.org/T108649#1740383 <3 [13:32:45] <_joe_> lol [13:34:31] legoktm: that's my standard answer for everything :) [13:35:06] <_joe_> ehehehh [13:35:26] I dream of a world in which our cp* hosts just terminate TLS and cache content according to RFC-standard headers and do generic normalization/cleanup/protection tasks, but don't know anything application-specific [13:36:43] (oh and handle analytics and /beacon/, those can't realistically move out of VCL) [13:38:27] <_joe_> I dream of a world where war is a thing of the past and all men are equals [13:38:42] <_joe_> it's almost as likely to happen during my lifespan :P [13:41:47] <_joe_> bblack: our vcl code changed a LOT since I last looked [13:41:56] <_joe_> you and ema clearly work way too much [13:43:01] I think we have a realistic medium-term target (as in, next 6 months or so) of getting to a state where only the text cluster has its own custom VCL files like the ones in templates/varnish/ [13:43:27] <_joe_> that sounds reasonable [13:43:35] maps, misc, and upload, the few cluster-specific things they do can be turned into tunables and/or feature-flags enabling bits in the common VCL [13:44:11] <_joe_> https://gerrit.wikimedia.org/r/295493 if you have time to take a look [13:44:21] and a lot of the "reasonable" app-specific things can be auto-generated from declarative app_backends data (this set of URLs maps to this backend, with these parameters and basic cache policy, etc) [13:47:03] _joe_: arguably we could/should support beta there too and they'll behave the same instead of proxy-into-self on beta. but it would mean declaring the shortner + metawiki hostnames in hiera, kinda like bits_domain. [13:47:37] <_joe_> bblack: ok I can do it [13:48:23] hmmm [13:48:31] the existing bits_domain sucks because it's half-refactored, too :) [13:48:41] it's in r::c::base, really should be in r::c::text I guess [13:48:56] <_joe_> ok [13:49:16] <_joe_> I'll try to do that now, before https://phabricator.wikimedia.org/T137908 happens [13:49:55] I can do it too, really most of those should be refactored [13:50:23] static_host and upload_domain are wrongly in base, too [13:50:51] <_joe_> I am fixing it now, at least for this specific instance [13:51:00] ok [13:51:24] just do what you need to do in whichever place really, can cargo-cult existing base-class stuff too [13:51:32] and I'll fix/move all the moveable ones after [13:53:08] <_joe_> yeah I'm going to use the base class again [13:53:12] <_joe_> for consistency [13:53:15] <_joe_> even in error :P [13:59:51] <_joe_> ok I did it, I hope it's correct [14:00:01] <_joe_> I'll be back in ~ 2 hours [14:13:26] bblack: fixed [15:23:26] 10Traffic, 06Operations, 13Patch-For-Review: Investigate TCP Fast Open for tlsproxy - https://phabricator.wikimedia.org/T108827#2399537 (10ema) A few more things tools-wise. TFO support has been added to curl in version 7.49.0: [[ https://curl.haxx.se/changes.html#7_49_0 | --tcp-fastopen]]. Unfortunately De... [16:11:42] ema and/or paravoid? https://gerrit.wikimedia.org/r/#/q/topic:traffic-sysctls,n,z [16:12:02] some cleanup of old mysterious sysctls, raising a few, etc... plus a couple of new ones near then end, and applying the netdev ones similarly to LVS. [16:12:23] I was looking into the mysterious ones yesterday :) [16:12:26] as usual gerrit orders the topic kinda randomly, seems to be upside down on PS1 (first patches at the bottom) [16:12:33] the comment was particularly nice [16:13:13] nice! [16:13:16] # Ancient and mysterious, these can either have absolutely no [16:13:16] # effect on modern kernels or could be the very things that keeps [16:13:16] # the fabric of the space time continuum (and our wikis) alive. [16:13:29] 10Traffic, 06Operations, 13Patch-For-Review: Decrease max object TTL in varnishes - https://phabricator.wikimedia.org/T124954#1970940 (10Krinkle) How does the cache ttl of Varnish interact with the concept of 304 renewals? I remember in the past we often had bugs where a cache object had expired (but not ye... [16:20:11] bblack: puppetlint didn't like the LVS change https://gerrit.wikimedia.org/r/#/c/295544/ [16:20:41] ./modules/lvs/manifests/balancer.pp:64 WARNING indentation of => is not properly aligned (arrow_alignment) [16:20:44] ./modules/lvs/manifests/balancer.pp:65 WARNING indentation of => is not properly aligned (arrow_alignment) [16:20:47] ./modules/lvs/manifests/balancer.pp:66 WARNING indentation of => is not properly aligned (arrow_alignment) [16:23:10] heh [16:23:21] ok, will fixup arrow-align bullshit later :) [16:28:52] 10Traffic, 06Operations, 13Patch-For-Review: Decrease max object TTL in varnishes - https://phabricator.wikimedia.org/T124954#2399786 (10BBlack) Varnish 3 and 4 may differ a bit on 304 basics, and Varnish 4 clearly does a better job of managing grace-mode in general, and using it for 304-refreshes, and my cu... [16:30:26] 10Traffic, 06Operations, 13Patch-For-Review: Decrease max object TTL in varnishes - https://phabricator.wikimedia.org/T124954#2399812 (10BBlack) I should have noted above: our current maximum grace is 1 hour beyond whatever the TTL is. Basically we're really not using grace very effectively today, but it's... [16:33:20] bblack: great stuff! I love the comments un-mysteriousing the settings [16:33:48] yes, I lol'd [16:33:52] it was yuvi (i had to check) [16:41:57] must be reviewed to the tune of https://www.youtube.com/watch?v=HGDmBLAPikU [16:42:16] bblack: thanks for the review, what does it take for a +2? :) [16:49:24] bblack, ema: after a chat with one of the varnish dev I finally managed to repro the timeout, simply adding -p debug=+syncvsl to varnishd [16:51:32] "Make VSL synchronous" [16:51:34] heh [16:51:57] what I've got so far: when the vsl_buffer in varnish fills up, it is flushed to shmlog. It may happen that a request get partially flushed, for example a Begin timestamp is in the shmlog but the rest is still in the buffer or it hasn't been logged yet (long req). [16:52:54] any reader of the shmlog, like vk or varnishlog, waits by default 120 seconds before giving up [16:53:22] but you can tune it with the -T parameter (not in vk yet) [16:55:44] so what I am going to do on my side is to add the -T paramter to vk, together with the VSL formatter to have the possibility to log VSL warnings/errors if we wish [16:56:16] not sure what is best for the varnish vsl* settings [16:56:59] ema: I know it is kinda cheating but it worked :P [19:31:57] 10Traffic, 10MediaWiki-extensions-UrlShortener, 06Operations, 13Patch-For-Review, 05Wikimania-Hackathon-2016: Configure apache rules for UrlShortener - https://phabricator.wikimedia.org/T133485#2400220 (10Legoktm) 05Open>03Resolved a:05Dereckson>03Joe