[06:24:56] 10Traffic, 10Operations: Setup a new PKI software as an alternative to the puppet CA for managing services certificates - https://phabricator.wikimedia.org/T194031 (10elukey) p:05High>03Normal a:03Joe [08:00:13] hello people [08:00:38] I've merged a change to aggregate the vk alarms by instance type (webrequest text, upload, misc, statsv, eventlogging) [08:00:51] rather than having them on each cp host [08:01:13] this should be better to avoid storm of alarms if vk for some reason misbehaves [08:01:50] it is also tuned to raise alerts more often than before, so if you see any weird alarm during the next days ping me or bare in mind that I've changed them today :) [08:09:46] <_joe_> cool [09:08:37] 10Traffic, 10Operations, 10ops-esams: cp3037 is currently unreachable - https://phabricator.wikimedia.org/T196974 (10ema) [09:33:48] 10Traffic, 10DNS, 10Operations: rack/setup/install dns200[12].wikimedia.org - https://phabricator.wikimedia.org/T196493 (10ema) [09:37:38] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install LVS200[7-10] - https://phabricator.wikimedia.org/T196560 (10ema) [09:37:47] 10Traffic, 10Operations, 10Patch-For-Review: Enable numa_networking on all caches - https://phabricator.wikimedia.org/T193865 (10ema) [09:37:58] ema: do you need help cleaning/restoring traffic tasks? [09:38:01] 10Traffic, 10Operations, 10Performance-Team, 10Patch-For-Review: Refactor varnishospital and varnishslowlog - https://phabricator.wikimedia.org/T193489 (10ema) [09:41:20] vgutierrez: I think I should be done :) [09:41:31] let me know if I've missed anything though of course [09:41:53] I just cleaned tags here: https://phabricator.wikimedia.org/T197763 [09:42:06] but that's the only thing missing apparently [09:49:45] 10Traffic, 10Operations, 10Patch-For-Review: Enable numa_networking on all caches - https://phabricator.wikimedia.org/T193865 (10MacFan4000) p:05High>03Normal [09:57:08] 10Traffic, 10netops, 10DNS, 10Operations, 10ops-codfw: switch port configuration for dns200[1-2] - https://phabricator.wikimedia.org/T197697 (10ema) p:05High>03Normal [10:03:31] 10netops, 10Operations, 10ops-codfw: switch port configuration for graphite2003 - https://phabricator.wikimedia.org/T198119 (10CommunityTechBot) a:03Papaul [10:03:33] 10netops, 10Operations, 10ops-codfw: switch port configuration for graphite2003 - https://phabricator.wikimedia.org/T198119 (10CommunityTechBot) p:05High>03Normal [10:03:36] 10netops, 10Operations, 10ops-codfw: switch port configuration for graphite2003 - https://phabricator.wikimedia.org/T198119 (10CommunityTechBot) [10:03:39] 10netops, 10Operations, 10ops-codfw: switch port configuration for graphite2003 - https://phabricator.wikimedia.org/T198119 (10CommunityTechBot) [10:03:42] 10netops, 10Operations, 10ops-codfw: switch port configuration for graphite2003 - https://phabricator.wikimedia.org/T198119 (10CommunityTechBot) 05Open>03Resolved [10:44:55] ema: quick q: where is the VCL code that handles {{lang}}.m.wp.org url rewrites? [10:58:24] mobrovac: https://github.com/wikimedia/puppet/blob/production/modules/varnish/templates/text-frontend.inc.vcl.erb#L114 that one? [10:59:55] yup, that one, thnx vgutierrez! [11:00:20] now i have to figure out how x-dt-host and host are used [11:00:21] :P [11:00:29] :) no problem! [11:26:51] mobrovac: there's https://github.com/wikimedia/puppet/blob/production/modules/varnish/templates/text-backend.inc.vcl.erb#L27 too [11:27:06] va bene [11:27:08] thnx [11:38:30] ema: so varnish sets host to the desktop domain, and x-subdomain to the mobile one if present? [11:39:26] basically i'm interested in knowing the way the request for mobiles looks like when it reaches the appservers [11:39:39] mobrovac: X-Subdomain is either 'M' or 'Z' (mobile/zero) [11:41:59] mobrovac: right before reaching the appserver the host header is set to x-dt-host for mobile requests [11:43:17] ok so for en.m.wp.org/wiki/Blah we get uri: en.wp.org/wiki/Blah and headers: { x-subdomain: 'M' } ? [11:43:44] right, the host header is rewritten from en.m.wp.org to en.wp.org [11:43:54] kk cool [11:43:55] I'm not entirely sure whether we strip x-subdomain or not, let me check [11:44:27] the appservers need to have a way of differentiating them, so it shouldn't be stripped [11:44:43] correct, it is sent to the applayer [11:45:55] ok great [11:45:58] thnx ema! [11:48:39] hmm can't find anything about x-subdomain in the codebase [11:48:43] * mobrovac keeps digging [12:12:50] mobrovac: so, the Host header is actually left intact (eg: en.m.wp.org), while x-dt-host is set to en.wp.org [12:14:07] the exception is restbase, for which we don't differentiate between desktop and mobile and we do set Host to x-dt-host [12:17:14] the above is true at the varnish frontend layer [12:18:15] then, at the backend-most varnish backend, right before sending the request to the applayer, we do the mobile hostname mangling thing and rewrite Host [12:18:19] example: [12:18:32] - BereqHeader Host: en.m.wikipedia.org [12:18:38] - BereqHeader X-Subdomain: M [12:18:43] - BereqHeader x-dt-host: en.wikipedia.org [12:18:49] [...] [12:18:55] - BereqUnset Host: en.m.wikipedia.org [12:18:55] - BereqHeader host: en.wikipedia.org [12:25:40] mobrovac: and yes, I can confirm that "X-Subdomain: m" is indeed used by mediawiki to know that the mobile version of a page needs to be generated [12:27:50] mobrovac: https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/mobile.php#L12 [12:27:55] ok great [12:28:02] thnx ema! thsi was really helpful [12:30:47] mobrovac: aaaand https://github.com/wikimedia/mediawiki-extensions-MobileFrontend#wgmfmobileheader [12:31:05] :) [12:35:42] yaaay [12:35:44] hehe [12:37:12] ok, so to recap, when a req reaches the appservers, the uri is {mwapi_srv_lvs}/wiki/Title, and headers: { host: en.wp.org, x-subdomain: 'M', x-dt-host: en.wp.org } ? [12:37:26] i doubt x-dt-host is needed [12:37:33] but put it there for completeness [12:39:05] yes [12:39:27] I also doubt x-dt-host is useful at the applayer, we could perhaps strip it [12:39:37] probably, yes [12:39:53] perfect! thnx ema [12:40:13] np! [12:40:51] in general, we are abusing headers because of the lack of support for variables in vcl [12:41:13] in this case though we need to pass information among different varnish layers, so using x-dt-host does make sense [12:42:18] all this reminds me that I wanted to create a task about using vmod_var(3) instead of temporary headers :) [14:16:14] yeah x-dt-host is definitely not a standard (even in-house), it's just an accidentally-exposed internal variable [14:16:40] relatedly on the forever wishlist: [14:17:20] 1) Get MW (and others) to parse host headers on their own, so we don't set X-Subdomain and don't mangle the host header and it does the /\.m\./ check and output variance all on its own. [14:18:26] 2) Get RB to look at host headers too, so we don't have to mangle the public URL en.wp.o/api/rest_v1/ -> /en.wp.o/v1/ [14:27:15] vgutierrez: what's the "as discussed on IRC, v6 records can be added automatically"? [14:28:08] bblack: so.. as papaul explained to me, ipv6 records can be added automatically with some script and he's using that [14:28:32] oh as a separate commit then, with the diff made by the script? [14:28:39] right [14:28:41] ok [14:40:52] ema: one follow-up question: I suppose that if send a req from the outside with the x-subdomain header for a desktop domain, varnish will strip it before passing it on? [14:41:53] mobrovac: yes we do unset x-subdomain at the frontend layer [14:42:04] kk thnx [15:03:41] bblack: BTW, I've just realised that our auth DNS records, ns[012].wikimedia.org are missing AAAA records, bug or feature? [15:03:49] feature [15:04:17] ook [15:04:35] considering that geoip database data is not as good yet for v6 as it is for v4 [15:05:02] we force client resolvers to use v4 to not worsen experience by having less optimal cache dc selected [15:05:07] but it's probably slowly getting better over time [15:07:35] would be interesting to test that some day [15:27:51] hmmm [15:27:54] https://gerrit.wikimedia.org/r/#/c/443442/ [15:28:35] bblack, ema: I can merge that and won't be a problem with the existing authdns::servers till I don't update puppet/modules/role/manifests/authdns/data.pp, right? [15:35:39] vgutierrez: won't be a problem I think. I wouldn't be shocked if puppetization only suceeds 99% of the way unti data.pp changes, but that's ok too. [15:42:00] bblack: ack :) [20:04:35] 10netops, 10Operations, 10fundraising-tech-ops: NAT and DNS for fundraising monitor host - https://phabricator.wikimedia.org/T198516 (10cwdent)