[09:59:11] bblack: Do you have T154026 / https://gerrit.wikimedia.org/r/#/c/394902/ on your backlog? If so I'll just wait patiently, just want to be sure whether you're aware. :) [09:59:14] T154026: On mobile, http://wikipedia.org/wiki/Foo redirects to https://www.m.wikipedia.org/wiki/Foo which does not exist - https://phabricator.wikimedia.org/T154026 [12:42:20] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3830715 (10Joe) With my latest patch, all redirects (including the mowiki and mowiktionary ones) work as expected. It will take so... [13:15:31] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3830791 (10Joe) @MarcoAurelio in my tests the redirects all work now, can you confirm they're ok? If so, should we close this tick... [13:21:35] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3830792 (10Strainu) I can confirm the redirects work as intended. [13:49:45] eddiegp: no, I hadn't looked at that ticket in a bit, will today! :) [13:51:05] bblack: Thanks :) [14:33:17] bblack: (if you have time) - just to triple check, I'd proceed with https://gerrit.wikimedia.org/r/#/c/397539 that will introduce the double A/PTR records for notebook1002/kafka1023 [14:33:40] just temporarily, then we'll remove it after the rename [14:34:19] well, it's double A/PTR just for the mgmt hostname, right? [14:34:25] the main hostname/IP just switch immediately [14:35:13] yes indeed [14:35:43] elukey: also, you'll probably want to purge the dns caches of the affected names, after you've done that authdns-update and before anything else, to avoid various confusing problems. Maybe restart install1002 dhcpd as well, not sure if that's necessary or just paranoia [14:38:04] sudo cumin A:dns-rec 'rec_control wipe-cache kafka1023.eqiad.wmnet kafka1023.mgmt.eqiad.wmnet notebook1002.eqiad.wmnet 14.5.64.10.in-addr.arpa 104.3.65.10.in-addr.arpa' [14:38:11] elukey: ^ probably that, to purge dns caches [14:38:47] seems like it will work :) [14:39:06] bblack: I was going to wait for the ttl to expire because I didn't want to mess with the DNS cache, but I can surely do it :) [14:42:23] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3831090 (10EddieGP) [14:42:40] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3413551 (10EddieGP) [14:45:58] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3831128 (10EddieGP) 05Open>03Resolved a:03EddieGP The redirect part is done and working as intended (thanks to @Joe !). What... [14:48:00] eddiegp: your analysis seems pretty thorough :) [14:49:07] bblack: I've learned that I usually break something if I'm not thorough ;) [14:49:13] so one thing: [14:49:31] just a little above the line you're editing, there's also: set req.http.MobileHost = regsub(req.http.MobileHost, "^(www\.)?(mediawiki|wikisource|wikidata)\.", "m.\2."); [14:49:54] which isn't really the problem we're talking about in the ticket with "www.m" [14:50:18] but it does mean we rewrite www.wikisource.org to m.wikisource.org before we reach the alter regexen [14:50:54] I don't know if that's a problem, I haven't though it through completely yet [14:50:59] s/alter/later/ :) [14:51:49] Oh yeah, wikisource being in both regexes is weird. [14:53:51] I don't think those should be overlapping. [14:55:14] right [14:55:45] but I wonder why we have a www.wikisource -> m.wikisource rewrite there for mobile (which then the third regex already ignores because "m" was already in the negative lookahead list before you added "www") [14:56:31] There's the old wikisource at www.wikisource.org and there's the newer ones following our scheme of en.wikisource/de.wikisource/sv.wikisource/... [14:56:42] I guess the intention was to handle those differently. [14:57:33] so looking at the live results today for various wikisource things: [14:57:56] https://wikisource.org/ redirects to https://wikisource.org/wiki/Main_Page [14:58:17] https://m.wikisource.org/ redirects to https://m.wikisource.org/wiki/Main_Page [14:58:45] https://www.wikisource.org/ redirects to https://wikisource.org/ [14:58:55] and www.m doesn't exist in DNS [14:58:59] and then with a mobile UA [15:00:05] hmmm [15:00:50] oh of course, it's different if there's a real /wiki/Foo in the path heh [15:01:51] but the www links "work" for mobile as they are today for wikisource [15:02:20] https://www.wikisource.org/wiki/Foo redirects via varnish to https://m.wikisource.org/wiki/Foo which works correctly [15:03:47] So if we remove the first line, apache would rewrite https://www.wikisource.org/wiki/Foo to https://wikisource.org/wiki/Foo instead, right? [15:04:03] s/remove the line/remove wikisource from the line/ [15:04:07] which wouldn't be correct, I think we're supposed to rewrite it as m.wikisource for mobile [15:04:23] Yeah, so it has to stay in the first regex [15:04:25] I think basically your patch as-is is correct [15:04:39] there's some redundancy, but it's not easily fixable without making things way more complicated [15:04:43] Else we wouldn't reach the m.wikisource at all. [15:05:15] And we need wikisource in the third line because of $lang.wikisource, right? [15:05:20] right [15:05:34] So then both have to stay. [15:05:44] adding the "www" in the third line is doing nothing for wikisource, since that case is already rewritten, but it is doing its thing for all the other cases [15:06:22] Yeah, I'd say that the patch should be fine then :) [15:06:34] do you want to use a deploy window for this, or just merge and go? [15:07:18] I'm fine with either, you probably know better than me what the correct process would be ;) [15:07:30] But I'd rather not merge it right now. [15:08:30] I'll be away in a few until 16UTC. [15:08:46] from a safety-analysis perspective, the change in the patch prevents the possibility of a rewrite of www. to www.m., where none of have a www.m in DNS at all to begin with [15:09:21] so I don't think it's capable of harm, assuming the regex isn't completely borked, but we're not changing the regex semantics, either. Just adding another to the list of prefixes for negative-lookahead that already existed. [15:09:57] eddiegp: ok, just ping me later when you're ready [15:10:01] I'd like to be around while and after it's deployed in case something breaks. [15:10:06] Yeah, I'll do :) [15:11:51] in the meantime, I'm going to sockpuppet-suggest that someone needs to create the language-code "www" for wikipedia in support of an obscure language called WallaWistWop spoken only in my hometown. [15:12:56] :) [15:15:13] there's "new", which is pretty cool [15:15:16] new.wikipedia.org [15:15:38] Nepal Bhasa, Newari [15:16:09] https://tools.wmflabs.org/bash/quip/AU_JvX6F1oXzWjit5hxZ applies [15:17:41] I really think should push an effort to get rid of $lang subdomains, and the m-dot subdomains. We'll have to keep the DNS/TLS support around for many years afterwards anyways for compatibility with all the links out there in the world. But some future version of us will thank us later when they finally die. [15:17:52] I just don't have the time and energy to push it [15:18:57] switch all the canonical paths to e.g. https://wikipedia.org/en/wiki/Foo , and have the mobile detection and differentiation done on Vary stuff involving Varnish+MediaWiki and suppress/enable cookies or whatever as it is now, but not involving an explicit m-dot subdomain visible to users. [15:20:13] technically, we could do all of the above at the Varnish level without MW/apache's knowledge or involvement, but it would be a lot smoother and simpler to have rel=canonical from the backend change as well [15:29:09] I'd imagine a process where we'd first add support for the new canonical form at the MW level and set rel=canonical to it, and have Varnish rewrite (not redirect) the old-style URLs internally so the cache is shared and users don't experience redirects. [15:29:53] then give it some months (or years?) for the bulk of traffic stats to be showing the use of the new URLs before switching to a 301 redirect of the old. Then some years later maybe it can be removed. [15:30:11] in the intervening time, we could at least not add new language subdomains to the fixed list, since there would be no old URLs in the wild for them. [15:31:30] search engines would switch to the new canonical fairly quickly, and a lot of our traffic comes from there anyways. talking to a few other major sites about link cleanup could solve a lot of the rest to get past that first stage. [16:19:09] we're getting tls stats generated by varnishmtail into prometheus: https://grafana.wikimedia.org/dashboard/db/tls-ciphers-by-data-center [16:19:40] next step is to add aggregation rules so that the global prometheus instances also get the data [16:20:12] that will allow us to plot global stats instead of per-DC ones, if I understood godog correctly :) [16:20:48] that's the plan yea [16:21:04] bblack: Ready when you are :) [17:05:18] bblack: Did you see the ping earlier? [17:15:34] eddiegp: yeah I'm stuck in another meeting though [17:17:26] bblack: That's fine, just wanted to make sure you saw it. :) [17:33:00] bblack,ema - if you have time for https://gerrit.wikimedia.org/r/#/c/397765 later on, not sure what is the best place to put the cergen TLS client certificates for varnishkafka [17:33:23] (so we'll be able to progress it tomorrow EU time :) [17:34:44] my 30 second opinion is use a fresh directory, because both debian and puppetization of existing manual certs and LE are all complicated and probably don't want to interfere with them [17:35:06] but be careful that permissions are managed appropriately in that fresh directory (root/500 for privkey dir, etc) [17:36:21] for the letsencrypt stuff we ended up using /etc/acme/ to avoid touching /etc/ssl/ [17:38:18] super, we were thinking about /etc/varnishkafka/ssl [17:44:52] elukey: yeah so reading the commentary there a bit deeper and responding specifically: [17:45:18] I tend to disagree with otto about the lack of a need for separate dirs for private+public [17:45:50] I'd make separate subdirs under /etc/varnishkafka/ssl/, and I'd set the privkey directory and the privkey file both to very restrictive permissions [17:46:01] e.g. root/500 for the dir, root/400 for the file [17:46:48] but you may need to tone that down to match the realities of vk reading the keys, too... [17:46:59] either way, the dir+perms needs explicit mgmt [17:47:48] all right [17:48:08] taking a quick peek at a live varnish box, apparently VK runs as root anyways, so yeah you don't need to lower those permissions to account for anything. [17:48:35] I was about to say that, maybe in the future we want to review it? [17:48:43] (I mean vk running as root) [17:49:11] anyhow, will come up with a new code patch tomorrow and bother ema in the EU daytime :) [17:52:36] well if in the future we want vk to not run as root, we'd either have to give it a special group that has read on the keyfiles and execute on the key dir [17:52:48] or alternatively, have the vk code itself manage the privdrop and load the key before it drops the root priv [17:54:15] not looking forward to touch that part of the code :D [17:56:52] eddiegp: ready to give it a spin whenever you are [18:07:18] bblack: I'm here, happy to escape from paperworks :) [18:07:28] ok [18:07:40] need any warning before pulling the trigger, or ready to go? [18:07:54] Gooooooo! :) [18:12:42] Seems to work for me, at least for wikibooks. Testing ... [18:13:02] eddiegp: it's live everywhere :) [18:13:17] (before that statement, it was live most-of-everywhere) [18:16:09] curl -H "User-agent: mobi" -I https://www.{wikipedia,wiktionary,wikinews,wikisource,wikiquote,wikibooks,wikiversity,wikivoyage}.org/wiki/Foo | grep -i location; says it works as expected for all domains (redirects to en.$project.org/wiki/Foo) [18:17:09] awesome [18:17:51] bblack: Thanks for your help :) [18:22:25] np! [18:32:06] 10Wikimedia-Apache-configuration, 10Operations, 10Mobile, 10Patch-For-Review, and 2 others: On mobile, http://wikipedia.org/wiki/Foo redirects to https://www.m.wikipedia.org/wiki/Foo which does not exist - https://phabricator.wikimedia.org/T154026#3831883 (10EddieGP) 05Open>03Resolved a:03EddieGP Thi... [21:24:01] 10Traffic, 10Operations, 10ops-ulsfo: replace ulsfo aging servers - https://phabricator.wikimedia.org/T164327#3832487 (10RobH)