[01:52:08] 10Traffic, 10Operations, 10Reading List Service, 10Reading-Infrastructure-Team-Backlog, 10Patch-For-Review: PUT blocked by Varnish - https://phabricator.wikimedia.org/T182825#3835819 (10Dzahn) related ticket that allowed PUT on the "misc" cluster: T62350 [10:21:28] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3450565 (10Verdy_p) Moldavian: about the renaming/redirect from "mo.wiki(pedia|dictionary).org" to "ro.wiki(pedia|dictionary).org"... [10:25:23] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3836865 (10Strainu) The transliterator is not active on the Romanian projects. [10:26:07] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3836866 (10Verdy_p) Note also that "uselang=mo" really displays a Romanian-Cyrillic UI, but "uselang=ro-latn" just renders as an E... [10:31:51] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3836873 (10Strainu) @Verdy_p , where is ro-latn defined and why do you expect it to work? [10:34:23] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3836880 (10Verdy_p) So now with the redirect being active, you've dropped completely the capability of contributing and displaying... [10:38:23] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3836884 (10Strainu) You seem to be behind on the news: the language committee has decided that mo.wp should be deleted, not just c... [10:39:17] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3836886 (10Verdy_p) "ro-latn" and "ro" are the same. I don't want this addition as a separate locale (these can be simply aliases... [10:42:42] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3836899 (10Verdy_p) The comity can decide what they want. It was NOT requested to *delete* Moldavian but merge it to Romanian. Dro... [10:57:27] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3836917 (10Strainu) >>! In T169450#3836899, @Verdy_p wrote: > The comity can decide what they want. It was NOT requested to *delet... [11:10:21] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3836937 (10Verdy_p) The consensus was only to join the two communities so they create a joint content without dividing it, Blockin... [11:23:12] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3836955 (10EddieGP) >>! In T169450#3836846, @Verdy_p wrote: > Moldavian: about the renaming/redirect from "mo.wiki(pedia|dictionar... [12:19:35] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3837101 (10Verdy_p) About redirects keeping the title name: as the moldavian past page is most probably written in Cyrillic, forwa... [12:48:35] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3837176 (10Aklapper) Please contact the [[ https://meta.wikimedia.org/wiki/Language_committee | Language Committee ]] when it come... [13:02:34] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3837238 (10EddieGP) >>! In T169450#3837101, @Verdy_p wrote: > So I suggest redirecting user pages and Wiktionary without change of... [13:12:29] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3837258 (10Verdy_p) It's not complicate to add "mo-x-old" to direct to the readonly archive of the "mo" wiki, now that "mo" redire... [13:33:59] I hope DOH doesn't become "normal", it's kind of a tragic answer to what is a real problem [13:34:48] we have workable standards stuff for transport-layer security of requests from user->cache and cache->auth, they just don't have the traction they deserve. [13:36:08] (dns-over-tls for the general case, dnscurve (which needs a few minor usability fixups from its original form) for cache->auth, and dnscrypt (which is an offshoot of dnscurve for client->cache)) [16:02:29] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3837829 (10StevenJ81) As LangCom clerk, I think before this goes much further I should step in and make a couple of things clear.... [16:22:32] bblack: I knew you were going to respond to that :P [16:46:50] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3837929 (10Reedy) All projects are dumped at https://dumps.wikimedia.org/backup-index.html [17:08:41] we are almost ready to make the switch between kafka1018/23 with https://gerrit.wikimedia.org/r/#/c/398255 [17:08:56] kafka producer/consumer wise there shouldn't be any issue, but I am wondering about ipsec [17:10:42] elukey: so basically, you're going to need to break that change up a bit and overlap things with the ipsec switch [17:11:09] I'm not actually sure what controls the ipsec config/deploy on kafka10xx [17:11:13] I'd have to look [17:11:41] bblack: I think it is in the kafka role itself, but I can double check [17:11:51] from the caches' perspective though, we need to first add 1023 to the set of ipsec peers hieradata/common/cache/ipsec/kafka.yaml and puppet that out so that they establish ipsec with the new node [17:12:03] then make the kafka broker switchover, then remove 1018 from the peers list [17:12:31] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3837981 (10StevenJ81) Thanks for that. [17:13:09] bblack: so add 1023 to hieradata/common/cache/ipsec/kafka.yaml first, then apply the big change, then remove 1018 from hieradata/common/cache/ipsec/kafka.yaml [17:13:31] when provisions/configures ipsec on 1023? the same list? [17:14:17] just checked in site.pp we have role(ipsec) among the others for kafka10* [17:14:57] ah I see [17:15:11] so yeah, the breakup is a little more tricky, right? [17:15:27] from my ignorant perspective [17:15:44] https://gerrit.wikimedia.org/r/#/c/398255/3/hieradata/common.yaml [17:15:47] 1) add role(ipsec) to kafka1023 (that runs role system spare now) and add 1023 to hieradata/common/cache/ipsec/kafka.yaml [17:16:04] ^ so this link above, is what tells varnishkafka which brokers to use, right? [17:16:14] yep exactly [17:16:41] 2) apply the kafka roles etc.. to 1023 [17:16:50] 3) remove 1018 leftovers [17:16:57] and kafka1018 is actually hardware-dead, so it's not really seeing any local changes to itself now, right? [17:17:37] correct, but I'll power it down via serial console just to be sure, it should be stuck in early boot now [17:17:40] err, let me rewind a bit, though [17:18:12] https://gerrit.wikimedia.org/r/#/c/398255/3/hieradata/common.yaml is really just hints, right? as soon as kafka1023 joins the cluster, varnishkafka clients might connect to it because it's now in zookeeper dynamic lists? [17:18:58] so yeah [17:19:24] change#1 should be: add kafka1023 to hieradata/common/cache/ipsec/kafka.yaml and add ipsec role to spare::system on kafka1023 [17:19:25] so basically 1023 will takeover the '18' kafka broker id, that was previously held by kafka1018. The first thing that it will do is to figure out what things to replicate from other brokers etc.., but it will not be a leader for any topic/partition [17:20:10] well [17:20:21] we should remove the dead node too, so ipsec gets healthy again for checking [17:20:38] change#1 should be: add kafka1023 to and remove kafka1018 from hieradata/common/cache/ipsec/kafka.yaml and add ipsec role to spare::system on kafka1023 [17:20:39] yep, maybe last step? [17:21:08] and then force agent run on all the caches + live kafka10* and that the ipsec health checks all come back green now [17:21:21] all right [17:21:32] change#2 should be, basically, the rest of your change [17:21:39] and then you're done, I think [17:21:53] what could possibly go wrong? :D [17:22:10] going to make the first change [17:37:25] I was wondering why ipsec config wasn't changing on cp1048/cp1008 [17:37:30] then I realized they are in eqiad [17:37:48] cp3034.esams.wmnet looks better :D [17:39:59] https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=all&type=detail&servicestatustypes=16&hoststatustypes=3&serviceprops=6 [17:40:25] ^ basically all of those should get un-stuck from their 1018 failure (and not have new 1023 failure, once all relevant hosts are puppetized) [17:41:03] bblack: I think that they'll get a 1023 failure for v6, kafka1023 does not include interface::add_ip6_mapped { 'main': } now [17:41:36] Strongswan CRITICAL - ok: 52 connecting: kafka1023_v4,kafka1023_v6 [17:41:42] ^ that's cp3034 [17:42:00] I guess could add interface::add_ip6_mapped early alongside the ipsec role? [17:42:09] and take it away during the second commit that brings it in more-naturally [17:42:10] yep doing it now [17:44:14] https://gerrit.wikimedia.org/r/398295 [17:44:30] kafka1023 currently holds inet6 2620:0:861:104:92b1:1cff:fe06:c9a2/64 [17:44:50] jenkins really doesn't like you today :) [17:47:01] he hates me a lot but I can understand it today :D [17:47:07] uh right [17:47:20] yeah we might have to manually fix up the interface [17:48:36] apply the new patch there first, and then we can delete the old IP [17:49:15] ssh seems stuck to kafka1023, mmm [17:49:40] use "ssh -4" [17:49:44] lack of ipv6 matching DNS [17:50:28] (because the mapped address is already in DNS, and ssh wants to use it, but no add_ip6_mapped yet) [17:50:41] anyways, I'm running the agent there now, will fix it either way [17:50:58] thanks :) [17:51:56] look better now! [17:52:25] puppet agent itself got stuck when the v6 IP flipped out from under it lol [17:52:28] re-running now [17:53:17] seems no need for manual fixup after the agent run twice [17:55:11] heh [17:55:15] so next layer of problem... [17:55:36] is kafka nodes have iptables, which doesn't have a rule to allow ipsec on 1023 yet ... [17:56:23] modules/role/manifests/kafka/analytics/broker.pp: include ::ferm::ipsec_allow [17:56:42] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Language-setup, 10Patch-For-Review, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3838168 (10EddieGP) a:05EddieGP>03None [17:57:21] argh [17:58:08] I think that I need to make Jenkins angry again [18:08:16] almost all the ips secs criticals are gone [18:16:31] ready for the last step: https://gerrit.wikimedia.org/r/#/c/398301/1 [18:18:43] bblack: good to merge from your side? [18:22:02] yeah [18:22:10] as long as ipsec is healthy first, seems so [18:22:21] then we don't leak traffic while switching the new broker on [18:22:37] clearly, the existing puppetization is not well-suited to kafka cluster changes :) [18:22:55] but hopefully it will all go away soon, at least for the kafka case [18:23:25] we are ready to test TLS on cp1008, next step after this one :) [18:51:59] all right kafka1023 is bootstrapping, all good! [18:52:03] thanks a lot for the help! [18:54:53] np! [19:28:01] 10Traffic, 10Operations, 10ops-ulsfo: decom cp40(09|1[078]) - https://phabricator.wikimedia.org/T178815#3838385 (10RobH) a:05RobH>03None [19:28:07] 10Traffic, 10Operations, 10hardware-requests, 10ops-ulsfo, 10Patch-For-Review: Decom cp4005-8,13-16 (8 nodes) - https://phabricator.wikimedia.org/T176366#3838386 (10RobH) a:05RobH>03None [20:11:37] bblack: chachapoly rename https://gerrit.wikimedia.org/r/#/c/398311/ and X-CP-Full-Cipher removal https://gerrit.wikimedia.org/r/#/c/398314/ prepped [20:12:45] varnishxcps (python version) is not using the logs yet, I thought I'd updated it already! It's still parsing XCPS. Will work on that tomorrow [20:13:47] o/ [20:42:28] 10Traffic, 10Operations, 10Reading List Service, 10Reading-Infrastructure-Team-Backlog, 10Patch-For-Review: PUT blocked by Varnish - https://phabricator.wikimedia.org/T182825#3838538 (10Tgr) 05Open>03Resolved a:03Tgr