[15:29:41] topranks and I are going to merge the authdns v6 change. this only specific to the hosts for now, we are not updating the records yet [15:29:51] any pages related to that we will take care (plus topranks is on on-call ;) [15:32:07] to pranks, indeed [15:34:31] topranks: merging on dns7001 (depooled for everything) [15:35:01] I want to test the coupling of the ns1 v4 and v6 service states. basically they are one for all purposes and I guess that's fine like we discussed [15:35:09] but just want to make sure it works as intended [15:35:54] not sure I understand - the v4 and v6 service states? this for the anycast-healthchecker? [15:35:58] yeah [15:36:23] {"dns7001.wikimedia.org": {"weight": 100, "pooled": "no"}, "tags": "dc=magru,cluster=dnsbox,service=authdns-ns2"} [15:36:42] for example, if we say depool authdns-ns2 for a host, we will depool both the v6 and v4 now [15:36:50] clean puppet run [15:36:58] ok yeah gotcha [15:37:05] I think that's probably ok for now? [15:37:19] topranks: tbh I am a bit split but yeah, for now I guess [15:37:58] ok, going to pool it and see if bird starts advertising the v6 [15:38:53] ok [15:39:39] topranks: [15:39:40] define ACAST6_PS_ADVERTISE = [15:39:40] [ [15:39:40] 2001:db8::1/128, [15:39:40] 2a02:ec80:53::1/128 [15:39:53] it seems happy [15:39:53] please check on the switch side :) [15:39:57] https://www.irccloud.com/pastebin/nyMIPF6r/ [15:40:17] * sukhe breathes [15:40:35] check v4s as well, please [15:40:47] will do yeah, just running homer to configure the v6 session [15:40:50] thanks [15:40:53] 10.3.0.5/32, [15:40:56] 10.3.0.1/32 [15:41:04] 198.35.27.27/32, [15:42:10] looks good for both [15:42:16] https://www.irccloud.com/pastebin/y1k3dF6b/ [15:42:42] all hail topranks [15:43:13] sukhe@dns7001:~$ ip -6 route [15:43:14] 2620:0:860:53::1 dev lo proto kernel metric 256 pref medium [15:43:14] 2620:0:861:53::1 dev lo proto kernel metric 256 pref medium [15:43:14] 2a02:ec80:53::1 dev lo proto kernel metric 256 pref medium [15:43:14] 2a02:ec80:700:1::/64 dev eno12399np0 proto kernel metric 256 pref medium [15:43:16] fe80::/64 dev eno12399np0 proto kernel metric 256 pref medium [15:43:18] default via fe80::429e:a402:c78c:6c00 dev eno12399np0 proto ra metric 1024 expires 597sec hoplimit 64 pref medium [15:43:25] https://www.irccloud.com/pastebin/4Adrn7q3/ [15:43:54] phew. ok, checking one more thing, please confirm [15:44:03] https://www.irccloud.com/pastebin/n08G99Zz/ [15:44:45] there should be no 2a02:ec80:53::1/128, or 198.35.27.27/32 [15:44:46] can you confirm? [15:46:37] I will wait for that before moving on [15:47:20] * 10.3.0.1/32 195.200.68.4 64605 I [15:47:21] * 10.3.0.5/32 195.200.68.4 64605 I [15:49:51] topranks: looks good to me [15:50:16] * 2a02:ec80:53::1/128 2a02:ec80:700:1:195:200:68:4 64605 I [15:50:54] sorry I got distracted [15:51:10] what do you mean "there should be no 2a02:ec80:53::1/128, or 198.35.27.27/32" ? [15:51:29] topranks: sorry, I depooled authdns-ns2 and wanted to check if both routes were withdrawn successfully or not [15:51:37] and they were, I checked on asw1-b3 [15:51:50] ah ok sorry [15:52:57] topranks: everything looks good on your end? [15:53:26] yeah all working fine [15:53:52] btw I was gonna make a patch for the PTR entry for the ns2 IP but realised it probably doesn't make sense before you add the AAAAs [15:54:01] yeah [15:54:03] https://phabricator.wikimedia.org/P88509 [15:54:12] if it saves you working it out when it comes to it [15:54:46] topranks: weird thing is that some SREs report connectivity issues to 2a02:ec80:53::1 [15:55:01] basically, can't connect to the v6, though you can and I can [15:55:25] slyngs and volans both, even though they have working v6 connections [15:55:45] are they discussing this elsewhere/ [15:55:47] taavi as well [15:56:00] I pinged them here [15:56:20] * volans here [15:56:34] is this about the glue records? [15:56:37] I see the route via Telefonica at least [15:56:38] no [15:56:44] volans: no, we haven't updated those yet [15:56:52] we are simply advertising the v6 from dns7001 [15:56:57] magru [15:57:10] well, yes glue records are related but sukhe was saying you have a routing problem to 2a02:ec80:53::1 ? [15:57:19] yes can't ping/traceroute [15:57:37] mtr shows it getting to et-0-0-48.asw1-b3-magru.wikimedia.org and ??? [15:57:54] that's odd [15:57:59] I have 1 hop only and then I'm stuck [15:58:11] https://phabricator.wikimedia.org/P88512 [15:58:16] maybe volans and taavi can share their mtrs in private or something [15:58:26] trying mtrs [15:58:28] right so volans is more what I expected - ISP has no route to it - but that's odd, I'm definitely seeinig Telia learning it amongst others [15:58:52] for me: [15:58:54] 13.|-- et-0-0-48.asw1-b3-magru.wikimedia.org 0.0% 10 118.5 117.2 114.9 123.6 3.5 [15:58:58] 14.|-- 2a02:ec80:53::1 0.0% 10 114.5 114.5 114.4 114.6 0.1 [15:59:08] taavi: can you DM me your V6 IP ? [15:59:37] sent to you both my mtr [15:59:43] sent [16:00:24] if you want better mtr options let me know, I just run mtr -6 2a02:ec80:53::1 [16:01:33] topranks: would it help if we just announced this from everywhere and see if things work? [16:01:38] I mean we can do that if we roll this out [16:04:40] mtr to 2a02:ec80:53::1 works fine for me [16:06:20] as a comparison, I can happily ping/mtr 2a02:ec80:600:ed1a::1 [16:07:02] sukhe: that's not really possible with the current way we generate the aggregate routes that get announced into BGP [16:07:11] like it's of course possible but would require quite a lot of changes [16:08:17] topranks: yeah I meant if we roll this out and advertise the IP from everywhere (all DNS hosts), would it help? [16:08:48] we actually might hide the current issue, so actually probably best to try and tease out what is happening [16:08:57] true [16:09:01] fwiw taavi's requests get to dns7001, and it responds, but he is not getting the response [16:09:55] ah wait [16:10:18] mtr to taavi works just fine - sourced from public1-b3-magru IP - but from the ns2 IP it doesn't get anywhere.... [16:10:30] this may be uRPF on cr2 possibly? [16:13:05] is there a unicast address for the same service I could try? [16:13:28] not right now, no. once this is rolled out everywhere, you can try the ns01 unicast [16:13:42] puppet is just disabled on dnsbox other than dns7001 [16:14:10] since I can reach any other service I've tried (text-lb, bastion ssh) on magru over ipv6 just fine [17:36:11] re-enabling puppet on A:dnsbox and rolling out the changes everywhere [17:36:59] * volans watching closely :-P [17:37:58] we get a full suit if this breaks, not even a tshirt [18:00:05] it's ok, we've got everyone trained now. just blame the DNS and everyone will be like "oh, it's always DNS, nothing you can do about that" [18:01:22] let's hope it doesn't turn out to be technically true this time :P [18:32:06] {◕ ◡ ◕} [18:48:13] It seems the ops@ lists config has changed such that "from" no longer reflects the sender address. [18:48:55] This means filters for git@ to puppet-private stop working, but also means Gmail attributes emails to "Taavi via Ops" and "Ori via Ops" etc [18:49:17] afaik no other mailing list is configured this way, but maybe this fixes something else? [18:49:47] it changed Jan 27 it seems [19:11:42] [Listadmins-announce] Change: We updated the listservs to handle DMARC enforcement (mail sent Jan 23) [19:11:46] Krinkle: ^^ [19:13:28] topranks: we are all done on the DNS hosts, puppet enabled everywhere [19:13:47] we can check everything today and maybe tomorrow pursue the glues for ns1. thanks so much for all the help <3 [19:14:05] (that part is on me to be clear, once you have run homer, just let me know) [19:14:31] I am happy to verify it everywhere on the switches/routers [19:14:42] sukhe: fwiw I'm now getting the dns answer from dns6002-auth, as expected [19:14:48] volans: nice :) [19:14:52] good confirmation, thanks [19:16:53] sukhe: ok looks good I've updated routers/switches at the other sites too [19:17:01] too kind! [19:17:16] we are announcing the new /48 everywhere, my own resolution going to Amsterdam [19:17:40] nice [19:28:06] https://lists.wikimedia.org/hyperkitty/list/listadmins-announce@lists.wikimedia.org/thread/C4JEQDKLSC5V6RTRI3QZLFVIRWAVU3NA/ [19:28:11] Thanks apergos [19:28:24] yw [19:34:39] I would have thought there's some way to verify the authenticity through the long signed hash in the headers but I guess that doesn't work due to mailing lists often changing the subject and appending a footer to messages. Hence mailing lists break but a simple email forwarding service works fine? Seems like there's a way to make that work. Eg I use forwardemail.net to forward jquery.com emails to a individual addresses of the team and [19:34:39] that's without munging and without dkim failures [19:35:41] I'm guessing Mailman 3 always modifies something/enough of the message to break that. Or maybe there's a way for this to be resolved for some lists.. [19:35:46] https://dmarcian.com/mailing-lists-dmarc/ [19:36:02] I'm missing something as to why one can work but not the other. [19:42:57] Krinkle: sorry about the breakage, the other option is to preserve the message so that the dkim signing remains valid, unfortunately mailman uses a python library that munges every message, https://gitlab.com/mailman/mailman/-/issues/1079 [19:43:37] I would prefer to go the dkim route as that provides better assurance that the email is truly from the author, but I didn't see a way forward with mailman [19:48:19] https://dmarc.org/wiki/FAQ#I_operate_a_mailing_list_and_I_want_to_interoperate_with_DMARC.2C_what_should_I_do.3F [19:52:34] Yeah so if always munges it today, that's sad. Although subject line prefixing would also be something I imagine people might miss. It's fine for peope that use folders based on mailing list ID headers, but afaik most email clients don't show that by default. You'd have to look at Reply-To to know where it came from. [19:53:06] Sub/unsub is standardised these days so we wouldn't need footers for that per se. [19:53:17] Thx for sharing. [19:54:07] yeah, you can add headers and not break dkim, so ideally the list headers could be added, to allow folks to unsub and filter, but still preserve dkim. [19:54:45] I've been tempted to take a look at the python library, but I also trust the mailman author that it would take quite a bit of work [19:55:08] Yeah, perhaps our grandchildren will have nice things. [19:56:28] I suppose one could think of (public) mailing lists as forum notifications which are naturally wrapped and sent in this way. It's a bit strange for private lists. [19:57:18] for sure [19:57:26] Oh well. I'll see if I can filter puppet-private based on the reply-to. Gmail doesnt support arbitrary header filters like Fastmail does [19:57:59] Or maybe we'll just not hire anyone named "Git" [19:59:10] :), my filter is from:git list:(ops.lists.wikimedia.org) [19:59:21] so I guess I made the same hope