[06:56:55] 06Traffic: DNSSec / wikidough issues on doh3006 - https://phabricator.wikimedia.org/T440830 (10Jelto) 03NEW [08:05:25] 06Traffic: DNSSec / wikidough issues on doh3006 - https://phabricator.wikimedia.org/T440830#12420426 (10Fabfur) FYI: @ssingh depooled doh3006 on Oct 11 @ 20.12UTC [09:31:52] 06Traffic: DNSSec / wikidough issues on doh3006 - https://phabricator.wikimedia.org/T440830#12420930 (10cmooney) doh3005 seems ok fwiw ` cathal@officepc:~$ dig +nsid +dnssec +https www.ietf.org @wikimedia-dns.org ; <<>> DiG 9.20.29-1~deb13u1-Debian <<>> +nsid +dnssec +https www.ietf.org @wikimedia-dns.org ; (2... [09:47:07] 06Traffic: DNSSec / wikidough issues on doh3006 - https://phabricator.wikimedia.org/T440830#12420954 (10dkertesz) I'm not familiar with these systems so take this with a big grain of salt: it seems that pdns only knows about the old key (`rec_control get-tas`) and is //not// loading `/etc/powerdns/recursor.lua`,... [09:51:02] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps, 06SRE, 13Patch-For-Review: Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439713#12420972 (10MoritzMuehlenhoff) >>! In T439713#12416592, @fnegri wrote: > I think this task can be resolved, no references... [10:04:27] 06Traffic: DNSSec / wikidough issues on doh3006 - https://phabricator.wikimedia.org/T440830#12421059 (10cmooney) >>! In T440830#12420954, @dkertesz wrote: > I'm not familiar with these systems so take this with a big grain of salt: it seems that pdns only knows about the old key (`rec_control get-tas`) and is //... [10:27:41] dkertesz: hey thanks for the review on that patch :) [10:27:50] topranks: hello, thakns for fixing it :) [10:27:52] and hello - thanks for the feedback on the issue [10:28:02] and to dkertesz for suggesting the fix! [10:28:05] sukhe: are you not off and what the hell time is it over there hahaha [10:28:13] haha, it's all good, and yeah I will be off soon [10:28:16] it's just 6:30 AM [10:28:23] well you're around now - I was going to say we need to roll this out [10:28:30] topranks: hey np! I took the chance to learn more about this system :) [10:28:37] root zone TTL is less than an hour on doh3005 for example - after that it will also be broken [10:28:50] so I think we need to merge that before all of them break [10:29:07] yeah I am not being pedantic about this but just thinking if we should make this conditional in the recursor module [10:29:54] tbh I don't really think it's an issue to load that file regardless (even if dnssec is disabled), however the recursor.lua file is not present on the dns boxes so I wrapped it in a conditional [10:30:02] but I see your point, basically off and process no validate will not do DNSSEC anyway so we should be ok [10:30:11] I don't see a reason we would have another variable controlling it - other than if dnssec is enabled or not [10:30:16] btw that's all it does right now [10:30:41] https://www.irccloud.com/pastebin/BVoWRLnq/ [10:30:52] btw 4.8.8 which we're running doesn't ship with the new key, but likely more recent versions do... it's possible that upgrading to trixie would also indirectly fix the issue [10:31:12] dkertesz: yeah, until the next key-signing key rollover [10:31:13] (not that I suggest that as a fix :D ) [10:31:39] topranks: yep looks good, I guess when we do 5.x, we can remove this and set dnssec.trustanchors. instead [10:31:42] ok [10:31:42] I think what's broken now is we are relying on the in-built trust anchors, when we should use the ones the "dns-root-data" package is managing which will update over time [10:31:45] +1ing [10:32:15] yeah I guess the gap was that I was always suspected we used that. which is when I saw that news, I was like bleh, doesn't affect us :P [10:32:16] sukhe: yeah I think so, we can definitely review over time but just wanted to flag there is a bigger problem than one box incoming [10:32:33] yep sounds good. +1. [10:33:01] pdns-rec doesn't automatically restart on doh* [10:33:06] (intentionally) [10:33:17] so we will need to restart that to pick up the changes [10:33:52] oh ok good info I didn't realise [10:35:18] topranks: I know you love pressing buttons, so all yours but let me know if you need anything [10:35:19] moritz just warned me to hold off on the patch so I am going to revert the merge [10:35:24] the patch won't work [10:35:28] moritzm: oh why? [10:35:36] it patched the legacy config mechanism, not the current config file [10:35:47] I tested it on doh3006, it definitely works [10:35:56] moritzm: which current config you mean? [10:36:01] we need to fix recursor.yml.erb, not recursor.conf.erb [10:36:19] moritzm: no, so wikidough is still using pdns-rec 4.x [10:36:28] dns* hosts are using 5.x which uses yml but which does not do dnssec [10:36:38] dnsXXX are using yaml, but dohXXX are using .conf if that's what you mean [10:36:56] https://phabricator.wikimedia.org/P96667 [10:37:15] sukhe: but all roles use profile::dns::recursor::use_new_pdns_cfg: trie [10:37:17] sukhe: but all roles use profile::dns::recursor::use_new_pdns_cfg: true [10:37:20] moritzm: I get where you're coming from, the doh boxes seeem to be using the old conf file version though [10:37:27] this is unrelated to v4 vs v5? [10:37:50] topranks: the config file is the same, but the template is a different one? [10:37:53] yep but only dnsbox does, and that's dnsXXX [10:38:02] modules/profile/manifests/dns/recursor.pp: Boolean $use_new_pdns_cfg = lookup('profile::dns::recursor::use_new_pdns_cfg', {'default_value' => false}), [10:38:15] hieradata/role/common/dnsbox.yaml:profile::dns::recursor::use_new_pdns_cfg: true <-- dnsXXX [10:38:19] pcc shows the change for the doh boxes if you check the patch [10:38:26] ah, right. [10:39:02] got misled by the implicit class default [10:39:29] yeah it's primarily because of how the dnsrecursor module is shared by so much stuff [10:39:34] doh, dns, cloud dns-rec hosts [10:39:39] so yes, the patch is fine and let's proceed [10:39:44] yeah no worries, honestly I was a bit confused I worked back from what was on the box, the yaml vs conf did confuse me [10:40:00] I assumed dnsbox (which is an awful generic term) also applied to doh* [10:40:17] but turns out it's the role for the main dns* nodes only [10:40:19] where's the fun in that? :D [10:40:47] cloud seems to be using the new conf file / yaml version, so I assume this patch won't affect it? [10:40:59] topranks: correct [10:41:07] new config only uses YAML [10:41:10] there is no recursor.lua file in /etc/powerdns/ on the cloudservices nodes, so it would fail if tehy were [10:41:11] ok [10:41:55] and once you are done, cookbook sre.dns.roll-restart-reboot-wikimedia-dns --alias wikidough --reason "pdns-rec change" restat_daemons should take care of the restarting [10:42:06] (letting you do it since again you love pressing buttons :P) [10:42:53] plus it's good, more wikidough maintainers \m/ [10:43:41] sukhe: ok, should I wait for puppet to run across the board and then do that? or will I force a puppet run first with cumin? [10:43:58] forcing it better I guess, given what you said [10:44:04] and no harm in doing so, so A:wikidough [10:45:25] looks like it works anyway: https://phabricator.wikimedia.org/P96668 [10:45:34] \m/ [10:50:55] moritzm: yeah that's a fair point on A:dnsbox. as you may recall, once upon a time we had dns-rec and dns-auth separate [10:51:02] and so we combined them into one role which led to dnsbox [10:51:16] but it does not include wikidough since we weren't even sure wikidough would be around [10:51:57] (that was 2020 :) [10:52:34] well the DNS hosts are "boxes" and the DoH are VMs... but yeah it is kind of confusing [10:53:04] I remembered shortly after and I'm extremely happy we no longer have separate dns auth and rec :-) [10:54:44] I also checked cloudservices and their use of pdns-rec has been migrated to v5/trixie, so the patch by Cathal should be enough [10:56:44] topranks: thanks for taking care of it with dkertesz. see you tomorrow <3 [10:57:37] sukhe: yep, enjoy your day off! [10:58:31] moritzm: yep, and they appear to have dnssec validation switched off too so shouldn't have any fallout due to the ksk rollover [11:11:48] 06Traffic, 13Patch-For-Review: DNSSec / wikidough issues on doh3006 - https://phabricator.wikimedia.org/T440830#12421219 (10cmooney) 05Open→03Resolved a:03cmooney Ok I think this should be sorted now. Root cause was that we were relying on the in-built trust anchors powerdns-recursor ships with, rat... [11:13:00] dkertesz: I closed that task just now but I just remembered dns3006 still has puppet disabled and is depooled. I think we can undo that at any point but no crazy rush it can also wait until Su khbir is back tomorrow [11:24:43] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859 (10cmooney) 03NEW p:05Triage→03Medium [11:24:51] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12421273 (10cmooney) [11:24:56] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: eqiad row A/B switch refresh configuration - https://phabricator.wikimedia.org/T439535#12421274 (10cmooney) [11:44:20] topranks: good call, thank you! I'll take a look (or note it down for tomorrow) [11:47:35] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Support channelized / breakout interfaces on SR-Linux - https://phabricator.wikimedia.org/T439537#12421368 (10cmooney) 05Open→03Resolved Closing this now, the patch has been merged to support this in our automation, and some validat... [11:52:04] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12421401 (10cmooney) @ssingh @Vgutierrez please advise if this is the correct approach to be taking here. I know we have made a lot... [12:09:24] topranks: it appears that when the cookbook restarts pdns and dnsdist it also restarts bird, which re-pools the server; it's also serving traffic according to the metrics, and puppet is enabled... so I think it's all good! [12:10:32] dkertesz: heh ok TIL thanks for the info. there are probably scenarios that would have caught us out but all ok in this case [13:26:32] 10netops, 06Infrastructure-Foundations: Equinix Ashburn IXP: replace 2x10G with 1x100G - https://phabricator.wikimedia.org/T440876 (10ayounsi) 03NEW p:05Triage→03Low [13:57:27] 10netops, 06Infrastructure-Foundations: Equinix Ashburn IXP: replace 2x10G with 1x100G - https://phabricator.wikimedia.org/T440876#12421915 (10ayounsi) [14:28:19] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-esams, 06SRE: esams router and switch maint - https://phabricator.wikimedia.org/T440349#12422047 (10cmooney) p:05Triage→03Medium [14:54:51] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06ServiceOps, and 2 others: Add missing wikikube workers to conftool-data - https://phabricator.wikimedia.org/T420729#12422177 (10JMeybohm) 05Open→03Resolved [15:11:09] FIRING: LVSHighRX: Excessive RX traffic on lvs3008:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs3008 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [15:14:43] FIRING: [6x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [15:16:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs3008:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs3008 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [15:19:43] FIRING: [10x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [15:29:43] RESOLVED: [10x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [15:39:25] FIRING: SystemdUnitFailed: wmf_auto_restart_varnish-frontend-fetcherr.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:39:25] FIRING: SystemdUnitFailed: wmf_auto_restart_varnish-frontend-fetcherr.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:39:25] FIRING: SystemdUnitFailed: wmf_auto_restart_varnish-frontend-fetcherr.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed