[00:09:46] 06Traffic, 10Prod-Kubernetes, 06serviceops, 07Kubernetes, 13Patch-For-Review: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10201349 (10ssingh) Continuing on the above: I was wondering if we should even worry about setting `dont-query` to `!10.0.0.0/8` or negating any other pri... [01:33:40] FIRING: [2x] VarnishHighThreadCount: Varnish's thread count on cp5027:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [01:34:09] FIRING: [8x] LVSHighCPU: The host lvs5005:9100 has at least its CPU 0 saturated - https://bit.ly/wmf-lvscpu - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs5005 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighCPU [01:38:40] FIRING: [8x] VarnishHighThreadCount: Varnish's thread count on cp5025:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [01:39:09] RESOLVED: [8x] LVSHighCPU: The host lvs5005:9100 has at least its CPU 0 saturated - https://bit.ly/wmf-lvscpu - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs5005 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighCPU [01:43:40] FIRING: [9x] VarnishHighThreadCount: Varnish's thread count on cp5025:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [01:58:40] FIRING: [9x] VarnishHighThreadCount: Varnish's thread count on cp5025:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [02:03:40] FIRING: [9x] VarnishHighThreadCount: Varnish's thread count on cp5025:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [02:18:40] FIRING: [13x] VarnishHighThreadCount: Varnish's thread count on cp5026:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [02:23:40] FIRING: [11x] VarnishHighThreadCount: Varnish's thread count on cp5026:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [02:38:40] RESOLVED: [6x] VarnishHighThreadCount: Varnish's thread count on cp5026:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [03:05:09] FIRING: [8x] LVSHighCPU: The host lvs5005:9100 has at least its CPU 0 saturated - https://bit.ly/wmf-lvscpu - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs5005 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighCPU [03:10:09] RESOLVED: [8x] LVSHighCPU: The host lvs5005:9100 has at least its CPU 0 saturated - https://bit.ly/wmf-lvscpu - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs5005 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighCPU [07:08:27] 06Traffic, 10Prod-Kubernetes, 06serviceops, 07Kubernetes, 13Patch-For-Review: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10201601 (10cmooney) >>! In T344171#10201349, @ssingh wrote: > Continuing on the above: I was wondering if we should even worry about setting `dont-query`... [07:40:56] hey folks! [07:41:28] I had a chat with Faidon about stream.wikimedia.org (eventstreams), and IIUC every 5 mins the clients are all disconnected [07:53:53] vgutierrez: not sure if we discussed this before or not [07:54:28] in terms of qos on our network it would be great if Liberica could set the DSCP/TOS field on IPIP encapsulated packets it sends to the same value as on the inner packet it received [07:54:38] Katran seems to support this: [07:54:39] https://github.com/facebookincubator/katran/blob/main/katran/lib/bpf/balancer_consts.h#L303 [07:54:53] although a quick search isn't revealing to me how one can enable it [08:05:15] well... that's support for copying the DSCP value, not for setting it [08:05:29] and it's enabled just like that, via a macro on compilation time [08:06:32] My terminology probably wasn't precise enough [08:06:39] yeah we just want to copy it if we can [08:06:52] yup, no problem with that [08:06:54] is this something we can enable do you think? [08:06:56] ok great! [08:06:59] dunno about IPVS though [08:07:02] huh that was easier than I thought [08:07:14] I'll see if I can see any options for that [08:07:30] the framework isn't used much right now so we can probably live without it in IPVS if needs be [08:08:35] how is that managed on the linux kernel networking stack? [08:08:41] for non-IPIP encap IPVS it will be ok, as original header remains the same minus the Ethernet addr [08:09:02] on the non-lvs hosts we set DSCP with iptables/nftables [08:09:47] most hosts only have a default rule marking as "normal" for now [08:09:50] cmooney@dns1004:~$ sudo iptables -L POSTROUTING -v --line -n -t mangle [08:09:50] Chain POSTROUTING (policy ACCEPT 372M packets, 66G bytes) [08:09:50] num pkts bytes target prot opt in out source destination [08:09:50] 1 372M 66G DSCP 0 -- * * 0.0.0.0/0 0.0.0.0/0 DSCP set 0x00 [08:10:00] all high-traffic load balancers are using IPIP encapsulation at the moment [08:10:15] of course you can't do that on cp hosts cause iptables isn't available [08:12:28] I don't see a lot of tunning available for linux kernel encapsulation [08:12:33] yeah on those we gotta classify on the network side which isn't ideal [08:12:37] (cp hosts) [08:13:01] but biggest use for marking traffic high or low prio (diff than normal) would be for internal flows, not user-bound [08:13:08] so we can live with it I think [08:13:12] topranks: also... https://github.com/facebookincubator/katran/blob/ca38bf872ac0b9720f7e712c7136299dec31285f/katran/lib/bpf/balancer_consts.h#L303 impacts forwarded traffic from users [08:13:25] healthchecks won't be impacted by that [08:13:46] as those are encapsulated using the good old linux kernel networking stack [08:13:49] the healthchecks are done through normal kernel connections? [08:14:04] yeah gotcha makes sense, and that var is for the ebpf program / encap [08:14:49] healthcheck forwarding is done by an eBPF program, but it only matches healthcheck traffic and forwards it to the IPIP encapsulation device targetting a specific realserver [08:15:17] hmm ok [08:15:28] I see in the "ip tunnel" man there there is this, but strangely v6 only [08:15:33] dscp inherit [08:15:33] (only IPv6 tunnels) Inherit DS field between inner and outer header. [08:24:16] topranks: kernel source code could explain that [08:24:59] ip6_tnl_xmit gets the outer dsfield value as a parameter [08:25:11] https://github.com/torvalds/linux/blob/0c559323bbaabee7346c12e74b497e283aaafef5/net/ipv6/ip6_tunnel.c#L1069 [08:26:38] nothing like that for ipv4: https://github.com/torvalds/linux/blob/0c559323bbaabee7346c12e74b497e283aaafef5/net/ipv4/ipip.c#L282 [08:28:20] TOS is "hardcoded" like this: nla_put_u8(skb, IFLA_IPTUN_TOS, parm->iph.tos) [08:29:09] (the third parameter of nla_put_u8 is the value BTW: `int nla_put_u8 (struct nl_msg *msg, int attrtype, uint8_t value)` [08:33:10] so in there `parm` is `struct ip_tunnel_parm_kern *parm` [08:33:37] and apparently that's "/* Kernel-side variant of ip_tunnel_parm */" [08:34:31] so it looks to me that for IPv4 you could set the TOS to an arbitrary value but the kernel implementation doesn't have any way of inheriting it from the inner packet [08:35:45] thanks for the info! [08:35:55] yeah looking through it that looks to be the case :( [08:36:34] I guess it is what it is. Perhaps will turn into an incentive for services to move to Liberica [08:37:28] well.. first stage of liberica will use IPVS as the forwarding plane [08:37:33] though the health-check is a complication there :( [08:37:37] yeah [08:37:38] oh yeah. ah sure [08:37:49] healthchecking will keep using the kernel stack [08:38:08] of course we could set the TOS value there [08:38:36] but we should sync those values manually rather than just having the inner value inherited [08:39:00] yeah I think that is achievable, we'd need to be able to define the value in the service definition [08:39:52] hmm it won't be by service but by load balancer? [08:40:03] we aren't setting tunnel devices per service [08:42:08] actually load balancers don't have any IPIP device configured at the moment [08:43:58] yeah, just looking on lvs1013, there is no 'tos inherit' flag for ipip (that exists for other tunnel types) [08:43:59] only the receiving side (realservers) have ipip0/ipip60 devices [08:44:06] you could mangle the outgoing packet with iptables maybe [08:44:18] gotcha [08:44:32] hmmmm this is weird [08:44:47] https://www.irccloud.com/pastebin/SZgHCrQp/ [08:45:06] TX packets == 0 (as expected) but 3527 TX errors? [08:45:17] that's on a realserver? [08:45:21] cp6001 [08:45:24] hmm [08:45:35] something tried to use ipip60 as an outbound device? [08:45:45] could be some odd protocol trying to send packets on every int? but fails on this cos it's only configured in a way that allows rx? [08:46:10] lldp or something maybe, but I think that works at L2 so shouldn't be trying to use a non-Ethernet int [08:46:31] we also have that on ganeti guests like ncredir6001 [08:46:41] oh.. lldp is also running in those [08:50:21] this is only on the ipv6 ints? [08:50:30] yep [08:50:47] could be router solitications or something like that [08:52:08] not sure what this value means: [08:52:09] net.ipv6.conf.ipip60.router_solicitations = -1 [08:52:21] man page says: [08:52:27] https://www.irccloud.com/pastebin/mzZTSg1k/ [08:59:39] they seem to increment about every 50 mins [08:59:52] topranks: -1 == inifinte? [08:59:55] *infinite [08:59:58] aka never stop trying [09:00:11] https://w.wiki/BR5b [09:00:21] https://github.com/torvalds/linux/blob/0c559323bbaabee7346c12e74b497e283aaafef5/net/ipv6/addrconf.c#L4039 [09:01:10] hmm ok. I'm not 100% if that setting necessarily enables them however, could be a red herring [09:01:23] I'd expect they'd be more frequent than in the graph if that was what they were [09:01:59] yeah... given traffic is dropped we cannot capture it with tcpdump [09:02:06] chaning that query to increase: [09:02:07] https://w.wiki/BR5d [09:02:10] two packets every time [09:02:19] and yeah can't get with tcpdump I think [09:02:34] but I think we could dump them with pwru [09:03:30] what is this magic ??? [09:03:50] looking closer at the graph it actually seems to be trying to do something for like a whole minute or so, then stops [09:05:22] actually scrap that, goes up by 1 each time [09:05:22] https://w.wiki/BR5q [09:06:43] so it would be fair to assume that it won't be IP6IP6 traffic? [09:08:46] sorry I don't get you... my guess would be it's some traffic from the ipip60 int's link-local IP [09:10:19] so targeting dst host ff02::1 would be a valid pcap-filter? [09:11:58] or src host fe80::f41c:daff:fe42:643d for ncredir6001 [09:12:02] it could be to any of those tbh [09:12:12] i.e. ff02::1, ff02::2 [09:12:19] I think capturing eveyrting on the int would be best [09:12:29] and if you need a filter use that src host [09:12:32] that's kinda expensive with pwru [09:12:37] hence the filter :) [09:12:42] well there is only 1 packet every 50 minutes [09:12:47] but I'm not sure how it works [09:13:04] 1 TX packet [09:13:14] RX packets are slightly higher [09:13:24] oh yeah you don't want to get them [09:13:48] looking up pwru on google just shows me a picture of Machu Picchu :D [09:13:58] https://github.com/cilium/pwru [09:14:03] yeah the src host sounds like the way to go [09:14:32] nice, will read up, suspected it was some ebpf tracing thing [09:17:44] yeah.. pretty useful to see what's going on under the hood [09:17:58] especially if you're mangling packets with eBPF [09:18:54] heh [09:18:56] yeah it's pretty cool [09:19:12] have it running on my laptop here, nice and easy to work with :) [09:19:29] I've used it quite a bit while debugging stuff like tcp-mss-clamper [09:20:03] cool, looks very useful to really get to grips with the kernel network flow [09:20:36] yep... given it's able to track traffic via skb it's able to follow NATs and so on [09:21:15] that's --filter-track-skb [09:21:51] so you can filter by dst 1.1.1.1 and if a NAT rule changes the destination to 2.2.2.2 you still get that output [09:22:00] ok nice, so even if the packet changes and filter rule no longer matches [09:22:02] yeah nice [09:28:28] so around ~11:50 CEST I should have some output on pwru [09:28:40] lol ok nice :) [09:29:10] topranks: look how you derailed my whole morning lol [09:29:16] proper nerd snipping [09:32:51] sorry man :) [09:33:18] you noticed the v6 int stats all on your own though - I'm innocent on that one! [09:53:04] yeah.... I have a weird ability to find weird behaviors in production systems [09:58:18] it's a gift. super-power even :) [09:58:24] did it turn up anything yet? [10:00:38] 06Traffic: Study the viability of using pki.goog (Google Trust Services) as a 2nd ACME CA - https://phabricator.wikimedia.org/T376459 (10Vgutierrez) 03NEW [10:02:34] topranks: yes [10:02:57] https://www.irccloud.com/pastebin/dhdW7GrK/ [10:03:30] topranks: it looks like router discovery traffic [10:04:37] 06Traffic: Test acme-chief ability to use pki.goog - https://phabricator.wikimedia.org/T376460 (10Vgutierrez) 03NEW [10:04:48] haha, for once in my life I was right! [10:06:55] vgutierrez: I suspect disabling this might help: net.ipv6.conf.ipip60.autoconf = 1 [10:08:48] not 100% sure though [10:09:10] why not setting net.ipv6.conf.ipip60.router_solicitations to 0? [10:09:38] we still need an IPv6 address on ipip60, if we turn the autoconf to 0 that would become a manual step [10:09:43] sorry - just re-read the man page on that one [10:09:54] yeah net.ipv6.conf.ipip60.router_solicitations=0 is probably what we need [10:10:41] I was assuming with autoconf off it just disabled SLAAC for a gloabl IP, but you may well be right it could also disable generating the link local [10:52:30] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: dns/netbox: integrate PTR support for 2a02:ec80:a100::/48 - https://phabricator.wikimedia.org/T376462 (10aborrero) 03NEW [11:04:05] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: dns/netbox: integrate PTR support for 2a02:ec80:a100::/48 - https://phabricator.wikimedia.org/T376462#10202145 (10cmooney) This patch covers the delegation for the openstack-managed ranges, I think it's correct? https://g... [11:14:24] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: dns: integrate PTR support for 2a02:ec80:a100::/48 - https://phabricator.wikimedia.org/T376462#10202157 (10aborrero) [11:16:06] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: dns: integrate PTR support for 2a02:ec80:a100::/48 - https://phabricator.wikimedia.org/T376462#10202161 (10cmooney) To be more clear, you need to make sure these two zones are working on the openstack authdns: ` 0.0.0.0.0.... [11:34:55] 06Traffic: Test acme-chief ability to use pki.goog - https://phabricator.wikimedia.org/T376460#10202195 (10Vgutierrez) after a quick patch: `lang=diff vgutierrez@carrot:~/gitlab.wikimedia.org/sre/acme-chief$ git diff acme_chief/acme_requests.py diff --git a/acme_chief/acme_requests.py b/acme_chief/acme_requests... [11:43:17] 06Traffic: Test acme-chief ability to use pki.goog - https://phabricator.wikimedia.org/T376460#10202203 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez On a second attempt after wiping remaining challenges on the DNS server using acme-chief-designate-tidyup.service, acme-chief issued both certificates... [11:54:48] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, and 2 others: dns: integrate PTR support for 2a02:ec80:a100::/48 - https://phabricator.wikimedia.org/T376462#10202237 (10aborrero) before and after merging the tofu-infra patch above: `lang=shell-session arturo@nostromo:~ $ dig SO... [12:00:14] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, and 2 others: dns: integrate PTR support for 2a02:ec80:a100::/48 - https://phabricator.wikimedia.org/T376462#10202238 (10aborrero) 05Open→03In progress p:05Triage→03Medium [13:50:42] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: codfw:frack:servers migration task - https://phabricator.wikimedia.org/T375151#10202535 (10Papaul) [14:27:15] 06Traffic: Get a WMF/SRE/Traffic GCP account - https://phabricator.wikimedia.org/T376477 (10Vgutierrez) 03NEW [20:49:40] 06Traffic, 10Thumbor: investigate ThumbnailRender volume 2024-09-20 til 2024-10-04 (thumbnail, thumbor) - https://phabricator.wikimedia.org/T376509 (10jeremyb) 03NEW [22:16:54] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade Management routers to 23.4R2-S2 - https://phabricator.wikimedia.org/T369504#10204075 (10Papaul) [22:30:59] 10Domains, 06Traffic, 06SRE, 13Patch-For-Review: Acquire enwp.org - https://phabricator.wikimedia.org/T332220#10204094 (10BCornwall) Okay, the patch has approval and will be merged if/when we get the domain into our MarkMonitor account - otherwise automation (ncmonitor) will be unhappy and try to remove it... [22:55:29] 10Wikimedia-Apache-configuration, 06SRE, 06Traffic-Icebox, 13Patch-Needs-Improvement, 10Wiki-Setup (Delete / Redirect): redirect sco.wiktionary.org/wiki/(.*?) -> sco.wikipedia.org/wiki/Define:$1 - https://phabricator.wikimedia.org/T249648#10204155 (10BCornwall) @Dzahn You had hesitations on the Gerrit CR... [23:45:43] 10Wikimedia-Apache-configuration, 06SRE, 06Traffic-Icebox, 13Patch-Needs-Improvement, 10Wiki-Setup (Delete / Redirect): redirect sco.wiktionary.org/wiki/(.*?) -> sco.wikipedia.org/wiki/Define:$1 - https://phabricator.wikimedia.org/T249648#10204219 (10Pppery) Although Alemannic already does that... [23:49:24] 10Wikimedia-Apache-configuration, 06SRE, 06Traffic-Icebox, 13Patch-Needs-Improvement, 10Wiki-Setup (Delete / Redirect): redirect sco.wiktionary.org/wiki/(.*?) -> sco.wikipedia.org/wiki/Define:$1 - https://phabricator.wikimedia.org/T249648#10204217 (10Pppery) ... but it's worth pointing out that bar.wikti...