[06:59:09] FIRING: [8x] LVSHighCPU: The host lvs6001:9100 has at least its CPU 1 saturated - https://bit.ly/wmf-lvscpu - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs6001 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighCPU [07:04:09] RESOLVED: [8x] LVSHighCPU: The host lvs6001:9100 has at least its CPU 1 saturated - https://bit.ly/wmf-lvscpu - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs6001 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighCPU [08:10:30] 06Traffic, 06collaboration-services, 10MinT, 10LPL Essential (LPL Essential 2025 Feb-Mar), 13Patch-For-Review: MinT: Fails to download models/files from peopleweb.discovery.wmnet - https://phabricator.wikimedia.org/T383750#10556111 (10Nikerabbit) p:05High→03Medium [08:11:12] 06Traffic, 06collaboration-services, 10MinT, 10LPL Essential (LPL Essential 2025 Feb-Mar), 13Patch-For-Review: MinT: Fails to download models/files from peopleweb.discovery.wmnet - https://phabricator.wikimedia.org/T383750#10556113 (10Nikerabbit) [09:16:49] 10netops, 06Infrastructure-Foundations, 10observability, 10Prod-Kubernetes, and 3 others: Prevent BGP alerts triggering when K8s host maintenance is being done - https://phabricator.wikimedia.org/T384731#10556218 (10JMeybohm) From lunch discussion in Atlanta: It would be ideal if we could create a recordin... [09:31:46] 10netops, 06Infrastructure-Foundations, 10observability, 10Prod-Kubernetes, and 3 others: Prevent BGP alerts triggering when K8s host maintenance is being done - https://phabricator.wikimedia.org/T384731#10556225 (10fgiunchedi) Since we have to overwrite `instance` with the host instead of the router, that... [11:29:20] 06Traffic, 06SRE: Define an event stream and schema for haproxy_requestctl analytics pipeline ingestion - https://phabricator.wikimedia.org/T383392#10556727 (10Fabfur) 05Open→03Resolved [14:45:46] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [14:48:13] 10netops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 06serviceops: WikiKube clusters close to exhausting Calico IPPool allocations - https://phabricator.wikimedia.org/T375845#10557253 (10akosiaris) [14:55:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [16:02:50] 06Traffic, 06SRE: Anycast ns1.wikimedia.org - https://phabricator.wikimedia.org/T366193#10557396 (10cmooney) >>! In T366193#9851085, @cmooney wrote: > - Most major resolvers/dns providers appear to be 'smart' and pick the lowest-latency server (as per [[ https://datatracker.ietf.org/doc/html/rfc4697#section-2.... [17:01:22] 06Traffic: Set CPU affinity for haproxykafka process - https://phabricator.wikimedia.org/T378758#10557520 (10Fabfur) 05Open→03Resolved Removed override from cp4037 (on cp3066 this has been already removed on 05/11/2024) Metrics showed no benefits from this settings [21:51:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [22:01:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX