[07:22:35] 06Traffic: update haproxy to version 2.8.14 - https://phabricator.wikimedia.org/T386751#10562300 (10Vgutierrez) 05Open→03In progress [07:35:28] 06Traffic: update haproxy to version 2.8.14 - https://phabricator.wikimedia.org/T386751#10562306 (10Vgutierrez) [07:37:35] 06Traffic: update haproxy to version 2.8.14 - https://phabricator.wikimedia.org/T386751#10562317 (10Fabfur) a:03Fabfur [08:10:39] 06Traffic: liberica control plane fails to upgrade metrics on administrative depool of a realserver - https://phabricator.wikimedia.org/T386785 (10Vgutierrez) 03NEW [08:18:44] 06Traffic: update haproxy to version 2.8.14 - https://phabricator.wikimedia.org/T386751#10562402 (10Vgutierrez) [08:39:59] 06Traffic: haproxykafka features - https://phabricator.wikimedia.org/T374128#10562452 (10Fabfur) [08:58:12] 06Traffic: update haproxy to version 2.8.14 - https://phabricator.wikimedia.org/T386751#10562505 (10Vgutierrez) [09:36:14] 06Traffic: update haproxy to version 2.8.14 - https://phabricator.wikimedia.org/T386751#10562683 (10Vgutierrez) [10:00:40] 06Traffic: update haproxy to version 2.8.14 - https://phabricator.wikimedia.org/T386751#10562721 (10Vgutierrez) [10:02:16] 06Traffic, 10Prod-Kubernetes, 06serviceops, 07Kubernetes: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#10562723 (10cmooney) Agreed. If the POD (both ends of the veth linking it to the main netns) has a lower MTU, but the K8s host physic... [10:06:20] 06Traffic, 10Prod-Kubernetes, 06serviceops, 07Kubernetes: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#10562736 (10Vgutierrez) Thanks @cmooney, do you have any idea of when could you proceed with this @JMeybohm? [10:35:58] 06Traffic, 07Upstream: HAProxy 2.8.13 crashes after backend server goes away - https://phabricator.wikimedia.org/T386756#10562774 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez as mentioned by wtarreau on https://github.com/haproxy/haproxy/issues/2869#issuecomment-2668087238 it looks like it was a m... [10:38:19] 06Traffic: Evaluate HAProxy 3.1 - https://phabricator.wikimedia.org/T386796 (10Vgutierrez) 03NEW [10:38:23] 06Traffic: Evaluate HAProxy 3.1 - https://phabricator.wikimedia.org/T386796#10562789 (10Vgutierrez) p:05Triage→03Medium [10:39:18] 06Traffic: update haproxy to version 2.8.14 - https://phabricator.wikimedia.org/T386751#10562790 (10Vgutierrez) [10:51:37] 10netops, 06Infrastructure-Foundations: cr2-esams:interface ae1 present under protocol ospf but not configure - https://phabricator.wikimedia.org/T386766#10562845 (10ayounsi) a:03Papaul Thanks! You can remove the now obsolete references from the `ospf` section in https://github.com/wikimedia/operations-homer... [10:57:56] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade network devices to Junos 20+ - https://phabricator.wikimedia.org/T316539#10562858 (10ayounsi) 05Open→03Resolved a:03ayounsi Nop, thanks for the ping. There is now {T364092} [11:13:12] 06Traffic: haproxykafka features - https://phabricator.wikimedia.org/T374128#10562928 (10Fabfur) [11:23:03] 06Traffic, 13Patch-For-Review: tune haproxykafka message_buffer config value - https://phabricator.wikimedia.org/T386753#10562958 (10Fabfur) 05Open→03Resolved The new configuration (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1120924) is deployed everywhere [11:34:48] 06Traffic: Split MessageBuffer configuration for different processing channels - https://phabricator.wikimedia.org/T386801 (10Fabfur) 03NEW [11:53:11] 06Traffic: update haproxy to version 2.8.14 - https://phabricator.wikimedia.org/T386751#10563032 (10Fabfur) [11:53:53] 06Traffic: update haproxy to version 2.8.14 - https://phabricator.wikimedia.org/T386751#10563034 (10Fabfur) [12:20:53] 10netops, 06Infrastructure-Foundations, 06SRE: Gaps in gNMI network statistics in eqiad - https://phabricator.wikimedia.org/T386807 (10cmooney) 03NEW p:05Triage→03Low [12:20:59] 10netops, 06Infrastructure-Foundations, 06SRE: Gaps in gNMI network statistics in eqiad - https://phabricator.wikimedia.org/T386807#10563148 (10cmooney) [12:21:01] 10netops, 06Infrastructure-Foundations, 06SRE: Productionize gnmic network telemetry pipeline - https://phabricator.wikimedia.org/T369384#10563149 (10cmooney) [12:22:35] 06Traffic: update haproxy to version 2.8.14 - https://phabricator.wikimedia.org/T386751#10563150 (10Fabfur) [13:42:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [13:57:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [14:11:01] 06Traffic: update haproxy to version 2.8.14 - https://phabricator.wikimedia.org/T386751#10563512 (10Fabfur) 05In progress→03Resolved [14:50:48] 06Traffic, 10conftool, 13Patch-For-Review: Integrate requestctl haproxy rules into our TLS terminator - https://phabricator.wikimedia.org/T370745#10563670 (10Joe) 05Open→03Resolved @Fabfur I have no idea where we are on this task, specifically regarding the worry chris expressed above. Given the func... [14:52:29] 06Traffic, 10conftool, 13Patch-For-Review: Integrate requestctl haproxy rules into our TLS terminator - https://phabricator.wikimedia.org/T370745#10563682 (10Fabfur) >>! In T370745#10563670, @Joe wrote: > @Fabfur I have no idea where we are on this task, specifically regarding the worry chris expressed a... [14:54:17] 10netops, 06Infrastructure-Foundations, 10observability, 10Prod-Kubernetes, and 3 others: Prevent BGP alerts triggering when K8s host maintenance is being done - https://phabricator.wikimedia.org/T384731#10563685 (10ayounsi) Thanks ! >>! In T384731#10556225, @fgiunchedi wrote: > Since we have to overwrite... [16:00:28] 10netops, 06Infrastructure-Foundations, 06SRE: Gaps in gNMI network statistics in eqiad - https://phabricator.wikimedia.org/T386807#10564100 (10cmooney) [16:26:50] 10netops, 06Infrastructure-Foundations, 10observability, 10Prod-Kubernetes, and 3 others: Prevent BGP alerts triggering when K8s host maintenance is being done - https://phabricator.wikimedia.org/T384731#10564274 (10cmooney) >>! In T384731#10563685, @ayounsi wrote: >>! In T384731#10556225, @fgiunchedi wrot... [16:45:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [16:55:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [16:59:40] 06Traffic, 06SRE: Define an event stream and schema for haproxy_requestctl analytics pipeline ingestion - https://phabricator.wikimedia.org/T383392#10564431 (10Fabfur) >>! In T383392#10560361, @Ottomata wrote: > @Fabfur {T383914} has been deployed, so it should be possible to remove the `meta.domain` field... [18:01:13] 10Domains: URL can use another script - https://phabricator.wikimedia.org/T32766#10564862 (10Pppery) [19:08:39] 10netops, 06Infrastructure-Foundations, 10observability, 10Prod-Kubernetes, and 3 others: Prevent BGP alerts triggering when K8s host maintenance is being done - https://phabricator.wikimedia.org/T384731#10565308 (10cmooney) >>! In T384731#10563685, @ayounsi wrote: > Is it possible to duplicate the metric,... [19:41:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [19:46:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [19:59:37] 06Traffic, 06DC-Ops, 10ops-esams, 10ops-magru, and 2 others: CPU temperature issues in cp hosts - https://phabricator.wikimedia.org/T373993#10565447 (10BCornwall) I don't see any change in performance - The throttling notifications only come sparingly so I doubt we'd see much of a difference until resource...