[02:41:43] FIRING: HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=ulsfo&var-instance=cp4039&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [02:51:43] RESOLVED: HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=ulsfo&var-instance=cp4039&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [08:34:54] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06ServiceOps, 07Kubernetes: Migrate Wikikube k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420436#12440563 (10JMeybohm) >>! In T420436#12438719, @ssingh wrote: > Sorry for re-opening this folks but it seems like Sophroid is still on IPIP... [08:35:33] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06ServiceOps, 07Kubernetes: Migrate Wikikube k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420436#12440567 (10JMeybohm) 05Open→03Resolved [09:01:55] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12440636 (10Vgutierrez) >>! In T440859#12438737, @ssingh wrote: > Filed T441352 for `inference` and `inference-staging` move and T441... [09:29:27] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 06SRE: lsw1-c1-eqiad: mgmt0 traffic not working after upgrade to SR-Linux 26.7.3 - https://phabricator.wikimedia.org/T440631#12440710 (10ayounsi) Looking a bit more into this, in eqiad we have 3 models: * NETGEAR GS748T - purchased in 2020 * NETGEAR FS750T2... [09:52:06] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 06SRE: lsw1-c1-eqiad: mgmt0 traffic not working after upgrade to SR-Linux 26.7.3 - https://phabricator.wikimedia.org/T440631#12440789 (10cmooney) >>! In T440631#12440709, @ayounsi wrote: > They first try DHCP then default their IP to 192.168.0.239/24 so if... [09:59:10] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 06SRE: lsw1-c1-eqiad: mgmt0 traffic not working after upgrade to SR-Linux 26.7.3 - https://phabricator.wikimedia.org/T440631#12440804 (10cmooney) Actually ignore the above table. Checking the LLDP info on msw1-eqiad only one of the three problematic ports... [10:44:02] 10netops, 06Infrastructure-Foundations, 06SRE: Netbox: modify move-server script to allow for 1G->10G upgrade during process - https://phabricator.wikimedia.org/T440999#12440979 (10cmooney) Sorry @RobH @Jclark-ctr if I wasn't clear here what I had imagined we use this for is to help us get T440696 over the l... [10:53:35] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12441086 (10cmooney) Thanks for the help here guys! It certainly will be easier if we don't have to add them, but at the same time i... [11:22:02] 10netops, 06Infrastructure-Foundations, 06SRE: Create cookbook to make Netbox changes when upgrading server connection speed - https://phabricator.wikimedia.org/T441408 (10cmooney) 03NEW p:05Triage→03Low [12:00:30] 06Traffic, 06SRE Observability, 07Sustainability (Incident Followup): alert on server_ip NEL field for each DNS-pooled PoP - https://phabricator.wikimedia.org/T426301#12441286 (10hnowlan) [12:18:42] 10netops, 06Infrastructure-Foundations, 06SRE: Create cookbook to make Netbox changes when upgrading server connection speed - https://phabricator.wikimedia.org/T441408#12441358 (10cmooney) [13:31:13] 06Traffic: Migrate Sophroid to IPIP - https://phabricator.wikimedia.org/T441429 (10ssingh) 03NEW [13:31:15] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06ServiceOps, 07Kubernetes: Migrate Wikikube k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420436#12441715 (10ssingh) >>! In T420436#12440563, @JMeybohm wrote: >>>! In T420436#12438719, @ssingh wrote: >> Sorry for re-opening this fol... [13:31:48] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12441724 (10ssingh) >>! In T440859#12440636, @Vgutierrez wrote: >>>! In T440859#12438737, @ssingh wrote: >> Filed T441352 for `infere... [13:32:37] 06Traffic: Migrate Sophroid to IPIP - https://phabricator.wikimedia.org/T441429#12441726 (10ssingh) a:03ssingh [14:13:05] 06Traffic: Upgrade to HAProxy 3.2.26 - https://phabricator.wikimedia.org/T441435 (10Vgutierrez) 03NEW [14:22:48] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Datacenter-Switchover: Investigate high saturation in codfw - https://phabricator.wikimedia.org/T438896#12441928 (10CDanis) [14:26:25] 06Traffic, 06ServiceOps: Trial of weighted dnsdisc for mw-web-ro switchover (gradual ro traffic shift) - https://phabricator.wikimedia.org/T441436#12441944 (10CDanis) [14:28:05] 06Traffic, 06ServiceOps: Trial of weighted dnsdisc for mw-web-ro switchover (gradual ro traffic shift) - https://phabricator.wikimedia.org/T441436#12441954 (10CDanis) [14:33:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp2047 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2047 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [14:38:43] RESOLVED: HaproxyKafkaExporterDown: HaproxyKafka on cp2047 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2047 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [15:25:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp2047 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2047 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [15:30:43] RESOLVED: HaproxyKafkaExporterDown: HaproxyKafka on cp2047 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2047 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [15:44:47] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12442303 (10cmooney) [15:46:52] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12442349 (10cmooney) [16:23:10] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12442586 (10cmooney) [16:26:24] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12442591 (10cmooney) [17:00:13] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12442756 (10cmooney) [17:50:49] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12442950 (10cmooney) [17:54:52] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12442981 (10cmooney)