[11:32:40] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: CloudVPS: IPv6 in codfw1dev - https://phabricator.wikimedia.org/T245495#10185344 (10taavi) [11:32:47] 10netops, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: Cloud IPv6 subnets - https://phabricator.wikimedia.org/T187929#10185345 (10taavi) [11:33:09] 10netops, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: netbox: create IPv6 entries for Cloud VPS - https://phabricator.wikimedia.org/T374712#10185346 (10taavi) [11:33:29] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: openstack: work out IPv6 and designate integration - https://phabricator.wikimedia.org/T374715#10185348 (10taavi) [11:33:34] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: cloudgw: add support and enable IPv6 - https://phabricator.wikimedia.org/T374716#10185349 (10taavi) [11:33:41] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: cloudsw: codfw: enable IPv6 - https://phabricator.wikimedia.org/T374713#10185350 (10taavi) [11:33:49] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847#10185351 (10taavi) [12:22:37] 10netops, 10Cloud-VPS, 06Infrastructure-Foundations, 10cloud-services-team (FY2024/2025-Q1-Q2): cloud: edge network suffers downtime if one cloudsw is down - https://phabricator.wikimedia.org/T375259#10185394 (10taavi) [12:45:24] 10netops, 06cloud-services-team, 10Data-Services, 06Infrastructure-Foundations: clouddb: evaluate moving them into cloud-private - https://phabricator.wikimedia.org/T357543#10185443 (10taavi) [20:29:50] FIRING: [2x] PyBalBGPUnstable: PyBal BGP sessions on instance lvs5005 are failing - https://wikitech.wikimedia.org/wiki/PyBal#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPyBalBGPUnstable [20:34:49] FIRING: [3x] PyBalBGPUnstable: PyBal BGP sessions on instance lvs5004 are failing - https://wikitech.wikimedia.org/wiki/PyBal#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPyBalBGPUnstable [21:10:30] 10netops, 06Infrastructure-Foundations, 06SRE: cr2-eqsin disk failure Sept 2024 - https://phabricator.wikimedia.org/T375961 (10cmooney) 03NEW p:05Triage→03High [21:17:00] 10netops, 06Infrastructure-Foundations, 06SRE: cr2-eqsin disk failure Sept 2024 - https://phabricator.wikimedia.org/T375961#10185783 (10cmooney) Actually looking at the output in more detail BGP to the LVS servers / PyBal is down. ` Peer AS InPkt OutPkt OutQ Flaps Last Up/Dw... [21:54:50] RESOLVED: [3x] PyBalBGPUnstable: PyBal BGP sessions on instance lvs5004 are failing - https://wikitech.wikimedia.org/wiki/PyBal#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPyBalBGPUnstable [22:09:18] 10netops, 06Infrastructure-Foundations, 06SRE: cr2-eqsin disk failure Sept 2024 - https://phabricator.wikimedia.org/T375961#10185804 (10cmooney) The obvious thing I didn't at first spot was the config on the router was seriously out of date. The PyBal group had the old lvs500[1-3] configured, which have lon...