[04:11:53] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw:frack:rack/install/configuration new switches in rack F5 - https://phabricator.wikimedia.org/T405618#11920464 (10Papaul) [05:03:35] 10netops, 06Infrastructure-Foundations, 06ServiceOps new: codfw: rack A2 maintenance - https://phabricator.wikimedia.org/T426199#11920516 (10Marostegui) @ayounsi @FCeratto-WMF dbprpxy hosts will need a `systemctl reload haproxy` after the maintenance is done, even if they are not active. [09:57:47] 06Traffic, 07Sustainability (Incident Followup): Ensure the pre-repooling checklist includes to restart liberica services whenever realserver IPs has changed - https://phabricator.wikimedia.org/T426299 (10hnowlan) 03NEW [10:04:27] 06Traffic, 07Sustainability (Incident Followup): alert on server_ip NEL field for each DNS-pooled PoP - https://phabricator.wikimedia.org/T426301 (10hnowlan) 03NEW [10:21:02] 10netops, 06Infrastructure-Foundations, 06SRE: Create single Homer BGP group template to cover all variants - https://phabricator.wikimedia.org/T349116#11921022 (10cmooney) 05Open→03Declined Closing this one for now. We do need to look a this, but also we need to review in light of having both Junip... [10:24:42] 10netops, 06Infrastructure-Foundations, 06ServiceOps new: codfw: rack A2 maintenance - https://phabricator.wikimedia.org/T426199#11921066 (10jcrespo) @ayounsi not urgent, but please ping me with a time of start of maintenance when you have it so I can do what I mention above in advance. [10:33:56] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: ASW single-point of failure for LVS VIPs at POPs - https://phabricator.wikimedia.org/T362772#11921091 (10cmooney) 05Open→03Resolved a:03cmooney I'm going to close this one. I think everyone is agreed cross-rack links to have an LVS peer... [10:44:42] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Network telemetry - collect device sub-interface statistics with gnmic - https://phabricator.wikimedia.org/T424683#11921120 (10cmooney) 05Open→03Resolved Ok this is rolled out and working. I have tried to update our dashboards wher... [10:49:19] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Network telemetry - collect device sub-interface statistics with gnmic - https://phabricator.wikimedia.org/T424683#11921125 (10cmooney) {F81386939 width=600} [11:15:41] 10netops, 06Infrastructure-Foundations: Apply egress Source Address Validation on the Wikimedia core routers - https://phabricator.wikimedia.org/T372158#11921238 (10cmooney) This is basically [[ https://datatracker.ietf.org/doc/html/rfc2827 | BCP38 ]] right? >>! In T372158#10097098, @Southparkfan wrote: >>>... [13:04:03] 06Traffic, 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: k8s changes needed to allow article topic (and other future isvcs) to use the kserve v2 inference protocol (and gRPC) - https://phabricator.wikimedia.org/T424049#11921513 (10isarantopoulos) [13:08:12] 06Traffic, 06ServiceOps new, 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: k8s changes needed to allow article topic (and other future isvcs) to use the kserve v2 inference protocol (and gRPC) - https://phabricator.wikimedia.org/T424049#11921528 (10isarantopoulos) [13:16:29] 06Traffic, 10Wikimedia-Site-requests, 13Patch-For-Review: Request throttle exemption of IP addresses for ESEAP Conference 2026 - https://phabricator.wikimedia.org/T426295#11921549 (10Reedy) While the throttle may be helpful for onwiki stuff, depending on amount of traffic, it may start triggering other rate... [13:54:10] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw:frack:rack/install/configuration new switches in rack F5 - https://phabricator.wikimedia.org/T405618#11921702 (10Jhancock.wm) [14:08:22] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw:frack:rack/install/configuration new switches in rack F5 - https://phabricator.wikimedia.org/T405618#11921799 (10Jhancock.wm) [14:08:32] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw:frack:rack/install/configuration new switches in rack F5 - https://phabricator.wikimedia.org/T405618#11921800 (10Jhancock.wm) [14:12:08] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw:frack:rack/install/configuration new switches in rack F5 - https://phabricator.wikimedia.org/T405618#11921818 (10Jhancock.wm) [14:42:53] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: FY2526 Q3 ulsfo: switch refresh - https://phabricator.wikimedia.org/T408510#11921972 (10RobH) [15:15:57] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw:frack:rack/install/configuration new switches in rack F5 - https://phabricator.wikimedia.org/T405618#11922147 (10Jhancock.wm) [16:43:40] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: POPs - free up 2xQSFP ports - https://phabricator.wikimedia.org/T424611#11922700 (10cmooney) @RobH we've disabled the following links as part of the work on this, and will need to remove the current fibres and optic modules before we re-use the port... [17:06:00] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Install new line card in cr2-eqiad slot 0, move card from slot 1 to cr1-eqiad slot 0 and configure - https://phabricator.wikimedia.org/T426343 (10cmooney) 03NEW p:05Triage→03Medium [17:06:38] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Install new line card in cr2-eqiad slot 0, move card from slot 1 to cr1-eqiad slot 0 and configure - https://phabricator.wikimedia.org/T426343#11922831 (10cmooney) [20:12:45] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw:frack:rack/install/configuration new switches in rack F5 - https://phabricator.wikimedia.org/T405618#11923273 (10Papaul) [20:34:59] 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-04-24 - 2026-05-15): Create depool hiera keys for cirrussearch hosts - https://phabricator.wikimedia.org/T426228#11923343 (10bking) 05Open→03In progress p:05Triage→03Medium a:03bking [20:42:06] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw:frack:rack/install/configuration new switches in rack F5 - https://phabricator.wikimedia.org/T405618#11923368 (10Papaul) [22:26:08] 06Traffic, 05Bot detection and mitigation (WE4.10 hCaptcha), 07Documentation, 06Product Safety and Integrity (Sprint lily-of-the-valley (May 4 - May 22)): hcaptcha proxy: update wikitech page - https://phabricator.wikimedia.org/T411131#11923667 (10Dreamy_Jazz) Per the above comment for #psi to add addition... [22:45:43] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw:frack:rack/install/configuration new switches in rack F5 - https://phabricator.wikimedia.org/T405618#11923791 (10Papaul) [22:46:35] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw:frack:rack/install/configuration new switches in rack F5 - https://phabricator.wikimedia.org/T405618#11923792 (10Papaul) 05Open→03Resolved This is complete. thanks to @Jhancock.wm and @Jgreen