[07:26:22] 06Traffic, 06DBA, 06Infrastructure-Foundations: Move orchestrator (dborch) to private ipaddrs + CDN - https://phabricator.wikimedia.org/T317179#11915792 (10MoritzMuehlenhoff) dborch1003 is currently unreachable, possibly due to broken Puppet config? If this is not actively being worked on ATM, let's set the... [08:02:11] 06Traffic, 06DBA, 06Infrastructure-Foundations: Move orchestrator (dborch) to private ipaddrs + CDN - https://phabricator.wikimedia.org/T317179#11915907 (10FCeratto-WMF) @MoritzMuehlenhoff dborch1003 is powered down while not in use pending the CDN configuration. It's currently using: ` grep nftables hierad... [08:12:17] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 5 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11915999 (10Aklapper) [08:25:38] 06Traffic, 13Patch-For-Review: Test HAProxy 3.2 with AWS-LC libraries - https://phabricator.wikimedia.org/T419825#11916040 (10Fabfur) [11:00:40] 06Traffic, 13Patch-For-Review: Test HAProxy 3.2 with AWS-LC libraries - https://phabricator.wikimedia.org/T419825#11916809 (10Fabfur) [11:52:32] 06Traffic, 10ConfirmEdit (CAPTCHA extension), 05Bot detection and mitigation (WE4.2 hCaptcha editing trial), 06Product Safety and Integrity (Sprint lily-of-the-valley (May 4 - May 22)): [hCaptcha] CORS error on jawiki/enwiki Special:CreateAccount (fails ... - https://phabricator.wikimedia.org/T423039#11917000 [11:52:45] 06Traffic, 10ConfirmEdit (CAPTCHA extension), 10hCaptcha, 05Bot detection and mitigation (WE4.10 hCaptcha), and 2 others: hCaptcha: Failed to load resource: Origin (domain) is not allowed by Access-Control-Allow-Origin - https://phabricator.wikimedia.org/T426178#11917003 (10kostajh) [12:48:29] FIRING: HAProxyRestarted: HAProxy server restarted on cp7001:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/gQblbjtnk/haproxy-drilldown?orgId=1&var-site=magru%20prometheus/ops&var-instance=cp7001&viewPanel=10 - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [12:49:46] ^^ it's me [12:53:29] RESOLVED: HAProxyRestarted: HAProxy server restarted on cp7001:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/gQblbjtnk/haproxy-drilldown?orgId=1&var-site=magru%20prometheus/ops&var-instance=cp7001&viewPanel=10 - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [13:05:08] 06Traffic, 06SRE, 13Patch-For-Review: TCP FastOpen not working since at least December 2025 - https://phabricator.wikimedia.org/T415454#11917265 (10SLyngshede-WMF) For testing I suggest rolling out to e.g. MAGRU first. We can then test that cookies get set with ` $ curl -I -4 connect-to en.wikipedia.org:4... [13:29:33] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#11917426 (10ayounsi) [13:30:11] 10netops, 06Traffic, 06Infrastructure-Foundations: 2026 Junos upgrade - https://phabricator.wikimedia.org/T416444#11917431 (10ayounsi) [13:30:13] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#11917430 (10ayounsi) [13:35:32] 10netops, 06Infrastructure-Foundations: codfw: rack A2 maintenance - https://phabricator.wikimedia.org/T426199 (10ayounsi) 03NEW p:05Triage→03Medium [13:36:01] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#11917492 (10ayounsi) [14:00:08] 06Traffic, 10ConfirmEdit (CAPTCHA extension), 10hCaptcha, 05Bot detection and mitigation (WE4.10 hCaptcha), and 2 others: hCaptcha: Failed to load resource: Origin (domain) is not allowed by Access-Control-Allow-Origin - https://phabricator.wikimedia.org/T426178#11917614 (10kostajh) p:05Unbreak!→03High... [14:12:01] 06Traffic: Test HAProxy 3.2 with AWS-LC libraries - https://phabricator.wikimedia.org/T419825#11917678 (10Fabfur) [14:21:43] 10netops, 06Infrastructure-Foundations: codfw: rack A2 maintenance - https://phabricator.wikimedia.org/T426199#11917759 (10ayounsi) [14:22:10] 10netops, 06Infrastructure-Foundations: codfw: rack A2 maintenance - https://phabricator.wikimedia.org/T426199#11917776 (10ayounsi) [14:22:49] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#11917781 (10cmooney) [14:23:00] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#11917782 (10cmooney) [14:23:47] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#11917784 (10cmooney) [14:24:42] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#11917793 (10cmooney) [14:30:13] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#11917850 (10cmooney) In terms of the Nokia configuration for the ports connecting to the CRs set them up like this to create the two needed sub-interfaces... [14:32:36] 10netops, 06Infrastructure-Foundations, 06ServiceOps new: codfw: rack A2 maintenance - https://phabricator.wikimedia.org/T426199#11917886 (10ayounsi) [14:34:09] 10netops, 06Infrastructure-Foundations, 06ServiceOps new: codfw: rack A2 maintenance - https://phabricator.wikimedia.org/T426199#11917903 (10jcrespo) backup2015 is part of the media backup hosts, I can stop media backups for codfw before the maintenance on my own. [14:40:39] 10netops, 06Infrastructure-Foundations, 06ServiceOps new: codfw: rack A2 maintenance - https://phabricator.wikimedia.org/T426199#11917931 (10jijiki) [15:15:34] fabfur: everything looks the same on the host level but the disk I/O is quite lower o.O https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=cp7001&var-datasource=000000026&var-cluster=cache_text&from=2026-05-13T09:52:18.081Z&to=2026-05-13T15:13:17.445Z&timezone=utc [15:15:39] specifically https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=cp7001&var-datasource=000000026&var-cluster=cache_text&from=2026-05-13T09:52:18.081Z&to=2026-05-13T15:13:17.445Z&timezone=utc&viewPanel=panel-35 [15:16:40] don't know if this is related to awslc specifically (it shouldn't if I have to guess) but I've seen happening in the past with haproxy/haproxykafka restart [15:16:50] probably too early to tell either way [15:20:22] noted [15:34:29] FIRING: HAProxyRestarted: HAProxy server restarted on cp7009:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/gQblbjtnk/haproxy-drilldown?orgId=1&var-site=magru%20prometheus/ops&var-instance=cp7009&viewPanel=10 - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [15:38:38] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#11918235 (10cmooney) Overall the other info in this task makes sense to me. I think we can do all the vlan renames in advance. So when we set up the swi... [15:39:13] 06Traffic: Test HAProxy 3.2 with AWS-LC libraries - https://phabricator.wikimedia.org/T419825#11918240 (10Fabfur) [15:39:29] RESOLVED: HAProxyRestarted: HAProxy server restarted on cp7009:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/gQblbjtnk/haproxy-drilldown?orgId=1&var-site=magru%20prometheus/ops&var-instance=cp7009&viewPanel=10 - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [15:45:24] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#11918255 (10cmooney) [16:38:59] 06Traffic: Purge dhcpcd-base from traffic hosts - https://phabricator.wikimedia.org/T426224 (10BCornwall) 03NEW [16:39:13] 06Traffic: Purge dhcpcd-base from traffic hosts - https://phabricator.wikimedia.org/T426224#11918523 (10BCornwall) p:05Triage→03Low [16:52:57] 06Traffic: Purge dhcpcd-base from traffic hosts - https://phabricator.wikimedia.org/T426224#11918598 (10BCornwall) Purged it from ncmonitor1001 and cp7001, will do the rest later out of an abundance of caution. [16:58:31] 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-04-24 - 2026-05-15): Create depool hiera keys for cirrussearch hosts - https://phabricator.wikimedia.org/T426228 (10bking) 03NEW [16:59:03] 10netops, 06Infrastructure-Foundations, 06ServiceOps new: codfw: rack A2 maintenance - https://phabricator.wikimedia.org/T426199#11918618 (10bking) @ayounsi Sorry for the trouble, confirming that the `depool` and `repool` commands are enough for `cirrussearch` hosts. [19:15:53] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 5 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11919194 (10Ladsgroup)