[07:21:07] 06Traffic: Observability for ephemeral ports on cp servers - https://phabricator.wikimedia.org/T440164#12399047 (10dkertesz) @BCornwall I must have not made it clear enough, but this ticket is not related to the port clashes, it's more a "since we're talking about ephemeral ports...". My rationale is that epheme... [07:49:10] 06Traffic, 10HaproxyKafka: Support HAProxy json logging - https://phabricator.wikimedia.org/T440268 (10Fabfur) 03NEW [07:51:23] 06Traffic, 10HaproxyKafka: Support HAProxy json logging - https://phabricator.wikimedia.org/T440268#12399155 (10Fabfur) p:05Triage→03Low [07:52:31] 06Traffic, 10HaproxyKafka: Introduce JSON input support in HaproxyKafka - https://phabricator.wikimedia.org/T440270 (10Fabfur) 03NEW [09:03:13] 06Traffic: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667#12399518 (10dkertesz) [09:07:02] 06Traffic: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667#12399529 (10dkertesz) 05In progress→03Resolved all done; double checked running `dpkg -l haproxy` with cumin on `A:cp`: ` ii haproxy 3.2.25-1~bpo13+1 amd64 fast and reliable load balancing reverse... [10:59:43] 06Traffic, 06Collaboration-Services, 06SRE Observability, 10Wikimedia-Mailing-lists, and 2 others: Consider disabling paging for lists.wikimedia.org - https://phabricator.wikimedia.org/T436049#12399966 (10hnowlan) 05Stalled→03Resolved a:03hnowlan Alert is live [11:15:21] dkertesz: FYI, I'm puppet-merging your "varnish: fix race condition in flaky test" patch along [11:15:44] doh... moritzm thanks, I forgot about it! [11:16:58] you've only really started into the WMF SRE job the first time you forget to puppet-merge :-) [11:19:42] need to mark that on my calendar :D [13:13:02] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: WMCS switch refresh 2026 (C8 & D5) - https://phabricator.wikimedia.org/T440305 (10cmooney) 03NEW p:05Triage→03Medium [13:14:14] 06Traffic, 10HaproxyKafka, 13Patch-For-Review: Support HAProxy json logging - https://phabricator.wikimedia.org/T440268#12400495 (10Fabfur) [13:34:09] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Datacenter-Switchover: Investigate high saturation in codfw - https://phabricator.wikimedia.org/T438896#12400701 (10Gehel) [13:35:58] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: WMCS switch refresh 2026 (C8 & D5) - https://phabricator.wikimedia.org/T440305#12400724 (10cmooney) [13:38:32] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: WMCS switch refresh 2026 - rack and cable new switches - https://phabricator.wikimedia.org/T440313 (10cmooney) 03NEW p:05Triage→03Medium [13:38:48] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: WMCS switch refresh 2026 - rack new switches - https://phabricator.wikimedia.org/T440313#12400750 (10cmooney) [13:43:20] https://wikitech.wikimedia.org/wiki/LVS#Configure_the_load_balancers tells me to give a headsup, would now be an okay time to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1352041 ? [13:47:03] yeah, let us know if you want to run any of these steps [13:50:29] ok, I'll merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1352041 in 5 mins and then run the steps from https://wikitech.wikimedia.org/wiki/LVS#Configure_the_load_balancers [14:08:21] elukey: is https://pki.discovery.wmnet/bundles/network_devices.pem being down expected? [14:08:52] vgutierrez: yeah I am fixing it with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1352109 [14:14:09] should be fixed in 5 mins, aaand after that it will be with https :) [14:14:14] no more vhost port 80 [14:20:58] should be fixed [14:34:09] elukey: <3 [15:04:53] FYI, ldap-ro-test deployed in lvs_setup step and seems all fine [15:23:48] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-esams, 06SRE: esams router and switch maint - https://phabricator.wikimedia.org/T440349 (10RobH) 03NEW [15:28:56] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-esams, 06SRE: esams router and switch maint - https://phabricator.wikimedia.org/T440349#12401819 (10RobH) [15:33:41] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-esams, 06SRE: esams router and switch maint - https://phabricator.wikimedia.org/T440349#12401861 (10RobH) [15:34:46] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-esams, 06SRE: esams router and switch maint - https://phabricator.wikimedia.org/T440349#12401863 (10RobH) [15:43:27] 06Traffic: Observability for ephemeral ports on cp servers - https://phabricator.wikimedia.org/T440164#12401907 (10BCornwall) It's 100 lines of maintenance, testing, and then there's puppetization. IMO this juice is not worth the squeeze, and work on this would be more effective as contributions upstream to node... [16:00:58] 06Traffic, 06Infrastructure-Foundations, 06SRE, 07LDAP: Add a new LVS ldap-ro-test service - https://phabricator.wikimedia.org/T439730#12402103 (10MoritzMuehlenhoff) ldap-ro-test has been rolled out to the LVS servers (step: lvs_setup) [17:53:09] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: WMCS switch refresh 2026 - rack new switches - https://phabricator.wikimedia.org/T440313#12402958 (10Jclark-ctr) These rails were used to replace the broken ones from the A/B install. [19:29:12] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad A/B switch cabling documentation - https://phabricator.wikimedia.org/T418018#12403321 (10Jclark-ctr) @cmooney @RobH Finished A8 LSW, spine management, and SCS cabling. Updated the SCS port labels and Google Sheet. Adjusted the... [20:23:23] 10netops, 06Infrastructure-Foundations: eqiad: upgrade routers (2026) - https://phabricator.wikimedia.org/T417873#12403481 (10Papaul) @ayounsi i can take it. [20:24:59] 10netops, 06Infrastructure-Foundations, 06SRE: InboundInterfaceErrors alerts firing for Nokia switches on v25.10.1 - https://phabricator.wikimedia.org/T412733#12403488 (10Papaul) We are waiting to upgrade the Nokia swatches in codfw and we can resolve this task.