[07:14:59] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12383679 (10cmooney) >>! In T417871#12382617, @Papaul wrote: > I spoke with @ssingh and @BCornwall to see if we can schedule the core routers Junos upgrade next week Wednesday at 9am CT, @ssingh wi... [07:32:56] hello? [07:51:21] hello team would anyone be able to give a thumbs up on this one? https://gerrit.wikimedia.org/r/c/operations/dns/+/1346553 :) [08:00:52] dpogorzelski: I don't see in production the CDN configuration to handle that hostname [08:20:52] you mean this? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1346557 [08:21:17] sorry, forgot to link [08:31:21] dpogorzelski: no caching and a timeout of 10 minutes per request? :( [08:32:09] you can't cache llm responses :P caching of static assets might be ok but there won't be many as it's just a chat web ui :P [08:32:44] the 10 minute timeout might not be needed but downstream uses this setting for response streaming [08:33:02] so this is just to match that setting [08:33:26] btw, this is just for WMF/NDA ldap users [08:33:34] no public access [08:36:21] dpogorzelski: ack, you need to fix the TLS cert on the appservers [08:36:27] liftwing-studio.wm.o needs to be there as well [08:36:39] roger [08:36:53] the DNS change is fine but I'll aprove it after the CDN is able to handle that traffic [08:46:42] 06Traffic: Avoid heap allocations related to prometheus metrics on purged hot path - https://phabricator.wikimedia.org/T439234#12383830 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez [08:46:46] 06Traffic: Reduce heap allocations on purged TCPPurger.Send() - https://phabricator.wikimedia.org/T439254#12383833 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez [08:46:59] 06Traffic: Reduce goroutine churn in purged - https://phabricator.wikimedia.org/T439361#12383836 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez [09:15:21] this is the updated chart change: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1346563 which i can rollout immediately after the dns changes [09:16:07] well not chart itself but it's instance [09:35:55] you don't need the DNS change in place to deploy that [09:55:02] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps, 06SRE, 06tools-platform-team: Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439713#12384184 (10Jelto) The legacy CNAME `url-downloader` URL was removed from wmf-navigator with the patch above. [10:02:24] 06Traffic: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667#12384213 (10dkertesz) [10:17:43] 06Traffic, 06Infrastructure-Foundations, 06SRE, 07LDAP: Add a new LVS ldap-ro-test service - https://phabricator.wikimedia.org/T439730#12384261 (10MoritzMuehlenhoff) [10:18:00] 06Traffic: Refactoring and some other work on purged - https://phabricator.wikimedia.org/T350396#12384262 (10Vgutierrez) [10:35:35] 06Traffic, 10Liberica, 13Patch-For-Review: Provide a cookbook that validates IPIP/IP6IP6 capabilities on a given realserver - https://phabricator.wikimedia.org/T419873#12384345 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez cookbook has been available for a while: ` vgutierrez@cumin1004:~$ sudo -i... [10:41:11] 06Traffic, 10HaproxyKafka: HAProxy sometimes does not apply host normalization - https://phabricator.wikimedia.org/T407796#12384365 (10dkertesz) nothing new to report except that this is still happening, albeit with low frequency; e.g. past 3 months on Turnilo: https://w.wiki/W2j4 [11:31:56] vgutierrez: admin_ng and service changes are synced [11:51:16] 06Traffic, 06Infrastructure-Foundations, 06SRE, 07LDAP: Add a new LVS ldap-ro-test service - https://phabricator.wikimedia.org/T439730#12384563 (10MoritzMuehlenhoff) Initial steps: 1. Patch for conftool-data is at https://gerrit.wikimedia.org/r/1347474 2. No discovery record is used/needed 3. VIPs were cr... [11:52:20] if anyone has a moment: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1347474 (this is for https://phabricator.wikimedia.org/T439730), following the steps from https://wikitech.wikimedia.org/wiki/LVS#Add_a_new_load_balanced_service [12:02:56] 10netops, 06Infrastructure-Foundations, 06SRE: Missing series for BGP session_state from eqiad CRs since upgrade to 23.4R2-S8.7 - https://phabricator.wikimedia.org/T435909#12384669 (10cmooney) [13:06:55] 06Traffic, 10HaproxyKafka: HAProxy sometimes does not apply host normalization - https://phabricator.wikimedia.org/T407796#12384946 (10dkertesz) 05Open→03Resolved a:03dkertesz looks like this ticket is a duplicate of T434766 where they actually implemented a fix (quite recently), so I think we can cl... [14:02:21] 06Traffic: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667#12385278 (10dkertesz) [14:04:10] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps, 06SRE: Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439713#12385299 (10aputhin) [14:20:39] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06ServiceOps, and 2 others: Migrate Wikikube k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420436#12385464 (10Jelto) [14:26:22] 10netops, 06Infrastructure-Foundations: Junos: investigate BGP rib sharding - https://phabricator.wikimedia.org/T320264#12385509 (10ayounsi) [14:26:47] 06Traffic, 06Infrastructure-Foundations, 06SRE, 07LDAP, 13Patch-For-Review: Add a new LVS ldap-ro-test service - https://phabricator.wikimedia.org/T439730#12385512 (10MoritzMuehlenhoff) And: 4. https://gerrit.wikimedia.org/r/c/operations/puppet/+/1347624 to add it to the service catalogue 5. https://ger... [14:27:23] 10netops, 06Infrastructure-Foundations: eqsin: upgrade routers (2026) - https://phabricator.wikimedia.org/T416563#12385519 (10ayounsi) Let's use the upgrade opportunity to enable BGP RIB sharding on cr2-eqsin {T320264} [14:27:37] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12385522 (10ayounsi) Let's use the upgrade opportunity to enable BGP RIB sharding on one of the two routers {T320264} [14:27:58] 10netops, 06Infrastructure-Foundations: eqiad: upgrade routers (2026) - https://phabricator.wikimedia.org/T417873#12385524 (10ayounsi) [14:28:54] 10netops, 06Infrastructure-Foundations: eqiad: upgrade routers (2026) - https://phabricator.wikimedia.org/T417873#12385527 (10ayounsi) [14:29:28] 10netops, 06Infrastructure-Foundations: eqiad: upgrade routers (2026) - https://phabricator.wikimedia.org/T417873#12385529 (10ayounsi) The new JTAC recommended version for the SRX300 is now Junos 24.4R2-S3. [14:31:26] 10netops, 06Infrastructure-Foundations: eqiad: upgrade routers (2026) - https://phabricator.wikimedia.org/T417873#12385550 (10ayounsi) a:05cmooney→03Papaul @papaul, are you (or someone else in DCops) ok to take care of mr1-eqiad? [14:58:46] moritzm: sorry this was missed [14:58:54] we will take up the reviews [15:04:24] cheers [15:13:03] vgutierrez: is https://gerrit.wikimedia.org/r/c/operations/dns/+/1346553 good at this point? [15:13:10] should be the only thing remaining [15:21:36] moritzm: reviewing it now - is there a better bug report than the test issue? [15:22:43] vgutierrez@carrot:~$ curl --connect-to ::$(dig +short dyna.wikimedia.org):443 https://liftwing-studio.wikimedia.org -s |fgrep "" [15:22:43] <vgutierrez> <title>LibreChat [15:22:54] dpogorzelski: ^^ that looks as the expected content? [15:24:19] yea [15:25:15] cheers! [15:33:05] brett: https://phabricator.wikimedia.org/T439730 is for the new LVS service (and the wider work is https://phabricator.wikimedia.org/T331699) [19:19:55] 06Traffic, 10DNS, 06Fundraising-Backlog, 10fundraising-tech-ops, and 4 others: Investigate having combowiki on donate.wikipedia.org - https://phabricator.wikimedia.org/T433108#12387089 (10XenoRyet) We have legal and security sign-off. Jeff and Traffic team have a plan for the implimentation that is relativ...