[01:07:17] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO:Switch refresh diagram - https://phabricator.wikimedia.org/T408511#11625878 (10Papaul) [01:32:53] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO:Switch refresh diagram - https://phabricator.wikimedia.org/T408511#11625923 (10Papaul) @ayounsi I need you input here. et-0/0/1 on cr3/4-ulsfo are connected to asw2-22/23 the goal was to wait until phase 2 to move et-0/0/1 to the... [09:08:27] hey traffic folks, could you please advise on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1239651 ? [10:19:25] ok I simplified it https://gerrit.wikimedia.org/r/c/operations/puppet/+/1239651, if I could get a +1 to move forward that would be grand [10:24:33] vgutierrez: I fixed the trixie pbuilder hook and kicked off a test build of tcp-mss-clamper on build2002, it's working fine now (up to the point where it complains about golang-gitlab-wikimedia-sre-qemutest-dev and golang-github-u-root-u-root-dev missing) [10:25:16] and best to move your things to build2002, we'll retire build2001 in the coming 1-2 months (but after a proper announcement to ops@ ofc) [10:25:43] 10netops, 06Infrastructure-Foundations: ulsfo: upgrade routers (2026) - https://phabricator.wikimedia.org/T416562#11626918 (10ayounsi) a:03Papaul Assigning this to Papaul, to be bundled at some point with the switch replacement (not urgent, but a good opportunity). [10:25:45] 10netops, 06Infrastructure-Foundations: eqsin: upgrade routers (2026) - https://phabricator.wikimedia.org/T416563#11626920 (10ayounsi) a:03Papaul Assigning this to Papaul, to be bundled at some point with the switch replacement (not urgent, but a good opportunity). [10:35:44] 06Traffic, 06cloud-services-team, 10Data-Services, 10Datasets-General-or-Unknown, 13Patch-For-Review: Move dumps.wikimedia.org HTTP service behind CDN edge - https://phabricator.wikimedia.org/T306550#11626966 (10ayounsi) On the netops side the main advantage is to not require real servers to use public I... [11:09:06] hey folks, I am a wee bit blocked on 1239651 getting merged (avoid unnecessary alerts), I would appreciate it [11:10:47] So, I am facing issues with DoH :( [11:11:04] I have it setup on Firefox in India and it shows US servers instead of one hosted in SG [11:26:40] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Trixie switches rp_filter from strict (1) to loose (2) for all interfaces - https://phabricator.wikimedia.org/T417632#11627180 (10JMeybohm) @ayounsi suggested we could remove `linux-sysctl-defaults` from our nodes and copy the useful setting... [11:31:48] moritzm: thx for the heads up [11:34:20] effie: have you addressed scott comments? [11:34:36] effie: those still show up as unresolved and unreplied [11:42:49] let me see if I failed to hit submit [11:43:37] I had updated the code, but didt submit the responses [11:45:05] I will resolve the outstanding one before merging [12:30:48] hiya: hi. thanks for reporting! can you open a phabricator task with more information? [12:31:15] sukhe: I will do after shift. Thank you! [12:31:50] thanks. please add Traffic to it and we will take a look. [12:50:48] is 1239651 good to go and it only needs to let puppet run? [13:35:56] 06Traffic, 06MW-Interfaces-Team, 06ServiceOps new, 07Epic, and 3 others: Epic: Enforce API rate limits (WE5.1.3c) - https://phabricator.wikimedia.org/T412585#11627662 (10daniel) [13:36:32] 06Traffic, 06MW-Interfaces-Team, 06ServiceOps new, 07Epic, and 3 others: Epic: Enforce API rate limits (WE5.1.3c) - https://phabricator.wikimedia.org/T412585#11627666 (10daniel) [13:40:54] 06Traffic, 06MW-Interfaces-Team, 05FY2025-26 KR 5.1, 06MediaWiki-Platform-Team (Radar), 07OKR-Work: haproxy: strip x-wmf-* headers from responses - https://phabricator.wikimedia.org/T417781 (10daniel) 03NEW [13:41:44] 06Traffic, 06MW-Interfaces-Team, 06MediaWiki-Platform-Team (Radar), 07OKR-Work: haproxy: strip x-wmf-* headers from responses - https://phabricator.wikimedia.org/T417781#11627694 (10daniel) [13:45:14] effie: from pybal's PoV yes [13:46:03] 06Traffic, 06serviceops, 07Epic, 05FY2025-26 KR 5.1, 07OKR-Work: Log rate limits from rest-gateway in webrequests - https://phabricator.wikimedia.org/T414349#11627718 (10daniel) I just filed {T417780}, now I'm wondering if the two tasks should be combined, and if they are the same. Do we need a separate... [13:59:21] 06Traffic, 06serviceops, 07Epic, 05FY2025-26 KR 5.1, 07OKR-Work: Log rate limits from rest-gateway in webrequests - https://phabricator.wikimedia.org/T414349#11627779 (10Clement_Goubert) The `x-requestctl` I'm proposing would put it in the place we look for in Turnilo to understand what rate limit rules... [14:17:19] effie: in theory yes. if you want, you can also disable puppet on A:lvs and then enable on lvs1020 or 2014, merge and check and then proceed [14:17:23] let us know if you need any help [14:29:26] 06Traffic, 06SRE: Anycast ns[01].wikimedia.org for IPv4 - https://phabricator.wikimedia.org/T366193#11627887 (10ssingh) Thanks for the comment! Some more thoughts, based on the previous discussions and recent observations: - In our own traffic per netflow data for the month of January 2026, we see that ns2 ge... [14:35:52] 06Traffic, 06SRE: Anycast ns[01].wikimedia.org for IPv4 - https://phabricator.wikimedia.org/T366193#11627934 (10ssingh) TL;DR: > [bblack] To me, the more-important meta-point is that we get all our public authdns IPs (IPv4 + IPv6) switched to Anycast-able (not site-specific) IPs, and that they're deployed in... [14:41:35] sukhe: if there are concerns, I would appreciate it if someone from traffic merges it [14:46:18] effie: it should be as easy as running PCC against lvs-low-traffic [14:46:32] as long as it's a noop for pybal config file you're golden [14:51:06] 06Traffic, 10Prod-Kubernetes, 06ServiceOps new, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#11628022 (10JMeybohm) >>! In T352956#11623492, @JMeybohm wrote: > @ayounsi / @cmooney do you see any issues... [14:54:58] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11628047 (10Vgutierrez) [16:50:37] 06Traffic, 10Liberica, 13Patch-For-Review: liberica-fp doesn't error/refuse to start if the detected MAC Address for the gateway is invalid - https://phabricator.wikimedia.org/T417306#11628542 (10Vgutierrez) 05Open→03Resolved fixed by: - Retrying if we can't fetch the MAC address - Reporting the conf... [17:00:46] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: FY2526 Q3 ulsfo: switch refresh - https://phabricator.wikimedia.org/T408510#11628604 (10RobH) [17:07:08] 06Traffic, 07OKR-Work, 06Test Kitchen (Experiment Platform Sprint 19): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11628647 (10ssingh) >>! In T407570#11601202, @Sfaci wrote: > Hi @ssingh! > Just a friendly reminder of what we... [17:17:47] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO:Switch refresh diagram - https://phabricator.wikimedia.org/T408511#11628686 (10Papaul) Created ticket Case Order #01144222 for initial racking and wiring of the new Nokia switches. [18:33:58] 06Traffic, 07OKR-Work, 06Test Kitchen (Experiment Platform Sprint 19): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11628938 (10dr0ptp4kt) Hi all - just finished some meetings ( CC @JVanderhoop-WMF @BBlack @Vgutierrez @KReid-WM... [19:33:41] 06Traffic, 06ServiceOps new: Test Nokia switches BGP config for k8s workers - https://phabricator.wikimedia.org/T417817 (10Raine) 03NEW [19:49:09] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11629471 (10BCornwall) [23:38:58] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 2 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11630302 (10Krinkle) >>! In T414805#11623347, @Joe wrote: >>>! In T414805#11612457, @Krinkle wrote: >> >> @MatthewVernon Based on di...