[03:39:29] <wikibugs>	 06Traffic, 06SRE: TCP FastOpen not working since at least December 2025 - https://phabricator.wikimedia.org/T415454#11837399 (10Naruse_shiroha) Any update on this after one month...?
[07:51:43] <wikibugs>	 06Traffic, 06collaboration-services, 10Gerrit, 06Release-Engineering-Team, and 2 others: gerrit: Adapt timeouts to avoid 502 errors in CI jobs - https://phabricator.wikimedia.org/T421827#11837765 (10ABran-WMF) I've tried increasing `initial_connection_window_size` and `initial_stream_window_size`, but stil...
[08:52:30] <wikibugs>	 06Traffic, 06Commons: No thumbnail after overwriting SVG file: too many requests - https://phabricator.wikimedia.org/T423811#11838013 (10AlexisJazz) Can't reproduce this anymore so might as well close it.
[08:52:59] <wikibugs>	 06Traffic, 06Commons: No thumbnail after overwriting SVG file: too many requests - https://phabricator.wikimedia.org/T423811#11838014 (10AlexisJazz) 05Open→03Resolved
[08:53:01] <wikibugs>	 06Traffic, 10ConfirmEdit (CAPTCHA extension), 072026-user-javascript-incident, 07ContentSecurityPolicy, and 2 others: [hCaptcha] CORS error on jawiki/enwiki Special:CreateAccount (fails to load secure-api.js), but works on mediawikiwiki - https://phabricator.wikimedia.org/T423039#11838015 (10kostajh) > hCa...
[09:33:36] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10observability, 10Prod-Kubernetes, and 4 others: Collect calico network metrics - https://phabricator.wikimedia.org/T423851 (10Blake) 03NEW
[09:34:41] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10observability, 10Prod-Kubernetes, and 4 others: Add calico network alerting - https://phabricator.wikimedia.org/T423852 (10Blake) 03NEW
[10:34:18] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10observability, 10Prod-Kubernetes, and 4 others: Collect calico BGP metrics - https://phabricator.wikimedia.org/T423851#11838312 (10JMeybohm)
[12:25:05] <wikibugs>	 06Traffic, 06collaboration-services, 10Gerrit, 06Release-Engineering-Team, and 2 others: gerrit: Adapt timeouts to avoid 502 errors in CI jobs - https://phabricator.wikimedia.org/T421827#11838692 (10ABran-WMF) I've tried to disable http2 on envoy but it also triggered: https://integration.wikimedia.org/ci/...
[13:29:26] <wikibugs>	 06Traffic, 06collaboration-services, 10Gerrit, 06Release-Engineering-Team, and 2 others: gerrit: Adapt timeouts to avoid 502 errors in CI jobs - https://phabricator.wikimedia.org/T421827#11838973 (10ABran-WMF) After checking Envoy's logs, I'll disable http2 again, Envoy was not fully reloaded for the previ...
[14:21:02] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#11839250 (10ayounsi) p:05Triage→03Medium
[15:41:03] <Amir1>	 hi, is it okay if I deploy this? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1272869?tab=comments like what's the process? just puppet merge or i should disable puppet and do the dance.
[15:42:08] <sukhe>	 hi!
[15:42:25] <sukhe>	 yeah, puppet disable on all hosts (A:cp-text is fine), test on one host, and run on all other
[15:42:28] <sukhe>	 want me to do this dance?
[15:43:40] <Amir1>	 that would be great since I'll probably will do it with shaky hands
[15:45:52] <sukhe>	 you will do great but happy to do it
[15:45:52] <sukhe>	 on it
[15:55:27] <Amir1>	 Thank you <3
[15:58:32] <inflatador>	 y'all aren't having any problems with IPIP on Trixie are you? I just reimaged `cloudelastic1012` to trixie and I'm having some weird problems where I can ping other hosts, but I can't get the host to respond to inbound TCP or UDP so it can't rejoin the cluster
[15:59:51] <sukhe>	 inflatador: that would be unlikely given all cp hosts are on trixie right now
[16:00:03] <sukhe>	 as in, problems with trixie and IPIP, or some combination thereof
[16:00:16] <sukhe>	 inflatador: is it just that host or other hosts as well?
[16:01:06] <inflatador>	 sukhe no, this is the only host that has the problem. With any tcp or udp service, I see the packets hit the interface but it never responds
[16:02:16] <inflatador>	 I've been testing from `relforge1010` which is in the same rack/layer 2 network, same issue 
[16:02:48] <wikibugs>	 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 6 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11840075 (10Aklapper)
[16:04:28] <sukhe>	 Amir1: all done
[16:04:50] <Amir1>	 Amazing. thanks!
[16:06:22] <sukhe>	 inflatador: I can help take a look after lunch; it's unlikely that this is related to IPIP especially if you can't reach the host directly
[16:06:48] <inflatador>	 sukhe sure, it's non urgent. This is a set host and I've depooled all services
[16:08:56] <inflatador>	 One interesting thing is that I changed the hieradata for LVS to match the new service name in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1275435 , and that changed the LVS IP. Not sure how all that's wired together, but it was already broken before so I don't know if this matters much
[17:13:20] <wikibugs>	 06Traffic, 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Data-Platform-SRE (2026-03-27 - 2026-04-17), 13Patch-For-Review: Surge in webrequest validation check - https://phabricator.wikimedia.org/T422030#11840498 (10Ahoelzl) a:03JAllemandou
[17:19:21] <jasmine_>	 inflatador: perhaps same as [0]? ended up needing to just re provision twice (first with legacy) then again with the UEFI default following the trixie upgrade
[17:19:21] <jasmine_>	 [0] - https://phabricator.wikimedia.org/T390861#11822788 
[17:20:44] <inflatador>	 Thanks jasmine_ ! I can get the host to boot but it won't respond to inbound connections on its main services. I just reimaged it to rule out some stuff
[17:22:05] <wikibugs>	 06Traffic, 10SRE-swift-storage: OpenSSL 3.x performance issues - https://phabricator.wikimedia.org/T352744#11840551 (10Ladsgroup) >>! In T352744#9413282, @MoritzMuehlenhoff wrote: >>>! In T352744#9413140, @jhathaway wrote: >> wolfssl is packaged in Debian, so that may be a possible option longer term, https://...
[17:28:56] <wikibugs>	 06Traffic, 10SRE-swift-storage: OpenSSL 3.x performance issues - https://phabricator.wikimedia.org/T352744#11840575 (10ssingh) >>! In T352744#11840551, @Ladsgroup wrote: >>>! In T352744#9413282, @MoritzMuehlenhoff wrote: >>>>! In T352744#9413140, @jhathaway wrote: >>> wolfssl is packaged in Debian, so that may...
[19:22:32] <wikibugs>	 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Revert lvs1017 Mellanox NIC to Broadcom - https://phabricator.wikimedia.org/T421421#11841270 (10wiki_willy) @Jclark-ctr & @VRiley-WMF - can you provide a status on this one?
[19:25:27] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Servers exposing incorrect LLDP info - https://phabricator.wikimedia.org/T250367#11841294 (10elukey) @ayounsi I think that iDRAC 10 hosts don't support the new LLDP code :( T418899#11840735
[19:29:36] <wikibugs>	 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Revert lvs1017 Mellanox NIC to Broadcom - https://phabricator.wikimedia.org/T421421#11841318 (10ssingh)
[19:30:29] <wikibugs>	 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Revert lvs1017 Mellanox NIC to Broadcom - https://phabricator.wikimedia.org/T421421#11841319 (10ssingh) Clarified the scope of work, "Set up lvs1017 with new NIC" is DC Ops and then Traffic is responsible for the other bits in the task ("Promote lvs1017").
[21:56:38] <wikibugs>	 06Traffic, 10Continuous-Integration-Config: Purge frontend cache when publish new coverage report under https://doc.wikimedia.org/cover - https://phabricator.wikimedia.org/T423951#11841694 (10Reedy)
[22:07:53] <wikibugs>	 06Traffic, 06Product Safety and Integrity, 06Security-Team, 072026-user-javascript-incident, and 5 others: Deduplicate CSP between VCL and MediaWiki - https://phabricator.wikimedia.org/T420604#11841739 (10sbassett)
[22:08:09] <wikibugs>	 06Traffic, 06Product Safety and Integrity, 06Security-Team, 072026-user-javascript-incident, and 4 others: Deduplicate CSP between VCL and MediaWiki - https://phabricator.wikimedia.org/T420604#11841740 (10sbassett)
[22:27:38] <wikibugs>	 06Traffic, 06Product Safety and Integrity, 06Security-Team, 072026-user-javascript-incident, and 4 others: Deduplicate CSP between VCL and MediaWiki - https://phabricator.wikimedia.org/T420604#11841836 (10sbassett) Hey @ssingh - We deployed https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/127...
[22:29:49] <wikibugs>	 06Traffic, 06Product Safety and Integrity, 06Security-Team, 072026-user-javascript-incident, and 4 others: Deduplicate CSP between VCL and MediaWiki - https://phabricator.wikimedia.org/T420604#11841847 (10Catrope) Furthermore, I verified that the correct CSP headers appear at https://en.wikipedia.beta.wmcl...