[03:59:17] pretty odd, yeah, these numbers have been much lower than we observed jn October, it's why we're trying to collect them in a few ways to rule out browser and routing oddities [10:22:07] milimetric: thanks for pinging us... eventgate looked ok from ATS PoV BTW: https://grafana.wikimedia.org/goto/uDhweuHvR?orgId=1 [10:22:36] 10% increase on avg in TTFB [11:01:59] wow, that's amazing! [11:11:24] jayme, akosiaris which firewall is in place on the kube workers? [11:11:34] ferm? nothing? [11:16:18] ferm [11:25:59] thx moritzm :D [13:17:08] 10netops, 06Infrastructure-Foundations, 06SRE: cr2-codfw alarm: FPC 5 power is unstable - https://phabricator.wikimedia.org/T416691 (10cmooney) 03NEW p:05Triage→03High [14:03:50] 10netops, 06Infrastructure-Foundations, 06SRE: cr2-codfw alarm: FPC 5 power is unstable - https://phabricator.wikimedia.org/T416691#11591499 (10ayounsi) If we only do `request chassis fpc offline slot 5` it will come back up automatically. Not tested but I think we need to set `cr2-codfw# set chassis fpc 5 p... [14:19:20] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 2 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11591581 (10Ladsgroup) >>! In T414805#11589974, @Tacsipacsi wrote: > #mediaviewer got broken by this. ☹ For example, https://commons.... [14:26:49] 10netops, 06Traffic, 06Infrastructure-Foundations: 2026 Junos upgrade - https://phabricator.wikimedia.org/T416444#11591620 (10ssingh) Thanks for letting us know @ayounsi. Let us the specific dates you have in mind and we can plan around that. [14:27:55] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 2 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11591625 (10Ladsgroup) This image is in a standard size and passes through our rate limit: https://upload.wikimedia.org/wikipedia/com... [15:09:30] 10netops, 06Infrastructure-Foundations, 06SRE: cr2-codfw alarm: FPC 5 power is unstable - https://phabricator.wikimedia.org/T416691#11591784 (10cmooney) >>! In T416691#11591499, @ayounsi wrote: > If we only do `request chassis fpc offline slot 5` it will come back up automatically. Yeah I seen that before a... [15:18:04] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 2 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11591828 (10Ladsgroup) I‌ think this should fix it: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1237517 but I‌ nee... [15:39:45] 10netops, 06Infrastructure-Foundations, 06SRE: Nokia SR-Linux DHCP Relay Bug - https://phabricator.wikimedia.org/T411054#11591967 (10cmooney) 05Open→03Resolved I'm gonna close this one for now. We have not seen a repeat of this since we have adjusted the config to deal with the ARP resolution bug, t... [15:41:08] 10netops, 06Infrastructure-Foundations, 06SRE: Nokia SR-Linux ARP resolution bug on v24.10.x+ - https://phabricator.wikimedia.org/T409178#11591971 (10cmooney) 05Open→03Resolved Closing this. The work-around is working well and we will upgrade the OS version in eqiad over the coming months. [15:47:21] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Decom Asw Switches in Rows C & D - https://phabricator.wikimedia.org/T412525#11592001 (10cmooney) [16:34:05] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 2 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11592156 (10Tacsipacsi) >>! In T414805#11591625, @Ladsgroup wrote: > Can you get the 429 response body? I don’t get 429 anymore. I c... [16:36:56] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 2 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11592158 (10Ladsgroup) I have reverted the rate limit for "medium" browser score before the weekend to reduce disruptions to people.... [17:46:46] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10SRE-tools: Reboot cookbook workflow leaves Puppet disabled - https://phabricator.wikimedia.org/T410944#11592363 (10CDobbins) 05Open→03Resolved [18:01:09] 10netops, 06Infrastructure-Foundations, 06SRE: codfw expansion: configure new Nokia switches in rows E/F - https://phabricator.wikimedia.org/T402590#11592446 (10cmooney) 05Open→03Resolved Things are working ok here now. [23:49:04] 06Traffic, 07OKR-Work, 06Test Kitchen (Experiment Platform Sprint 19): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11593311 (10Sfaci)