[00:03:43] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11704894 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp4043.ulsfo.wmnet with OS trixie executed with errors: - cp4043 (**FAIL**) - Downtimed on Icinga/Alertmanager - D... [00:10:56] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11704903 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp4040.ulsfo.wmnet with OS trixie completed: - cp4040 (**PASS**) - Removed from Puppet and PuppetDB if present and d... [00:11:50] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11704904 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp4043.ulsfo.wmnet with OS trixie [00:41:50] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11704951 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp4042.ulsfo.wmnet with OS trixie completed: - cp4042 (**WARN**) - Removed from Puppet and PuppetDB if present and d... [00:43:23] RESOLVED: ErrorBudgetBurn: varnish-combined codfw - https://slo.wikimedia.org/?search=varnish-combined - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn [00:46:00] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11704954 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp4044.ulsfo.wmnet with OS trixie [00:46:01] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11704955 (10CDobbins) [01:05:12] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11704962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp4043.ulsfo.wmnet with OS trixie completed: - cp4043 (**PASS**) - Removed from Puppet and PuppetDB if present and d... [01:10:01] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11704963 (10CDobbins) [01:23:02] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11704965 (10BCornwall) [01:41:42] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11704979 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp4044.ulsfo.wmnet with OS trixie completed: - cp4044 (**WARN**) - Downtimed on Icinga/Alertmanager - Disabled Pup... [01:43:26] 06Traffic, 06DC-Ops, 10ops-eqsin, 06SRE: cp5022 is unreachable - https://phabricator.wikimedia.org/T414411#11704980 (10RobH) Tech is running late, their dispatcher called me to let me know. They were set to be onsite at 7AM, but it will now be closer to 10:30AM / 19:30 Pacific [02:51:55] 06Traffic, 06DC-Ops, 10ops-eqsin, 06SRE: cp5022 is unreachable - https://phabricator.wikimedia.org/T414411#11705018 (10RobH) Tech is onsite and performing the hw power distro board swap on cp5022 [03:39:53] FIRING: ErrorBudgetBurn: varnish-combined codfw - https://slo.wikimedia.org/?search=varnish-combined - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn [07:40:08] FIRING: ErrorBudgetBurn: varnish-combined codfw - https://slo.wikimedia.org/?search=varnish-combined - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn [10:12:14] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 2 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11705635 (10MatthewVernon) @Reedy you did the 1.43 backports (at least according to gerrit), can you have a look at this, please? I c... [10:43:13] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 2 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11705753 (10TheDJ) [10:59:02] 06Traffic, 06Data-Platform-SRE (2026-03-06 - 2026-03-27): Prevent HaproxykafkaNoMessages alerts from being generated due to standard maintenance operations - https://phabricator.wikimedia.org/T419829#11705877 (10BTullis) a:03BTullis [11:40:08] FIRING: ErrorBudgetBurn: varnish-combined codfw - https://slo.wikimedia.org/?search=varnish-combined - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn [11:54:02] 06Traffic, 06Data-Platform-SRE (2026-03-06 - 2026-03-27), 13Patch-For-Review: Prevent HaproxykafkaNoMessages alerts from being generated due to standard maintenance operations - https://phabricator.wikimedia.org/T419829#11706094 (10BTullis) I found that the HaproxyKafkaNoMessages alert in the `team-traffic`... [12:44:53] RESOLVED: ErrorBudgetBurn: varnish-combined codfw - https://slo.wikimedia.org/?search=varnish-combined - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn [12:58:50] vgutierrez: FIY, we've removed the rate limit on POSTs for cspreports through the action API, that was the bulk of rate limited POSTs, so the new connection rate should lower significantly [12:59:03] claime: nice :D [12:59:35] We'll try to find a way to make envoy not send a Connection: close on rate limiting POSTs though [12:59:54] claime: my understanding is that you would need to buffer the POST body [13:00:10] vgutierrez: or just erase the header before sending the response back [13:00:18] that's not safe [13:00:30] you need to consume the POST body [13:00:32] Ah :/ [13:00:39] or discard the connection [13:00:58] I have something else to deal with rn, but we can try and discuss it next week maybe [13:01:08] cool [13:45:07] 10netops, 06Infrastructure-Foundations, 06SRE: InboundInterfaceErrors alerts firing for Nokia switches on v25.10.1 - https://phabricator.wikimedia.org/T412733#11706596 (10cmooney) @papaul please tell them to keep the case low as they have not yet fixed it [15:19:01] 06Traffic, 03Hackathon-Northwestern-Europe-2026: Intermittent rate limiting at hackathon-northwestern-europe-2026 - https://phabricator.wikimedia.org/T420011#11707242 (10A_smart_kitten) [15:50:20] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11707402 (10BCornwall) [17:11:50] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11707877 (10BCornwall) [18:12:09] 10netops, 06Infrastructure-Foundations, 06SRE: InboundInterfaceErrors alerts firing for Nokia switches on v25.10.1 - https://phabricator.wikimedia.org/T412733#11708155 (10Papaul) @cmooney yes can do [18:20:55] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11708170 (10BCornwall) [20:56:20] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11708810 (10BCornwall) [21:03:45] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11708821 (10BCornwall)