[12:45:04] 10Traffic, 10Beta-Cluster-Infrastructure, 10Operations, 10Release-Engineering-Team-TODO: Beta cluster seems to be extremely slow for logged in user during page navigation - https://phabricator.wikimedia.org/T267435 (10ema) >>! In T267435#6609895, @thcipriani wrote: > @ema I noticed you'd done some work on... [13:14:34] 10Traffic, 10Desktop Improvements, 10Operations, 10Product-Infrastructure-Team-Backlog, and 4 others: Connection closed while downloading PDF of articles - https://phabricator.wikimedia.org/T266373 (10CDanis) Zero NEL reports of `http.response.invalid.content_length_mismatch` on PDFs since 2020-11-18T13:28... [13:15:14] 10netops, 10Operations, 10ops-codfw: Switch on rack C7 in codfw is down - https://phabricator.wikimedia.org/T267865 (10Gehel) [13:58:32] 10Traffic, 10Operations, 10serviceops, 10Platform Team Workboards (Green): MW REST API should be routed to api_appserver MW cluster - https://phabricator.wikimedia.org/T268043 (10Pchelolo) 05Open→03Resolved a:03Pchelolo Tutum, now it seems to work: ` curl -i https://en.wikipedia.org/w/rest.php/v1/pa... [14:14:56] 10Traffic, 10Desktop Improvements, 10Operations, 10Product-Infrastructure-Team-Backlog, and 4 others: Connection closed while downloading PDF of articles - https://phabricator.wikimedia.org/T266373 (10Emdosis) Headers and Footers are still missing and the font is still Liberation instead of Georgia.. Here... [14:16:26] 10Traffic, 10Desktop Improvements, 10Operations, 10Product-Infrastructure-Team-Backlog, and 4 others: Connection closed while downloading PDF of articles - https://phabricator.wikimedia.org/T266373 (10CDanis) @Emdosis This task relates to very low-level issues stopping the PDFs from being delivered at all.... [14:17:33] 10Traffic, 10Desktop Improvements, 10Operations, 10Product-Infrastructure-Team-Backlog, and 4 others: Connection closed while downloading PDF of articles - https://phabricator.wikimedia.org/T266373 (10Emdosis) @CDanis Ok. [14:56:07] 10Traffic, 10Operations: TCP traffic increase for DNS over TLS breached a low limit for max open files on authdns1001/2001 - https://phabricator.wikimedia.org/T266746 (10BBlack) Various related gdnsd fixes were deployed to production with version 3.4.1 of upstream. What remains here is possibly spinning up so... [14:57:58] bblack: hi, I am rebuilding the CI container for dnslint :] [14:59:41] thanks! [15:39:57] 10Traffic, 10Operations, 10Technical-blog-posts: 2nd part of blog post series: the evolution of Wikimedia's Content Delivery Network - https://phabricator.wikimedia.org/T266857 (10ema) >>! In T266857#6628308, @srodlund wrote: > Take a look at the suggestions and accept or decline what you like. Done, thank... [16:17:43] 10netops, 10Operations, 10ops-codfw: ripe-atlas-codfw is down - https://phabricator.wikimedia.org/T267714 (10CDanis) a:05Papaul→03faidon Today we tried powercycling the anchor while I was watching on serial console. It didn't output a thing. As far as I can tell, we need replacement hardware. [16:18:25] 10Traffic, 10Operations: Broken package state on cp4032 - https://phabricator.wikimedia.org/T268243 (10ema) [17:02:28] 10Traffic, 10Operations, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10ema) >>! In T264398#6596427, @Gilles wrote: > We should probably wait to have at least a full week of post-warmup data for confirmation. This... [17:53:47] 10Traffic, 10Operations, 10Technical-blog-posts: 2nd part of blog post series: the evolution of Wikimedia's Content Delivery Network - https://phabricator.wikimedia.org/T266857 (10srodlund) Great! I'm going to prepare it today and publish it on Wed 25 November. [17:58:33] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10mforns) Hi @ayounsi, The new data is already in Hive's netflow table. And I'm about to enable Druid loading for the new fie... [18:31:50] 10Traffic, 10Operations, 10Technical-blog-posts: 2nd part of blog post series: the evolution of Wikimedia's Content Delivery Network - https://phabricator.wikimedia.org/T266857 (10srodlund) @ema do you have any preference for a featured image? If not, let me know, and I will pick one. [19:23:30] 10Traffic, 10Beta-Cluster-Infrastructure, 10Operations, 10Release-Engineering-Team-TODO: Beta cluster seems to be extremely slow for logged in user during page navigation - https://phabricator.wikimedia.org/T267435 (10thcipriani) >>! In T267435#6640481, @ema wrote: > I did, but the issue does not seem rela... [19:33:36] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10ayounsi) I don't understand the difference :) Why can't we do the same as T231339#6612105 ? [20:12:44] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10mforns) We could do the same as T231339#6612105 if necessary. However, netflow's datasource in Druid is already big (1.9TB)... [21:05:19] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10mforns) More thoughts... The addition of a field to a data set can have different effects to the data size. 1. If the field... [21:22:04] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10mforns) And still more thoughts :) After playing a bit more with the datasource, I think the data is very granular, it is l...