[08:11:45] 10Traffic, 10Analytics-Clusters, 10Operations: varnishkafka 1.1.0 CPU usage increase - https://phabricator.wikimedia.org/T264074 (10ema) The function `VUT_Main` is the main loop of VUT programs. The [[ https://github.com/varnishcache/varnish-cache/blob/6d4df3639725bbec6d1657b07867ec44f4ba14f8/lib/libvarnish... [08:21:41] 10netops, 10Operations: cr3-esams linecard diversity issue - https://phabricator.wikimedia.org/T262524 (10ayounsi) Steps to go directly with the most sustainable option. No need for a site depool if done carefully. {F32383060} [] Configure cr2/3:ae0 with `link-speed mixed` [] Disconnect cable #20042 between... [09:18:12] ema: is it okay if I remove this dashboard? https://grafana.wikimedia.org/d/000000612/frontend-responses-nginx-vs-varnish?orgId=1&var-site=All I can move its graphs to another dashboard if you use/need them [09:19:39] Amir1: ah, good catch, that's a blast from the past [09:19:48] please remove it [09:20:08] done [09:20:18] we now have the equivalent data (and more) on https://grafana.wikimedia.org/d/000000479/frontend-traffic?orgId=1&refresh=1m [09:22:25] 10Traffic, 10Operations, 10observability, 10Performance-Team (Radar), 10Sustainability (Incident Followup): Document and/or improve navigation of the various HTTP frontend Grafana dashboards - https://phabricator.wikimedia.org/T253655 (10Ladsgroup) [09:24:28] Moving forward with this: https://grafana.wikimedia.org/d/dh8FDiNWk/varnish-http-errors-datacenters?orgId=1 [09:25:02] https://grafana.wikimedia.org/d/Q985DmNWk/varnish-http-errors-datacenters?orgId=1 [09:25:07] In favor of this [09:25:24] (Should we remove "500" errors, it seems empty) [09:25:33] (not 5xx ones) [09:26:56] 10Traffic, 10Operations, 10observability, 10Performance-Team (Radar), 10Sustainability (Incident Followup): Document and/or improve navigation of the various HTTP frontend Grafana dashboards - https://phabricator.wikimedia.org/T253655 (10Ladsgroup) [09:27:50] Amir1: first time I see varnish-http-errors-datacenters [09:27:51] 10Traffic, 10Maps, 10Operations, 10Product-Infrastructure-Team-Backlog: Limit maps serving to Wikimedia hosted sites only - https://phabricator.wikimedia.org/T261424 (10Elitre) @JMinor @JKatzWMF please give your Go to get this done. Thanks! [09:28:09] haha, this is chaos [09:42:53] 10Traffic, 10Operations, 10observability, 10Performance-Team (Radar), 10Sustainability (Incident Followup): Document and/or improve navigation of the various HTTP frontend Grafana dashboards - https://phabricator.wikimedia.org/T253655 (10Ladsgroup) After talking to @ema, I'm cleaning this up a bit. Delet... [09:51:09] 10netops, 10Operations, 10Patch-For-Review, 10Security, 10User-jbond: Review default ferm INPUT policy - https://phabricator.wikimedia.org/T264888 (10JMeybohm) p:05Triage→03Medium [09:59:33] 10netops, 10Operations, 10homer: Homer: Netbox driven switch interfaces - https://phabricator.wikimedia.org/T250429 (10ayounsi) [10:40:11] 10Traffic, 10Operations, 10observability, 10Patch-For-Review, 10User-fgiunchedi: Aggregated metrics for ats-tls <-> clients ttfb percentiles - https://phabricator.wikimedia.org/T263536 (10fgiunchedi) >>! In T263536#6522137, @fgiunchedi wrote: > With 50 percentile added I'm considering this closed! > > A... [10:41:03] 10Traffic, 10Operations, 10observability, 10Performance-Team (Radar), 10Sustainability (Incident Followup): Document and/or improve navigation of the various HTTP frontend Grafana dashboards - https://phabricator.wikimedia.org/T253655 (10fgiunchedi) I wanted to mention/add that as part of {T263536} there... [10:50:33] 10Traffic, 10Analytics-Clusters, 10Operations, 10Patch-For-Review: varnishkafka 1.1.0 CPU usage increase - https://phabricator.wikimedia.org/T264074 (10ema) >>! In T264074#6537906, @ema wrote: > So varnishkafka seems to be correctly looping continuously in the do-while part of VUT_Main. Why is VSM_Status b... [12:42:35] 10netops, 10Operations, 10Patch-For-Review, 10Security, 10User-jbond: Review default ferm INPUT policy - https://phabricator.wikimedia.org/T264888 (10jbond) > no sure why but some host take a looong time While debugging this i noticed that we where receiving a lot of messages like the following (avalible... [12:43:03] 10Traffic, 10Operations, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10Gilles) I've captured 30 minutes of data using varnishlog simultaneously on cp3052 and cp3054, using 4 variants of this command for hit-front... [13:04:16] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10mforns) @ayounsi > Let me know when it's fine to merge the relevant change (for src_net + dst_net at least). Please, merge... [13:30:02] 10Traffic, 10Maps, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Epic: Support maps serving for affiliate sites via an allow list - https://phabricator.wikimedia.org/T261694 (10MSantos) @CDanis and @Dzahn as per T261424#6538173, is there anything else to be done for the 3rd party block in the tr... [13:40:55] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10ayounsi) Merged, note that it's not in a CIDR notation, so `src_mask` + `dst_mask` would be needed to generate the CIDR form. [13:45:06] https://etherpad.wikimedia.org/p/Traffic-2020-10-13 [16:56:55] 10Traffic, 10Maps, 10Operations, 10Product-Infrastructure-Team-Backlog: Limit maps serving to Wikimedia hosted sites only - https://phabricator.wikimedia.org/T261424 (10JMinor) Yes, I think everyone who requested to be added to the allow list has been added. There were a couple questions on the mailing lis... [17:42:48] 10Traffic, 10Maps, 10Operations, 10Product-Infrastructure-Team-Backlog: Limit maps serving to Wikimedia hosted sites only - https://phabricator.wikimedia.org/T261424 (10CDanis) >>! In T261424#6539494, @JMinor wrote: > Yes, I think everyone who requested to be added to the allow list has been added. There w... [17:43:00] 10Traffic, 10Maps, 10Operations, 10Product-Infrastructure-Team-Backlog: Limit maps serving to Wikimedia hosted sites only - https://phabricator.wikimedia.org/T261424 (10CDanis) a:03CDanis [20:06:40] 10Traffic, 10Operations, 10Technical-blog-posts: Blog post series: the evolution of Wikimedia's Content Delivery Network - https://phabricator.wikimedia.org/T264729 (10srodlund) @ema are you the sole author would you like additional authors added? [20:34:04] 10Traffic, 10Operations, 10Technical-blog-posts: Blog post series: the evolution of Wikimedia's Content Delivery Network - https://phabricator.wikimedia.org/T264729 (10srodlund) @ema For the main image, I went with the earth from space: https://commons.wikimedia.org/wiki/File:North_America_from_low_orbiting_... [21:02:26] 10netops, 10Operations, 10ops-codfw: patch in FB peering into cr2-eqdfw - https://phabricator.wikimedia.org/T265412 (10RobH) p:05Triage→03Medium [21:02:55] 10netops, 10Operations, 10ops-codfw: patch in FB peering into cr2-eqdfw - https://phabricator.wikimedia.org/T265412 (10RobH) [21:03:12] 10netops, 10Operations, 10ops-codfw: patch in FB peering into cr2-eqdfw - https://phabricator.wikimedia.org/T265412 (10RobH) [22:41:12] 10netops, 10Operations, 10ops-codfw: patch in FB peering into cr2-eqdfw - https://phabricator.wikimedia.org/T265412 (10RobH)