[08:14:38] 06Traffic: Consider using Intel Xeon CPUs with QAT - https://phabricator.wikimedia.org/T400324 (10Vgutierrez) 03NEW [08:14:47] 06Traffic: Consider using Intel Xeon CPUs with QAT - https://phabricator.wikimedia.org/T400324#11030138 (10Vgutierrez) p:05Triage→03Low [08:17:24] 06Traffic: Consider using Intel Xeon CPUs with QAT - https://phabricator.wikimedia.org/T400324#11030142 (10Vgutierrez) @jcrespo this could be interesting for backups as well, compression using Zstd can be accelerated with QAT as well, see https://community.intel.com/t5/Blogs/Tech-Innovation/Artificial-Intelligen... [09:08:25] XioNoX, after disabling BGP for a host on netbox do I need to sync it somehow? [09:17:53] vgutierrez: run homer on its switch or local core routers [09:18:32] probably local switch given it's sitting on eqiad E row [09:19:25] FIRING: SystemdUnitFailed: prometheus_liberica_cp_checks.service on lvs1013:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:19:35] that's expected :) [09:19:43] yep.. it's talking BGP with irb-1031.lsw1-e1-eqiad.eqiad.wmnet. [09:21:05] vgutierrez: yep [09:22:36] 06Traffic, 10Liberica, 13Patch-For-Review: Stop using lvs1013 as a liberica canary - https://phabricator.wikimedia.org/T400259#11030322 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1002 for host lvs1013.eqiad.wmnet with OS bookworm [09:36:18] INFO:homer:Homer run completed successfully on 1 devices: ['lsw1-e1-eqiad.mgmt.eqiad.wmnet'] [09:36:19] XioNoX: done [09:57:29] 06Traffic, 10Liberica: Stop using lvs1013 as a liberica canary - https://phabricator.wikimedia.org/T400259#11030394 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1002 for host lvs1013.eqiad.wmnet with OS bookworm completed: - lvs1013 (**PASS**) - Downtimed on Icinga/A... [09:59:06] 06Traffic, 10Liberica: Stop using lvs1013 as a liberica canary - https://phabricator.wikimedia.org/T400259#11030402 (10Vgutierrez) 05In progress→03Resolved [10:03:53] nice! [12:32:41] 06Traffic, 10Hiddenparma, 06SRE: Browser behaviour detection at the edge - https://phabricator.wikimedia.org/T400270#11030895 (10Vgutierrez) We might need to perform some of this work on HAProxy given it has direct access to the client connection and its properties [12:53:43] 06Traffic, 10Liberica: Test/benchmark Mellanox NICs for LVS usage - https://phabricator.wikimedia.org/T399502#11030990 (10Vgutierrez) 05Open→03In progress a:03Vgutierrez [13:31:09] 06Traffic, 13Patch-For-Review: Consider using the alternate chain of Google Trust Services certificates - https://phabricator.wikimedia.org/T398596#11031135 (10Ottomata) FYI: This change encountered this problem: https://wikitech.wikimedia.org/wiki/Event_Platform/Schemas/Guidelines#Complex_array_element_and_m... [13:33:03] 06Traffic, 06Data-Engineering: Fix Hive event.development_network_probe table - https://phabricator.wikimedia.org/T400360 (10Ottomata) 03NEW [13:33:29] 06Traffic, 06Data-Engineering: Fix Hive event.development_network_probe table - https://phabricator.wikimedia.org/T400360#11031165 (10Ottomata) [13:34:00] 06Traffic, 06Data-Engineering: Fix Hive event.development_network_probe table - https://phabricator.wikimedia.org/T400360#11031167 (10Ottomata) [14:30:09] 10Domains, 06Moderator-Tools-Team, 06The-Wikipedia-Library: wikipedialibrary.org isn't working - https://phabricator.wikimedia.org/T400367 (10Samwalton9-WMF) 03NEW [17:15:19] 10Domains, 06Traffic, 06Moderator-Tools-Team, 06The-Wikipedia-Library: wikipedialibrary.org isn't working - https://phabricator.wikimedia.org/T400367#11032339 (10jsn.sherman) This is a configuration issue for whatever is serving out the redirects (assuming mark monitor?) when you try to access an http only... [18:39:12] 10Domains, 06Traffic, 06Moderator-Tools-Team, 06The-Wikipedia-Library: wikipedialibrary.org isn't working - https://phabricator.wikimedia.org/T400367#11032568 (10Dzahn) fwiw, the more standard way to do this would be to ask SRE to add this domain to our own DNS servers, certs and redirect configs. And the... [19:21:34] 10Domains, 06Traffic, 06Moderator-Tools-Team, 06The-Wikipedia-Library: wikipedialibrary.org isn't working - https://phabricator.wikimedia.org/T400367#11032699 (10jsn.sherman) >>! In T400367#11032568, @Dzahn wrote: > fwiw, the more standard way to do this would be to ask SRE to add this domain to our own DN... [19:30:31] 10Domains, 06Traffic, 06Moderator-Tools-Team, 06The-Wikipedia-Library: wikipedialibrary.org isn't working - https://phabricator.wikimedia.org/T400367#11032708 (10BCornwall) 05Open→03In progress p:05Triage→03Low a:03BCornwall @jsn.sherman: I'll just retitle/reword this and get to work on transferr... [19:31:27] 10Domains, 06Traffic, 06Moderator-Tools-Team, 06The-Wikipedia-Library: Import wikipedialibrary.org into wmf redirection services - https://phabricator.wikimedia.org/T400367#11032713 (10BCornwall) [20:26:07] 10Domains, 06Traffic, 06Moderator-Tools-Team, 06The-Wikipedia-Library, 13Patch-For-Review: Import wikipedialibrary.org into wmf redirection services - https://phabricator.wikimedia.org/T400367#11032844 (10BCornwall) @Samwalton9-WMF, @jsn.sherman A few things have popped up here. For context, it looks lik... [23:34:27] 06Traffic, 06SRE: Outdated link in message that is sent for forbidden connections - https://phabricator.wikimedia.org/T400421#11033311 (10bd808) `lang=shell-session $ git grep https://meta.wikimedia.org/wiki/User-Agent_policy modules/profile/files/wmcs/services/maintain_dbusers/maintain_dbusers.py: # https:... [23:50:40] 06Traffic, 06SRE, 13Patch-For-Review: Outdated link to User-Agent Policy in Varnish 403 and 429 responses - https://phabricator.wikimedia.org/T400421#11033332 (10bd808)