[00:28:36] 10Traffic, 10Operations, 10wikitech.wikimedia.org: Wikitech page views sometimes default to MobileFrontend - https://phabricator.wikimedia.org/T220567 (10bd808) [05:40:08] 10Traffic, 10Operations, 10Patch-For-Review: Move cache text cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231627 (10Vgutierrez) [10:30:28] 10Traffic, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar): ATS doesn't support X-Wikimedia-Debug - https://phabricator.wikimedia.org/T237687 (10ema) 05Open→03Resolved The functionality is now deployed to production, a brief illustration follows. Valid XWD header: ` $ curl -s -v -H "X-Wik... [10:37:18] 10Traffic, 10Operations: Remove debug proxies once all Varnish backends are gone - https://phabricator.wikimedia.org/T237932 (10ema) [10:37:23] 10Traffic, 10Operations: Remove debug proxies once all Varnish backends are gone - https://phabricator.wikimedia.org/T237932 (10ema) p:05Triage→03Normal [10:46:34] 10Traffic, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar): ATS doesn't support X-Wikimedia-Debug - https://phabricator.wikimedia.org/T237687 (10ema) [10:46:38] 10Traffic, 10Operations: Replace Varnish backends with ATS on cache text nodes - https://phabricator.wikimedia.org/T227432 (10ema) [12:57:50] https://blog.acolyer.org/2019/11/11/snap-networking/ [12:58:05] thought it might be interesting for this audience [13:27:47] 10Traffic, 10Operations, 10Patch-For-Review: Replace Varnish backends with ATS on cache text nodes - https://phabricator.wikimedia.org/T227432 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts: ` ['cp3050.esams.wmnet'] ` The log can be found in `/var/log/wm... [13:31:39] godog: very interesting [13:32:00] hehe, the paper has more authors than pages :) [14:02:59] 10Traffic, 10Operations, 10Patch-For-Review: Replace Varnish backends with ATS on cache text nodes - https://phabricator.wikimedia.org/T227432 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp3050.esams.wmnet'] ` and were **ALL** successful. [15:26:59] 10Traffic, 10Analytics, 10Operations: Create replacement for Varnishkafka - https://phabricator.wikimedia.org/T237993 (10elukey) [15:27:17] there you go, added some thoughts to --^ [19:17:41] 10Traffic, 10Operations, 10Wikidata, 10Wikidata-Query-Service, 10Patch-For-Review: LDF service does not Vary responses by Accept, sending incorrect cached responses to clients - https://phabricator.wikimedia.org/T232006 (10Gehel) [22:58:43] vgutierrez: cp3065 crashed, I've depooled and power-cycled it. Nothing stands out in kern.log.1 and SEL as far as I can tell, we should give a closer look tomorrow