[06:46:28] 10Traffic, 10Operations: OOM killer killed varnihsd cache-main on cp3053 - https://phabricator.wikimedia.org/T247195 (10Vgutierrez) p:05Triage→03Medium [09:32:01] 10Traffic, 10MediaWiki-ResourceLoader, 10Operations, 10Performance-Team, and 2 others: load.php?modules=startup miss rate tripled on 2020-02-05 - https://phabricator.wikimedia.org/T247020 (10ema) >>! In T247020#5948063, @ema wrote: > I suspect this is due to the fact that we are unsetting `Accept-Encoding... [10:20:53] 10Traffic, 10MediaWiki-ResourceLoader, 10Operations, 10Performance-Team, 10Wikimedia-Incident: load.php?modules=startup miss rate tripled on 2020-02-05 - https://phabricator.wikimedia.org/T247020 (10ema) 05Open→03Resolved a:03ema The hitrate is now recovering after applying the patch: {F31672603}... [10:26:32] 10Traffic, 10Operations, 10Patch-For-Review: Investigate side-effects of enabling KA between ats-tls and varnish-fe - https://phabricator.wikimedia.org/T244464 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez KA between ats-tls and varnish-fe is working successfully and enabled globally in the caching c... [10:26:34] 10Traffic, 10Operations: ulsfo varnish-fe vcache processes overflow on FDs - https://phabricator.wikimedia.org/T243634 (10Vgutierrez) [10:44:29] 10Traffic, 10Operations: lua related crash on ats-be @ cp4032 - https://phabricator.wikimedia.org/T247232 (10Vgutierrez) [10:44:58] 10Traffic, 10Operations: lua related crash on ats-be @ cp4032 - https://phabricator.wikimedia.org/T247232 (10Vgutierrez) p:05Triage→03High [12:52:04] 10Traffic, 10Operations, 10Performance-Team, 10SRE-swift-storage, and 2 others: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10Gilles) @dpifke this is something I might be interested in handing off to you and that you might want to consider for Q4. Let me k... [13:51:59] 10Traffic, 10Operations: lua related crash on ats-be @ cp4032 - https://phabricator.wikimedia.org/T247232 (10ema) [13:53:32] 10Traffic, 10Operations: traffic_server crash upon Lua reload: attempt to concatenate a table value - https://phabricator.wikimedia.org/T242952 (10ema) [13:53:34] 10Traffic, 10Operations: lua related crash on ats-be @ cp4032 - https://phabricator.wikimedia.org/T247232 (10ema) [13:55:10] 10Traffic, 10Operations: traffic_server crash upon Lua reload: attempt to concatenate a table value - https://phabricator.wikimedia.org/T242952 (10ema) T247232 is another example of this bug, with a different Lua error (string length overflow). [17:47:56] 10Traffic, 10Operations, 10observability: prometheus2004 not scraping lvs2007 & lvs2008 - https://phabricator.wikimedia.org/T246860 (10elukey) I am investigating a similar problem for some new elasticsearch nodes (like elastic2055). If I create a ssh tunnel for prometheus2003 and 2004, I can see the metrics... [18:17:04] 10Traffic, 10Analytics, 10Operations, 10Performance-Team (Radar): Send X-Analytics information from Varnish to Hadoop with VCL_Log - https://phabricator.wikimedia.org/T196558 (10Krinkle) 05Open→03Resolved [18:17:07] 10Traffic, 10Operations, 10SRE-swift-storage, 10Performance-Team (Radar): Reduce amount of headers sent from web responses - https://phabricator.wikimedia.org/T194814 (10Krinkle) [18:19:12] 10Traffic, 10Operations, 10SRE-swift-storage, 10Performance-Team (Radar): Reduce amount of headers sent from web responses - https://phabricator.wikimedia.org/T194814 (10Krinkle) [21:16:03] 10Traffic, 10Operations, 10observability: prometheus2004 not scraping lvs2007 & lvs2008 - https://phabricator.wikimedia.org/T246860 (10CDanis) 05Resolved→03Open a:05CDanis→03None I'll take a look at this tomorrow in my day, unless @fgiunchedi beats me to it.