[08:11:28] 10Domains, 10Traffic, 10Operations: nameserver change for wikimedia.sk - https://phabricator.wikimedia.org/T241084 (10Luky001) 05Stalled→03Open Hello, the name servers are as following: ` ns1.websupport.sk ns2.websupport.sk ns3.websupport.sk ` Everything should be working but please keep in mind that I... [10:04:49] 10Traffic, 10Elasticsearch, 10Operations, 10Discovery-Search (Current work), and 2 others: Sustained periods (2-4h) of bad latency on production-search eqiad - https://phabricator.wikimedia.org/T241421 (10ema) p:05Triage→03High [10:10:05] 10Traffic, 10Elasticsearch, 10Operations, 10Discovery-Search (Current work), and 2 others: Sustained periods (2-4h) of bad latency on production-search eqiad - https://phabricator.wikimedia.org/T241421 (10jcrespo) [10:58:18] 10Traffic, 10Analytics, 10Operations, 10User-Elukey: Add VSL error counters to Varnishkafka stats - https://phabricator.wikimedia.org/T164259 (10elukey) 05Open→03Declined Given than we are moving to ATS, I'd decline this task :) [11:08:23] 10Traffic, 10Elasticsearch, 10Operations, 10Discovery-Search (Current work), and 2 others: Sustained periods (2-4h) of bad latency on production-search eqiad - https://phabricator.wikimedia.org/T241421 (10ema) >>! In T241421#5768632, @dcausse wrote: > I believe this is caused by a bot sending a large amoun... [11:46:10] 10Traffic, 10Operations, 10Patch-For-Review: two failing upload VTC tests - https://phabricator.wikimedia.org/T241653 (10ema) 05Open→03Resolved a:03ema ` [*] Finding cluster... cp3051.esams.wmnet is a cache_upload host [*] Running varnishtest (this might take a while)... sudo varnishte... [13:14:38] 10Traffic, 10Operations: servers freeze across the caching cluster - https://phabricator.wikimedia.org/T238305 (10Marostegui) [15:03:54] ema: ahh, thanks for fixing those tests :) [15:05:27] cdanis: ty for the patch! The ACL name also changed meanwhile, it's now "public_cloud_nets" to distinguish between wmf cloud and the cloudy clouds outside [15:06:28] oh you meant the upload tests probably [15:09:48] ah yes, I did [15:10:20] well ty for the other patch nonetheless :) [15:10:44] yeah, still am sad about my dumb mistake on 12/31, finishing that postmortem today [15:11:06] thanks for taking a look at the search API issue as well :) [19:31:27] 10Traffic, 10Operations: servers freeze across the caching cluster - https://phabricator.wikimedia.org/T238305 (10wiki_willy) In going through all the affected systems in this task, I'd like to treat db2125 and backup2001 separately, since they seem like one-offs and could very well be hardware issues. (db2125... [19:44:27] 10Traffic, 10Operations: Add more detailed instructions to the "sec-advice" page - https://phabricator.wikimedia.org/T241309 (10herron) p:05Triage→03Normal [20:34:18] 10Traffic, 10DNS, 10Operations: redirect non-existing wikimania2020.wikimedia.org to wikimania.wikimedia.org - https://phabricator.wikimedia.org/T240341 (10herron) p:05Triage→03Normal [23:50:27] 10Domains, 10Traffic, 10Operations: nameserver change for wikimedia.sk - https://phabricator.wikimedia.org/T241084 (10CRoslof) I've updated the nameservers. Let me know if there are any issues and you would like me to change them back.