[18:00:38] it seems like something is going on with Wikidough, it's retuning lots of Servfail ( brouberol noticed it) [18:00:44] https://grafana.wikimedia.org/goto/sgf68j?orgId=default [18:27:08] * andrewbogott wonders if turnilo or superset have dns stats [18:53:41] looks like the recursors in esams are flapping quite a bit. Can't tell if that correlates to those servfails though... [19:10:29] Hello folks. I _think_ one of your public DoH servers is having issues with a missing DNSSEC key: doh3006. About an hour ago i started getting some weird results, some pages loading some returning the various incarnations of SERVFAIL depending on the application. Doing some sleuthing i found that it was nothing in my DNS stack but it was coming from my upstream, which is [19:10:31] https://wikimedia-dns.org/dns-query. [19:17:07] i've got a quick reproducer posted here https://paste.debian.net/hidden/cb129546 just a little more than half the time i seem to end up on 3006 and get a failure. Just wanted to let you know in case it flew under the radar. [19:17:45] And take the opportunity to thank you for the service. It's desperately needed, as everything you do. [19:55:03] 06Traffic, 10Pywikibot, 06SRE, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error on Wikidata - https://phabricator.wikimedia.org/T421642#12419904 (10ArthurPSmith) " Maybe I would implement some kind of priority aging (gradually increasing the priority of requests the longer they have been wait... [20:05:44] mind: thanks for reporting. I am not sure right now what's up but something is not right for sure so I am going to try to depool doh3006 and let 3005 pick up traffic and then take it for there for at least right now [20:06:43] XioNoX: thanks, doh3006 is acting up, not sure why [20:07:35] doh3005 is fine so I will pick it up on Tuesday [20:11:07] andrewbogott: it was most certainly the doh3006 SERVFAILs, https://grafana.wikimedia.org/goto/s452d6?orgId=default [20:11:28] NOERRORs going up as they should and SERVFAILs going down. doh3006 does get a lot more traffic in general so there's that but yeah [20:11:38] * sukhe heads out now and hopes it's calm [20:16:55] thanks sukhe! I wasn't sure if that warranted a ping but I'm glad you appeared and sorted it :) [21:23:26] thank you sukhe! much appreciated. [22:07:27] tnx!