[01:44:20] 10Traffic, 10Analytics, 10Operations, 10Research: Wikipedia Accessibility, check false positives and false negatives of traffic alarms - https://phabricator.wikimedia.org/T245166 (10Nuria) [07:50:52] 10Traffic, 10Operations, 10Goal, 10Patch-For-Review, 10Performance-Team (Radar): Support TLSv1.3 - https://phabricator.wikimedia.org/T170567 (10Vgutierrez) Sites running nginx or apache outside the caching cluster that have been upgraded to buster are now offering TLSv1.3: a few examples: ` $ /usr/local... [07:56:10] vgutierrez, ema: could I get a second sanity check for https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/571928/ [07:56:24] I don't know if I can provide sanity but... [07:57:29] looks good to me [07:58:55] thanks :-) [08:09:06] :) [11:10:13] 10netops, 10Operations: RRDP status alert - https://phabricator.wikimedia.org/T245121 (10jbond) >https://rrdp.ripe.net/notification.xml=-1 it works with out `=-1` (i.e. https://rrdp.ripe.net/notification.xml) is that a typo? [12:45:57] moritzm: sir, could you revisit your review of https://gerrit.wikimedia.org/r/c/operations/puppet/+/532348? [12:45:59] O:) [12:46:41] will do in ~ 10m! [12:47:26] no rush :) [13:04:56] \o/ [13:14:47] 10Traffic, 10Analytics, 10Operations, 10Research: Wikipedia Accessibility, check false positives and false negatives of traffic alarms - https://phabricator.wikimedia.org/T245166 (10jbond) p:05Triage→03Medium [13:40:30] 10Traffic, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar): Improve ATS backend connection reuse against origin servers - https://phabricator.wikimedia.org/T241145 (10ema) It turns out that cache_text nodes send quite a lot of RSTs to `appservers-rw`. See for example cp3054: ` 11:13:59 ema@c... [13:41:15] 10Traffic, 10Discovery, 10Operations, 10Wikidata, and 3 others: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service) - https://phabricator.wikimedia.org/T243701 (10Ladsgroup) >>! In T243701#5880988, @Bugreporter wrote: > The only way to resolve the issue is increase... [13:55:25] 10Traffic, 10Discovery, 10Operations, 10Wikidata, and 3 others: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service) - https://phabricator.wikimedia.org/T243701 (10Bugreporter) Currently Widar use a policy of sleeping 3x+1 second if the lag (x second) is higher than... [14:02:18] 10Traffic, 10Discovery, 10Operations, 10Wikidata, and 3 others: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service) - https://phabricator.wikimedia.org/T243701 (10Strainu) >>! In T243701#5884801, @Ladsgroup wrote: > I have been thinking about this and I think I have... [14:05:40] 10Traffic, 10Discovery, 10Operations, 10Wikidata, and 3 others: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service) - https://phabricator.wikimedia.org/T243701 (10Bugreporter) By default pywikibot will do one edit every 10 seconds (or longer if you use parallel proc... [15:56:21] bblack: taking suggestions on names for the new dyna [15:56:40] maps-addrs would be consistent :) [15:56:51] oh, I meant the DNS name [15:57:19] oh I figured you would just change the prod one. it's not like your first attempt will make it worse [15:57:33] and you can't really tell the results without real traffic either heh [15:58:10] oh sure [16:01:51] cdanis: FYI you might need to change a bit also the known hosts script, it treats dyna specially (to skip them) [16:08:46] bblack: speaking of not being able to tell the results -- have you done any mapping analysis using analytics data? I looked at it for this purpose but webrequest_sampled_128 doesn't have US state (although in theory it could) [16:52:43] bblack: are you not-opposed to this? https://gerrit.wikimedia.org/r/c/operations/dns/+/567163 [16:53:28] not-opposed would be a good way to put it :) [16:53:44] hehe, that's why i phrased it that way:( [16:53:54] i mean :) [16:54:10] vg is working on making all of these concerns obsolete soon, so it almost doesn't matter either way :) [16:54:45] we're aiming for a state where all domains the foundation formally owns (have been xferred to wmf legal and managed by our MarkMonitor registrar account) [16:55:33] that is great, yes, you mentioned this at allhands [16:55:36] all of that set will be polled from an API at the registrar, and aside from the canonical cases we really care about, all will be in our DNS and all will be in the ncredir service and properly TLSd and redirected to somewhere (most of them, to some default foundation domain) [16:56:14] :) ok, i will abandon that one with a comment about this [16:56:31] very nice to get to that state [16:58:55] well.. or merge it since i got the +1 now.. but then stop worrying about it :) [17:00:14] either way is fine with me :) [17:05:18] done. merged. thanks, i figured it was something for a Friday [17:07:10] 10netops, 10Operations: RRDP status alert - https://phabricator.wikimedia.org/T245121 (10ayounsi) It's because Grafana reports Routinator pulling data from https://rrdp.ripe.net/notification.xml as a -1 on its graph. Where I think -1 means timeout. So the correct URL is https://rrdp.ripe.net/notification.xml...