[00:10:47] 10Traffic, 10Operations, 10TechCom-RFC, 10Patch-For-Review, and 3 others: Harmonise the identification of requests across our stack - https://phabricator.wikimedia.org/T201409 (10Tgr) One issue brought up in {T113817} was that we might want to avoid connecting too many things for privacy reasons (specifica... [01:20:39] 10Traffic, 10Operations, 10Performance-Team, 10Wikimedia-General-or-Unknown, and 2 others: Search engines continue to link to JS-redirect destination after Wikipedia copyright protest - https://phabricator.wikimedia.org/T199252 (10Tbayer) >>! In T199252#4595021, @Imarlier wrote: > @Nemo_bis Regarding your... [06:52:43] 10Traffic, 10Operations, 10TechCom-RFC, 10Patch-For-Review, and 3 others: Harmonise the identification of requests across our stack - https://phabricator.wikimedia.org/T201409 (10Joe) As long as a request ID doesn't get associated with PII-worthy data, I don't see it being a privacy issue. Since it's a per... [10:10:57] 10Traffic, 10Operations: Update certspotter - https://phabricator.wikimedia.org/T204993 (10MoritzMuehlenhoff) Adding the Debian maintainer :-) This seems fixed in 0.9-1 so updating stretch-backports to 0.9 could fix this. [10:11:02] 10Traffic, 10Operations: Update certspotter - https://phabricator.wikimedia.org/T204993 (10MoritzMuehlenhoff) p:05Triage>03Normal [12:02:08] So is Valentin back today? [13:54:54] https://blog.cloudflare.com/encrypted-sni/ + https://blog.cloudflare.com/esni/ fwiw :) [14:00:21] damn, I waiting to see someone around to drop it... you beated me :) [14:00:55] <_joe_> paravoid: cool so my interview question is now invalid :P [14:22:38] :) [17:00:52] bblack, hey, got a question about gdnsd and delegations out to other nameservers, in particular I'm interested in the handling of 56.15.185.in-addr.arpa. - is now an okay time to talk about that? [17:05:35] Krenair: sure [17:05:49] so here's the situation [17:06:09] we've got upstream provider's auth DNS servers delegating this to prod auth DNS servers [17:06:22] we've got a faulty delegation from prod auth DNS servers to labs auth DNS servers, where the records actually lie [17:06:35] ok give me a sec to catch up and look at the data [17:06:38] ok [17:06:57] now I think I know why the templates/56.15.185.in-addr.arpa. file doesn't work like one may expect [17:07:31] ok so [17:07:51] 1) Yes, this zone is still delegated from RIPE to ns[012].wikimedia.org [17:08:05] the actual behaviour seen from prod NSes is https://phabricator.wikimedia.org/T199374#4552010 [17:08:50] they just serve the SOA they have (rather than the NS records out to the desired nameservers) and I think I get why they do that [17:08:59] 2) The change merged a few weeks ago: https://gerrit.wikimedia.org/r/c/operations/dns/+/445303 is not how you re-delegate a zone, so that was faulty. [17:09:12] it has nothing to do with gdnsd specifics, it's just wrong [17:09:12] yeah [17:09:29] I think I understand that [17:09:48] so, in general you can't re-delegate a whole zone in the DNS [17:10:00] you can delegate out subzones of what's delegated to you, but you can't just pass off the whole thing [17:10:03] the key thing really is that queries for *.56.15.185.in-addr.arpa PTR records get put through to labs-ns* servers [17:10:12] now as I see it there are probably 2 or 3 ways to achieve this [17:10:31] 1) tell RIPE to change their NS records to point directly to labs-ns* instead of ns* [17:10:58] the best way, if supported by RIPE, would be to ask RIPE to change the delegation, but only for this 1x /24 (which is part of a contiguous /22 they gave us, so I'm not sure of that's possible) [17:11:11] 2) have prod put in an NS records for all 1-255 subdomains that point at labs-ns* (?) [17:11:50] 2 is probably a bad idea, although it may kinda-work in some limited ways [17:11:57] 3) give the prod auth DNS servers a zone file for 15.185.in-addr.arpa and make a 56 IN NS record pointing at labs-ns* [17:12:21] 3 won't work at all on a fundamental level (nobody asks our servers for delegations for that zone, they ask RIPE) [17:12:22] (probably not good since we don't actually control the whole of 185.15.0.0/16, but may work?) [17:12:44] so, there's a fourth possibility [17:13:20] ok [17:13:24] that's good to hear :) [17:13:25] which is to use classless delegation from https://tools.ietf.org/html/rfc2317 even though we're delegating off the whole zone [17:13:50] :/ I was hoping we wouldn't need that. ok. [17:14:02] or we can of course invent our own stupid variant of classless delegation, but better to follow the Standard Way [17:14:07] it's not that awful [17:14:45] basically we'd delegate a subzone named, literally, "0/24.56.15.185.in-addr.arpa" using a proper delegation with NS records [17:15:21] and then put in 256 CNAME records that are all of the form "42.56.15.185.in-addr.arpa CNAME 42.0/24.56.15.185.in-addr.arpa" [17:15:37] yeah [17:15:42] so this is like what we did with 208.80.155.128/25 [17:15:46] and you tell yout labs-nsX servers that the zone they should put their data in is: 0/24.56.15.185.in-addr.arpa instead of 56.15.185.in-addr.arpa [17:16:14] ok [17:16:28] yeah we can do this [17:16:51] had been thinking we wouldn't need to [17:17:18] alright [17:17:34] steps on the labs side [17:17:48] create 0-24.56.15.185.in-addr.arpa. zone [17:18:37] 0/24 is the zone name, not 0-24 [17:18:56] well [17:19:07] previous one was done with a dash instead of a slash [17:19:10] but ok [17:19:14] oh? [17:19:25] yeah 208.80.155.128/25 [17:19:42] had a 128-25.155.80.208.in-addr.arpa setup [17:20:04] completely necessary in that case because it was a /25, and the other /25 was actually prod stuff [17:20:22] right, so what's described by RFC 2317 is the 128/25 style, but IIRC the RFC isn't inventing anything new in the standard, just codifying a best practice that would've worked no matter what you stuck in the "0/24" slot of the naming scheme [17:20:39] okay [17:20:45] well let's try doing it with a slash this time I guess [17:21:00] I don't get your last statement though, I don't see why it would ever be necessary to deviate except maybe "our nameserver can't handle zones with slashes in them" [17:21:12] sorry [17:21:15] I mean [17:21:20] the classless delegation was necessary [17:21:26] it probably could've handled slashes [17:21:31] ok [17:21:33] yeah [17:21:40] maybe peek at the commits that set that up, maybe there's a reason [17:22:11] re-reading the RFC a bit, it even says: [17:22:13] The examples here use "/" because it was felt to be more visible and pedantic reviewers felt that the 'these are not hostnames' argument needed to be repeated. We advise you not to be so pedantic, and to not precisely copy the above examples, e.g. substitute a more conservative character, such as hyphen, for "/". [17:22:30] uh oh [17:22:30] so I guess whatever, use the dash :) [17:22:31] yeah here we are [17:22:34] https://gerrit.wikimedia.org/r/#/c/operations/dns/+/299513/ [17:22:48] Alex Monk [17:22:48] Jul 20, 2016 [17:22:49] ↩ [17:22:49] Patch Set 3: Code-Review-1 [17:22:49] Designate doesn't want to create a zone with a forward slash, will use hyphen [17:23:26] dash it is [17:23:52] hah, ok :) [17:24:28] gdnsd has issues with it too, because all zones in gdnsd are the names of files in a filesystem, and while you could escape a / in a filename, it would be awkward and annoying. [17:24:40] alright thanks for your help bblack [17:25:06] but gdnsd's answer was to support auto-translation of @ -> / when scanning zonefile names (if you name the file 0@25.foo, the effective zone name is 0/25.foo) [17:32:46] https://phabricator.wikimedia.org/T199374#4611710 [17:40:57] 10Traffic, 10MediaWiki-ResourceLoader, 10Operations, 10Performance-Team: Investigate source of 404 Not Found responses from load.php - https://phabricator.wikimedia.org/T202479 (10Krinkle) >>! In T202479#4580307, @ema wrote: >>>! In T202479#4575922, @Krinkle wrote: >> 2. Hostnames we route to text-lb that... [17:42:13] Krenair: yeah looks sane. Maybe bump up the delegation + CNAME record TTLs all to 1D (or maybe do that in a followup after confirming working, but since the existing one doesn't work either it's not much risk to do it right off) [17:42:53] 10netops, 10Operations, 10Patch-For-Review: Evaluate NetBox as a Racktables replacement & IPAM - https://phabricator.wikimedia.org/T170144 (10Volans) [17:43:05] yeah the existing one isn't working yet so np [18:03:00] bblack, I guess one question is left in my mind about the DNS protocol and what clients expect [18:03:46] is it simply not valid for, after RIPE has pointed at ns0 with an NS record, for ns0 to then give an NS record to a different server? [18:04:37] you said "in general you can't re-delegate a whole zone in the DNS" [18:05:06] so will a client error if RIPE says `56.15.185.in-addr.arpa. 172800 IN NS ns0.wikimedia.org.` and then ns0 says `56.15.185.in-addr.arpa. 172800 IN NS labs-ns0.wikimedia.org.` [18:05:27] I guess there has to be some limit to it otherwise DNS servers could have clients on a wild goose chase [18:17:41] it's not really about goose-chasing, it's just that delegations don't actually work that way. [18:18:46] so what would a client do if a DNS server tries to respond like that? [18:18:48] The delegation from .org to example.org has to involve NS records for "example.org" that are fetched from the "org" zone on the org zone's nameservers [18:19:12] it wouldn't do anything, it would never look for any delegation records at that place (the NS records inside of our zone, at its own root) [18:19:29] in practice, the root-of-zone NS records you see at the top of every zonefile are functionally useless [18:19:43] it's good practice to define them, they can be used for debugging or confirmation [18:19:54] but nothing looks at them to know where to resolve names [18:21:25] (and in traditional DNS servers configured in traditional ways, they're also tacked onto the bottom of every normal response from the zone, in the authority section. But this is also pointless/useless in practice, which is why even bind has option to not do it, and gdnsd just never does it. [18:21:30] ) [18:23:02] the crux of understanding this is that a delegation response (as in: Hi I'm the server for "org", and I delegate "example.org" to nameserver blah) is not a normal response. It has different features than, say, just a query for NS records and a response saying "here's some NS records". [18:23:11] so a client would ignore the NS records in the response and treat it as getting a NOERROR with an empty answer? [18:23:29] a client would normally never even look for those NS records [18:24:09] delegations aren't something a client explicitly asks about. A blank-slate client starting with only the root-servers list assumes that the internet's 13 root servers can answer any query about any name directly. [18:24:45] so it asks a.root-servers.net "What's the address for en.wikipedia.org?", which the root server could actually answer directly if it wanted to, protocol-wise. [18:25:22] but instead it says "I donno, but I delegate the org zone to a0.org.afilias-nst.info., go ask them" [18:26:02] then the client goes and asks that server, and it could also reply with our IP if it wanted to, but instead says "I donno, but I delegate the wikipedia.org zone to ns0.wikimedia.org, go ask them" [18:26:51] the client never asks for NS records, it just gets them anyways in special delegation responses like these, which are different from a response for a direct query for type=NS [18:28:01] the important but not directly relevant implication is a server cannot delegate to itself [18:28:34] e.g. we cannot define zone data for both wikipedia.org and subz.wikipedia.org on ns0.wikimedia.org, and have the wikipedia.org data delegate the subz to ns0 (same server as self) [18:29:16] because answering questions about "subz" requires two different kinds of responses to the same question in that case. You're either the one delagating, or the one delegated to, you can't be both. [18:30:11] and for similar reasons, you can't provide a delegation response for the zone you've been delegated for. doing a delegation always involves moving one label down the hierarchy. [18:31:16] [hmmm I should note everywhere I use "client" above, I meant "client of an authserver". Such clients are (usually caching) recursor servers, not end-clients] [18:31:29] sure [18:33:00] I knew most of this [18:37:44] the no-delegation-to-self rule is a surprising implication sometimes if you haven't hit the problem or stared at it before. It can crop up where the owners of example.org delegate foo.example.org to a different organization to administer, and then both companies choose the same 3rd-party DNS hosting service [18:37:55] who places them on the same DNS servers and it all breaks down [18:38:46] Well in designate we have a wmflabs.org domain and various sub-domains all on the same nameservers [18:38:56] if you're a larger DNS hosting company though, there's standard ways you can mitigate that (e.g. have different sets of NS servers you put different pools of customers in, and one of the factors that goes into it can be "set A does domains with an even-numbered count of labels, and set B odd", so that there's no direct parent->child relationship on one server set. [18:39:07] Krenair: yes, but not via delegation [18:39:14] hmm [18:39:19] yes [18:39:37] whereas in the cross-organisation case you might try delegation [18:39:41] some servers will accept a set of zones/zonefiles that look like a self-delegation, but in practice they treat it like $INCLUDE files and it's all one zone. [18:39:42] interesting [18:40:09] and yeah, I recall AWS Route 53's nameserver sets being strange at first [18:40:51] huge number of different nameservers that could be allocated when you went to create a zone [18:41:02] yeah, given their focus on virtualization they probably use fairly small pool sizes and huge numbers of distinct sets of virtual nameserver IPs. [18:41:26] I donno though, I haven't ever used it. [18:43:34] the bit I'm missing is if a0.org.afilias-nst.info. says "I donno, but I delegate the wikipedia.org zone to ns0.wikimedia.org, go ask them", what is actually stopping ns0.wikimedia.org. from saying (for example) "I donno, but I delegate the wikipedia.org zone to labs-ns0.wikimedia.org, go ask them" ? some protocol limitation only allowing delegation of subdomains? The client noticing that it attempted to re-delegate and deciding the server is [18:43:35] a witch? The client looking through the response and not finding anything it was expecting, so treating it as effectively empty? I know the client won't go looking for NS records [18:47:19] I'm not sure off the top of my head, but I know there's a reason I've run into before. [18:47:22] hmmm [18:50:03] Krenair: because RFC1034 says so. In section 5.3.3 (which defines the lookup algorithm), when dealing with following delegations it says: [18:50:08] If the response shows a delegation, the resolver should check to see [18:50:08] that the delegation is "closer" to the answer than the servers in SLIST [18:50:11] are. This can be done by comparing the match count in SLIST with that [18:50:14] computed from SNAME and the NS RRs in the delegation. If not, the reply [18:50:17] is bogus and should be ignored. If the delegation is valid the NS [18:50:19] -- [18:50:21] delegation RRs and any address RRs for the servers should be cached. [18:50:23] The name servers are entered in the SLIST, and the search is restarted. [18:50:39] So basically, if you did a delegation-type response that didn't go any deeper in the tree, the RFC says the resolver is to treat it as bogus and ignore it. [18:51:03] so it's not so much that they didn't provide a mechanism for re-delegation [18:51:11] they wrote in a clause pretty much prohibiting it? [18:51:17] right [18:51:56] probably for decent reasons, it seems like there would probably be unresolveable bad edge-cases that could cause with other parts of the spec, but I can't think of any deadly ones myself at the moment. [18:52:21] well you could have servers point at each other [18:52:29] send clients around in loops [18:52:32] well sure, but that problem already exists with CNAMEs [18:52:41] true [18:53:15] also, you could easily infinite-loop legal delegations with just two distinct sets of nameservers and some hacky software to implement it [18:53:32] well, up to the limit they query for anyways [18:54:21] I guess that could've been part of the reason: recursive delegation is limited to a step-count equal to the count of labels in the name, but if you allowed re-delegation without descent there wouldn't such a limit. [18:54:31] yes [18:54:46] but you'd think there'd be a better reason than that, since these are the same people that defined CNAME the way they did :P [18:55:07] :) [18:55:29] as of today, if i touch trafficserver/backend.yaml does that already affect reality in production? [18:55:44] mutante: no, it's just in testing [18:56:06] ok, i will feel ok to self-merge a change then.. like replacing mwmaint1001 with 1002. thanks [18:56:22] just CC e.ma so he sees it when he gets back pls [18:56:48] ok, i have him on Gerrit as reviewer and sending him a message. ack [18:58:57] and one more thing i have in actual cache/text.yaml which would have been "misc" until recently. but it's a commented line that will change things only when we switch back: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/462036/1/hieradata/role/common/cache/text.yaml [18:59:11] bblack, so I think this answers my question anyway - the spec says that clients should treat this as bogus. thanks for your time [19:00:12] np! [19:01:11] (well also, because clients would treat such a wire-response as bogus, nameservers don't implement it that way either, so there's no way to even try it without writing custom software or patching it. When you define NS records at the root of the zone, they're just considered local data, not a delegation) [20:47:46] 10netops, 10Operations: analytics1-a VLAN has no DNS for gateway addresses to match other analytics VLANs - https://phabricator.wikimedia.org/T205340 (10chasemp) p:05Triage>03Low [20:50:59] I didn't read 80% of the backlog [20:51:10] but we can set whatever NSes we want, on whichever zones we want at RIPE [20:51:21] it's super easy [20:52:03] paravoid: also subset of a subnet for PTRs? IIRC it was about a /24 inside a /21 or /22 [20:53:24] yeah, zones are not necessarily 1:1 with route objects which in turn are not necessarily 1:1 with inetnums [20:54:25] it that case it might be a simpler solution :) [20:56:33] fixed [21:00:27] paravoid, wait so you've done it? [21:00:29] yes [21:00:31] lol [21:00:33] ok [21:01:42] and just pushed https://gerrit.wikimedia.org/r/462574 [21:02:21] cool [21:02:40] alex@alex-laptop:~$ host eqiad1.bastion.wmflabs.org [21:02:40] eqiad1.bastion.wmflabs.org has address 185.15.56.13 [21:02:40] alex@alex-laptop:~$ host 185.15.56.13 [21:02:40] 13.56.15.185.in-addr.arpa domain name pointer eqiad1.bastion.wmflabs.org. [21:02:40] bblack: I considered leaving an otherwise empty file instead with a comments '; this zone is ours, but is delegated directly to labs-ns0/1' but I'm not sure how gdnsd will behave [21:03:10] it'll see the file so it'll think it serves the zone, but no SOA will be provided, so maybe that will result in an error? no idea [21:03:13] Krenair: what, no IPv6!?! [21:03:15] :) [21:03:16] I think gdnsd will baulk at a zone file not containing an SOA and refuse to start [21:03:33] :) [21:09:25] yes, that [21:10:09] my only minor complaint about direct delegation from RIPE, is that labs-ns[0123] aren't set up the best, but if it's easy to change on a whim and it's just revdns, eh? [21:10:47] plus I guess labs-ns[0123] are already directly delegated for other zones [21:10:51] (wmflabs.org?) [21:11:06] yes, for wmflabs.org [21:11:08] sorry I keep saying 0123, it's just [01] today, but [23] are defined [21:11:19] I was about to ask that [21:11:34] if I should delegate to 23 too [21:11:46] I just followed what was attempted in that delegation commit [21:11:57] seems wmflabs.org points to only 01 too [21:12:01] but "aren't set up the best", I mean they're just the row-subnet public IPs of specific hosts that happen to be running the software (labservicesNNNN/labcontrolNNNN) [21:12:23] vs setting up IPs in an LVS-like range that are routed and can be manipulated easier [21:12:55] or floating IPs within WMCS, so that designate can run in WMCS VMs? :) [21:13:08] which avoids scenarios like "oh no labs-ns0 died, our authserver IP isn't responding" -> "oh let's spin it up on another machine" -> "oh wait it has to be in Row X" -> "oh there's some port/space capacity problem somewhere" [21:13:26] etc, etc [21:13:28] I don't know if forcing the delegation to go via prod would really help with regards to this sort of thing [21:13:49] well you could choose to account for that and keep the delegation short-TTL at least [21:14:14] the delegation is to an FQDN under wikimedia.org, not IPs [21:14:24] (NS labs-ns0.wikimedia.org etc.) [21:14:32] labs-ns0 seems to have a 1H TTL [21:14:43] delegations are always to IPs, not names. the name is an illusion. [21:14:53] what? [21:14:57] no it isn't :) [21:14:58] what matters is the glued IP of that name at the .org servers, not whatever we put in wikimedia.org. [21:15:02] yes, it is :P [21:15:02] it's not glued [21:15:12] this is not an in-zone delegation [21:15:36] this is true when you're delegating, say, wikimedia.org to ns0.wikimedia.org [21:15:47] not when you're delegating 56.15.185.in-addr.arpa to labs-ns0.wikimedia.org :) [21:15:57] it doesn't have to be glued, but it is: [21:16:11] dig @a0.org.afilias-nst.info. wmflabs.org A [21:16:18] ;; AUTHORITY SECTION: [21:16:19] wmflabs.org. 86400 IN NS labs-ns1.wikimedia.org. [21:16:19] wmflabs.org. 86400 IN NS labs-ns0.wikimedia.org. [21:16:26] ;; ADDITIONAL SECTION: [21:16:26] labs-ns0.wikimedia.org. 86400 IN A 208.80.155.117 [21:16:26] labs-ns1.wikimedia.org. 86400 IN A 208.80.154.12 [21:16:33] note those IPs don't come from our nameservers [21:16:33] oh I was talking about RIPE [21:17:07] Okay this is interesting [21:17:17] well even in the wmflabs.org case, it's not technically required to glue those IPs, but since the glue is emitted and within ".org", it will be believed [21:17:21] the org nameservers have just labs-ns0[01] set up for wmflabs.org [21:17:43] but labs-ns* lists labs-ns0[0-3] for it [21:18:46] Krenair: well, from my earlier rambling, remember that those are mostly just informative (the NS records inside the zone itself on labs-ns*), they don't affect delegation directly (although in some cases, some recursors may cache them and use them, indirectly and eventually) [21:18:55] rirght [21:18:59] the even wtf'er part is that I delegated 56.15.185.in-addr.arpa to only labs-ns0/ns1, and that's what WHOIS says [21:19:06] that's just where I notice they show up [21:19:08] but digging @pri.authdns.ripe.net gives all 4 [21:19:20] sounds like a ripe bug? [21:19:27] where would it even learn those? [21:19:37] run an NS query against what I gave it probably? [21:19:52] I only see the labs-ns[01] from the ripe dns servers [21:20:18] oh wait, pebkac [21:20:18] what query are you executing? [21:20:21] ok :) [21:20:42] (I had a -x there but queried 56.15.185.in-addr.arpa instead :) [21:20:59] anyway, RIPE DTRT and doesn't provide glues [21:21:24] Krenair: re the right/wrongness of listing labs-ns[23] in the local zone data NS records: I don't know designate well, but it could be tacking that information onto lots of responses, which will pollute caches with it quickly. [21:21:27] I don't know why .org does, it shouldn't, right? [21:21:54] yeah I think I'm gonna ask andrew about those [21:22:19] Krenair: but even if it doesn't include unecessary NS records with normal responses, if anyone does an explicit query for NS records through a recursive cache, it may learn them that way and try to use them. Depends on the cache implementation. [21:22:43] Krenair: so if ns[23] aren't meant to be used yet (testing and/or unreliable), they probably shouldn't be there JIC [21:22:56] Krenair: the disparity between what labs-ns think it's their NSes, and what .org and now RIPE thing is a bug either way, so I'd open a task :) [21:23:06] s/thing/think/ [21:23:20] yeah [21:24:11] I remember from many years ago forms by registrars where you can add glues yourself [21:24:19] maybe we did that with .org, if so we should remove them [21:24:41] it's a tricky subject, and arguably the wmflabs.org glue is legit. [21:24:50] why? [21:25:14] because it's in-zone, and it's more efficient that way than the other valid way. [21:25:36] haha I just found an email thread [21:25:42] but technically, you can define out-of-zone glue too. whether a recursor believes it may be some matter of debate or implementation choice, though. [21:25:57] HI Ryan, [21:25:57] These severs have not been registered with Versign. I will add the glue records and then you will be able to update to the new servers. [21:26:00] This should be done today. [21:26:03] No match for nameserver "LABS-NS0.WIKIMEDIA.ORG". [21:26:05] >>> Last update of whois database: Mon, 18 Jun 2012 17:11:16 UTC <<< [21:26:07] right [21:26:08] No match for nameserver "LABS-NS1.WIKIMEDIA.ORG". [21:26:09] > The requested change was: [21:26:11] >>> Last update of whois database: Mon, 18 Jun 2012 17:18:07 UTC <<< [21:26:14] this was in a response to [21:26:16] > [21:26:19] > Change nameservers from: [virt0.wikimedia.org, labsconsole.wikimedia.org] to: [labs-ns0.wikimedia.org, labs-ns1.wikimedia.org] [21:26:22] > [21:26:24] > The failure occurred due to: [21:26:27] > [21:26:29] > There was an error creating host labs-ns0.wikimedia.org at the registry. Any actions depending upon the successful creation of [21:26:30] for the gtlds it has always been the norm to just always save and emit glue, IIRC [21:26:32] this host have been aborted. [21:26:36] haha [21:26:39] I complained at the time [21:26:39] at least in-zone glue (in the TLD, I mean) [21:26:51] I don't understand: why do we need glue records in the first place? The nameservers in question are in a different domain than the one [21:26:54] they're serving requests for, and hence there is no chicken-and-egg loop to be broken with glue. [21:26:57] is what I said at the time :P [21:27:00] that was [21:27:00] Date: Mon, 18 Jun 2012 20:53:54 +0000 [21:27:14] glue is a complex subject, we could spend a lot of time on it :) [21:27:21] at the time it was all manual, via emails [21:27:23] but what they're doing isn't a bug, it's pretty normal [21:27:51] alright there's an obscure technical reason for these labs-ns[23] entries [21:27:53] gonna make a ticket anyway [21:27:57] note wikipedia.org is the same way (emits glued wikimedia.org servers) [21:28:14] well that's a bit different [21:28:25] not really [21:28:30] because ns0/1/2.wikimedia.org are already in the .org zone [21:28:37] ... [21:28:38] because of wikimedia.org [21:29:15] wikimedia.com doesn't respond with glues, though [21:29:31] it's hard to even ramble about this, because there's several different contexts you could discuss it in, which makes it very confusing [21:30:10] what's standards-allowed, what works on the wire, what various buggy implementors do or don't do and how we cope with them, the security implications that caches should be dealing with, the practical implications for when you shouldn't bother emitting certain things because secure caches wouldn't believe them anyways. [21:30:29] sure ok [21:30:32] and then of course operational practice of registrars and operators [21:30:36] I remember the whole "additional section" debacle [21:31:19] but the bottom line is, there's nothing inherently wrong with what the TLD nameservers are emitting in any of the cases we've looked at here, and in the wmflabs.org case, client caches will take the glue from the .org servers, not us, and that's how it's meant to be. [21:31:58] but it's unnecessary and as you originally pointed out could create issues if you want a shorter TTL [21:32:01] I suppose you could ask them to not emit it, but I'm not sure if that's an option (for them operationally, to configure that way)? [21:32:21] and why would we have the same information in two places if we can have it in just one :) [21:32:28] well "if you want a shorter TTL" is because our operational practices for labs-nsX are faulty in some sense. They'd say to fix that :P [21:32:44] DNS is full of pointless information in multiple places :) [21:33:33] (operational practices: they're all in one geographic location, and they're not even floating IPs. Luckily they are in two different /24 I guess, but adjenct ones we probably advertise in aggregate?) [21:34:58] my personal POV, within all the wiggle room on all things related to glue, is that as a default policy it's honestly better if all delegations carry glue. Really the NS records are the part that shouldn't exist, and delegations should've been to IPs if you go back and fix the design. [21:34:58] I don't disagree at all, but maybe it's worth pointing out that 100% of the contents of that zone are sitting not in just one geographic location, but in just one datacenter's row :P [21:35:11] but you can basically invent random hostnames for the NS records and glue them in the output and get the same effect [21:35:21] 10Traffic, 10Cloud-VPS, 10DNS: Inconsistent lists of labs-ns* nameservers - https://phabricator.wikimedia.org/T205344 (10Krenair) [21:35:46] ^ [21:35:53] well if that was the case, then for us to change the IP of e.g. ns2.wikimedia.org we'd have to go to N TLDs/registrars and update it [21:36:06] paravoid: yeah but now you're getting into negative cache TTLs vs time-to-fix row. Either way, DNS is meant to be fairly reliable. [21:36:30] paravoid: yes, we've been through that before, but markmonitor does it for us [21:36:31] 10Traffic, 10Cloud-VPS, 10DNS, 10Operations: Inconsistent lists of labs-ns* nameservers - https://phabricator.wikimedia.org/T205344 (10Paladox) [21:36:46] what do you mean? [21:36:56] huh [21:36:57] i didn't edit anything. [21:36:58] (sorry speaking to wikibugs) [21:37:38] delegations from other TLDs don't include glues I think, so what do you mean it does it for us? or do you mean that registrars could in this theoretical scenario do it for us? [21:37:47] paladox, it notices when subscribers change unfortunately :) [21:37:54] or possibly, because of herald [21:37:54] oh [21:38:07] i suspect it adding the "ops" tag did it :) [21:38:14] the latter I guess, although honestly I don't know which TLD/2LD we have domains within (lots) do or don't emit out-of-zone glue (and I don't care so long as markmonitor fixes it all) [21:38:47] (also, wouldn't negative cache TTL kick in only in NXDOMAIN rather than the servers being unresponsive? :) [21:38:56] 10Traffic, 10DNS, 10Operations, 10WMF-Communications, and 4 others: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches - https://phabricator.wikimedia.org/T188776 (10Varnent) [21:39:17] but again from the "how it should be" POV, which is implementable as best-practice without changing protocols: ideally all delegations should carry glue regardless of the hostname used for the NS record, and on top of that even better practice is to always put the arbitrary NS hostname in-zone for the zone being delegated to. [21:39:44] paravoid: yeah no idea what the retry policy is there, various by cache I guess? [21:40:06] you mean create ns0.wikimedia.com ns0.wikipedia.org etc. for all of our zones? [21:40:47] I think for our case, in practice, it makes little difference, but yes that's the idea [21:41:30] it's the IPs that matter, not the names. And putting them all in-zone makes everyone emit glue regardless of policy/operations/whatever. [21:41:51] I'd bet that markmonitor would not have a way to update all of these automatically with one form/email/whatever [21:42:06] so that would make changing an nsN's IP about two hundred times more complicated :P [21:43:23] I'm honestly unsure still why you prefer that over non-glued domains (except the one in which NSes reside), but just pointing out complexities that I foresee with this scheme [21:43:40] in markmonitor's case, that's what they're paid to do is manage all that [21:43:53] i.e. entertaining the idea, even though I don't fully understand why we'd do such a thing [21:43:54] and then yes you have more updates to make [21:44:19] well, I haven't proposed that we do it, because it doesn't have a huge positive benefit for us in particular [21:44:30] but as general policy for The World, it's better [21:45:01] in this World, or some alternative universe where everyone does that and registrars etc. are accustomed to it? :P [21:45:30] in this world. it's not that hard. [21:45:51] the reason it's better general-purpose policy is outlined in the examples and problems pointed out in the upper sections of ;; AUTHORITY SECTION: [21:45:54] 56.15.185.in-addr.arpa. 172800 IN NS labs-ns1.wikimedia.org. [21:45:54] bleh [21:46:05] bad paste, in the upper sections of: http://cr.yp.to/djbdns/notes.html [21:50:03] anyways, the TL;DR is the world would be a more reliable and better place with the general policy to always put NS hostnames in-zone and thus delegations also always emit glue. [21:50:55] it's not such a practical issue here, since everything is under one organization's control anyways [21:51:21] well ok, DJB seems to disagree with an RFC or two and you seem have the same opinion as a DJB [21:51:40] but you can imagine at various commercial entities that own hundreds of brands and domains and routinely transfer control of them as business units and/or brands are bought and sold. [21:51:41] which is not very surprising tbh :P [21:52:19] these glue situations and who controls what get out of control quickly [21:52:44] as in the example where www.w3.org was effectively controlled by a compromiseable DNS server that probably nobody at w3.org had ever heard of or suspected. [21:52:59] how sure are you that this labs-ns0 glue is accepted and not filtered-out by (new/current) recursors? [21:53:35] I'm not sure, but I don't see any fundamental reason why they would, since it's within org and the answer came from the authserver for org [21:55:09] "disagree with an RFC" is questionable. There's a lot to dislike about the DNS RFCs, but this is a more-subtle matter. The RFCs allow a lot of latitude, and this is an argument that one should restrict what they're doing to a subset of that latitude that makes it much harder to hang yourself. [21:56:15] RFC 1537 (granted, super old and also informational), titled "Common DNS Data File Configuration Errors" says "Glue records need only be in a zone file if the server host is within the zone and there is no A record for that host elsewhere in the zone file." [21:56:52] I went there because DJB linked to it [21:57:12] I'd have to look at the context in the rest of that RFC to understand exactly what they're saying there [21:57:15] where he says "For referrals to out-of-bailiwick DNS servers, however, RFC 1034 says that glue is unnecessary [...]", basically saying how he disagrees with the RFCs :) [21:57:45] but either way, that's not what this hinges on [21:58:18] the argument is that all NS records should point in-zone, and thus glue will always be required by implication. There is no RFC recommendation on where the hostname in the NS record should be in the DNS tree. [21:58:32] oh [21:58:33] sure, yeah [21:58:59] I was referring to the labs-ns0 glue, but yeah, in the ns0.wikimedia.com etc. case it's implicitly immaterial indeed [21:59:09] if you follow that one simple rule universally, it's impossible for the gluelessness and "omg random servers I neve rheard of control my stuff" problems to ever arise. [21:59:48] welllll [21:59:55] in *this* particular case... [22:00:07] in this particular case it's all controlled by us [22:00:13] nope :) [22:00:18] all of *.wmflabs.org is not [22:00:38] what do you mean by that? [22:00:55] that the contents of wmflabs.org are VMs that third-parties (for the most part) control [22:01:19] so an "ns0.wmflabs.org" hostname could conceivably be created by a third-party [22:01:31] the bulk of the hostnames within wmflabs.org, yes. [22:02:05] but the zone is defined by the designate software running on labs-ns0 or whatever, which has to define zone-level details like SOA and NS (and probably other static data) aside from the injected address/cname stuff for VMs. [22:02:15] no I mean [22:02:25] perhaps delegating wmflabs.org to something not under wmflabs.org is a good idea because of that? [22:02:36] if it's stupid enough to let a created VM override the hostname->IP mapping of the NS record for the zone itself, that's not really the DNS's bug [22:02:49] that's designate's awful bug :P [22:03:03] sure :) [22:03:13] and I guess you'll have the glue tying this to an IP you control too [22:03:16] but then again, either way, those NS records should be rarely seen [22:03:36] ok [22:03:39] another question for you [22:03:40] because since they're in-zone, the .org delegator sends glue the cache believes [22:03:51] and there's no point emitting the NS records any other time, unless directly queried [22:03:58] inthis scheme, does this mean we should create ns0.56.15.185.in-addr.arpa A/AAAAs as well? :) [22:04:39] yes, I think if you applied the rule universally it would mean precisely that, but I think in practice in-addr.arpa probably doesn't even accept glue. [22:04:47] in any case, in-addr.arpa is kind of a special-case zone :) [22:05:00] maybe? I doubt anyone has tried it :) [22:05:08] (accept glue) [22:05:36] from a philosophical point of view, once you've accepted all the other opinion above, the next logical conclusion is that NS hostnames don't actually matter, only the IPs do, in such a world [22:06:06] you could generate fake random hostnames every time you reload the zone, and different ones every time you send any random update about the zone to your registrar. as long as they're in-zone. [22:06:31] and as long as the chain completes from zone->NS->A and is glued up in the delegation [22:07:07] which is effectively just a way of think of NS records pointing at IPs like they should have in the first place, just we're sending other pointless bytes too. [22:07:35] throughout this conversation I'm confused between your "ideal world" proposals vs. best operational practices for this world :) [22:07:41] but this is again jumping between the many context within which one can discuss glue [22:07:45] yes [22:08:25] there's even differing definitions of ideal above I think [22:08:35] i.e. if you could change what the specs/registrars/registries/root zone operators do, vs. what you think an individual zone operator like us should be doing [22:08:44] it's a murky, murky area, everything about glue. answers depend entirely on context. [22:09:43] for wikimedia, for the stuff in our ops/dns repo, because all of the domains we're talking about are all owned by us, and all serviced by the same set of servers, and all in our same domain of organizational control or whatever, I don't think it matters much which way you do things. [22:09:51] my PoV fwiw, is that needless glues are rare and kinda a pain in the butt right now, and I'd prefer to not have them to avoid sudden surprises around TTLs and IP changes and what not [22:10:07] from a cursory search it seems like org -> labs-ns0 is the exception rather than the rule [22:11:07] but I haven't ever had to implement a DNS server or recursor and deal with all the murky areas of the specs, so I may have an overly practical view of things :P [22:11:27] re: cursory search: in-TLD but out-of-delegated-zone glue is fairly common actually. [22:11:56] dig @a.gtld-servers.net foo.verisign.com A -> glued IPs for av[1234].nstld.com. [22:12:14] examples abound, it's pretty common, at least within a TLD [22:12:21] rare for us I meant! [22:12:47] but yeah, that's useful information too, I hadn't checked whether .com behaves like .org [22:12:47] it's there for all of our domains that are in .org, where ns[012].wikimedia.org are [22:12:54] I think! [22:13:20] e.g. wikiversity.org lookups show ns0.wikimedia.org glue [22:13:38] so back in that 2012 thread, markmonitor "created" a ns0.wikimedia.org "server" [22:13:43] separately from a delegation [22:13:49] and then linked the domain with that "server" [22:14:05] which sounds terrible to me as a model [22:14:16] but would explain why they have this flattened view of the world regarding additional sections [22:14:22] anyways, re: "my PoV ... avoid sudden surprises" - the delegation glue is common, and it's always more important than actual address data for nameserver hostnames queried directly. [22:15:02] paravoid: yeah but that's been the traditional standard practice for the old TLD operators, I don't think it ever really changed. [22:15:09] yeah [22:15:15] (to define "server" objects for glue) [22:15:40] so an "ns0.wmflabs.org" hostname could conceivably be created by a third-party [22:15:42] caches may eventually, indirectly, learn your real A records for hostnames that are in NS records, and then actually use that as part of lookups [22:15:48] but that really means that the ns0.wm.org <-> IP association exists (and can only be updated) only once [22:15:51] there's very few people who can create stuff directly under the wmflabs.org zone [22:15:59] but from a fresh state (or anytime the wrong bits fall off of TTL or LRU), it's the delegation glue that matters more. [22:16:23] most projects just have their subdomain [22:16:50] everything else has to be a web proxy [22:16:51] paravoid: right, the complication is if we own 200 domains, even all within .org, and do a hardcore in-zone-NS policy, if we want to change the IP of ns2 we have to ask for 200 changes. [22:17:05] yeah [22:17:17] and the opposite side of this is [22:17:22] which would point at our proxy server [22:17:31] and wouldn't serve their DNS records [22:17:33] that there is only one view of ns0.wikimedia.org, not a per domain one [22:17:42] if we update that once, it gets updated for all of our (.org) domains [22:17:56] one view of ns0.wikimedia.org in .org I mean [22:18:39] .org NSes won't (probably?!) respond with a different value for ns0.wm.org A in domain A's additional section and domain B's additional value [22:18:47] s/value$/section/ [22:21:01] they won't, but that's an implementation detail [22:21:21] (a good one!) [22:24:08] anyways, the wmflabs.org situation is similar to the wikiversity one, so in that sense it's "normal" [22:24:12] yeah ok [22:24:21] well, sort of :P [22:24:39] it's not invalid, but it would work equally if .org TLDs didn't have a glue for labs-ns0/1 [22:24:40] caches could choose not to believe the glue and go separately ask for a wikimedia.org delegation and find it [22:25:07] but since the .org servers are technically-trustworthy for everything under .org, it would seem pointlessly pedantic and inefficient for a cache to choose that. [22:25:07] equally well* [22:25:42] well, "equally", performance would be worse without that shortcut [22:26:36] for an empty cache, without (or disbelieving) the glue in that case, it would get wmflabs.org -> labs-ns0 delegation, then go back to .org and ask for labs-ns0's IP and get the wikimedia.org -> ns0.wikimedia.org delegation, then ask ns0.wikimedia.org for labs-ns0's IP [22:26:50] yeah [22:27:13] but if you're going to believe, in step two, org's response for where labs-ns0's IP delegates to, why wouldn't you believe the glue it gave for the same name? [22:27:26] (in security/trust terms, and avoid the inefficiency) [22:28:26] sure, but why let the parent zone dictate e.g. the TTL, which in this case is not even user-controllable [22:28:52] but note also, in thinking on all related things, that the A-record for labs-ns0 which exists in the org servers as glue, is only ever emitted as glue, not as a direct response [22:28:53] it's not wrong if you want the perf benefit, these days they may have called it "NS stapling" :P [22:28:59] may had* [22:29:27] if I ask org-server for foo.wmflabs.org/A, I get a delegation that includes the glue labs-ns0 IP [22:29:34] yeah sure [22:29:48] if I ask org-server for labs-ns0.wikimedia.org/A, I don't get the glued IP, I get the wikimedia.org delegation info [22:30:09] yeah that was what I expected :) [22:30:26] so in that sense in particular, the glue address is a sort of ephemeral quasi-record. It's not directly queryable, and only emitted to link up the two records in the delegation response packet [22:30:43] (pointlessly. it could be any in-zone name and do that, or the NS record could've just had an IP instead of a hostname) [22:32:09] any disagreements on https://gerrit.wikimedia.org/r/#/c/operations/dns/+/462574/ ? [22:32:20] the only reason it has meaning outside of the delegation response itself, is that caches notoriously cross-pollute data from anywhere else they can learn, and they could in fact indirectly learn a different IP for that hostname and re-target. [22:32:53] think it's kind of obvious, would have been nice to have a way to signal in the repo that we haven't forgotten it but it's actually delegated elsewhere, but oh well [22:34:03] just to wrap this up :P [22:34:12] (in my mind, even when the cross-pollution is security-valid, it's unecessarily complex to even think about) [22:35:08] paravoid: technically, you should wait out RIPE's standard TTLs first so that those which previously cached it aren't getting a REFUSED from our servers and calling it a lame delegation, but since the zone was useless before anyways, I doubt anyone will notice. [22:35:21] exactly [22:35:39] exactly my train of thought, I mean :) [22:36:57] so tying the last points above back to earlier about controlling the local TTL of labs-ns0's IP or whatever [22:37:15] (a) such glue is common in the TLD [22:38:00] (b) the glue at the TLD matters more than our A-records for the same name, for delegation purposes (clean caches go to the glue first, others might eventually cross-learn a differen TTL and/or IP from us, but unreliably and subject to cache expiry/eviction) [22:38:30] (c) So even having the separate A-record over in wikimedia.org is just a complicating factor sometimes. The IP and TTL at the delegation is what matters. [22:39:27] sure, I knew as much [22:39:57] we could just have a text file in that repo containing docs for the other zones which aren't controlled in that repo [22:39:58] my point was that... if we can remove the labs-ns0/1 glues from .org, then that would work untangle that, at the expense of performance (an extra query) [22:41:21] CI for ops/dns broken? [22:41:25] I guess that's true, but then as a general broader pov on the world of DNS, you're moving away from rather than towards glue [22:41:49] anyways [22:41:50] doesn't seem to have voted on my change at all [22:41:53] am I missing something? [22:42:01] it worked last I used it, but it's been a couple days [22:42:05] change number isn't showing on zuul [22:42:06] oh it did on PS1 [22:42:21] but not on PS2, submitted 7 minutes ago [22:42:29] it's given PS2 a V+2 [22:42:38] just did [22:42:46] so just lagging behind ok [22:42:56] haven't pushed a DNS commit in a long while, sorry :) [22:43:13] wasn't sure if it was lag, or something else [22:44:16] so authdns::lint is I think what CI uses to install the basics, and just does ensure=>installed for the package [22:44:33] it voted, I submitted, merged, authdns-updated, all is good [22:44:35] do those repuppet from scratch or autoupdate, or do we have to go log into them to get a newer package? [22:44:45] oh no idea, likely the latter [22:44:49] I mean in general, so we can start CI-ing with the new versoin we're running [22:44:59] for that matter, are the even stretch? :) [22:45:03] *they [22:45:15] once CI is updated we can start using fancier features. at least real CAA records