[06:39:06] templates/wikimedia.org:wikimedia.org. 5M IN CAA 0 iodef "mailto:dns-admin.org" [06:39:13] etc., for all zones [06:39:31] mediawiki.org is the only one with IN CAA 0 iodef "mailto:dns-admin@wikimedia.org" [06:39:44] I'm guessing this is accidental :) [07:56:48] <_joe_> volans: I think that if we agree something is not great, we should add to CI a job taking a /delta/ fo warnings, and maybe vote -1 if we're adding any [08:31:01] paravoid: it looks like a too eager replacement [08:33:07] introduced in b5f2af161e888f69820b14aecd961b5ab104d1dd / I856b161bb65802f0359ee14132e4753232015d5f [08:41:43] 10Traffic, 10Operations: Fix CAA iodef tags - https://phabricator.wikimedia.org/T211860 (10Vgutierrez) [08:42:17] 10Traffic, 10Operations: Fix CAA iodef tags - https://phabricator.wikimedia.org/T211860 (10Vgutierrez) p:05Triage>03Normal a:03Vgutierrez [08:42:37] would probably be quicker to push a patch than a task :) [08:47:40] OCD issues? no Bug line on the commit message :( [08:47:55] https://gerrit.wikimedia.org/r/#/c/operations/dns/+/479394/ [09:11:34] _joe_: you're late :-P I sent this yesterday (ok tonight :D ) https://gerrit.wikimedia.org/r/c/operations/dns/+/479358 [11:04:33] 10netops, 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): Renumber cloud-instance-transport1-b-eqiad to public IPs - https://phabricator.wikimedia.org/T207663 (10aborrero) Thanks! I love diagrams, they help me better understand topology and architectures. Please, confirm the following ar... [11:59:39] 10netops, 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): Renumber cloud-instance-transport1-b-eqiad to public IPs - https://phabricator.wikimedia.org/T207663 (10aborrero) For the D day: `lang=shell-session # create new subnet root@cloudcontrol1004:~# neutron subnet-create --gateway 208.... [12:48:02] 10Traffic, 10Operations: Fix CAA iodef tags - https://phabricator.wikimedia.org/T211860 (10Vgutierrez) 05Open>03Resolved [13:14:21] thanks! [13:14:31] (for fixing the CAA thing) [13:15:27] XioNoX: re: policies and infra IPs, I guess there's maybe some misunderstanding of the term "reserve" yesterday possibly. But either way, if we're going to use some IPs for infra and give them PTRs, we should probably give them forward names too. [13:16:13] XioNoX: but I haven't looked at them all to figure out if/how policy should be on the names (i.e. if there's good reasons to put them in the "wrong" zone, like private IPs with names in wikimedia.org and/or public IPs with names in wmnet) [13:17:20] are infra IPs (by that I mean e.g. router interfaces and vips, etc) going to be something that eventually comes from netbox as well, or (my guess) more likely to be manual still for the foreseeable future. [13:17:24] ? [13:19:26] I think both :P [13:19:56] i.e. eventually come from netbox, but we're a bit far from that atm, so manual for now? [13:20:03] I guess it depends how far is "foreseeable" :) [13:20:41] in any case these IPs aren't special in any way, whatever we do for the rest [13:20:43] sure [13:21:19] the special-est cases that need policy decisions really are all the intentional duplicates [13:21:52] duplicates in what sense? [13:22:23] e.g. the ones where we have a pair of A records for "machinename" and "servicename" pointing at the same IP today [13:22:45] templates/wikimedia.org:civi1001 1H IN A 208.80.155.11 [13:22:45] templates/wikimedia.org:civicrm 1H IN A 208.80.155.11 [13:22:49] that kind of thing [13:22:56] ah [13:23:10] we can just decide that in such cases we should use CNAME, it makes things make more logical sense [13:23:10] -s TOO_MANY_PUBLIC_NAMES ;) [13:23:50] really in that case you're using CNAME to make up for the zonefiles' lack of metadata about hostnames vs logical names, while netbox might have the metadata. [13:23:55] CNAME or DYNA/DYNC? [13:24:03] e.g. [13:24:03] wikimedia.org:332 git-ssh.eqiad.wikimedia.org. AAAA 2620:0:861:ed1a::3:16 [13:24:06] they're not really dynamic [13:24:06] wikimedia.org:334 git-ssh.wikimedia.org. AAAA 2620:0:861:ed1a::3:16 [13:24:35] but you could also take a deeper stance that in most cases these are "wrong" [13:25:01] git-ssh isn't a server hostname, I guess that's a different class of error [13:25:08] than the one you were just talking about [13:25:19] yeah but similar in nature [13:25:49] I don't think, as far as I know, that git-ssh is super warm where we could fail it over trivially anyways [13:26:00] but anyways, a CNAME would suffice just as well in this case [13:26:22] if there are no plans to multiDC it, why do we even have git-ssh.eqiad.wikimedia.org? :) [13:26:30] anyway, getting a bit too specific I guess [13:26:42] well the examples illuminate, it's ok to be specific :) [13:26:55] you're talking about the forest and I started talking about one of the trees :) [13:27:05] I suspect the idea with putting names like that in place is "we'll eventually multi-dc this, so get ready early" or something [13:27:33] I thikn that kind of thinking is probably ok, but maybe this is an over-application of it [13:27:45] note there is a git-ssh.codfw.wikimedia.org [13:27:52] I just don't know how warm that really is [13:28:10] ah hadn't seen that [14:16:03] 10Traffic, 10Operations: Migrate most standard public TLS certificates to CertCentral issuance - https://phabricator.wikimedia.org/T207050 (10Vgutierrez) 05Open>03Resolved [14:16:17] \o/ [14:17:37] :) [14:18:11] you still have 17 days left in the quarter, we'll have to make up something awful to do with the time! :) [14:18:43] * vgutierrez reopens the issue [14:18:52] O:) [14:19:25] * bblack files a task to translate all traffic puppetization to Catalan naming for code comments, class, and variable names. [14:19:34] catalan? seriously? [14:19:55] I guess I have a pretty small blocker for that one, I do not speak catalan [14:20:48] it's ok just google search your way through it, isn't that how we get anything done in any language? :) [14:21:09] hahah [14:21:36] I'm wondering if I can come with a test for the renewal process on certcentral [14:22:31] pybal rebiews! [14:22:33] reviews [14:22:36] : [14:22:37] :) [14:22:39] vgutierrez: DNS fixes for all the warnings [14:22:40] * mark needs coffee [14:22:55] damn... it's the last time I mark a goal as resolved [14:23:06] * vgutierrez newbie :( [14:23:14] brandon: could you mark it as done on the wiki? :) [14:23:41] yesterday I was in a meeting reporting it as "nearly done" ;) [14:25:40] could get a head start on next quarters stuff [14:26:04] there's some open certcentral tasks and stuff for review [14:26:29] yep... [14:26:36] vgutierrez: jokes apart if you have spare time I could use some help for the spicerack goal ;) [14:31:34] so yeah, one of the meta-things coming up for all things certcentral [14:32:20] is to integrate it and sslcert::certificate a little more seamlessly in certain consumers that need both (e.g. so that caches can deploy acme and non-acme certs alongside each other trivially) [14:32:45] that may need no changes on the CC side, just on the p::cache::ssl::unified side [14:32:59] and make sure that tlsproxy still works for SAN-driven multi-certs [14:33:30] and make sure we can trivially puppetize OCSP for the static + acme cases as well [14:34:31] all that leads to being able to use CC for various traffic cases: wikiba.se cert, 3rd unified for wildcards (which we might not turn on at runtime anywhere initially, but least have as a tertiary backup!), and working on the noncanonical redirect service. [14:36:34] 10Traffic, 10Operations, 10Wikidata, 10wikiba.se, and 2 others: [Task] move wikiba.se webhosting to wikimedia cluster - https://phabricator.wikimedia.org/T99531 (10Addshore) As far as I remember from an IRC discussion, the next step here is to wait for the WMF to get a cert for the domain, then we can move... [14:40:56] 10Traffic, 10Operations, 10Wikidata, 10wikiba.se, and 2 others: [Task] move wikiba.se webhosting to wikimedia cluster - https://phabricator.wikimedia.org/T99531 (10BBlack) There's still a couple of things that can be done serially at present, one of which is necessary for the cert issuance later: 1. Switc... [14:41:43] also, there's the whole "rename CC" topic heh [14:42:09] yeah, there were some suggestions in the task [14:42:10] I really don't have strong opinions on what the name is, just not the conflicty one we have now [14:42:21] Krenair: could you review them and add your own? [14:42:23] ah I hadn't seen the suggestions, I'll go look [14:43:01] but I think that right now it's a good moment to go through the renaming process [14:43:02] ok [14:51:53] ema: i just updated to grafana 5.4.2 which fixes the typing in filter box bug, btw :) [14:52:18] mark: done [14:54:45] tx :) [14:55:59] cdanis: it does indeed! ty [14:56:18] i also filed https://github.com/grafana/grafana/issues/14486 upstream as well, haha [14:56:23] so many problems with such a little dropdown box [15:56:59] 10Traffic, 10Operations, 10Wikidata, 10wikiba.se, and 2 others: [Task] move wikiba.se webhosting to wikimedia cluster - https://phabricator.wikimedia.org/T99531 (10Dzahn) For part 2. of this please contact @CRoslof [17:22:11] 10netops, 10Operations, 10cloud-services-team (Kanban): labtestn neutron router is accesible from the internet - https://phabricator.wikimedia.org/T211901 (10aborrero) p:05Triage>03Normal [19:01:19] XioNoX: I'm questioning my sanity now: have we always only routed in port 53 public authdns for UDP and not TCP? [19:01:46] bblack: ECANTPARSE [19:02:13] the rules on the routers that forward e.g. the ns0.wikimedia.org IP -> authdns1001 IP, and whatever filtering is applied on inbound traffic for those IPs/ports [19:02:17] 10Domains, 10Traffic, 10Operations, 10WMF-Legal, 10Patch-For-Review: Move wikimedia.ee under WM-EE - https://phabricator.wikimedia.org/T204056 (10CRoslof) @Dzahn Yes, we plan to renew all our current .ee domain registrations. [19:02:18] if you're mentioning routing only, then it's for the full IP not per port [19:02:22] it seems like TCP connections aren't being allowed in [19:02:32] but I'm now unused whether that's always been the case [19:02:38] err s/unused/unsure/ [19:03:31] but when I look at e.g. cr1-eqiad config, I do see tcp seemingly allowed for port53 of the nsX IPs [19:03:33] bblack: and border-in4 allows tcp and udp on port 53 [19:03:48] maybe it's just me, let me find more test points [19:03:56] I am on unfamiliar wifi today [19:04:18] oh yeah, that's it, nevermind! [19:04:32] for some reason this network doesn't let me make outbound tcp on port 53 or whatever [19:04:45] that was very confusing for a while! [19:04:47] (don't offer to fix their wifi, it's a trap) [19:05:00] it's a double-trap, it's family wifi! [19:05:08] hehe [19:05:09] never offer to help solve family technical issues! :) [19:05:47] bblack: for my own knowledge... is dig +tcp @ns0.wikimedia.org en.wikipedia.org to check this? [19:05:54] *enough to check [19:06:58] volans: yeah [19:08:39] thx, was checking my sanity :D and yeah that works fine [19:09:49] you can also trip through into it naturally with a truncated response [19:09:51] e.g. for now: [19:09:58] dig @ns0.wikimedia.org wikipedia.org ANY [19:10:07] (which will do UDP, get a truncated response, then retry over TCP) [19:13:26] yep, one had to recall which reply is long enough to get truncated :D [19:15:37] technically that one isn't long enough by traditional rules [19:15:59] but we have a reflection defense that, regardless of output size, ANY-requests just get universally truncated over to TCP [19:16:37] got it [19:16:55] also, i think dig automatically does EDNS with a 4K bufsize, and gdnsd in our (default) config limits that to 1410 over IPv4 [19:17:12] so you'd have to find a response bigger than 1410 to get dig to truly get a non-ANY truncation from us [19:17:20] or use dig's +bufsize=512 or something [19:17:50] I don't think we have any responses that are big at all, but I'm not 100% [19:18:12] ehehe [19:18:18] makes sense [19:18:23] thanks for the info [19:33:54] 10netops, 10Cloud-VPS, 10Operations: Create cloud-in4 filter for labtestn - https://phabricator.wikimedia.org/T211921 (10chasemp) [19:34:02] 10netops, 10Cloud-VPS, 10Operations: Create cloud-in4 filter for labtestn - https://phabricator.wikimedia.org/T211921 (10chasemp) p:05Triage→03Normal [19:37:45] and now we have nsid for future debugging of routing/anycasting [19:37:49] e.g. [19:37:54] dig +nsid @ns0.wikimedia.org en.wikipedia.org [19:38:08] (among normal output includes): [19:38:11] ; NSID: 61 75 74 68 64 6e 73 31 30 30 31 ("authdns1001") [19:39:16] and cookies! :) [19:41:13] 10Domains, 10Traffic, 10Operations, 10WMF-Legal, 10Patch-For-Review: Move wikimedia.ee under WM-EE - https://phabricator.wikimedia.org/T204056 (10Dzahn) Ok, thanks @CRoslof [20:13:03] 10netops, 10Operations: Add eqsin routing special cases to jnt - https://phabricator.wikimedia.org/T211930 (10ayounsi) p:05Triage→03Normal [20:37:32] 10Traffic, 10Operations, 10Performance-Team (Radar), 10Services (designing), and 3 others: Harmonise the identification of requests across our stack - https://phabricator.wikimedia.org/T201409 (10Krinkle)