[00:04:51] putting JSON inside a response header with VCL, and then writing VTC for that, is, uh, really something [00:21:59] https://gerrit.wikimedia.org/r/c/operations/puppet/+/627629/7/modules/varnish/templates/wikimedia-frontend.vcl.erb#165 [04:45:11] 10Domains, 10Traffic, 10Analytics, 10Operations, 10Wikimedia-General-or-Unknown: Blocking all third-party storage access requests - https://phabricator.wikimedia.org/T262996 (10RolandUnger) [04:46:38] 10Domains, 10Traffic, 10Analytics, 10Operations, 10Wikimedia-General-or-Unknown: WMF third-party cookies rejected - https://phabricator.wikimedia.org/T262882 (10RolandUnger) [05:51:34] 10netops, 10Operations: Audit Juniper EX snapshots version - https://phabricator.wikimedia.org/T262290 (10ayounsi) [06:42:33] 10Traffic, 10netops, 10Operations: Wikimedia projects not reachable for some Telecom Italia users - https://phabricator.wikimedia.org/T262869 (10Nemo_bis) >>! In T262869#6464839, @Andyrom75 wrote: > ~15min ago connection has been restored. I'll test it again tomorrow. [A blog](https://www.optimagazine.com/2... [07:58:27] 10Traffic, 10Gerrit, 10Operations, 10Release-Engineering-Team-TODO, and 2 others: Enable avatars in gerrit - https://phabricator.wikimedia.org/T191183 (10hashar) 05Open→03Declined The summary is: We can't use the third party service gravatar.com since that leaks personal information to a third party.... [08:12:47] cdanis: :) [08:38:52] 10Acme-chief, 10Traffic, 10Operations: Let's Encrypt transitioning to ISRG's Root - https://phabricator.wikimedia.org/T263006 (10Vgutierrez) [08:48:34] 10Acme-chief, 10Traffic, 10Operations: Let's Encrypt transitioning to ISRG's Root - https://phabricator.wikimedia.org/T263006 (10ema) p:05Triage→03High [09:15:40] 10netops, 10Operations: Audit Juniper EX snapshots version - https://phabricator.wikimedia.org/T262290 (10ayounsi) 05Open→03Resolved All done! [09:53:03] 10Acme-chief, 10Traffic, 10Operations: Let's Encrypt transitioning to ISRG's Root - https://phabricator.wikimedia.org/T263006 (10ema) I have backported the patch to python-acme-0.31.0, here it is for review and future reference. ` Description: Backport of upstream patch "certbot: add --preferred-chain" Auth... [13:36:31] 10Traffic, 10Operations: Don't set cookies for api.wikimedia.org at the caching layer - https://phabricator.wikimedia.org/T260943 (10eprodromou) @Nuria we split the conversation here into T259296 and this ticket. @hnowlan is taking care of making sure the API server doesn't pass through cookies, so it's really... [13:37:27] 10Traffic, 10Operations, 10Platform Team Initiatives (API Gateway), 10Platform Team Sprints Board (Sprint 1), and 2 others: Client Developer has a cookie-free API call - https://phabricator.wikimedia.org/T258748 (10eprodromou) I'm moving this to blocked until we work out how to get the cookies out of the A... [14:34:58] so ema -- the patch still needs work (and I'm about to get back to that), but, did the overall structure seem reasonable to you? [14:39:39] cdanis: the structure does look good, "returning JSON from VCL" and "reasonable" should likely not be part of the same sentence though [14:40:13] ahah yes, "as reasonable as possible" I'll take [14:40:49] cdanis: is there a task that we can mention in the commit log / a VCL comment? [14:40:53] ema: I was going back and forth yesterday if I wanted to maybe-overcomplicate things by allowing the JSON values to be specified in hiera -- probably one for each wiki deployment group -- and writing ERB code to then translate to VCL, but, I think that's not necessary [14:40:56] yes ofc! [14:41:07] https://phabricator.wikimedia.org/T257527 [14:41:12] if you haven't seen it, worth a quick skim [14:41:37] I have not, reading [14:45:12] > Build a reasonably-nice Logstash dashboard [14:45:13] :) [14:45:44] yeahhhhhh [14:50:00] very cool [15:13:26] bblack: by any chance you around? [15:16:31] volans: somewhat, but in a meeting [15:16:41] what's up? [15:17:42] no hurry, can wait. Wanted your input if you have any preference/caveat on how to proceed with the DNS migrations. First patch is https://gerrit.wikimedia.org/r/c/operations/dns/+/627605 (see also my comments/questions there) [16:04:57] 10Acme-chief, 10Traffic, 10Operations, 10Patch-For-Review: Let's Encrypt transitioning to ISRG's Root - https://phabricator.wikimedia.org/T263006 (10Vgutierrez) So I've prepared a 0.29 release shipping https://gerrit.wikimedia.org/r/q/topic:%22T263006%22+(status:open%20OR%20status:merged) I've tested it m... [16:34:17] 10Acme-chief, 10Traffic, 10Operations, 10Patch-For-Review: Let's Encrypt transitioning to ISRG's Root - https://phabricator.wikimedia.org/T263006 (10BBlack) @Vgutierrez Yes, sounds good! [16:35:53] volans: I guess the main question is how we handle wikimedia.org, and then also sub-questions about the existing poorly-placed lvs per-vlan hostnames [16:36:20] (they're also lacking AAAA) [16:36:52] for the subquestion it's more for netops to decide, I'll happily adapt to any needed use case :) [16:37:12] well us really, it's all LVS [16:37:26] yeah I meant to include traffic in that netops [16:37:28] they're just convenience labels for allocations in theory [16:37:40] we never allocated v6 and use auto-addresses for those [16:37:52] wikimedia.org part is tricky [16:38:04] what is tricky? [16:38:05] I guess there's no way to partially deploy the wikimedia.org include? [16:38:14] as per-dc? [16:38:20] yeah or something of that nature [16:38:36] not right now, we were discussing with chaomodus earlier of that possibility inferring it from te prefixes and hence the vlans [16:38:54] but all this additional work would make sense only for the migration, and be kinda useless after that AFAICT [16:39:21] it does seem error-prone to deploy reverse via netbox while leaving forward manual, for these per-dc wikimedia.org hostnames [16:39:23] but if we feel strongly about it can probably do it [16:39:47] how much they change? we think within few weeks, less than a month to be able to migrate everything [16:39:48] unless we have something at least checking that it doesn't get messed up for the duration of the migration(s) [16:40:19] I've added myuself as reviewer to all dns changes since a while fwiw :) [16:40:23] ok :) [16:40:32] volans(tm) - the ultimate CI script! [16:40:37] rotfl [16:42:44] I can have a look tomorrow on how much work would that be and report back [16:42:55] maybe it's a quick win if all the data is already there [16:58:30] ack [17:00:10] in a sense error prone==additional checksum [17:15:10] 10netops, 10Operations, 10ops-codfw: (Need by: ) codfw:rack/setup/new management switches - https://phabricator.wikimedia.org/T253154 (10Papaul) [17:45:45] bblack: and would you split by DC or by prefix? also any proposal for the zone name? like codfw.wikimedia.org (but that looks like a real zonename) [17:52:13] prob wikimedia.org-codfw makes more sense [17:52:20] so it's clearly not a real zone name [18:00:37] 10netops, 10DC-Ops, 10Operations, 10ops-eqiad: Telia IC-361191) patch - https://phabricator.wikimedia.org/T261791 (10RobH) I've been working with Chris on this today, and after testing, it was determined our optic in xe-3/3/7 was not outputting correctly. It measured a -1.x output from the module, but a p... [18:04:49] 10netops, 10DC-Ops, 10Operations, 10ops-eqiad: Telia IC-361191) patch - https://phabricator.wikimedia.org/T261791 (10RobH) > Telia Implementation Team, > We ended up having to swap the optic on our end of the link, as its TX out was not acceptable.  Its now showing a much better TX light out (-5) at our dm... [18:14:55] we have few weird stuff too, like 208.80.154.221 (eqiad block) named gr-0-1-0-1.cr2-esams.wikimedia.org [18:15:36] that will endup in wikimedia.org-eqiad [18:19:26] but it's doable, I've a local patch working [18:27:36] it's reasonable for that to end up in eqiad given what i've seen of these things [18:28:00] i think the esams more shows its a connection to esams. [18:49:56] patch sent [19:06:04] right anywhere we have a link-local thing between DCs you'll see cases like that [19:06:16] there has to be a network for the link itself, which exists between them :) [19:15:07] 10Traffic, 10MediaWiki-Stakeholders-Group, 10Operations, 10TechCom-RFC, and 3 others: RFC: API-driven web front-end - https://phabricator.wikimedia.org/T111588 (10Krinkle) [19:17:10] 10Traffic, 10MediaWiki-Stakeholders-Group, 10Operations, 10TechCom-RFC, and 3 others: RFC: API-driven web front-end - https://phabricator.wikimedia.org/T111588 (10Krinkle) 05Open→03Declined Closing old RFC that is not yet on to [our 2020 process](https://www.mediawiki.org/wiki/Requests_for_comment) and... [19:18:44] so with the current patch I've sent we generated wikimedia.org-$dc and in wikimedia.org we have left for now only the nsa record because being anycast its parent prefix has no DC set in netbox [19:18:59] I think it's reasonable and should allow for the required flexibility [19:31:12] right [19:31:20] that subnet isn't actually part of ulsfo [19:31:42] (the anycast one) [19:31:46] ulsfo just has its /24 [19:32:09] the anycast one doesn't belong to any one DC. there will be a few of those [19:32:38] there's a private v4 one too at 10.3.0.0/24 [19:33:15] from an ipam perspective, it's probably easiest to define a special datacenter named "anycast" for all those cases [19:43:01] that's a possibiloty, but leaving their records into the zonename without suffix seems reasonable too no? [19:43:21] I can even hardcode one for those that don't have a site [19:45:11] the private one doesn't need special treatment, it's already in anycast.wmnet and related reverse [19:45:23] 0.3.10.in-addr.arpa [21:27:32] 10Traffic, 10netops, 10Operations: Wikimedia projects not reachable for some Telecom Italia users - https://phabricator.wikimedia.org/T262869 (10Andyrom75) Today, all is fine. I would consider this issue closed and solved. [21:28:28] 10Traffic, 10netops, 10Operations: Wikimedia projects not reachable for some Telecom Italia users - https://phabricator.wikimedia.org/T262869 (10CDanis) >>! In T262869#6468336, @Andyrom75 wrote: > Today, all is fine. I would consider this issue closed and solved. Thanks Andy, but we are going to keep it ope...