[14:17:14] vgutierrez: in case you didn't see it yourself: https://community.letsencrypt.org/t/acme-v1-v2-validating-challenges-from-multiple-network-vantage-points/112253 [14:17:38] that smells like more work [14:17:41] * vgutierrez hides [14:17:42] ;P [14:18:02] I think we're okand no new work needed [14:18:13] yeah [14:18:24] TL;DR is they're going to start hitting our acme dns challenge data multiple times from multiple places [14:18:43] apparently starting like yesterday [14:18:50] yup, bday present for me apparently [14:18:56] :) [14:19:16] so people restricts incoming DNS traffic to auth servers? [14:19:32] hopefully not! [14:19:44] I think the customer issues they're expecting are things like: [14:20:14] "oh we only ever published it to our one server in north america, because all your tests seemed to always land there, we didn't set up our infra to replicate acme challenges to our other auth servers!" [14:20:47] and also: "We were very silly and designed our solution to yank the challenge back off the servers after we've seen it queried once" [15:12:37] <_joe_> hi, I'm going to merge the final boss of my war against the old lvs::configuration, https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/572215/ [15:12:55] <_joe_> I got a +1 earlier from valentin, but I'm going to be cautious and run puppet on one lvs at a time [15:13:03] <_joe_> the compiler says a noop anyways, [15:13:16] <_joe_> but I'll disable puppet for a while on the lvs servers, FYI [15:13:20] ack [15:27:47] 125G [15:28:50] ? [15:29:01] vi command in the wrong place :) [15:37:48] or maybe it's bblack from the future with 125G wireless technology [15:41:40] So I only updated 2 of the esams hosts yesterday, got busy and paranoid about doing changes super late in day ;D [15:41:53] I'll continue them later this afternoon, just echoing in here in case of blockers. [15:42:08] (also if anyone notices anything odd with esams two hosts from yesterday, let me know to stop) [15:42:28] I wont start again until Pacific tz PM [15:45:38] robh: no problem [15:45:45] =] [15:46:03] I've double checked this EU morning and everything looked good [15:46:18] cool [17:12:30] 10netops, 10Operations, 10Patch-For-Review: Add monitoring for BGP peers exceeding prefix-limit - https://phabricator.wikimedia.org/T239256 (10ayounsi) This surfaced that the v6 sessions to the Equinix router servers in Dallas and Ashburn have been down for quite a while. [17:19:52] bblack: I just got an email from digicert regarding safari's policy for future use to not allow certs longer than 398 days. Since we never renew them that long, we are unaffected, just sharing. [17:22:37] yeah I saw, seems like a good thing really :) [17:23:01] indeed [17:52:57] 10netops, 10Operations, 10cloud-services-team (Kanban): CloudVPS: enable BGP in the neutron transport network - https://phabricator.wikimedia.org/T245606 (10aborrero) Additional tests related to this are blocked on missing backported packages for the stretck-pike combo: `python3-os-ken` and `neutron-dynamic-... [21:11:05] 10Traffic, 10ContentSecurityPolicy, 10Gerrit, 10Operations, and 2 others: Add gerrit.wikimedia.org to the Phabricator CSP - https://phabricator.wikimedia.org/T218308 (10chasemp)