[00:09:38] bblack ^ heh meant to say, "thanks so so much" ahhh it's too late for more coffee [00:10:22] :) [12:16:15] 10Traffic, 10InternetArchiveBot, 10SRE, 10Platform Team Workboards (Clinic Duty Team): IAbot sending a huge volume of action=raw requests - https://phabricator.wikimedia.org/T269914 (10jijiki) @Cyberpower678 here is a sample: https://grafana.wikimedia.org/d/5E7tdiGWz/xxxx-effie?viewPanel=38&orgId=1&from=16... [12:48:16] 10Traffic, 10Desktop Improvements, 10SRE, 10Bengali-Sites, and 4 others: CDN cache revalidation on several wikis for desktop improvements deployment pt 2 - https://phabricator.wikimedia.org/T274784 (10ovasileva) 05Open→03Resolved a:03ovasileva >>! In T274784#6893902, @BBlack wrote: > ^ There was a la... [14:43:40] 10Traffic, 10DNS, 10SRE, 10serviceops, and 3 others: DNS for GitLab - https://phabricator.wikimedia.org/T276170 (10wkandek) gerrit.wikimedia.org lives on a second IP address on gerrit1001. Should we follow that model here as well? [20:30:21] 10Traffic, 10DNS, 10SRE, 10serviceops, and 3 others: DNS for GitLab - https://phabricator.wikimedia.org/T276170 (10Dzahn) >>! In T276170#6896563, @wkandek wrote: > gerrit.wikimedia.org lives on a second IP address on gerrit1001. Should we follow that model here as well? It would be appropriate in the "exa... [22:11:44] cergen / TLS question: I'm updating the existing certificate for `wdqs.discovery.wmnet` and one of the early steps in https://wikitech.wikimedia.org/wiki/Cergen#Update_a_certificate is to do a `puppet cert clean wdqs.discovery.wmnet` - but wouldn't this revoke the cert and thus cause issues for wdqs until the new cert is published? [22:14:19] ryankemper: I think that's likely true, and I suspect the answer is that you need to disable puppet all the affected hosts until you're 'done' and trust that the new cert works [22:19:08] yes, that's how I did the last one. disable puppet on all the affected hosts, do the cergen steps on the puppetmaster, copy out the newly-generated public key to the public-repo files/ssl/whatever . [22:19:25] commit+merge the public and private updates, then go re-enable puppet and run it on the affected hosts [22:20:24] probably starting with just one host to make sure it works :) [22:21:06] bblack / cdanis: and the affected hosts are the wdqs hosts here, but not any of the `cp*` right? in other words, `cp*` will keep presenting the old certificate, but with puppet disabled the `wdqs*` hosts won't realize that the cert's been revoked? [22:21:39] I am not at all confident in this but I don't think we do any CRL stuff with the puppet CA [22:22:03] so yeah, disable puppet on the wdqs hosts; they'll keep presenting the old certificate, which the cp* hosts will still accept, even if puppet runs on them after your change [22:22:11] yeah I don't think they get actively revoked in realtime [22:24:53] got it, that all makes sense [22:25:22] now that I think about it it might make sense just to make a net-new cert anyway and leave the existing `wdqs.discovery.wmnet` as is [22:25:44] since this is effectively a new service and we already have a different cert for `wdqs-internal.discovery.wmnet` [22:36:20] 10Traffic, 10InternetArchiveBot, 10SRE, 10Platform Team Workboards (Clinic Duty Team): IAbot sending a huge volume of action=raw requests - https://phabricator.wikimedia.org/T269914 (10jijiki) And today as well, so we can say that this happens every day. {F34148488} [23:33:51] 10Traffic, 10DNS, 10SRE, 10serviceops, and 3 others: DNS for GitLab - https://phabricator.wikimedia.org/T276170 (10wkandek) Let's go with the simpler solution and use the CNAME.