[01:08:18] 10Wikimedia-Apache-configuration, 10Continuous-Integration-Infrastructure, 10Pywikibot-core, 10Release-Engineering-Team, and 2 others: Pywikibot documentation showing broken directory listing - https://phabricator.wikimedia.org/T132136#4215978 (10Dzahn) The reason for this is that Apache is explicitly told... [01:17:54] 10Wikimedia-Apache-configuration, 10Continuous-Integration-Infrastructure, 10Pywikibot-core, 10Release-Engineering-Team, and 2 others: Pywikibot documentation showing broken directory listing - https://phabricator.wikimedia.org/T132136#4215979 (10Dvorapa) @Dzahn Weird is, that other projects (like oojs men... [01:26:36] 10Wikimedia-Apache-configuration, 10Continuous-Integration-Infrastructure, 10Pywikibot-core, 10Release-Engineering-Team, and 2 others: Pywikibot documentation showing broken directory listing - https://phabricator.wikimedia.org/T132136#4215991 (10Dvorapa) Some projects also return `403 Forbidden`, but othe... [07:47:17] 10Traffic, 10Operations, 10Wikimedia-Hackathon-2018: Create and deploy a centralized letsencrypt service - https://phabricator.wikimedia.org/T194962#4216086 (10Krenair) a:03Krenair [07:48:33] bblack, hey, you around? [07:50:49] got a question about T194962 [07:50:49] T194962: Create and deploy a centralized letsencrypt service - https://phabricator.wikimedia.org/T194962 [07:51:07] why the two different APIs for pulling secrets down? [13:38:09] 10Traffic, 10Operations, 10Wikimedia-Hackathon-2018: Create and deploy a centralized letsencrypt service - https://phabricator.wikimedia.org/T194962#4216999 (10Krenair) Some of my work on this is being blocked by T195059 [13:49:25] 10Traffic, 10Operations, 10Availability (MediaWiki-MultiDC): Create HTTP verb and sticky cookie DC routing in VCL - https://phabricator.wikimedia.org/T91820#4217093 (10tstarling) I explained ChronologyProtector to @Joe and @BBlack just now. They seemed happy with the idea of not sending a useDC cookie for no... [14:46:20] 10Traffic, 10Operations, 10Wikimedia-Hackathon-2018: Create and deploy a centralized letsencrypt service - https://phabricator.wikimedia.org/T194962#4217229 (10Krenair) https://krenair.hopto.org is running on a labs machine There's a central LE service on there which uses acme_tiny to request the cert from L... [15:15:28] 10Traffic, 10Operations, 10Wikimedia-Hackathon-2018: Create and deploy a centralized letsencrypt service - https://phabricator.wikimedia.org/T194962#4217296 (10BBlack) Some after-thoughts on design issues and such (I haven't looked at any code!): * We should look hard for a good abstract ACME library that a... [15:19:46] 10Traffic, 10Operations, 10Wikimedia-Hackathon-2018: Create and deploy a centralized letsencrypt service - https://phabricator.wikimedia.org/T194962#4217300 (10BBlack) Also: * We should assume by default we want all certificates to be dual-issued as ECDSA+RSA variants and served to clients in both forms (I... [15:23:43] 10Traffic, 10Operations: Setup a new PKI software as an alternative to the puppet CA for managing services certificates - https://phabricator.wikimedia.org/T194031#4186323 (10BBlack) @Joe - So we're looking at doing something just for the LetsEncrypt (ACME) use-case over in T194962. The idea is this will mana... [15:25:39] 10Traffic, 10Operations, 10Wikimedia-Hackathon-2018: Create and deploy a centralized letsencrypt service - https://phabricator.wikimedia.org/T194962#4214163 (10BBlack) Also: naming bikshedding stuff: we should name/implement this as a generic ACME tool rather than LE-specific, and just make LE be the default... [15:29:16] 10Traffic, 10Operations, 10codfw-rollout: Enable VCL applayer datacenter-switch via confd - https://phabricator.wikimedia.org/T127485#4217331 (10BBlack) 05Open>03declined At this point, we're pushing off commit-free DC switching to post-ATS (sometime during the latter part of next FY, probably). [15:38:54] 10Traffic, 10Operations, 10Availability (MediaWiki-MultiDC): Create HTTP verb and sticky cookie DC routing in VCL - https://phabricator.wikimedia.org/T91820#4217378 (10BBlack) Right. Just to re-state for clarity, the sort of logic we should be implementing in VCL (in the cache layers) will look like this ps... [15:39:09] 10Traffic, 10Operations, 10Wikimania-Hackathon-2018, 10Availability (MediaWiki-MultiDC): Create HTTP verb and sticky cookie DC routing in VCL - https://phabricator.wikimedia.org/T91820#4217380 (10BBlack) [15:45:11] 10Traffic, 10Operations, 10Wikimania-Hackathon-2018, 10Availability (MediaWiki-MultiDC): Create HTTP verb and sticky cookie DC routing in VCL - https://phabricator.wikimedia.org/T91820#4217391 (10Joe) About ChronologyProtector: - If ChronologyProtector kicks in, it should send back a specific header to v... [15:51:29] grate with puppet in our case, which greatly simplifies the puppet-level configuration for cert deployment to all the endpoint hosts (in terms of resolving puppet dependencies only when the files change, etc) [15:51:38] bleh I missed a line of output there [15:52:51] Krenair: re: the 2x protocols: the simple fetching one is easy and universal (e.g. for others reusing this code with other integrations). But having the puppet fileserver protocol (which is just a specific REST API using json responses) allows us to directly integrate with puppet in our case, which greatly simplifies the puppet-level configuration for cert deployment to all the endpoint hosts (i [15:52:57] n terms of resolving puppet dependencies only when the files change, etc) [15:57:14] 10Traffic, 10Operations, 10Wikimania-Hackathon-2018, 10Availability (MediaWiki-MultiDC): Create HTTP verb and sticky cookie DC routing in VCL - https://phabricator.wikimedia.org/T91820#4217406 (10BBlack) Right, I forgot, that was discussed as an optimization (vs having ChronologyProtector just timeout -> f... [16:20:41] 10Traffic, 10Operations, 10Wikimania-Hackathon-2018, 10Availability (MediaWiki-MultiDC): Create HTTP verb and sticky cookie DC routing in VCL - https://phabricator.wikimedia.org/T91820#4217453 (10tstarling) Currently, ChronologyProtector times out after 10 seconds, this is configurable. Timeout causes "lag...