[08:51:27] godog: I was learning about certspotter when I saw your CR regarding Let's encrypt certificates, maybe we could leverage certspotter to deliver metrics about certificate expiration on an automated way [09:09:11] vgutierrez: intriguing, how would that work? [09:10:52] something related we started is checking internal/external endpoints with blackbox_exporter and get prometheus metrics out of that https://phabricator.wikimedia.org/T169860 [09:11:43] for tls checks that includes expiration time as well, though we'd need to expand blackbox monitoring coverage to more tls endpoints [09:12:19] certspotter automatically watches for us issued certificates for WMF domains (https://github.com/wikimedia/puppet/blob/b2b749102a65f0a88d228a837fc57147148db5ca/modules/role/manifests/certspotter.pp) [09:12:57] so for new certificates we already get this information on an automated way [09:14:06] right now we are only getting an email with this information, but using it to trigger automatic alerts and/or building a calendar with that information should be "easy" [09:17:43] yeah for externally issued certs that'd be nice to have too [09:17:53] I just noticed certspotter does send a fair amount of cronspam [09:22:12] yup... and more to come with "short lived certificates" [09:25:36] but instead of getting "spam" we could use that information on a more effective way [09:27:01] <_joe_> you mean programmatically parsing certspotter's output into something usable? [09:27:15] yep [09:27:26] <_joe_> I thought we already did that :P [09:27:28] or leveraging certspotter state directory [09:27:34] really? [09:27:50] <_joe_> vgutierrez: it's totally possible we don't [09:28:21] <_joe_> I would go as far as say that I might have heard/read someone saying they were interested into doing something with that data [09:28:30] I've only found this https://phabricator.wikimedia.org/T155807 [09:28:50] basically the setup task and [09:29:10] <_joe_> "There is more to do in this area (better alerts etc.) but these are for a future task" heh [09:29:23] yep :) [09:29:23] <_joe_> ok, that might be what I did remember [09:31:49] 10netops, 10Operations: rhenium running out of disk space on / - https://phabricator.wikimedia.org/T187688#3982552 (10fgiunchedi) [10:53:47] 10Traffic, 10Operations, 10Patch-For-Review: Support TLSv1.3 - https://phabricator.wikimedia.org/T170567#3435393 (10Vgutierrez) Could be interesting for us to roll out openssl 1.1.0? (https://www.openssl.org/blog/blog/2018/02/08/tlsv1.3/). OpenSSL 1.1.1 should be ABI and API compatible with OpenSSL 1.1.0, so... [10:57:27] 10Traffic, 10Operations, 10Patch-For-Review: Support TLSv1.3 - https://phabricator.wikimedia.org/T170567#3982896 (10MoritzMuehlenhoff) Yeah, that's the most plausible option (and we're already using custom OpenSSL 1.1 packages on Debian jessie to support e.g. chacha), but 1.1.1 has only just seen it's first... [16:51:48] bblack: I think we should discard the old VCL in reload-vcl? https://gerrit.wikimedia.org/r/c/412737/ Not doing so currently causes the geoip file handles to never get closed [19:37:03] 10Traffic, 10Operations, 10Performance-Team, 10Performance: missing H2 coalesce for upload.wm.o for images ref'd in projects' page outputs - https://phabricator.wikimedia.org/T116132#3984187 (10Gilles) Now that HTTP/2 is a thing and Zero won't be anymore, could we put connection coalescing for upload.wikim... [21:57:03] 10Traffic, 10Operations, 10Performance, 10Performance-Team (Radar): missing H2 coalesce for upload.wm.o for images ref'd in projects' page outputs - https://phabricator.wikimedia.org/T116132#3984383 (10Gilles)