[08:39:24] <marostegui>	 akosiaris we are restarting m1 master (etherpad) in around 20 minutes
[08:44:43] <_joe_>	 marostegui: not sure alex is here rn
[08:44:49] <marostegui>	 ah, ok
[09:02:19] * akosiaris around. 
[09:25:15] <paravoid>	 akosiaris et al: https://lwn.net/Articles/843313
[09:30:04] <akosiaris>	 yup, saw it yesterday
[09:30:12] * akosiaris subscribed to the bug
[09:30:36] <akosiaris>	 I am not clear on what that means for bullseye though
[09:30:59] <akosiaris>	 it's on a soft freeze already, does this mean that kubernetes can make it in time for bullseye (if ready) or not?
[09:33:33] <jynus>	 quoting the article 'the actual response from Moritz Mühlenhoff was somewhat more nuanced than that... "and it would be presumptuous to pretend that we can seriously commit to fix security issues in k8s for longer than upstream"'
[09:33:42] <moritzm>	 this decision is only about keeping it in the archive in the current packaged form
[09:34:13] <moritzm>	 to keep it on stable someone need to reliably commit to rebase to the latest LTS releases once the former one EOLes
[09:34:29] <moritzm>	 and deals with potential bumps in Golang
[09:34:40] <moritzm>	 that's likely to happen, but still needs to be sorted out
[09:38:10] <paravoid>	 speaking of quotes, btw ema: you're quoted on LWN's "installing Debian on modern hardware" article from yesterday
[09:38:55] <ema>	 my mom always said I'd be famous one day!
[09:38:58] <jynus>	 do you have a link?
[09:39:32] <jynus>	 oh, subscription required
[09:40:41] <ema>	 jynus: https://lwn.net/SubscriberLink/843172/a082b27acdc0446b/
[09:51:23] <kormat>	 _joe_: huh. did i forget to add `bsection` to any profiles? doh.
[13:55:28] <_joe_>	 vgutierrez: so, I'm looking at the envoyproxy module
[13:55:48] <_joe_>	 you want to reuse the envoy::tls_terminator class?
[13:55:58] <vgutierrez>	 yes
[13:56:00] <_joe_>	 that has advantages and disadvantages :)
[13:56:21] <vgutierrez>	 I need to extend that a little bit for our purposes
[13:56:25] <_joe_>	 for one, I think you might end up needing a bit more options to be declared
[13:56:41] <_joe_>	 in the envoy config itself
[13:57:11] <_joe_>	 but yeah it's sensible overall
[13:57:39] <_joe_>	 oh btw, we should really move to the v3 api for the config :)
[13:57:59] <_joe_>	 https://phabricator.wikimedia.org/T265880
[13:58:34] <vgutierrez>	 maybe that's even a hard requirement for me
[13:58:50] <_joe_>	 ok, that's interesting
[13:59:13] <_joe_>	 we might want to first test the conversion in k8s, where it's easier to control the blast radius of the change
[14:01:21] <vgutierrez>	 it looks like it... type.googleapis.com/envoy.api.v2.auth.DownstreamTlsContext doesn't seem to support OCSP stapling
[14:01:33] <vgutierrez>	 and type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext does
[14:02:18] <_joe_>	 yeah expected.
[14:03:14] <vgutierrez>	 v2 doesn't seem to support the PROXY protocol and v3 does
[14:03:33] <vgutierrez>	 so yeah.. v3 is a hard requirement for us :)
[14:04:47] <_joe_>	 ok so, we can find someone from serviceops to work on converting to v3, but given the current envoy installations are very relevant, maybe not the best idea to start there
[14:05:12] <_joe_>	 you can add a "api_version" parameter to envoyproxy::tls_terminator I guess
[14:05:23] <_joe_>	 and to envoyproxy::cluster and envoyproxy::listener
[14:05:40] <vgutierrez>	 that's basically forking them :)
[14:06:12] <_joe_>	 well yes, only temporarily if you don't want to be constrained by our pace, which will be slower
[14:06:19] <vgutierrez>	 I'm wondering what's the approach here.. should I put some time in T265880?
[14:06:20] <stashbot>	 T265880: Upgrade envoy configuration to use the v3 API - https://phabricator.wikimedia.org/T265880
[14:07:08] <_joe_>	 vgutierrez: so, you need to write a v3 config anyways, and we need to do a controlled upgrade
[14:07:17] <_joe_>	 and if we had to pick, we'd start from k8s, not puppet
[14:07:34] <_joe_>	 where we also have better CI
[14:07:56] <vgutierrez>	 so we have some envoy config not managed by puppet used in k8s?
[14:08:18] <_joe_>	 yes
[14:08:36] <_joe_>	 basically our hope was to be done with envoy in puppet when we moved over mediawiki :P
[14:08:57] <vgutierrez>	 yeah.. I'm messing with your evil plans, sorry <3
[14:08:58] <_joe_>	 but the service proxy classes get configured via the same data structures
[14:09:26] <_joe_>	 in puppet and k8s
[14:09:37] <_joe_>	 that is of no interest to you obviously
[14:10:29] <_joe_>	 I mean the differences are not huge
[14:11:02] <vgutierrez>	 so I could define v3 types on puppet and make the class(es) accept both to render v2 or v3 configs
[14:11:15] <vgutierrez>	 it should be a NOOP for you and v3 configs for me
[14:11:23] <_joe_>	 but really given you need a v3 config, we can look just at the tls terminator and fork the templates for now
[14:11:31] <_joe_>	 and we will move over as soon as we're ready
[14:11:49] <_joe_>	 https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/common_templates/0.2/_tls_helpers.tpl#170 is the configuration in k8s btw
[14:16:33] <vgutierrez>	 _joe_: so it would be acceptable to refactor https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/envoyproxy/manifests/tls_terminator.pp#83 to something like Variant[Array[Envoyproxy::Tlsconfig, Envoyproxy::TlsconfigV3] $upstreams = [], and go on from there?
[14:18:59] <_joe_>	 vgutierrez: yeah, definitely
[14:20:14] <vgutierrez>	 awesome
[14:28:20] <_joe_>	 vgutierrez: your choice wether to fill the templates with ifs or just duplicate them for now
[15:00:37] <dcaro>	 volans: is there a way to disable clustershell showing the stdout when the command succeeds? (I just want to gather the output do do stuff with it, but showing a dump of everything seems too verbose)
[15:00:55] * dcaro talking about spicerack/clustershell completely out of the blue xd
[15:01:34] <volans>	 dcaro: hey, it's a long story, let me try the TL;DR version
[15:02:02] <_joe_>	 🍿
[15:02:20] <_joe_>	 dcaro: how many hours do you have?
[15:02:33] <dcaro>	 xS
[15:02:58] <volans>	 yes it will be possible fairly soon, no it was not possible before, for a while we had to disable it globally for a bug with some of the deps. If you *really* need it *right now* there's an hack to do that ;)
[15:03:58] <volans>	 btw good question/catch, it's the main techdebt point of cumin :/
[15:08:56] <dcaro>	 okok, I can bear with it, but let me know if I can help
[15:09:23] <volans>	 sure, thanks!
[15:14:00] <_joe_>	 dcaro: I usually just send stdout to /dev/null in the command btw
[15:14:32] <dcaro>	 _joe_: but then I would lose it and not be able to get it in the cookbook no?
[15:14:44] <volans>	 yes
[15:14:56] <_joe_>	 oh sorry, I misunderstood, you wanted to capture it *and* suppress it
[15:15:08] <_joe_>	 yeah it's one of my biggest pain points
[15:15:14] <_joe_>	 and the hack isn't pretty :)
[16:38:03] <marostegui>	 hnowlan: Are you creating the index on m2 master?
[16:50:26] <hnowlan>	 marostegui: gmodena just did a few minutes ago - did it cause issues?
[16:50:41] <hnowlan>	 marostegui: oh! I see the ticket :/
[16:50:49] <marostegui>	 yes, please do not create anymore for now
[16:50:57] <hnowlan>	 ack, will avoid.
[16:51:16] <marostegui>	 thanks
[19:29:25] <longma>	 Hi, I was wondering how long it should take before a helm chart gets published after merge to deployment-charts?
[19:34:20] <cdanis>	 longma: assuming you mean that /srv/deployment-charts on the deploy* hosts is updated -- looks like that happens once a minute
[19:36:00] <longma>	 my chart doesn't appear to have been published to the chart museum
[19:37:25] <longma>	 (https://helm-charts.wikimedia.org/stable/api/charts/rdf-streaming-updater)
[19:37:35] <cdanis>	 hm
[19:37:37] <cdanis>	 "New charts/chart versions from operations/deployment-charts repository are packed and pushed to ChartMuseum every 2 minutes via systemd timers on the ChartMuseum nodes."
[19:38:36] <longma>	 I wonder if there's an issue since it got merged over 20 min ago
[19:38:44] <cdanis>	 yeah it seems like
[19:38:56] <cdanis>	 https://phabricator.wikimedia.org/P13880
[19:39:02] <cdanis>	 I see log output that looks like this on both chartmuseum hosts
[19:39:07] <cdanis>	 so that's interesting
[19:39:30] <longma>	 hmm
[19:40:31] <cdanis>	 the 'stable' line on 'total chart versions' also increments around that time https://grafana.wikimedia.org/d/NY0yTkwiz/chartmuseum?viewPanel=29&orgId=1&refresh=1m&from=now-3h&to=now
[19:41:03] <longma>	 can you see the new package on the chartmuseum hosts?
[19:41:43] <cdanis>	 wait, now I see 0.0.2 on the helm-charts.wm.o link
[19:41:59] <cdanis>	 ahh, we cache charts in the CDN??
[19:42:09] <longma>	 oh, I have no idea lol
[19:42:33] <cdanis>	 yeah, we do
[19:42:35] <cdanis>	 hm
[19:43:52] <cdanis>	 although chartmuseum doesn't seem to send a cache-control resposne header?
[19:45:18] <longma>	 I don't see one in the response. I set no-cache in my request
[19:45:52] <cdanis>	 yeah, but, at the same time, I see this in the CDN response:
[19:45:57] <cdanis>	 < x-cache: cp2033 miss, cp2037 hit/3
[19:46:28] <cdanis>	 I suspect we probably do want to cache, but probably only for up to a minute or two?
[19:46:40] <cdanis>	 (I can file a task about that)
[19:46:56] <longma>	 okay, thanks for looking into it
[19:48:08] <cdanis>	 np :)
[19:48:14] <cdanis>	 longma: is the new one live from your pov now? if not I can purge the URL from the CDN
[19:48:55] <longma>	 no I still don't see it
[19:49:33] <longma>	 There it is
[19:49:35] <longma>	 Thank you!
[19:49:38] <cdanis>	 ✔️ cdanis@mwmaint1002.eqiad.wmnet ~ 🕒☕ echo https://helm-charts.wikimedia.org/stable/api/charts/rdf-streaming-updater | mwscript purgeList.php
[19:49:40] <cdanis>	 :)
[19:49:42] <cdanis>	 np!
[20:20:13] <cdanis>	 btw filed https://phabricator.wikimedia.org/T272633