[07:57:00] _joe_: morning, I have a patch pending for our python-build image in order to be able to reuse existing wheels. Would be nice to have and it is blocking me for zuul dependencies update :) https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/605653 [07:57:15] or maybe I can add some others to review / build the images [07:57:48] <_joe_> hashar: I thought alex did review that before leaving, clearly he didn't manage to do it in time [07:58:00] <_joe_> either me or jayme will take a look later [07:58:08] yeah as I understand it his last week on clinic duty have been a roller coaster [07:58:18] ok great :) [07:58:49] thank you [08:47:19] <_joe_> hashar: I did review your code, but that generated more work for you [08:47:26] <_joe_> I'm not sure that was the expected outcome :P [08:49:12] _joe_: well I got a review? ;]]] [08:49:38] the intent is merely avoid rebuilding wheels each time. But yeah I will look at your comments and address them. Thank you! [08:50:06] <_joe_> yeah my comments were in the direction of "let's make this the right way" [08:51:06] instead of rebuilding wheels, why not reinvent them! :) [08:52:04] kormat: we did both! [08:52:16] reinvent how to rebuild them :D [08:53:02] * _joe_ kneecaps volans [08:54:11] hashar: thanks for fixing puppet ci yesterday :) [08:54:49] kormat: you are welcome. To be honest the fault I made was to not test the new image after I have deployed the change, something I usually do [08:55:18] the good outcome is that we now use the proper Puppet version in the container (5.5.10 same as Buster) and upgraded rspec-puppet ;] [08:55:28] \o/ [09:34:56] is there a way to ask puppet to ensure packageX version >=Y is installed? [09:36:10] you can ensure latest, but it's a discouraged practice [09:36:59] kormat: https://puppet.com/blog/version-ranges-a-smarter-way-to-manage-packages-with-puppet/ :D [09:37:11] 6.15.0 release [09:37:17] good luck! [09:37:22] gdi [12:10:13] kormat: I think you can ensure an exact version, which might be enough in our case, but the usual method here is to do that a bit more manually [13:10:34] <_joe_> kormat: what do you want to do? I mean, what's the reason why you want to ensure a minimum version of a package? [13:15:04] _joe_: version 0.1 did not provide a certain script; version 0.2 and later do/will [13:17:52] <_joe_> ok, usually the way to ensure that is by modifying the apt configuration, but if the question is updating it everywhere, we are managing that with debdeploy, not via puppet [13:20:48] <_joe_> if you want to avoid to install your stuff on a server that has an older version, run an exec (as refreshonly => true) that checks the version of the software installed [13:21:33] <_joe_> I think we *used* to expose a fact with all installed packages and their versions [13:22:03] <_joe_> but usually what I'd do is: upload the newer package to all distros, upgrade everywhere, merge the puppet code [13:23:59] TIL of debdeploy [13:28:26] the docs are a bit.. light. what is it's actual purpose? [13:28:47] <_joe_> upgrading packages across the fleet [13:29:14] <_joe_> if the doc is bad, you can blame all the people who came in and never wrote a better one. Most SREs here have used it [13:29:18] * _joe_ looks at jayme [13:29:39] does this imply that packages don't get upgraded unless someone specifically says they should? [13:29:44] yes [13:29:58] or rather, the version that puppet installs is "whatever version was in the repo when it asks" [13:29:58] * jayme will look back in a sec [13:30:06] so reimages trigger such as well [13:30:11] and otherwise there's no upgrades [13:30:13] uff, right [13:31:07] in this case, i've uploaded version 0.2 of a package that i now realise i do not want to deploy [13:31:23] so don't reimage anything or debdeploy anything ;) [13:31:25] i'm trying to figure out what to do about this [13:31:38] do you want to restore the old version to the repo? or is this a temporary condition [13:32:14] i do, yeah. i'm wondering if that will break anything [13:32:22] (not from a debian PoV, that side is fine, but other systems/automation) [13:32:32] I believe that is fine, although I've never done it myself [13:37:26] hmm...I was quite okay with debdeploy docs (https://wikitech.wikimedia.org/wiki/Software_deployment) besides from what I did fix :) [13:37:56] I think kormat is more asking about the "why?" as opposed to the "how?" [13:38:12] +1 [13:38:45] the doc is "here's how you use it", it doesn't say why this tool exists (i'm guessing deficiencies in puppet), or what it's purpose is [13:38:55] part of the answer is that we've steered away, forcefully, from using puppet in any way that even smells like release management or 'orchestration' [13:39:10] and so, tools like debdeploy and cumin and spicerack for such [13:39:36] <_joe_> because puppet is *not* a tool that does anything remotely like those things [13:40:05] back to the topic, is there a reason not to keep 0.2 on cumin hosts, just not install it on dbs? [13:40:05] Ah, okay. For that I heard more or less what j.o.e just said from the exact same person I guess :D [13:40:18] <_joe_> but for packages, it's even worse. We prefer to have a centralized control of package upgrade state and use a specific tool, rather than having to change the openssl version [13:40:34] I do wish we used an apt repo that supported keeping multiple versions of the same package, btw [13:40:37] <_joe_> jayme: I can thing that person heard that from me :P [13:40:39] it would make rollbacks much easier [13:40:43] <_joe_> cdanis: +1 [13:41:03] <_joe_> cdanis: also, we need one that is not embarassingly complex to run [13:41:06] yes [13:41:13] 'opaque' is an understatement [13:41:27] re: rollbacks, they're rare, but when you need to do one, generally you really need to do it [13:41:32] <_joe_> cdanis: I mean things different from reprepro are IME much much worse [13:42:26] <_joe_> kormat: going back to the point I was making: we want to control when software upgrades happen, puppet removes that ability from you :) [13:42:41] <_joe_> also https://debmonitor.wikimedia.org/ [13:45:24] * volans reading backlog [13:53:23] _joe_: ack [13:54:20] godog: if by any chance you could have a last look at the cassandra patches I would plan to merge them today once u.random is online, test them on a canary host (also manually removing the old IP) [13:54:28] and then if all is good proceed with the rest [13:54:39] can totally be another day if ENOTIME [13:55:13] the only thing that changed are mv from stdlib to wmflib and rewrite of the tests because different modules use different rspec helpers (ofc!) [13:55:57] volans: ack, will do! [13:56:12] <3 [16:54:56] ahoyhoy- I'd like to bring up my first LVS service. would anyone have a few minutes for reviewing my changes for it? https://gerrit.wikimedia.org/r/c/operations/dns/+/619499/ and https://gerrit.wikimedia.org/r/c/operations/puppet/+/615512/ [16:57:56] hnowlan: why encryption: false ? [16:58:03] that's pretty unusual for new services [16:58:26] d'oh, just added tls support today for forgot to update that [16:59:04] cool :) [16:59:23] oh haha, and you already have the right check_command for https [17:03:27] hnowlan: my only other comment is to remove the `# this is the most important entry in your service definition! See below for details` [17:03:30] LGTM aside from that :) [17:03:41] thanks! [17:04:13] also, not sure of your schedule, but I can help with pybal stuff in USTZ [17:10:08] I'm about to clock off now, but I might take you up on that tomorrow [17:14:06] ok cool, e.ma / v.gutierrez can help in EUTZ as well :) [17:25:41] cdanis: thanks for adding the missing line in envoy for websockets. knew that but somehow still missed adding it. it's appreciated. will look into the following issue with listening on IPv6 [17:26:28] mutante: yeah np, seems like it was just missed when aphlict was split off. I didn't dig much beyond that though, aside from looking at `ss -tlp` output [19:01:31] What would be the right project/tags in Phabricator for requesting more memory for a Ganeti instance? [19:02:43] dpifke: I think https://phabricator.wikimedia.org/project/view/1234/ [19:11:57] Thanks. Filed T260192. [19:11:58] T260192: More RAM needed for webperf1002 and webperf2002 - https://phabricator.wikimedia.org/T260192