[10:53:12] mhh I can't remember if we had a task for the general problem of dealing with (rolling) upgrades of debian packages and multiple clusters upgrading at different times [10:53:29] and my search fu is failing me, even searching in gmail [10:54:02] an example being cassandra upgrades, where we (used to?) pin the version in puppet so that e.g. reimages don't suprise us during upgrades [10:54:23] that's addressed by the current component setup in our repo? [10:54:39] incompatible versions of elastic e.g. use different components [10:55:19] <_joe_> moritzm: yes but godog wants to automate a rolling upgrade I guess, rather than having to do 1 puppet patch per host, if I'm not understanding incorrectly [10:56:56] yeah the current minor version in component name structure addresses some of it I think, without requiring a version pin [10:57:20] was it described/addressed in a task ? I'm looking for more context fwiw [10:57:39] https://gerrit.wikimedia.org/r/c/operations/puppet/+/545867/1/modules/profile/manifests/elasticsearch.pp#96 [10:58:40] I'm wondering if being able to say to puppet "install version x of this package if the package isn't installed, otherwise leave it alone" would help [10:58:59] would help with rolling upgrades while addressing the reimage use case too [10:59:01] <_joe_> it's what require_package() does [10:59:29] <_joe_> it will install the version that you configured apt to use if the package is not present [10:59:40] <_joe_> while not touching whatever you have if it's present [10:59:55] <_joe_> same with package { 'something': ensure => present } [11:02:27] ok so it'd be strictly an apt thing in that case, rather than puppet [11:04:53] <_joe_> well puppet configures which component you use [11:05:14] <_joe_> you still need to pick that in puppet [11:05:27] <_joe_> but puppet doesn't control the installation of the new version [11:05:41] there's an older task to extend require_package with component logic: https://phabricator.wikimedia.org/T178575 [11:05:56] but we don't have/had a generic task about what godog asked for TTBOMK [11:06:01] <_joe_> I would rather kill require_package in its current form [11:06:28] <_joe_> it's a hack that creates various issues [11:06:29] moritzm: thanks! yeah that seems close enough [11:07:03] <_joe_> godog: I think the best way to automate all this is - when you want to do a rolling upgrade: [11:07:15] yeah, doesn't need specifically need to mimic the logic of require_package (with the early application), but we should have some simpler logic to add components, too much boilerplate currently [11:07:20] <_joe_> - change the component for the cluster you want to upgrade via hiera/whatever [11:08:38] <_joe_> - write a cookbook using the LBRemoteCluster to depool, upgrade, verify, repool each server one by one [11:08:55] <_joe_> else you can do the latter part by hand ofc [11:14:45] indeed, ok component is clearly the way to go [11:14:48] thanks _joe_ moritzm [20:02:39] herron, godog: https://phabricator.wikimedia.org/T203485 [20:03:05] see the question at the end -- we should probably update that task with a current status [20:06:53] thx, will chat with godog and add an update [20:08:00] thanks! [20:08:05] also, https://space.wmflabs.org/2019/10/29/testing-the-path-from-mailing-lists-to-wikimedia-space/ [20:08:13] very timely :) [20:15:50] hah seriously, and also https://lists.wikimedia.org/pipermail/listadmins/2019-October/thread.html [20:16:13] seems like mostly grumbling there