[16:58:20] I remember reading somewhere that to delete a file in puppet I should do it in two phases -- first change ensure => present to ensure => absent, merge and roll that out, and then actually delete the puppetry -- is that correct? [16:58:35] because otherwise the files stay in place and puppet just stops managing them? [16:58:55] that's right [16:59:07] you can also just nuke from puppet and clean it up afterwards manually [16:59:12] thanks 👍 [16:59:28] for some things (systemd services), ensure=>absent will be easier (don't have to orchestrate stop, disable, daemon-reload, etc) [16:59:40] for files, you can do whatever [16:59:44] if it's just 1 host i usually do the latter, if it's multiple then i would do the former [17:00:03] if its just a file i would probably just remove it from puppet and use cumin to delete it [17:01:35] a bunch of profiles also have a general $ensure parameter that removes f.e. a config file, a package and a cron all at once [17:03:37] should be pretty straightforward in this case (profile::mediawiki::web_testing, just a static file plus a static directory x 4 hosts) so I could clean up by hand pretty easily but I'll do it the Proper Way just for the exercise [17:03:46] thanks for the discussion <3 [17:04:41] rlazarus: if removing a dir you will need to use `ensure => absent, force => true` [17:04:46] (spitballing) we likely have absented resources in puppet that could be just removed, maybe sth like a report couple of times a year to do a cleanup will take care of not leaving those in the codebase forever [17:04:57] rlazarus: R:File = /path/... before the removal from puppet [17:05:13] than run cumin with directly the list of hosts that the first query gave you [17:05:21] also one thing to consider whether a given class could be used in Cloud VPS (unless you're also doing the cleanup with Cumin on the WMCS instances) [17:05:24] after the merge and 30m has passed [17:06:50] to (kind of) check whether a class is used in Cloud VPS, use: https://openstack-browser.toolforge.org/puppetclass/ [17:06:50] and yeah WMCS too [17:07:04] but there is no puppetdb there [17:07:12] kind of because that does not cover things with their own puppetmaster [17:07:53] it could still break beta [17:08:14] unfortunately it's super common that beta puppet is broken [17:08:56] oops, they totally do use it, thanks for the suggestion [17:13:51] jbond42, volans: ack to both, thanjks [17:13:54] *thanks [17:15:32] rlazarus: if it's used widely in wmcs, go for absent and 2 commits [17:19:18] WMCS hosts are also on the half-hour schedule? [17:20:12] cdanis: yes, but [17:20:25] I know many also have broken puppet [17:20:32] if you have a local puppetmaster, it syncs from global puppetmaster every 10m [17:20:53] so it's more 40m the normal time to be sure [17:20:57] and yes many have broken puppet [17:20:57] ahh [17:21:08] and to add to rlazarus's case [17:21:36] if many are using it in wmcs in theory we should leave it absented for months hoping that instances will actually get it's removal [17:21:49] or on the other side, just go the quick way [17:22:01] and delete with cumin both in prod and wmcs [17:22:11] with some more trick to get the right list of hosts in wmcs [17:22:26] unless you want to do rm -fv and be happy, depending on the path [17:22:30] and how custom it is [17:22:51] good news, I hate this :D [17:23:12] okay, I'm definitely doing two commits anyway, but I'll wait a day between them and hope for the best [17:23:37] afaict these are the only hosts affected https://openstack-browser.toolforge.org/puppetclass/role::deployment_server [17:23:53] k [17:23:57] yea, cumin is also a thing in wmcs. and imagine this before we had openstack-browser :) [17:23:58] but I guess "are there any others" is an unknowable question, so [17:24:35] rlazarus: if the path is unique enough [17:24:46] just rm -fv in the whole wmcs and you'll be good :D [17:27:53] haha I read that as "rm -fv the whole wmcs" and, that would also solve my problem, at the expense of creating a different one [17:29:40] ahahahah [17:33:24] hrhrr [17:49:37] SRE: Solve My Problem at the Expense of Creating a Different One [17:50:14] so let's rename to SMPECDO? :-P [17:51:04] so i think we need to translate it into latin and put it on our coat of arms [17:58:29] Seen that Icinga check "Ensure hosts are not performing a change on every puppet run"? As a side effect that also detects hosts now that are not in site.pp. Because we changed the default to fail with "No puppet role has been assigned" and that also triggers the "performing a change on every run" so they pop up there. That just made me notice some new ganeti hosts were not in site. I added the [17:58:35] "role(insetup)" and that fixed that and other things. [18:00:25] other hosts that have that or anoter puppet issue are: elnath.codfw.wmnet, cescout1001.eqiad.wmnet, cloudvirt2003-dev.codfw.wmnet, [18:06:34] ah that's what hatis about, thanks for doing that! [18:09:55] yeah, there is nothing on cescout1001 so far but that will change shortly [18:11:05] ack. ran puppet on ganeti1019-1022 now and that should soon clear half of that alert [18:11:17] sukhe: let me add role(insetup) to that as well [18:11:44] it also means they get firewall and can be downtimed in icinga and more [18:13:55] mutante: thank you, I will follow your patches and will start by adding the deb for cescout [22:29:39] one more puppet removal question: does anything silly happen if I remove a require_package()? I don't think I really care whether the package gets uninstalled or not, I just don't want to wedge it into a weird state [22:29:55] I *think* the answer should be "no, it's fine, just delete it" but... this is puppet [22:32:01] rlazarus: it will not remove the package, nothing will happen [22:32:18] I think it's just another puppet-stops-managing-it kind of thing [22:32:22] though you might want to actually remove it [22:32:24] it won't get installed on new servers [22:41:29] it will not be installed by that resource on new/reimaged hosts, but it might still be installed because it's a dependency of another debian package [22:42:00] totally safe to remove it from puppet, nothing happens on existing hosts, but FYI we had cases in which dependencies were not explicitly defined in all places [22:42:12] hm, okay [22:42:18] like 2 profiles that both require python3-requests and maybe only one of the two defines it [22:42:28] it's a bug, and it's ok that we'll find out at the next reimage [22:42:31] so if something else expects this package to be installed, but doesn't actually have a require_package, it'll break on new hosts when I delete this [22:42:32] when something will break [22:42:32] yeah [22:42:40] yep [22:43:04] if in the context you need it youre' sure it's not anymore needed, delete it [22:43:27] sgtm, thanks all [22:44:20] yw, anytime