[13:27:07] XioNoX: I have seen db2082 and ganeti2004 getting timeouts on their mgmt iface during the day they are in B5 and B6 racks, do they share the same mgmt switch? [13:28:01] each rack have a different mgmt switch [13:28:19] maybe they just reached their EOL and they're all failing at the same time [13:28:38] XioNoX: That sounds familiar :p [13:28:59] XioNoX: Anyways, not sure if it just something punctual or merits a task [13:29:46] marostegui: DCops is looking into replacing them, if they start failing it might help them prioritize it :) [13:33:49] Let's see if it keeps happening or just a one time thing - I would expect more servers to timeout [16:05:14] o/ [16:42:31] Anyone know much about docker in beta and how services are configured there? I see that we have stuff like deployment-docker-mathoid01.deployment-prep.eqiad.wmflabs but I can't quite figure out how mathoid is configured in puppet there [17:44:15] hnowlan, IIRC puppet defines systemd services which tell docker to run a container [17:46:58] there's role::beta::docker_services and a profile::docker::runner::service_defs hiera key [18:13:15] Can I get someone off of my team to sign off on https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/593769/? As best I can tell gnupg is only getting installed by luck or accident (via dependencies) on prod hosts, and I suddenly have a host that doesn't have it. [18:14:26] Normally i would ask Moritz but he's off I think [18:21:31] yep [19:06:40] chaomodus: the thing about pcc is that it shows a diff on every host since just defining the resource in puppet shows up as a diff [19:06:54] I don't think the pcc knows w/not a package is installed on a host via dependency [19:07:04] fair [19:19:09] yeah, it is untracked by puppet [19:19:19] you could use https://debmonitor.wikimedia.org/ if you wanted [19:19:28] i was sitting puzzling the possible negative outcomes [19:20:14] I would just do it, though [19:21:49] also ISTR that you don't need the requires=>Package['gnupg'] stanza there? I believe that require_package also implicitly adds a dependency on the package to any resources that follow. [19:22:38] oh, looks like for any other resource that's part of the current scope [19:23:47] hm [19:24:04] (I have seen numerous examples of that construction but i wasn't sure either way) [19:24:28] it's not harmful to add it, but require_package's docs do explicitly mention it does this [19:30:45] kayt that sounds like a +1 to me :) [19:32:07] * andrewbogott in a meeting but thanks for the +1 [20:35:29] crap [20:36:04] (Exec[/usr/bin/apt-key add /var/lib/apt/keys/openstack-rocky-buster-nochange.gpg] => Apt::Repository[openstack-rocky-buster-nochange] => Exec[openstack-rocky-buster-apt-upgrade] => Package[gnupg] => Class[Packages::Gnupg] => Apt::Repository[openstack-rocky-buster-nochange] => Exec[/usr/bin/apt-key add /var/lib/apt/keys/openstack-rocky-buster-nochange.gpg]) [20:36:21] * andrewbogott hopes that isn't happening everywhere [20:44:17] andrewbogott: doesn't look like it to me [20:44:38] yeah, it's only happening on things that use openstack-rocky-buster-apt-upgrade and openstack-rocky-stretch-apt-upgrade [20:44:46] which was the one thing I was trying to fix with my patch :) [20:45:03] So I need a way to tell it 'do this before every package EXCEPT for this one package' [20:45:16] Exec['openstack-rocky-buster-apt-upgrade'] -> Package <| |> [20:53:34] andrewbogott: you probably already know but it looks like puppet is failing to compile on cloud* nodes (widespread puppet failure alert in -operations) [20:54:31] yep, that's what ^^ is about [20:57:51] instead of doing the require_package, you could also ensure=>present a package object, might help break the dep loop [20:58:50] I expected this to work [20:58:50] Exec['openstack-rocky-buster-apt-upgrade'] -> Package <| name != 'gnupg' |> [20:58:53] but it seems not? [20:59:08] cdanis: if I ensure=>present then the package has to be defined someplace doesn't it? [21:03:22] heh woops [21:04:15] iirc if it's like apt::package { 'gnupg', ensure=>present } it's what cdanis is suggesting [21:04:51] but idk [21:08:30] -> Package <| title != 'gnupg' |> seems to work [21:08:32] just not 'name' [21:26:08] ah