[08:03:55] fyi, I'm going to upgrade LibreNMS shortly [08:14:37] seems like everything went smoothly [10:42:49] moritzm: ema and me were discussing about the best approach to begin building traffic specific packages (varnish, ats..) for buster. Specifically if we should bump versions or just rebuild and import them in the repo ignoring the distro version [10:43:36] with distro version you mean what's shipped in Debian buster? [10:44:58] hmm nope, I was referring to --ignore=wrongdistribution [10:44:58] with ATS we're already beyond what's in Buster and we'll most certainly always have a delta of some sort, so that will need to be a custom buster-wikimedia package anyway [10:45:01] ah [10:45:38] yeh.. I 'm talking about the current custom wmf packages.. we want to keep varnish and ats like that [10:45:49] varnishkafka and a few others as well [10:47:01] of you rebuild the packages anyway (and we'll need to due to rdeps like libssl), I'd at least change the version number in debian/changelog [10:47:27] vgutierrez: are you sure they are binary-compatible? I'd rebuild them with a new changelog anyway, it's cleaner [10:47:28] to e.g. 8.0.5-1wm11+deb10 or 8.0.5-1wm11+deb9 [10:47:33] what mori.tzm said :D [10:47:38] then you can keep them apart in debmonitor etc. [10:48:00] and if you change debian/changelog anyway for the version number, you can also simply change the target-distro [10:48:23] yeah.. ema was worried about the complexity of releasing patches in the transition between stretch and buster [10:48:27] but as long as you rebuild separately, there's no harm in --ignore=wrongdistribution and ignoring the target distro [10:49:04] so we were thinking in rebuilding them but not bumping version/distribution on the changelog [10:49:10] vgutierrez: if you need a helper script check /home/volans/debmonitor-release on buster [10:49:32] it's how we bulild the debmonitor client for all distros [10:50:09] question is: what if we bump d/changelog for ATS (buster), release the package, then while we're in the process of upgrading the cluster there's a new ATS version that needs to be released urgently (including the hosts still on stretch) [10:50:40] ema: that script changes locally the changlog only for building purposes [10:51:03] the git changelog has "unstable" as distro for example in that case [10:51:07] yeah, what Riccardo said, I also wouldn't bother with separate branches and similar sillyness [10:51:15] and it get's "targeted" only at build time [10:51:23] right, multiple branches is exactly what I wanted to avoid [10:51:38] check the script, it's few lines of bash ;) [10:51:41] yup [10:51:52] keep one version in git and during build type simply sed the target version and target distro [10:52:00] so we just upgrade d/changelog on build time and we don't touch the git repo [10:52:05] all this in the hope that no packaging changes are needed between distros :) [10:52:21] last famous words... [10:52:25] that's also what is used for debdeploy (which is support across all distros by definition) [10:52:36] cool :D [10:53:23] if we just rebuild for buster without bumping the version, reprepro isn't going to scream cause the version is already there? [10:54:03] yes 1 version == 1 binary [10:54:36] you can copy it between distros but must be the same binary, that works for like python packages usually (and not even always) [10:54:41] but is a bit hacky IMHO [10:55:15] https://debmonitor.wikimedia.org/packages/debmonitor-client is much nicer to see :) [10:55:46] ema: so if we need to bump the version.. I'd say we bump the distribution as well and if needed we just target stretch on build time with a similar approach to the one used in the debmonitor-release example [10:56:12] that sounds good to me [10:56:45] +1 [10:56:54] cool :D [10:57:00] thanks guys [11:00:59] we could also generalise the build script and install it on boron via Puppet, with a standardised placeholder like like TARGETDISTRO and TARGETVERSION [11:01:14] just a thought, not trying to talk anyone into that work :-) [11:05:09] * volans hides [16:37:03] akosiaris do you mind taking a look to https://gerrit.wikimedia.org/r/c/operations/puppet/+/559110? it's a new LVS service that's being added by jeh, it looks ok to me but another check would be nice [17:23:45] vgutierrez: I just had a look, I see it was merged [17:24:02] LGTM anyway, I hope everything went fine? [17:25:18] almost [17:25:22] the lvs side looks good [17:25:53] jeh needs to fix the loopback interface on the origin instances [17:26:11] but nothing to be worried about it [18:44:20] I know e.ma/traffic has done a lot of work with systemtap -- has anyone messed around with the newer ebpf tools? [23:31:36] Regarding the new LVS service I deployed, everything seems to be working as expected but I see an icinga warning on puppetmaster2001 for a stale template [23:31:46] How should I resolve that? conftool-merge sounds like what I might need, but it also says I shouldn't have to run it. https://wikitech.wikimedia.org/wiki/Conftool#The_tools [23:35:07] jeh: it's a somewhat common issue. happened to know when i saw the icinga alert because i ran into that as well [23:35:20] keyword 'stale confd templates' in SAL [23:35:29] [puppetmaster2001:/var/run/confd-template] $ sudo rm .cloudceph*.err [23:35:51] this happens every time we add a new service [23:35:55] and then i rescheduled the service check and now: [23:35:57] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=puppetmaster2001&service=Confd+template+for+%2Fsrv%2Fconfig-master%2Fpybal%2Feqiad%2Fcloudceph [23:36:00] green [23:37:34] cool, thanks for that info! looks like it's cleaned up now [23:37:59] ACK, when it happens.. delete the .err files for that service [23:41:05] just wrote brief docs and now linking to them from the alert [23:41:09] mutante: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/562654 [23:41:35] https://wikitech.wikimedia.org/wiki/Confd#Monitoring [23:42:41] nice! +1. good idea to add it right away [23:43:49] it's like the third time i've seen it come up in the last month, wish i had done it earlier 🙃