[07:16:19] <godog>	 heads up: I'm decomissioning weblog1001
[07:17:43] * _joe_ weeps
[07:18:32] <godog>	 heheh LMK if you run into problems with centrallog hosts instead
[08:17:35] <godog>	 and another heads up: will flip back netmon to eqiad shortly, cc XioNoX
[08:28:29] <_joe_>	 godog: the main issue will be my muscle memory
[08:28:48] <_joe_>	 in 3 months we'll have an outage and it will take me 30 seconds to realize :P
[08:35:36] <godog>	 heheh fair
[08:36:02] <_joe_>	 to be clear, I am not objecting to the change
[08:36:09] <_joe_>	 I just know that will happen
[08:36:12] <volans>	 godog: leave a cname with a comment, ; for _joe_
[08:36:22] <_joe_>	 it happened when we moved from oxygen to weblog :P
[08:37:30] <godog>	 an util in wmf-sre-laptop that patches /etc/hosts {{done}}
[08:37:46] <godog>	 and/or messes with ~/.ssh/config
[08:38:09] <volans>	 godog: that's already part of the known hosts generation script ;)
[08:38:23] <volans>	 ]$ ssh icinga.wikimedia.org
[08:38:24] <volans>	 Linux icinga1001
[08:38:36] <volans>	 since the beginning
[08:39:04] <godog>	 even better
[08:39:14] <volans>	 when it founds a CNAME that matches a host, it replicates the fingerprint for the cname too
[08:39:27] <volans>	 so that you can ssh with strict check to the CNAME
[08:40:05] <godog>	 very nice, TIL
[08:40:59] <volans>	 it's optional, you have to pass the path to the DNS repo to the script, but it's documented :D
[12:57:43] <kormat>	 phamhi: hey there. do you have any idea why the reverse dns entries for cloud VPSes have a different domain than the forward entries?
[12:58:25] <kormat>	 e.g. cumin.mariadb104-test.eqiad.wmflabs maps to 172.16.2.34, but 172.16.2.34 maps back to cumin.mariadb104-test.eqiad1.wikimedia.cloud
[12:59:18] <kormat>	 (according to 208.80.154.143, aka ns-recursor0.openstack.eqiad1.wikimediacloud.org.)
[12:59:43] <_joe_>	 kormat: because they're transitioning to the wikimedia.cloud names IIRC
[12:59:56] <kormat>	 _joe_: uff, k. it's breaking some stuff for me
[13:00:08] <_joe_>	 so eqiad.wmflabs is transitioning to eqiad1.wikimedia.cloud
[13:00:43] <volans>	 kormat: cumin.mariadb104-test.eqiad1.wikimedia.cloud works too
[13:00:49] <volans>	 not sure if it's safe to use it though
[13:01:18] <kormat>	 volans: that's not the hostname the VM has (or the hostname the puppet cert is for, which is the issue)
[13:01:31] <volans>	 eh...
[13:02:13] <volans>	 good question for #wikimedia-cloud-adminconnect I guess
[13:02:21] <volans>	 sorry #wikimedia-cloud-admin
[13:03:07] <vgutierrez>	 hashar: can you help me deciphering https://integration.wikimedia.org/ci/job/debian-glue-non-voting/2803/console?
[13:03:50] <_joe_>	 vgutierrez: 12:57:42 fatal: A branch named 'debian' already exists.
[13:04:02] <_joe_>	 12:57:42 error: Cannot delete branch 'debian' checked out at '/srv/jenkins/workspace/workspace/debian-glue-non-voting/source'
[13:04:05] <vgutierrez>	 yeah, that's expected AFAIK
[13:04:16] <vgutierrez>	 I mean that the debian branch exists
[13:04:19] <_joe_>	 "fatal" seemed like a serious problem :P
[13:12:37] <ema>	 the usual pessimist
[13:15:50] <vgutierrez>	 I guess that something has been updated on the debian-glue container
[13:16:00] <vgutierrez>	 and it isn't happy anymore with acme-chief's gbp.conf
[13:21:00] <vgutierrez>	 also.. for some f*cking reason it's trying to build acme-chief 0.26 instead of 0.27 :/
[13:23:06] <_joe_>	 did you push the tag?
[13:23:29] <vgutierrez>	 yes
[13:23:41] <vgutierrez>	 otherwise I'd get an error trying to checkout the tag
[13:24:09] <vgutierrez>	 https://gerrit.wikimedia.org/g/operations/software/acme-chief
[13:24:13] <vgutierrez>	 but yes, the tag 0.27 is there
[13:24:49] <vgutierrez>	 even better https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/acme-chief/+/refs/tags/0.27
[13:48:54] <hashar>	 vgutierrez: looking
[13:49:18] <hashar>	 hmm
[13:50:20] <hashar>	 but it uses git branch -D !
[13:50:23] <hashar>	 which forces the deletion
[13:51:00] <vgutierrez>	 I guess it needs a --really-delete-it
[13:52:03] <hashar>	 oh
[13:52:22] <hashar>	 or it is because the repository HEAD is set to debian
[13:52:29] <hashar>	 so that is the current branch being used
[13:52:39] <hashar>	 and git branch -D debian  refuses to delete it because ... that is the current branch
[13:53:20] <hashar>	 hmm no it is not
[13:53:31] <hashar>	 https://gerrit.wikimedia.org/r/admin/repos/operations/software/acme-chief,branches  shows HEAD points to master
[13:53:50] <vgutierrez>	 yup
[13:55:47] <hashar>	 but the patch targets the debian branch
[13:55:56] <hashar>	 so the job does: git -C ./source checkout -B debian FETCH_HEAD
[13:56:42] <hashar>	 it fails later with:
[13:56:42] <hashar>	 00:00:01.078 + git branch debian remotes/origin/debian
[13:56:42] <hashar>	 00:00:01.078 fatal: A branch named 'debian' already exists.
[13:58:43] <vgutierrez>	 so git branch debian with a starting point set to remotes/origin/debian, but debian already exists...
[13:58:50] <vgutierrez>	 there is branch -D debian in the middle of those two?
[13:59:38] <vgutierrez>	 yes.. and it's failing cause it's currently checked out branch
[13:59:39] <vgutierrez>	 12:57:42 error: Cannot delete branch 'debian' checked out at '/srv/jenkins/workspace/workspace/debian-glue-non-voting/source'
[13:59:41] <hashar>	 so the script we use is https://github.com/mika/jenkins-debian-glue/blob/master/scripts/generate-git-snapshot#L451-L462
[13:59:54] <hashar>	 that block tries to create in the local repository a debian and upstream branch
[14:00:00] <hashar>	 cause they are often used by git buildpackage
[14:00:03] <hashar>	 by convention
[14:00:24] <hashar>	 and if you lack a local 'upstream' branch, git buildpackage would not be able to craft the upstream tarball
[14:00:33] <hashar>	 though for repositories using pristine tar, it is not an issue really
[14:00:48] <hashar>	 so that part of the script ensure there is a debian and an upstream branch created locally if they exist remotely
[14:01:03] <vgutierrez>	 we just use upstream-tag instead of a pristine tar
[14:01:22] <hashar>	 yeah
[14:01:36] <hashar>	 so with upstream-tag, I don't think git buildpackage needs an upstream branch at all
[14:01:54] <hashar>	 anyway the script calls  create_local_branch debian
[14:01:58] <hashar>	 the code then is https://github.com/mika/jenkins-debian-glue/blob/master/scripts/generate-git-snapshot#L193-L204
[14:02:11] <hashar>	 which tries to delete the debian branch and ignore any failure ( || true )
[14:02:24] <hashar>	 then it tries to create it based on the remote one
[14:02:37] <hashar>	 but since the 'debian' branch already exists, it bails up
[14:03:06] <hashar>	 the big question is why we only notice this issue now? There are surely other repositories using a 'debian' branch for packaging *sigh*
[14:04:03] <vgutierrez>	 any change recently?
[14:04:07] <vgutierrez>	 I mean on debian-glue
[14:04:42] <vgutierrez>	 hashar: https://gerrit.wikimedia.org/r/c/operations/software/acme-chief/+/605582 it was happy a few weeks ago
[14:04:53] <vgutierrez>	 Jun 15th to be precise
[14:05:19] <hashar>	 AH
[14:05:36] <hashar>	 excellent ;]
[14:05:57] <hashar>	 and we still have those logs
[14:06:13] <vgutierrez>	 http://gph.is/1qGs8K0
[14:06:57] <hashar>	 that means something somehwere changed bah
[14:07:12] <hashar>	 I will compare the console logs ( https://integration.wikimedia.org/ci/job/debian-glue-non-voting/2788/consoleText vs https://integration.wikimedia.org/ci/job/debian-glue-non-voting/2803/consoleText )
[14:09:43] <hashar>	 vgutierrez: here is the root cause https://gerrit.wikimedia.org/r/c/integration/config/+/607990/2/jjb/operations-debs.yaml
[14:09:54] <hashar>	 the CI job has been made to   git checkout -B debian
[14:09:58] <hashar>	 thus creating the branch hehe
[14:32:35] <hashar>	 vgutierrez: i got the fix ;)
[14:33:35] <hashar>	 vgutierrez: https://gerrit.wikimedia.org/r/c/integration/config/+/617476 jjb: debian-glue should not checkout to a branch
[14:35:58] <hashar>	 https://integration.wikimedia.org/ci/job/debian-glue-non-voting/2804/console
[14:35:59] <hashar>	 that works
[14:36:00] <hashar>	 ;)
[14:36:12] <vgutierrez>	 that was fast :) thx :D
[23:39:28] <cmjohnson1>	 mutante: okay to merge your change?
[23:43:14] <mutante>	 cmjohnson1: eh.. which one?
[23:43:24] <cmjohnson1>	 +        require => Package['xhgui'],
[23:43:33] <mutante>	 oh
[23:43:40] <mutante>	 yea, merge it please
[23:43:43] <mutante>	 that's weird :)
[23:43:49] <cmjohnson1>	 okay
[23:43:50] <cmjohnson1>	 done
[23:43:58] <mutante>	 i thought i already saw the fix that this brought
[23:44:01] <mutante>	 thx