[07:08:12] <elukey>	 happy new year folks :)
[07:08:53] <XioNoX>	 happy new year to you too!
[07:31:52] <elukey>	 XioNoX: when you have a moment can you check https://phabricator.wikimedia.org/T271041 and let me know what you think?
[07:32:03] <elukey>	 I am a little confused about why the host behaves in that way
[07:32:08] <moritzm>	 happy new year everyone :-)
[07:38:26] <_joe_>	 "happy"
[07:38:33] <_joe_>	 I love your collective optimism
[07:54:24] <apergos>	 it is a wish for the new year to be happy, not an expectation! :-P
[07:57:28] <jayme>	 I'll join that with another "happy new year" o/
[08:13:40] <XioNoX>	 elukey: sure, looking
[08:14:02] * volans here too
[08:21:32] <XioNoX>	 elukey: how can I ssh to the host?
[08:22:26] <elukey>	 XioNoX: you can ssh to the host, it works, it may be intermittent
[08:22:30] <elukey>	 but it worked yesterday
[08:23:33] <XioNoX>	 elukey: ok, from the graphs looks like something happened at 10am UTC on the 30th
[08:24:28] <elukey>	 yes exactly, I see errors from that moment for the interface, but I have found no real clue about what's wrong
[08:28:49] <XioNoX>	 switch logs don't have anything useful
[08:29:55] <XioNoX>	 well, some cryptic logs about 1h before
[08:35:20] <elukey>	 XioNoX: is my understanding right that the default gw for ipv6 comes from RA?
[08:35:39] <XioNoX>	 elukey: yep
[08:35:56] <XioNoX>	 I can't ssh to the host
[08:36:03] <elukey>	 I can :(
[08:37:45] <volans>	 fwiw it was also listed in T250367 XioNoX
[08:37:46] <stashbot>	 T250367: Servers exposing incorrect LLDP info - https://phabricator.wikimedia.org/T250367
[08:38:11] <XioNoX>	 elukey: bast2002:~$ ping ms-be2050.codfw.wmnet doesn't work
[08:39:11] <XioNoX>	 but works from bast3004
[08:40:24] <elukey>	 very weird
[08:40:46] <elukey>	 could it be that this is due to the interface errors that we are seeing?
[08:41:00] <elukey>	 (I also don't have any idea what those are about)
[08:44:12] <XioNoX>	 what was this task about cloud hosts with broken NIC firmware?
[08:44:19] <godog>	 greetings, and happy new year !
[08:45:10] <godog>	 elukey: thanks for taking a look at ms-be2050 !
[08:45:21] <elukey>	 ciao godog!
[08:45:28] <XioNoX>	 elukey: https://phabricator.wikimedia.org/T269313
[08:46:55] <elukey>	 wow
[08:48:57] <XioNoX>	 looks like we're running 'firmware_version': '20.8.163/1.8.4 pkg 20.08.04.03, so not the same version
[08:51:12] <XioNoX>	 and the dropped packets on the switch side is weird too, but the symptoms are similar
[08:52:02] <elukey>	 Can a broken fiber between the 10g port of ms-be2050 and asw-d cause the drops?
[08:55:49] <XioNoX>	 elukey: looks unlikely
[08:56:01] <XioNoX>	 trying to figure out more on why they are discarded
[08:56:42] <elukey>	 ack ack
[08:57:48] <XioNoX>	 I'm seeing the multicast/broadcast TX counters increase
[08:59:00] <XioNoX>	 elukey: can I bounce the switch port?
[08:59:45] <elukey>	 XioNoX: from my point of view I think so, but let's ask to godog to see if this is ok or not (in theory yes since the swift backends can be rebooted anytime)
[09:01:25] <elukey>	 XioNoX: I think that we can proceed, it will be quick
[09:01:27] <godog>	 elukey XioNoX yes please
[09:01:32] <godog>	 good to go any time
[09:01:32] <elukey>	 ah there you go :D
[09:01:35] <elukey>	 <3
[09:03:26] <godog>	 LMK if I can help with ms-be2050
[09:03:49] <XioNoX>	 done, I can ssh now
[09:04:50] <XioNoX>	 it's still dropping multicast packets though
[09:06:12] <elukey>	 XioNoX: but now I can do "telnet -4 puppet.eqiad.wmnet 8140"
[09:06:30] <elukey>	 it worked also briefly after the reboot
[09:09:25] * arturo waves to everyone
[09:09:39] <elukey>	 o/
[09:12:02] <XioNoX>	 elukey: so yeah I'd say, try a different patch, then different switch port
[09:13:55] <XioNoX>	 elukey: and upgrade NIC firmware
[09:22:09] <elukey>	 XioNoX: /me ignorant, what do you mean with "patch" ?
[09:22:20] <XioNoX>	 elukey: I mean DAC
[09:22:24] <XioNoX>	 patch cable
[09:22:34] <elukey>	 ah yes yes
[09:22:42] <XioNoX>	 true, patch means so many things in this world :)
[09:23:26] <elukey>	 thanks a lot for checking, I'll ask to Papaul (if he is in today) to check the cable..
[09:25:01] <_joe_>	 elukey/Xionox can you set a priority on that task, please?
[09:25:28] <elukey>	 yep done
[09:29:15] <_joe_>	 <3
[19:24:12] <andrewbogott>	 herron:  interested in a walk down memory lane?  I just set up acme-chief in the cloudinfra project; we need to change the mail exchanges to use that rather than letsencrypt::cert::integrated
[19:24:35] <andrewbogott>	 (art.uro has already offered to work on it but it might be super quick for you if you remember how that's put together)
[19:32:39] <herron>	 andrewbogott: nice, should be pretty straightforward tbh.  essentially including the acme_chief::cert in that profile, describing the cert in the acme cheif hiera yaml for that environment, and pointing the exim config to the live cert with a path based on the acme cheif name e.g. /etc/acmecerts/mx/live/rsa-2048.chained.crt.  happy to try and answer questions just lmk
[19:33:01] <andrewbogott>	 great, we'll see how far I get
[19:42:34] <sukhe>	 andrewbogott: I am not sure if this makes sense but I thougth I should share in case it is helpful: git show 7ee80b5aab a551c82d7c 340e8f085
[19:42:52] <sukhe>	 (basically what herron said, but with the commits. I remembered I had done this so shared)
[19:43:28] <andrewbogott>	 thanks! I think I understand most of that
[19:46:09] <andrewbogott>	 I don't think I know what to use for key_group.
[19:46:51] <andrewbogott>	 oh nm, I can probably just leave it as it was
[19:48:28] <sukhe>	 the default is 'root'
[19:51:39] <andrewbogott>	 oh, it's just 'who owns this key'?
[19:53:08] <sukhe>	 yep, the group that owns the key
[20:03:14] <andrewbogott>	 sukhe and/or herron, maybe it's just this?  https://gerrit.wikimedia.org/r/c/operations/puppet/+/654295
[20:06:09] <sukhe>	 sorry, what's $cert_name here?
[20:06:17] <herron>	 andrewbogott: that, plus an entry in the cloudinfra equivalent of role/common/acme_chief.yaml and possibly just defining a static name instead of the cert_name variable
[20:06:24] <sukhe>	 hieradata/role/common/acme_chief.yaml should match it
[20:06:42] <andrewbogott>	 sukhe: cert_name is just $hostname in the cases I care about
[20:07:04] <andrewbogott>	 here's my hiera:
[20:07:06] <andrewbogott>	 https://www.irccloud.com/pastebin/1PCfryaV/
[20:09:15] <sukhe>	 yep, I think this is about it
[20:10:14] <herron>	 yes agreed, although the authorized_regexes are a bit permissive considering the cert is meant for one host only
[20:11:54] <andrewbogott>	 true — it's a secure project so I don't think it matters.
[20:12:30] <andrewbogott>	 the next thing is — I don't immediately know how to test whether it's working or not.  herron, do you have any interest in merging/applying/validating the change?
[20:16:07] <herron>	 sure although now that I'm thinking about deploying this a safer way to go would be to do it in two steps.  first patch adds acme_chief::cert, then we run puppet and inspect the cert , and a follow-up patch updates exim to use it once all looks good
[20:17:15] <andrewbogott>	 ah, good idea
[20:17:17] <andrewbogott>	 I'll break it up
[20:17:54] <herron>	 ok sounds good thx
[20:21:07] <herron>	 andrewbogott: ok patch looks good, and that hiera snippet is live in the horizon project hiera?
[20:21:23] <andrewbogott>	 yep.  I'm merging the first patch that creates the acme certs
[20:21:27] <herron>	 kk
[20:22:17] <andrewbogott>	 be warned that puppet is super noisy since letsencrypt::cert::integrated doesn't run anymore
[20:25:55] <andrewbogott>	 hm, looks like my chief is broken somehow
[20:31:22] <herron>	 yeah from first glance seems TLS validation is failing to cloudinfra-acme-chief-01.cloudinfra.eqiad1.wikimedia.cloud:8140
[20:32:05] <herron>	 sorry I have to go afk for a bit shortly, but can have another look later on tonight or tomorrow
[20:33:05] <andrewbogott>	 ok — thank you!