[07:59:30] <elukey>	 hello folks
[07:59:50] <elukey>	 I am going to disable puppet on install1003 to manually set
[07:59:51] <elukey>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/654192/5/modules/install_server/files/tftpboot/buster-installer/pxelinux.cfg/ttyS1-115200
[08:00:05] <elukey>	 and test a pxe rescue for sretest1001, to see if it works
[08:03:13] <moritzm>	 sounds good
[08:41:07] <elukey>	 moritzm: qq - can I restart atftpd on install1003? I suspect that new config files are not picked up
[08:46:59] <moritzm>	 sure anytime
[09:05:31] <elukey>	 very weird, the new label is not picked up when I test it
[09:05:39] <elukey>	 boot: rescue leads to "not found"
[09:09:37] <moritzm>	 having a look
[09:09:58] <_joe_>	 that part of the stack is absolute black sorcery to me, I can't really help with thats
[09:11:01] <elukey>	 there are some tftpboot config files on apt1001, lemme test if they are picked up for $weird_reasons
[09:11:50] <elukey>	 yep now it works
[09:11:57] <elukey>	 moritzm: --^
[09:12:06] <moritzm>	 yeah, I was about to say that
[09:12:24] <elukey>	 is it expected?
[09:12:25] <moritzm>	 these are sourced from apt1001, install* only does DHCP
[09:13:08] <elukey>	 ok then it definitely confuses me, atftpd is on install1003, together with /srv/tftboot configs
[09:15:24] <moritzm>	 maybe, I need to look up the finer details, some things changed with the split of the install servers for pop sites
[09:15:32] <_joe_>	 I'm sure there is documentation somewhere...
[09:15:54] <_joe_>	 you're supposed to laugh,  that was a joke (albeit a sour one)
[09:16:55] <kormat>	 it was a joe-ke
[09:17:29] <elukey>	 so the tftpboot file are puppetized on both apt1001 and install1003, I just tested it, so we are good consistency-wise
[09:17:40] <moritzm>	 ok
[09:17:50] <elukey>	 aaand the rescue label works!
[09:18:13] <elukey>	 it is not offering a fancy menu' etc.. but that can be done later on if we need it
[09:18:40] <elukey>	 going to update the code review
[09:19:44] <moritzm>	 IIRC rescue modes runs through the initial steps of the installer (so detect disks etc) and then drops to a shell, if you get that, all is fine
[09:23:21] <elukey>	 yep yep I saw the menu for rescue, and tried to select some options etc.. (it starts asking for the language and other stuff so it is very different from the d-i regular install, easy to spot)
[09:26:06] <moritzm>	 sounds good. we can tweak this and create a custom preseed_url for the rescue mode
[09:29:54] <elukey>	 yep definitely
[10:12:49] <marostegui>	 dcaro moritzm ok to push your puppet changes?
[10:13:20] <moritzm>	 yeah
[10:15:59] <marostegui>	 dcaro: yours too?
[10:16:48] <dcaro>	 marostegui:yep, you can go
[10:16:58] <marostegui>	 ok, done
[10:18:39] <dcaro>	 thanks!
[12:33:26] <arturo>	 moritzm: do you happen to know for sure if profile::mail::smarthost is only used inside WMCS?
[12:33:36] <arturo>	 puppetdb seems to agree
[12:35:46] <arturo>	 git grep too
[12:35:50] <moritzm>	 yeah, that seems to be specifically created for wmcs and it's unused in prod
[12:35:57] <arturo>	 ack
[12:35:58] <moritzm>	 https://phabricator.wikimedia.org/T41785 is the original task
[12:35:59] <arturo>	 thanks!
[12:51:14] <elukey>	 Added some info about PXE rescue to https://wikitech.wikimedia.org/wiki/Debian_installer_rescue_mode#Option_2%3A_pxe-bootable_rescue_image
[13:38:15] <sukhe>	 "urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='integration.wikimedia.org', port=443): Read timed out. (read timeout=10)". hopefully temporary
[13:43:21] <elukey>	 sukhe: see you broke our CI :D
[13:45:59] <sukhe>	 elukey: as is tradition!
[15:58:59] <jbond42>	 came accross this the other day, not played with it yet but thought some here may find it curious https://omar.website/tabfs/
[16:29:02] <andrewbogott>	 herron, arturo and I have made more progress with the mail exchanges and are back to the point of wondering if it's now correct or not.
[16:29:12] <andrewbogott>	 We can send ourselves emails from cloud VMs without it sounding alarm bells… are there more things we should test?
[16:30:16] <andrewbogott>	 this is why you should always define victory conditions before going to war :(
[16:31:35] <herron>	 yeah I'd say if mail is flowing while using the new certificate, and the queues and logs look clear it's in good shape
[16:31:57] <herron>	 is there monitoring of the client system mail queues in that environment?  that would be a good indicator
[16:33:17] <arturo>	 herron if I run `openssl s_client -connect localhost:25 --starttls smtp` I don't see the new cert
[16:33:39] <arturo>	 example:
[16:33:40] <arturo>	 https://www.irccloud.com/pastebin/jagAhVcS/
[16:34:12] <andrewbogott>	 herron: I don't know about monitoring; I've literally never thought about these mail exchanges before last night
[16:35:04] <arturo>	 we have https://grafana-labs.wikimedia.org/d/HcDsu-WGk/toolforge-email-dashboard?orgId=1 andrewbogott
[16:35:40] <herron>	 arturo: that is consistent with the contents of mx-out01:/etc/acmecerts/mx/live/rsa-2048.chained.crt
[16:35:58] <arturo>	 herron: so I think the problem is acme-chief is not generating the cert
[16:36:26] <arturo>	 herron: andrewbogott: see https://phabricator.wikimedia.org/T260834#6722402
[16:36:47] <arturo>	 acme chief cannot generate the cert because it cannot update the DNS zone
[16:37:21] <andrewbogott>	 hm, so something with the designate integration is broken?
[16:37:23] <andrewbogott>	 hmph
[16:39:43] <andrewbogott>	 'Unexpected return code spawning DNS zone updater: 1' is less information than I was hoping for
[16:42:53] <bblack>	 yeah the debuggability there is less than ideal.  probably those errors should be louder in some way too, so they don't just go by without notice
[16:43:05] <andrewbogott>	 https://www.irccloud.com/pastebin/foi3Jb8t/
[16:43:09] <andrewbogott>	 that's a little better...
[16:45:31] <andrewbogott>	 this is going to turn out to be because the zone for the cert isn't owned by cloudinfra I bet
[16:45:59] <andrewbogott>	 although if that's the case it shouldn't have worked with ::integrated either I think?
[16:47:54] <arturo>	 I think both wmcloud.org and wmflabs.org are owned by the cloudinfra tenant
[16:48:12] <arturo>	 or should be, per our own policy?
[16:49:05] <bblack>	 and wikimedia.cloud I guess
[16:49:15] <arturo>	 yes
[16:49:40] <bblack>	 the logic in that script is hard to follow, but I'm wondering if it has issues with too-deep level of subdomain in the name or something
[16:49:43] <andrewbogott>	 I think I need to create mx-out.wmcloud.org in cloudinfra
[16:49:45] <andrewbogott>	 working on that now
[16:50:01] <bblack>	 yeah maybe that
[16:50:27] <bblack>	 although it's probably not properly a 'zone', but maybe if there's no records at that name or below at all, it trips up the zone[0] bit
[16:50:34] <bblack>	 err potential_zones[0]
[16:50:44] <arturo>	 mmm makes sense, the record is something like `_acme-challenge.mx-out.wmcloud.org`
[16:57:06] <andrewbogott>	 I added mx-out.wmflabs.org and mx-out.wmflabs.org and mx-out01.wmflabs.org and mx-out02.wmflabs.org
[16:57:14] <andrewbogott>	 and now it's at least not erroring out in that step :)
[17:01:26] <andrewbogott>	 arturo, "openssl s_client -connect localhost:25 --starttls smtp" output looks right to me now, do you agree?
[17:01:49] <arturo>	 wonderful :-)
[17:02:03] <arturo>	 I agree
[17:02:47] <andrewbogott>	 cool, now I'm going to finally close that bug.  Thank you arturo, herron, et al
[17:03:08] <herron>	 +1, awesome!
[17:03:24] <herron>	 odd that a snakeoil cert is issued in that case, but glad its working now
[17:04:23] <andrewbogott>	 yeah, a noisier failure would probably be better?
[17:05:58] <arturo>	 well, not sure, at least it still kept serving emails
[17:07:48] <andrewbogott>	 true
[18:23:58] <elukey>	 herron o/ - after https://gerrit.wikimedia.org/r/c/operations/homer/public/+/654469 all the analytics nodes should be able to ship syslog to kafka logging, so traffic will probably increase.. shouldn't be a problem but if you see something weird blame me :D
[18:24:47] <herron>	 elukey: haha, great thx for the heads up will keep an eye