[09:03:54] <XioNoX>	 fyi, I'm gong to depool magru at 13:00 UTC for a Junos OS upgrade (cc jelto, vgutierrez)
[09:04:12] <vgutierrez>	 that's for the next shift then :)
[09:04:20] <vgutierrez>	 thanks for the heads up :D
[09:04:20] <XioNoX>	 true!
[09:04:35] <jelto>	 :)
[09:05:03] <jelto>	 I can add it to corto 
[10:51:15] <effie>	 I am restarting theh various redis-servers dear oncallers 
[10:51:39] <effie>	 I do not think anything will happen 
[10:52:09] <jelto>	 good luck, thank you 👍
[10:52:42] <effie>	 moritz made me do it, whatever comes, is his problem not mine
[13:01:16] <XioNoX>	 arnoldokoth, jhathaway, starting the magru depool/maintenance
[13:03:15] <jhathaway>	 Thanks
[13:14:05] <arnoldokoth>	 XioNoX: ack. Ty.
[13:18:48] <XioNoX>	 going to wait for the :20 mark to reboot cr1, that one shouldn't be impactful as the routers are redundant
[13:23:56] <XioNoX>	 it's rebooting
[13:47:50] <XioNoX>	 all good with cr1, rebooting cr2
[14:03:08] <XioNoX>	 routers are both online and healthy, going to reboot asw1-b3 soon, that's going to be more impactful as 50% of the servers are connected there
[14:03:15] <XioNoX>	 but they're all downtimed
[14:17:11] <XioNoX>	 switch is back up
[14:37:44] <XioNoX>	 and 2nd switch back up,
[14:39:07] <XioNoX>	 moritzm: could you have a quick look at the Ganeti cluster to check if it's healthy? but afaik it's all good
[14:39:47] <moritzm>	 looking
[14:40:37] <XioNoX>	 moritzm: as it's routed ganeti we could actually have moved all the VMs from one switch to the other before the work, I'll try to remember to do that for next time
[14:41:03] <moritzm>	 indeed
[14:41:10] <moritzm>	 looks all good
[14:41:27] <XioNoX>	 thx
[14:57:26] <XioNoX>	 magru work is done, just need to repool the site now (cc sukhe)
[14:57:54] <sukhe>	 XioNoX: thanks. checking stuff once.
[14:58:14] <sukhe>	 pooling DNS for magru first
[15:01:25] <sukhe>	 removing downtime for A:magru
[15:03:21] <sukhe>	 XioNoX: ready to pool if you are
[15:03:30] <sukhe>	 (go for it or let me know if you want to :)
[15:05:24] <XioNoX>	 sukhe: go for it
[15:06:00] <sukhe>	 verified BGP on the the DNS hosts as well, liberica looks happy too, proceeding
[15:07:06] <sukhe>	 done
[15:08:12] <sukhe>	 hmm 
[15:08:37] <sukhe>	 ok picking up
[15:11:23] <sukhe>	 XioNoX: looks good, nice and clean <3 to celebrate this, I will have some bird patches your way this week to support IPv6-only-configs :P
[15:14:42] <XioNoX>	 nice!
[16:27:46] <Emperor>	 Hi folks. I tried to decomm some nodes with sudo cookbook sre.hosts.decommission -t T404771 ms-be20[57-61].codfw.wmnet and it failed (after doing some initial work) with 'No hosts provided'. cf T404771
[16:29:31] <Emperor>	 like maybe it's done away with the puppet certs and now a subsequent check is failing?
[16:38:04] <elukey>	 Emperor: o/ from the logs it seems it is trying to get info from puppetmaster1001, so probably some leftover, lemme check
[16:40:30] <elukey>	 https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1239986
[16:42:28] <elukey>	 Emperor: ok if I run `test-cookbook -c 1239986 sre.hosts.decommission ms-be[2057-2061].codfw.wmnet -t T404771` to test it?
[16:44:44] <elukey>	 (stepping afk in a bit, you can run it if you want later on and report back!)
[16:46:45] <Emperor>	 elukey: sorry, I went away to open T417670 to document the problem. Err, yes, please do go ahead (or I will tomorrow morning if not)
[16:46:45] <stashbot>	 T417670: sre.hosts.decommission fails with >1 host, leaves hosts impossible to decommission - https://phabricator.wikimedia.org/T417670
[16:49:27] <elukey>	 yep it is working
[17:06:44] <elukey>	 Emperor: done!
[17:07:14] <elukey>	 (thanks volan*s for the quick review <3)
[17:08:26] <volans>	 anytime
[17:09:28] <Emperor>	 thanks :)