[07:26:19] hello folks! [07:26:32] I am preparing the k8s aux codfw cluster for the upgrade to 1.31 [07:52:23] elukey: nice! Is that the last one? [07:54:53] brouberol: I also need to do eqiad, but yes it is the last :) [07:55:04] woohoo [07:58:36] for aux-k8s-codfw, we shouldn't see an impact on mirrormaker, because the kafka-main-codfw<-->kafka-main-eqiad mirrors are located in both aux-k8s clusters [07:59:20] however, mirroring will be stopped for a while when you reinstall aux-k8s-eqiad, as some mirroring pipelines are only located in aux-k8s-eqiad themselves [07:59:37] which is fair game, as they will resume when you reinstall everything w/ charlie [08:40:12] brouberol: the aux codfw cluster is back up [08:45:07] brouberol: mirror maker looks good afaics! [08:45:14] great [08:48:51] all right all good! I'll do eqiad tomorrow morning, will prep the patches later on [12:39:24] topranks: moritzm: heads up that I'm adding new LVS services for dumps, should not page but still pinging just in case [12:44:59] taavi: ok thanks, good luck! [14:11:49] federico3: you used to work on Designate, right? Can I interest you in a brief walk down memory lane in #wikimedia-cloud-admin ? [14:27:18] @andrewbogott: uh, joining [14:27:30] elukey heads-up that we have had some issues detaching PVs after the k8s upgrade. We're still not 100% it's related to the upgrade, but there's some context/workarounds in T422262 [14:27:30] T422262: Investigate persistent volume detach/attach issues in dse-k8s - https://phabricator.wikimedia.org/T422262 [14:29:52] inflatador: ack thanks!