[08:00:10] Cteam: welcome to today 🦄! Don’t forget to post your update in thread. [08:00:10] Feel free to include: [08:00:10] 1. 🕫 Anything you'd like to share about your work [08:00:10] 2. ☏ Anything you'd like to get help with [08:00:10] 3. ⚠ Anything you're currently blocked on [08:00:10] (this message is from a toolforge job under the admin project) [12:30:19] Working on: [12:30:21] * T371486 wiki replicas view change, pinging people and trying to move it along [12:30:23] * writing docs for cloudvps admin groups/permissions [12:30:25] * having another look at T318479 redis connection timeouts [13:00:16] Done: [13:00:16] * [ceph] rebooted the switch on C8 successfully, we are undraining the nodes there now, it did cause a network outage that lasted ~10min, but not a ceph/storage one, so everything returned to normal (or we though) [13:00:16] * [openstack] Since yesterday reboot, it seems the rabbitmq cluster started dropping connections and made the whole of openstack unreliable, fixed by re-creating the rabbitmq cluster and restarting all the openstack services (and cleaning up the leftover errors) [13:00:17] * [ceph] Since yesterday too, ceph has been slow when replying with stats to prometheus (>10s), so prometheus started paging due to not having data from ceph, fixed by increasing the scrape timeout [13:00:17] * [toolforge] the network blip also made kyverno fail, what "luckily" allowed the stuck pods to get terminated, but might have allowed new malformed ones to get created too (we will see when they try to get deleted) [13:00:17] * [toolforge upgrade] The upgrade is done now, there's a few tasks left (improve kyverno policy to sip DELETE for example), but we are running 1.27 \o/ [13:00:17] Doing: [13:00:17] * [builds-builder/tekton] Working on the upgrade to tekton, it seems that it's going to be more complicated than I though, it uses something called conversion webhook for the CRDs that make the installation order tricky, as if you install the CRDs first, the current (old) webhook will start failing to convert them, but tekton needs the new CRDs to install the newer webook, so getting in a loop there [13:00:18] Blockers: [13:00:18] * Time