[07:10:37] <godog>	 greetings
[08:00:44] <godog>	 FYI I'll be rebooting cloudcumin hosts shortly for T422596
[08:00:44] <stashbot>	 T422596: Failing Trixie VM installations on routed Ganeti - https://phabricator.wikimedia.org/T422596
[08:17:19] <godog>	 {{done}}
[09:42:58] <moritzm>	 ok to go ahead now and switch Cloud VPS to deb.debian.org by deploying https://gerrit.wikimedia.org/r/c/operations/puppet/+/1273441 ?
[09:45:38] <taavi>	 moritzm: sure
[09:46:46] <moritzm>	 ack, merging
[13:00:35] <godog>	 andrewbogott: I was off Fri for oncall comp, though I took a look at zk in codfw today and LGTM for the most part https://phabricator.wikimedia.org/T422646#11838198
[13:02:02] <andrewbogott>	 I was wondering about that, but then couldn't think of a likely scenario where zookeeper goes down on e.g. cloudcontrol1011 but the services /using/ zookeeper on cloudcontrol1011 are still up :)
[13:02:23] <andrewbogott>	 But assuming that tooz is smart about having multiple backends it's harmless at worst to list them all.
[13:03:30] <andrewbogott>	 I will be disappointed if designate-produced still can't recover :(
[13:05:52] <andrewbogott>	 I will write the puppet to add the full list of backends unless you have already done that
[13:06:31] <godog>	 I have not no, the scenario I have in mind is roll-restart zk the service e.g. for updates
[13:07:07] <godog>	 updates to zk itself that is, or the jvm
[13:07:42] <andrewbogott>	 yep, makes sense
[13:12:30] <godog>	 and yes a bummer designate-producer can't recover on its own heh
[14:20:42] <godog>	 andrewbogott: essentially https://wikitech.wikimedia.org/wiki/Reprepro#Adding_a_new_external_repository should have everything
[14:20:52] <godog>	 re: osbpo
[14:22:47] <andrewbogott>	 great, I will past that on the task :)
[14:27:46] <godog>	 heheh yes I did the same
[14:43:13] <andrewbogott>	 Is the idea that we would manually copy over the upstream repo periodically?
[14:43:27] <andrewbogott>	 Maybe there's something in the docs about automating and I haven't gotten there yet
[15:01:12] <godog>	 yes basically 'reprepro --restrict ... checkupdate' to verify if there are updates then 'reprepro --restrict ... update' to apply them
[16:06:04] <andrewbogott>	 I'm in the twice-yearly Magnum dev meeting and things aren't as dire as I hoped. A dozen attendees, at least two orgs providing dev support. There also seems to be a consensus around what driver to use (not the driver I've been working with, alas) although everyone is being very careful not to actually endorse.
[16:06:31] <andrewbogott>	 There are also more real-world deployments and paying customers out there than I expected.
[16:06:48] <andrewbogott>	 *as dire as I feared heh
[16:08:46] <andrewbogott>	 And there are debian worker images!
[16:20:22] <dhinus>	 what's the new cool driver of the day? :)
[16:22:48] <andrewbogott>	 it's just the other capi driver
[16:22:54] <andrewbogott>	 should be very similar, I hope
[16:23:46] * dhinus didn't even know there were two capi drivers :)
[16:44:04] <lucaswerkmeister>	 (intentionally vague question) is the email that went to some admin-y @toolforge.org addresses around 14:13 UTC being discussed / handled somewhere?
[16:45:08] <taavi>	 lucaswerkmeister: I'm not sure I see the mail you're referring to
[16:45:22] <lucaswerkmeister>	 moving to DM
[18:16:51] <andrewbogott>	 anyone looking at Quarry?
[18:22:20] <andrewbogott>	 hm, seems to have recovered for now
[18:22:51] <andrewbogott>	 very slow though
[18:25:51] <lucaswerkmeister>	 re what I wrote above: now at T423940
[18:26:23] <andrewbogott>	 ty lucaswerkmeister