[09:04:48] <dcaro>	 yw!
[10:45:52] * dcaro lunch
[12:41:01] <arnaudb>	 👋 I'll cause a bit of trouble on replication on clouddbs (they'll be downtimed): https://phabricator.wikimedia.org/T372287#10062148
[12:43:38] <dcaro>	 ack
[13:18:30] <andrewbogott>	 Good morning! How has the switch been behaving?
[13:27:21] <dcaro>	 so far it seems stable, nothing out of the usual
[13:28:06] <andrewbogott>	 how long would you like to wait before repooling things?  (topranks, same questions to you)
[13:28:26] <dcaro>	 I'm ok starting to undrain already
[13:28:45] <dcaro>	 (if topranks thinks it's safe ;) )
[13:51:50] <dhinus>	 arnaudb: ok, thanks for the heads up!
[13:52:07] <arnaudb>	 its done dhinus!
[13:52:16] <dhinus>	 perfect!
[14:12:48] <topranks>	 dcaro, andrewbogott: just catching up... yep everything remains stable good to go on any repooling etc 
[14:13:05] <andrewbogott>	 great! thx
[15:00:03] <dcaro>	 jobo: I got no more votes :/
[15:00:11] <dcaro>	 (on the easy retro board)
[15:02:29] <dhinus>	 jobo: same, I think there's a limit on the number of votes
[15:02:42] <andrewbogott>	 dcaro: whoops, did I use everyone's votes?  I voted until I couldn't anymore
[15:02:53] <andrewbogott>	 I guess if we were all anonymous maybe we had the same vote pool
[15:03:39] <andrewbogott>	 dcaro: I'm about to run 'cookbook -c 1060173 wmcs.ceph.osd.undrain_node  --osd-hostname cloudcephosd1011 --cluster-name eqiad1' <- look right?
[15:03:45] <andrewbogott>	 Default batch size is 2 isn't it?
[15:04:24] <dcaro>	 andrewbogott: yep, though it might not go through, as the cluster is not in a healthy status, you might have to pass --force (and maybe --no-wait, as currently it's in a perpetual rebalancing)
[15:04:42] <andrewbogott>	 ok, let's see...
[15:05:02] <dcaro>	 note that that will go ahead and undrain the whole node
[15:05:12] <dcaro>	 (as it will not wait for rebalance)
[15:05:27] <dcaro>	 so maybe pass --osd-id and pick a couple
[15:05:40] <andrewbogott>	 yeah needed --force
[15:05:59] <andrewbogott>	 oh, it's doing a lot at a time :/
[15:06:09] * andrewbogott hits ctrl-c
[15:06:24] <dcaro>	 yep, as it's not waiting for them to get rebalanced
[15:06:30] <dcaro>	 there's 4 up now
[15:06:52] <dcaro>	 that fixed our backfill_too_full issue though xd
[15:07:06] <dcaro>	 so after it rebalances these 4 osds, the cluster will be healthy again
[15:07:06] <andrewbogott>	 yeah, should be able to use the normal command after this
[15:07:23] <dcaro>	 yep
[15:07:54] <dcaro>	 dhinus: where did the user report that the page was down?
[15:08:10] <dhinus>	 dcaro: #mediawiki-security
[15:08:21] <andrewbogott>	 dcaro: I'll try to keep an eye on this today, will leave you an update about how far it gets
[15:08:40] <dcaro>	 andrewbogott: ack thanks, I'll pick up on it tomorrow
[15:09:29] <dcaro>	 okok, gtg, I'll be around a bit later
[20:43:12] <bd808>	 a.ndrewbogott, d.caro: I've been thinking about T372353 and I have a hunch that we could have made the project name "deployment-prep-s3" and everything would have worked. The s3 stuff uses the Project ID rather than the Project Name, so now that we are creating new projects with UUID valued Project IDs the readable name shouldn't have anything to do with the s3 or Swift gateways.
[20:43:13] <stashbot>	 T372353: Request creation of deploymentpreps3 VPS project - https://phabricator.wikimedia.org/T372353
[20:45:13] <bd808>	 This thought brought to you by figuring out that I needed to use "675d02a3344846919fd7fdee700b53a2:opentofu" instead of "deploymentpreps3:opentofu" as my bucket name with OpenTofu.