[05:37:56] <andrewbogott>	 dcaro, I'm starting the drain of cloudcephosd1020 and then going to sleep.  I think we're going to wind up a bit over 80% usage so if there's an easy way to adjust the warning threshold that's probably worth doing. The drain cookbook is  running in screen session on cloudcumin1001.
[12:28:58] <dcaro>	 I'm fixing the issue with the vm backups (added an exception when it should be just a log)
[12:29:37] <dcaro>	 the ceph cluster is currently in warning status due to being too full, will leave it for the weekend to try to rebalance and address it on monday
[12:29:56] <dcaro>	 (the osds that are too full right now are actually the ones in D5, that will be drained)
[12:35:16] <dhinus>	 thanks dcaro! while it rebalances it shuld stay around 85%, correct?
[12:35:18] <dhinus>	 *should
[12:35:37] <dhinus>	 as in, it should not increase further and if it does there might be something wrong?
[12:36:11] <dcaro>	 yep, it should actually decrease to ~70, worst case, stay the same (if it decides there's nothing to rebalance), but definitely not increase
[12:36:19] <dhinus>	 ok!
[12:41:25] <dcaro>	 hmm.. the backups are getting postgres errors
[12:41:26] <dcaro>	 Aug 09 12:40:40 cloudbackup1003 wmcs-backup[263233]: psycopg2.ProgrammingError: named cursor isn't valid anymore
[12:44:21] <dcaro>	 this is nice, progress in the process name xd
[12:44:23] <dcaro>	 https://www.irccloud.com/pastebin/SOQVRcMJ/
[12:45:08] <dcaro>	 should finish now though, I'll check in later, cya in a bit!
[13:16:26] <andrewbogott>	 dcaro: what should we do about draining cloudcephosd1023 and cloudcephosd1024? Should I drain one if/when the cluster goes out of error state? Or do we need to reschedule the switch maintenance?
[13:18:25] <dcaro>	 andrewbogott: you can try starting by the osds that are fullest if you want, now it needs some hand holding 
[13:19:56] <dcaro>	 Check with `ceph osd df D5`, and pick the one fullest, then the next and so
[13:20:46] <andrewbogott>	 ok -- but only after it's out of error state?
[13:21:08] <andrewbogott>	 (Also -- is it going to keep trying to pack things into D5 because it thinks it needs replicas there?)
[13:33:13] <dcaro>	 we can do when it's in warning, as we are taking out the osd that's fullest and spreading the data around
[13:33:26] <andrewbogott>	 ok, I'll try one now then
[13:35:08] <dcaro>	 okok, /me around if needed
[13:35:16] <andrewbogott>	 I can't tell if it's doing anything :)  ceph status doesn't show any rebalancing
[13:36:10] <andrewbogott>	 oh because I typed the wrong command...
[13:36:51] <dcaro>	 which one did you take out?
[13:36:52] <dcaro>	 aaah xd
[13:37:33] <andrewbogott>	 I did
[13:37:36] <andrewbogott>	 https://www.irccloud.com/pastebin/tClwFgno/
[13:37:54] <andrewbogott>	 but it seems like maybe the cookbook won't forward the command to ceph if the cluster is unhealthy?
[13:38:04] <andrewbogott>	 '0:01:16.387144 have passed, but the cluster is still not healthy, waiting 0:00:10 (timeout=0:30:00)...'
[13:38:21] <dcaro>	 yep, by default it waits for it to be healthy, you can try --force
[13:38:33] * andrewbogott tries it
[13:39:08] <dcaro>	 (or take out the osd manually worst case, `ceph osd crush reweight osd.XXX 0` + `ceph osd out osd.XXX`
[13:39:10] <dcaro>	 )
[13:39:22] <dcaro>	 I see remapping started
[13:39:29] <dcaro>	 and recovery was triggered
[13:39:38] <andrewbogott>	 yeah, --force did something
[13:40:04] <dcaro>	 let's see where it places the data, should go out of the D5 rack
[13:41:58] <dcaro>	 if it does not help, I'd wait for monday to try undraining the rest of the D5 rack down at once, to force it to put the data on the other racks
[13:42:24] <dcaro>	 (given that it's friday, and a holiday)
[13:43:09] <dcaro>	 the total usage of D5 seems to be going down
[13:43:10] <dcaro>	 TOTAL   26 TiB   21 TiB   21 TiB  1.4 GiB   67 GiB  5.2 TiB  79.97
[13:43:15] <dcaro>	 TOTAL   26 TiB   21 TiB   21 TiB  1.4 GiB   67 GiB  5.3 TiB  79.80
[13:43:20] <dcaro>	 that's good
[13:43:33] <dcaro>	 6 nearfull osd(s)
[13:43:34] <andrewbogott>	 cool
[13:43:47] <dcaro>	 yep, seems to have forced it to move stuff out of D5, nice
[13:43:57] <dcaro>	 (we had more before)
[13:44:28] <andrewbogott>	 yeah, so I'll continue to drain the >80% nodes and see if that cheers things up
[13:44:57] <dcaro>	 yep, ping me if anything goes awry (telegram if I don't reply)
[13:45:02] <andrewbogott>	 thanks!
[13:45:07] <dcaro>	 np
[15:08:29] <andrewbogott>	 hm, seems like it's doing at least some rebalancing into D5 still. 176 is up to 89% now :(
[15:08:52] <andrewbogott>	 Guess I'll give it a break and see what happens