[13:16:07] <dhinus>	 I'm seeing some alerts from tools-prometheus-6 being down and now tools-prometheus-7
[13:16:18] <dhinus>	 -6 is back up but it flapped twice since this morning
[13:26:26] <dhinus>	 -7 is indeed unreachable, "virsh console" is also getting stuck
[13:28:50] <dhinus>	 I can see a big spike in server load for -6 this morning
[13:34:42] <dhinus>	 aaand -7 is back
[15:21:01] <dhinus>	 tools-prometheus-6 is not responding again (3rd time today). andrewbogott can you think of anything to check? maybe it's just excessive load, but it's quite a beefy vm
[15:23:35] <andrewbogott>	 dhinus: The only (unlikely) think I can think of is if things are too crowded on the hypervisor and it's getting CPU starved -- I squished everything together to make room for the switch restart.
[15:23:40] <andrewbogott>	 I'll move it and we'll see if that makes a difference.
[15:24:20] <dhinus>	 thanks, good idea!
[15:28:28] <andrewbogott>	 well, -6 now has a hypervisor all to itself and I still can't ssh
[15:28:43] <andrewbogott>	 although I guess it might've been broken already before I moved it
[15:29:27] <andrewbogott>	 Did you have to reboot it before?
[15:30:55] <andrewbogott>	 dhinus: should I reboot -6?
[15:34:56] * andrewbogott does
[15:38:06] <andrewbogott>	 the oomkiller was firing before I restarted it
[15:38:55] <andrewbogott>	 so not a noisy neighbor issue
[16:05:48] <dhinus>	 andrewbogott: sorry, didn't see your ping. I don't know much about that host, but I assume it's safe to reboot
[16:05:58] <dhinus>	 this morning it came back on its own after 1 hour of being unresponsive
[16:06:31] <andrewbogott>	 Ok. Will see how it goes
[16:06:52] <dhinus>	 where did you see the oomkiller was firing?
[16:08:05] <dhinus>	 could you ssh to it? I could not when I tried earlier
[16:24:19] <dhinus>	 now cloudinfra-cloudvps-puppetserver-1 is alerting :/
[16:33:06] <dhinus>	 ssh and "virsh console" are not working
[16:40:13] <dhinus>	 I'll try forcing a reboot
[16:45:41] <dhinus>	 "openstack server reboot" fixed it
[16:46:53] <dhinus>	 I'm looking at the journald log for the past hour and I see "sssd_nss[2528288]: Shutting down (status = 0)"
[16:52:37] <dhinus>	 I remember sssd caused some issues to other vms in the past, but I'm not sure this is the same issue
[16:53:10] <dhinus>	 if this keeps happening I will open a task tomorrow
[16:53:24] * dhinus off for today