[13:34:25] * dcaro paged
[13:34:26] <dcaro>	 looking
[13:35:13] <dcaro>	  Cloudvirt node cloudvirt1048 is down. #page
[13:36:37] <dcaro>	 redis seems to be readonly (according to toolschecker)
[13:37:13] <dcaro>	 three tools k8s workers are down too, probably were running on that cloudvirt
[13:38:32] <andrewbogott>	 I'm on the console, going to reboot it
[13:40:31] * andrewbogott now waits forever to see if it's actually going to boot
[13:41:05] <andrewbogott>	 ok, there it goes
[13:41:14] <dcaro>	 watching too
[13:42:51] <dcaro>	 host is back up
[13:43:54] <andrewbogott>	 VMs should come back shortly
[13:44:25] <dcaro>	 k8s nodes coming up again
[13:44:52] <dcaro>	 redis back up again
[13:45:35] <dcaro>	 everything look ok now
[13:47:00] <andrewbogott>	 UE memory read error
[13:47:12] <andrewbogott>	 did phault make us a bug for this or should I make one?
[13:48:05] <andrewbogott>	 ah, T373740
[13:48:09] <stashbot>	 T373740: NodeDown - https://phabricator.wikimedia.org/T373740
[13:48:49] <dcaro>	 yep, the logs are interesting
[13:48:55] <dcaro>	 Memory failure: 0x4d47b27: already hardware poisoned
[13:50:56] <andrewbogott>	 Oh, there's more in the log, I updated that paste on the phab task (if that's where you were reading)
[13:51:46] <dcaro>	 let's see if it's stable, we can gather the full logs on monday
[13:52:09] <andrewbogott>	 yeah
[13:52:26] <andrewbogott>	 I guess I'll take my laptop with me when I leave the house :)
[13:52:36] <andrewbogott>	 I'll send an email about affected VMs.
[13:53:30] <dcaro>	 I'll keep an eye also during EU time
[13:53:34] <dcaro>	 thanks!
[13:53:45] <andrewbogott>	 dcaro: want me to move the tools hosts off, just in case?
[13:54:13] <andrewbogott>	 https://www.irccloud.com/pastebin/6ny9EqZb/
[13:57:48] <dcaro>	 maybe just redis, the others is ok if they go away
[13:58:00] <dcaro>	 yep xd
[14:00:39] <andrewbogott>	 tools-redis-7 is moved, email sent, going to go back to housekeeping
[14:03:37] <dcaro>	 thanks! I'll be around a laptop for a few hours at least, if you want to stay disconnected for a bit
[14:04:09] <andrewbogott>	 great, thank you