I have a service request open with VMware on this, but right now, this is a head scratcher, and I thought I would post here to see if anyone else has any ideas.
I have an ESXi 4 cluster consisting of four Cisco UCS B250-M2 Blades, each with 12 cpu cores and 384GB of RAM. They are attached through a 10GB uplink to a Nexus 5K, and the storage is fiber channel LUN on an EMC NS-480.
We are running approximately 150 VM's consisting of a combination of Windows Server 2003 and 2008 servers.
What we noticed a couple months ago is that just about every night, we would have a couple of the Windows 2008 servers which would just shutdown and power back up. Not a BSOD or a graceful shutdown, but an unexpected shutdown. As we continued to deploy more VM's the problem seemed to get worse to where we would have 5-6 VM's shutting down every night, sometimes twice a night. It wasnt always the same VM's, but it seemed to always be a few from a pool of about 20 servers that kept doing it.
We investigated just about every aspect of the OS thinking it had something to do with antivirus or Windows updates kicking off at night, but we turned up nothing after about a week of exhaustive searching.
As an idea, I decided to move 10 of the Windows 2008 VM's which were constantly crashing to another cluster running ESXi 5. Once I did that, the crashes stopped. The interesting thing is that the servers I moved to ESXi5 stopped crashing, but we also stopped having crashes on the ESXi4 cluster.
I started thinking that maybe we were hitting a capacity issue of some kind. So last night I deployed 10 test Windows 2008 servers to the ESXi4 cluster, and waited. Sure enough, at night, we had about 6 servers (including 2 of the new test servers) start shutting down again.
So now I am trying to figure out what it is about this environment that is making these Windows 2008 servers shut down. It doesnt affect the 2003 servers, only the 2008 servers, and not all of them, just some of them.
Can anyone shed any light on this or give me something to look for?