Hi folks,
Had a strange problem start yesterday and continuing to today. Trying to log in to a single ESXi 4.1 server (not part of a vCenter setup), and the login screen passes the login stage and is grinding through the inventory load stage, when it loses the connection. I tried using vSphere clients on other PC's (including one plugged into the same switch as the server itself), but the same thing occurs. This server has 8GB RAM and had 18 VM's (mostly low activity, but almost all had 1GB RAM allocations). The VM's themselves are running fine it seems, no lack of CPU, and I don't get the impression there is heavy memory swapping going on because the disks aren't being thrashed and the VM's are responsive. The regular server HTTPS webpage is viewable and I can use the linked HTTPS datastore viewer. I had made no ESXi level setting changes just prior or for some time before this problem started, though I have been recently opening remote console windows to VM's and using the CD connect feature to access some ISO files.
I have tried restarting both the management agents and the management network at the server console, but that didn't fix things. There were sorta similar problems listed in google, that recommended changing the DNS settings to a known working server, but the currently set one is operating correctly ( I did try switching to another DNS server that didn't fix it either so I switched back).
Thankfully I had left SSH access on, so I went diving into /var/log/messages and I see a strange error about no buffer space during the time I attempted to access with the client.
Jan 27 23:25:01 Hostd: [2011-01-27 23:25:01.434 29281B90 verbose 'Proxysvc Req00174'] New proxy client SSL(TCP(local=10.0.1.192:3980, peer=10.0.1.32:443))
Jan 27 23:25:01 Hostd: [2011-01-27 23:25:01.531 FFD19B90 verbose 'HttpSvc.HTTPService'] User agent is 'VMware VI Client/4.0.0'
Jan 27 23:25:01 Hostd: [2011-01-27 23:25:01.531 FFD19B90 verbose 'HttpSvc.HTTPService'] HTTP Response: Client: NeedsContentLength: false UnderstandsChunking: true CanKeepAlive: true (PresetContentLength 374)
Jan 27 23:25:01 Hostd: [2011-01-27 23:25:01.531 FFD19B90 verbose 'HttpSvc.HTTPService'] HTTP Response: Auto-completing at 374/374 bytes
Jan 27 23:25:01 Hostd: [2011-01-27 23:25:01.531 FFD19B90 verbose 'HttpSvc.HTTPService'] HTTP Response: Complete (processed 374 bytes)
Jan 27 23:25:01 Hostd: [2011-01-27 23:25:01.531 FFD19B90 verbose 'HTTP server'] Sent response for HEAD /client/clients.xml (from /usr/lib/vmware/hostd/docroot)
Jan 27 23:25:02 dhclient-uw-4734: XMT: Solicit on vmk1, interval 117
Jan 27 23:25:05 Hostd: [2011-01-27 23:25:05.258 FFDDCB90 verbose 'Proxysvc Req00175'] New proxy client SSL(TCP(local=10.0.1.192:3981, peer=10.0.1.32:443))
Jan 27 23:25:05 Hostd: [2011-01-27 23:25:05.383 FFD19B90 verbose 'Proxysvc Req00176'] New proxy client SSL(TCP(local=10.0.1.192:3982, peer=10.0.1.32:443))
Jan 27 23:25:05 Hostd: [2011-01-27 23:25:05.541 29240B90 verbose 'HttpSvc.HTTPService'] User agent is 'VMware VI Client/4.0.0'
Jan 27 23:25:05 Hostd: [2011-01-27 23:25:05.541 29240B90 verbose 'HttpSvc.HTTPService'] HTTP Response: Client: NeedsContentLength: false UnderstandsChunking: true CanKeepAlive: true (PresetContentLength -1)
Jan 27 23:25:05 Hostd: [2011-01-27 23:25:05.542 FFA22E80 verbose 'HttpSvc.HTTPService' opID=A79BC4B6-00000002] HTTP Response: Complete (processed 1219 bytes)
Jan 27 23:25:06 Hostd: [2011-01-27 23:25:06.831 FFA22E80 verbose 'Proxysvc Req00177'] New proxy client SSL(TCP(local=10.0.1.192:3983, peer=10.0.1.32:443))
Jan 27 23:25:06 Hostd: [2011-01-27 23:25:06.835 FFDDCB90 verbose 'HTTP server'] Sent response for HEAD /client/clients.xml (from /usr/lib/vmware/hostd/docroot)
Jan 27 23:25:06 Hostd: [2011-01-27 23:25:06.865 FFD19B90 verbose 'HttpSvc.HTTPService'] User agent is 'VMware VI Client/4.0.0'
Jan 27 23:25:06 Hostd: [2011-01-27 23:25:06.865 FFD19B90 verbose 'HttpSvc.HTTPService'] HTTP Response: Client: NeedsContentLength: false UnderstandsChunking: true CanKeepAlive: true (PresetContentLength -1)
Jan 27 23:25:06 Hostd: pam_per_user: create_subrequest_handle(): doing map lookup for user "root"
Jan 27 23:25:06 Hostd: pam_per_user: create_subrequest_handle(): creating new subrequest (user="root", service="system-auth-generic")
Jan 27 23:25:06 Hostd: pam_env(system-auth-generic:setcred): Unable to open config file: /etc/security/pam_env.conf: No such file or directory
Jan 27 23:25:06 Hostd: Accepted password for user root from 10.0.1.192
Jan 27 23:25:06 Hostd: [2011-01-27 23:25:06.872 29281B90 info 'Vimsvc' opID=A79BC4B6-00000003] [Auth]: User root
Jan 27 23:25:06 Hostd: [2011-01-27 23:25:06.873 29281B90 info 'ha-eventmgr' opID=A79BC4B6-00000003] Event 8 : User root@10.0.1.192 logged in
Jan 27 23:25:06 Hostd: [2011-01-27 23:25:06.873 29281B90 verbose 'HttpSvc.HTTPService' opID=A79BC4B6-00000003] HTTP Response: Complete (processed 673 bytes)
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.037 29281B90 verbose 'HttpSvc.HTTPService'] User agent is 'VMware VI Client/4.0.0'
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.037 29281B90 verbose 'HttpSvc.HTTPService'] HTTP Response: Client: NeedsContentLength: false UnderstandsChunking: true CanKeepAlive: true (PresetContentLength -1)
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.039 FFA22E80 verbose 'HttpSvc.HTTPService'] HTTP Response: Complete (processed 12469 bytes)
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.041 29240B90 verbose 'Proxysvc Req00178'] New proxy client SSL(TCP(local=10.0.1.192:3984, peer=10.0.1.32:443))
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.044 FFDDCB90 verbose 'Proxysvc Req00179'] New proxy client SSL(TCP(local=10.0.1.192:3985, peer=10.0.1.32:443))
Jan 27 23:25:07 vmkernel: 61:18:59:13.226 cpu3:9530389)WARNING: Tcpip: 1313: socreate(type=2, proto=0) failed with error 55
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.078 27FFFB90 warning 'Proxysvc Req00179'] Connection to localhost:8307 failed with error N7Vmacore15SystemExceptionE(Connection reset by peer).
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.078 29281B90 warning 'Proxysvc Req00178'] Connection to localhost:8307 failed with error N7Vmacore15SystemExceptionE(Connection reset by peer).
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.086 FFD19B90 verbose 'Proxysvc Req00180'] New proxy client SSL(TCP(local=10.0.1.192:3986, peer=10.0.1.32:443))
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.116 29281B90 verbose 'Proxysvc Req00181'] New proxy client SSL(TCP(local=10.0.1.192:3987, peer=10.0.1.32:443))
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.119 FFD19B90 verbose 'HttpSvc.HTTPService'] User agent is 'No-User-Agent'
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.120 FFD19B90 verbose 'HttpSvc.HTTPService'] HTTP Response: Client: NeedsContentLength: false UnderstandsChunking: true CanKeepAlive: true (PresetContentLength 545)
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.120 FFD19B90 verbose 'HttpSvc.HTTPService'] HTTP Response: Auto-completing at 545/545 bytes
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.156 FFDDCB90 verbose 'Proxysvc Req00182'] New proxy client SSL(TCP(local=10.0.1.192:3988, peer=10.0.1.32:443))
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.160 27FFFB90 verbose 'HttpSvc.HTTPService'] User agent is 'No-User-Agent'
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.160 27FFFB90 verbose 'HttpSvc.HTTPService'] HTTP Response: Client: NeedsContentLength: false UnderstandsChunking: true CanKeepAlive: true (PresetContentLength 545)
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.160 27FFFB90 verbose 'HttpSvc.HTTPService'] HTTP Response: Auto-completing at 545/545 bytes
Jan 27 23:25:07 vmkernel: 61:18:59:13.323 cpu5:9534698)WARNING: Tcpip: 1313: socreate(type=2, proto=0) failed with error 55
Jan 27 23:25:07 vmkernel: 61:18:59:13.323 cpu5:9534698)WARNING: Tcpip: 1313: socreate(type=2, proto=0) failed with error 55
Jan 27 23:25:07 vmkernel: 61:18:59:13.323 cpu5:9534698)WARNING: Tcpip: 1313: socreate(type=2, proto=0) failed with error 55
Jan 27 23:25:07 vmkernel: 61:18:59:13.323 cpu5:9534698)WARNING: Tcpip: 1313: socreate(type=1, proto=6) failed with error 55
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.168 27FFFB90 verbose 'HttpSvc.HTTPService'] User agent is 'VMware VI Client/4.0.0'
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.175 FFDDCB90 warning 'Proxysvc Req00180'] Connection to localhost:8307 failed with error N7Vmacore15SystemExceptionE(No buffer space available).
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.189 27FFFB90 verbose 'HttpSvc.HTTPService'] HTTP Response: Client: NeedsContentLength: false UnderstandsChunking: true CanKeepAlive: true (PresetContentLength -1)
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.458 29240B90 verbose 'Proxysvc Req00183'] New proxy client SSL(TCP(local=10.0.1.192:3989, peer=10.0.1.32:443))
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.458 FFD19B90 verbose 'Proxysvc Req00184'] New proxy client SSL(TCP(local=10.0.1.192:3990, peer=10.0.1.32:443))
Jan 27 23:25:07 vmkernel: 61:18:59:13.607 cpu6:9656671)WARNING: Tcpip: 1313: socreate(type=2, proto=0) failed with error 55
Jan 27 23:25:07 vmkernel: 61:18:59:13.608 cpu6:9656671)WARNING: Tcpip: 1313: socreate(type=2, proto=0) failed with error 55
Jan 27 23:25:07 vmkernel: 61:18:59:13.608 cpu6:9656671)WARNING: Tcpip: 1313: socreate(type=2, proto=0) failed with error 55
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.460 292C2B90 warning 'Proxysvc Req00183'] Connection to localhost:8307 failed with error N7Vmacore15SystemExceptionE(No buffer space available).
Jan 27 23:25:07 vmkernel: 61:18:59:13.608 cpu6:9656671)WARNING: Tcpip: 1313: socreate(type=1, proto=6) failed with error 55
Jan 27 23:25:07 vmkernel: 61:18:59:13.609 cpu5:9531975)WARNING: Tcpip: 1313: socreate(type=2, proto=0) failed with error 55
Jan 27 23:25:07 vmkernel: 61:18:59:13.609 cpu5:9531975)WARNING: Tcpip: 1313: socreate(type=2, proto=0) failed with error 55
Jan 27 23:25:07 vmkernel: 61:18:59:13.609 cpu5:9531975)WARNING: Tcpip: 1313: socreate(type=2, proto=0) failed with error 55
Jan 27 23:25:07 vmkernel: 61:18:59:13.609 cpu5:9531975)WARNING: Tcpip: 1313: socreate(type=1, proto=6) failed with error 55
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.461 29240B90 warning 'Proxysvc Req00184'] Connection to localhost:8307 failed with error N7Vmacore15SystemExceptionE(No buffer space available).
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.462 FFA22E80 verbose 'Proxysvc Req00185'] New proxy client SSL(TCP(local=10.0.1.192:3991, peer=10.0.1.32:443))
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.462 FFD5AB90 verbose 'Proxysvc Req00186'] New proxy client SSL(TCP(local=10.0.1.192:3992, peer=10.0.1.32:443))
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.462 FFDDCB90 verbose 'Proxysvc Req00187'] New proxy client SSL(TCP(local=10.0.1.192:3997, peer=10.0.1.32:443))
Jan 27 23:25:07 vmkernel: 61:18:59:13.618 cpu5:9697632)WARNING: Tcpip: 1313: socreate(type=2, proto=0) failed with error 55
Jan 27 23:25:07 vmkernel: 61:18:59:13.618 cpu7:9534695)WARNING: Tcpip: 1313: socreate(type=2, proto=0) failed with error 55
Jan 27 23:25:07 vmkernel: 61:18:59:13.618 cpu5:9697632)WARNING: Tcpip: 1313: socreate(type=2, proto=0) failed with error 55
Jan 27 23:25:07 vmkernel: 61:18:59:13.618 cpu5:9697632)WARNING: Tcpip: 1313: socreate(type=2, proto=0) failed with error 55
Jan 27 23:25:07 vmkernel: 61:18:59:13.618 cpu5:9697632)WARNING: Tcpip: 1313: socreate(type=1, proto=6) failed with error 55
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.470 29740B90 warning 'Proxysvc Req00186'] Connection to localhost:8307 failed with error N7Vmacore15SystemExceptionE(No buffer space available).
Jan 27 23:25:07 Hostd: [2011-01-27 23:25:07.470 FFD5AB90 warning 'Proxysvc Req00187'] Connection to localhost:8307 failed with error N7Vmacore15SystemExceptionE(No buffer space available).
Jan 27 23:25:07 vmkernel: 61:18:59:13.619 cpu3:9534695)WARNING: Tcpip: 1313: socreate(type=2, proto=0) failed with error 55
Jan 27 23:25:07 vmkernel: 61:18:59:13.619 cpu3:9534695)WARNING: Tcpip: 1313: socreate(type=2, proto=0) failed with error 55
Jan 27 23:25:07 vmkernel: 61:18:59:13.619 cpu3:9534695)WARNING: Tcpip: 1313: socreate(type=1, proto=6) failed with error 55
Jan 27 23:25:13 dhclient-uw-4737: DHCPDISCOVER on vmk4 to 255.255.255.255 port 67 interval 6
Jan 27 23:25:15 dhclient-uw-4738: DHCPDISCOVER on vmk5 to 255.255.255.255 port 67 interval 10
Jan 27 23:25:19 dhclient-uw-4737: DHCPDISCOVER on vmk4 to 255.255.255.255 port 67 interval 15
Jan 27 23:25:25 dhclient-uw-4738: DHCPDISCOVER on vmk5 to 255.255.255.255 port 67 interval 11
Jan 27 23:25:32 dhclient-uw-4735: DHCPDISCOVER on vmk2 to 255.255.255.255 port 67 interval 8
Jan 27 23:25:34 dhclient-uw-4737: DHCPDISCOVER on vmk4 to 255.255.255.255 port 67 interval 15
Jan 27 23:25:36 dhclient-uw-4738: DHCPDISCOVER on vmk5 to 255.255.255.255 port 67 interval 7
Jan 27 23:25:40 dhclient-uw-4735: DHCPDISCOVER on vmk2 to 255.255.255.255 port 67 interval 12
Jan 27 23:25:43 dhclient-uw-4738: DHCPDISCOVER on vmk5 to 255.255.255.255 port 67 interval 12
Jan 27 23:25:48 Hostd: [2011-01-27 23:25:48.547 FFD5AB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Jan 27 23:25:49 dhclient-uw-4737: DHCPDISCOVER on vmk4 to 255.255.255.255 port 67 interval 13
Jan 27 23:25:52 dhclient-uw-4735: DHCPDISCOVER on vmk2 to 255.255.255.255 port 67 interval 7
Jan 27 23:25:54 Hostd: [2011-01-27 23:25:54.906 FFD5AB90 verbose 'App'] Timed out reading between HTTP requests. : Read timeout after approximately 50000ms. Closing stream TCP(local=127.0.0.1:50341, peer=127.0.0.1:8309)
Wireshark packet capture shows TCP reset packets coming from the server during the inventory load phase it seems, perhaps as a consequence of the socket failure.
ESXTOP is showing a CPU load average around 0.15
ESXTOP for memory is like so
12:23:54am up 61 days 19:58, 280 worlds; MEM overcommit avg: 1.55, 1.55, 1.55
PMEM /MB: 8181 total: 945 vmk, 6319 other, 917 free
VMKMEM/MB: 7765 managed: 465 minfree, 4486 rsvd, 3278 ursvd, high state
PSHARE/MB: 3225 shared, 302 common: 2923 saving
SWAP /MB: 526 curr, 550 rclmtgt: 0.00 r/s, 0.00 w/s
ZIP /MB: 510 zipped, 319 saved
MEMCTL/MB: 8746 curr, 8676 target, 11631 max
GID NAME MEMSZ GRANT SZTGT TCHD TCHD_W SWCUR SWTGT SWR/s SWW/s OVHDUW OVHD OVHDMAX
14080 ISL - TestSML08 1024.00 383.04 356.41 74.52 35.07 31.05 22.73 0.00 0.00 23.43 33.95 90.58
14081 ISL - TestSML09 1024.00 920.68 554.66 194.21 106.81 3.45 0.00 0.00 0.00 23.43 34.60 90.58
14082 ISL - TestSML10 1024.00 261.00 220.31 60.99 28.70 40.00 61.29 0.00 0.00 25.14 37.27 98.51
14088 ISL - XPJPSP3-O 1024.00 471.40 400.06 91.66 53.07 6.26 7.84 0.00 0.00 37.13 49.84 110.57
189477 ISL - TESTSML12 1024.00 246.46 226.98 60.99 21.53 74.39 83.10 0.00 0.00 40.61 54.19 126.39
1862 A-VC6A3-main 1024.00 1005.11 642.23 81.92 61.44 8.74 0.00 0.00 0.00 54.52 84.30 195.78
90069 ISL - SourceCod 1024.00 177.48 121.35 17.94 3.59 162.43 166.53 0.00 0.00 37.13 49.41 110.57
825689 ISL - TestSML11 1024.00 436.72 479.13 119.82 78.34 6.89 7.81 0.00 0.00 35.48 45.74 102.70
825690 ISL - TESTSML13 1024.00 318.21 322.20 100.46 64.58 6.84 4.11 0.00 0.00 40.61 54.96 126.39
825691 ISL - TESTSML14 1024.00 286.11 265.83 68.17 28.70 35.33 41.10 0.00 0.00 40.61 55.39 126.39
1245163 ISL - HINPOXENA 1024.00 312.15 311.67 147.18 82.57 23.98 12.37 0.00 0.00 37.19 48.37 110.63
2782198 ISL - HINPOXENA 1024.00 975.14 473.27 247.75 118.92 8.13 0.00 0.00 0.00 37.19 48.33 110.63
14074 ISL - HINPOXEND 1024.00 268.11 258.74 50.23 21.53 28.52 52.33 0.00 0.00 35.48 46.66 102.70
14076 ISL - EXCHANGE2 1024.00 679.22 686.25 246.81 119.88 1.33 0.00 0.00 0.00 37.19 49.86 110.63
14077 ISL - TESTSMLIN 1024.00 299.77 222.84 57.44 35.90 31.96 32.29 0.00 0.00 25.14 37.52 98.52
14078 ISL - TESTSML15 1024.00 312.20 374.27 136.32 107.62 4.52 3.62 0.00 0.00 35.48 46.23 102.70
14079 ISL - TestSML07 1024.00 274.05 220.43 60.99 28.70 51.55 55.84 0.00 0.00 25.14 37.09 98.51
2783868 Z-Diver 512.00 509.14 477.72 245.76 204.80 0.98 0.00 0.00 0.00 23.06 31.11 92.40
Some of the google results talked about reserved system memory size. This server was originally installed with 4GB of RAM but was later upgraded 8GB, but I never made changes to the reserved memory. My suspicion is perhaps the reserved memory may be too low, but I am not sure how to check or change this with just command line tools.
There is one related possibility that I don't know how to go about checking. A little before this problem started, from within a VM onboard this server, I had been using a network probe tool called "The Dude" http://www.mikrotik.com/thedude.php (version 4 beta 2) to try mapping out serveral remote subnets elsewhere, accessed via a Vyatta software router VM onboard the server through a single pNIC to the remote subnets. Why I mention this is, in the past when I used an older version 3 of The Dude, I had encountered a problem where various windows IIS HTTP servers became unresponsive for no obvious reason other than I had been using The Dude. Since the vSphere client is basically accessing a HTTPS server on ESXi, it makes me wonder if this is related. I can't say that is the true cause though, because I can still even now access the remote subnets fine via the Vyatta VM, and I had already stopped and shut down The Dude before I had resarted the ESXi managemnt network and agents.
Right now, I am thinking of restarting the server and trying to cut back on the number of VM's operating as a stopgap measure to fix this, and possibly increase the reserved system memory. Is there anything I can do to avoid restarting the ESXi server to fix connectivity, and/or are there other things/settings I can do prevent this from reoccuring. It's kinda scary to (sort of) lose control of an ESXi server, even if teh VM's are alright.