Having what appears to be an odd issue. The most easily observable symptom is copying approximately 12GB of data from one virtual disk drive to a second virtual disk drive. This process takes roughly 10 - 12 minutes to complete. The exact same copy process on an older physical server with a direct-attached array with (9) 10k disks takes about 2 minutes. This file copy consists of mostly smaller (1mb or less) files.
The VM disk files live on a Compellent array with (12) 15k disks attached to the ESX 4.1 host via 1GB iSCSI. The ESX host connects to the Compellent SAN with a dual port QLogic iSCSI HBA. Each port of the server HBA is connected through its own Cisco switch (i.e. two fault domains) and the volume has a total of 4 paths to the data store where the disk files live (configured in a Round Robin multi-path policy).
I've done a little background work and have monitored the copy process using a combination of the Compellent GUI, the vCenter GUI and ESXTOP. According to the Compellent system, I/O maxes out at around 400 IOPS during the process and is reading/writing about 100,000kbps. The vCenter GUI shows almost the same performance stats. ESXTOP stats look okay too with QUED remaining consistently at 0 and DAVG consistently running between 10 and 15ms.
It's odd to me that though there appears to be no bottleneck, I'm only getting about 400-500 IOPS out of a SAN which should produce something more in the 1800-2000 range. Could there be something within ESX which is throttling I/O back? A setting in the switches I should look at? Two things I have not done are to enable Jumbo Frames or to adjust the Queue Depth setting on the ESX server's HBA. Wanted a little more information before going down that road.
Any input would be appreciated.
Thanks,
Kenny