I am having very poor speeds with iscsi right now. Below is my hardware setup.
I have a Supermicro Blade Enclosure with one Twinblade Blade unit SBI-7126TG.
The Blade has the AOC-XEH-iN2 Mezzanine adapter using the Intel 82599 Niantic chip for 10Gbe.
The Blade Enclosure has a Supermicro SBM-XEM-X10SM 10Gbe Switch.
I have a Supermicro SAN SYS-6036ST-6LR
The SAN has a Brocade 1020 CNA for 10Gbe in each node.
Each node in the SAN has an identical RAID-10 of 4 SSD drives OCZ Agility 3 240GB each.
The 10GBe switch is only for iSCSI traffic, everything else is on a 1GBe switch on 1GBe NICs.
For the Software side.
Both blades and SAN have ESXi 5 installed.
The Blade has a VM of Windows Server 2008 R2 Enterprise x64.
The SAN has a VM on each node of Ubuntu Server 11.10 x64.
The condiguration side.
The Ubuntu VMs use Glusterfs 3 for a replicated file system across the two nodes in the SAN on the two RAID-10 SSD arrays.
The Ubuntu VMs use Heartbeat to create a Cluster and a Virtual IP to node 1. If it goes offline it switches it to node 2.
The Ubuntu VMs use iSCSI Enterprise Target for the iSCSI Target.
The file system on the array is EXT4.
I used dd to create a flat file to serve via iSCSI to windows. (dd if=/dev/zero of=/iscsi_SSD/SSD.img bs=1M count=80000)
I created a MBR type NTFS partition for the served iSCSI volume.
The Windows VM is using the MS iSCSI initiator to connect to the virtual IP.
On each ESXi host I created a separate vSwitch for the 10GBe traffic and set its MTU to 9000.
All NICs are set to MTU 9000 and so is the 10Gbe Switch.
Ok now to my problem.
Everything is working ok but just very poorly. If I copy a 2GB file to the iSCSI array it starts out at about 200mb/s and after about 1 sec goes to 150ish mb/s then again drops to 90 then 70 then 60 etc, etc. Near the end it levels at 18 mb/s. So it progressively gets worse. To do a test I disabled Gluster so no replicating was going on. When doing this I got about 400mb/s but it still progrsesivly went down. Much faster but still a problem. I also did a test with a non RAID target to the ubunto OS single SSD in the SAN and got about 150mb/s with progressive degrade.
Now if I do a copy back from the SAN I get a steady 80ish mb/s on either target. So it's steady but still very slow. If I go to the SAN directly and do a dd command to create a 2GB file on the array I get about 300mb/s with gluster on, and about 700mb/s with gluster off. So that's great and leads me to believe it is an iSCSI, or Network, or Windows, or VM/ESXi issue.
I'm at a lost of wha to do now to pin point the problem. Any sugesstions? Oh and I tried this same config and test on the 1GBe main switch with MTU 1500 and got the exact same results.
Thanks
Chris