I had a question asked the other day about iSCSI Software initiator queue depth.
While adjusting queue depth on a hardware HBA (FCP or iSCSI) is a common performance tweak – assuming you have discussed the change with your storage vendor first – tweaking the same setting for the iSCSI software initiator doesn’t have as much of an effect.
An HBA has separate physical hardware to hold the queue and process commands, if the queue backs up the VMKernel will have to hold the commands. If you can optimize the resources on the card for your specific storage layout you’ll keep the commands from queuing at the VMKernel.
The iSCSI software initiator on the other hand is managed/owned by the VMKernel. Any commands that don’t fit in the queue don’t have far to go – the VMKernel still holds them. Thus, I would not expect to see much improvement (if any) when tweaking this parameter. You certainly would not see nearly as much benefit as tweaking the HBA queue. However, perhaps you have a specific reason or test, so here’s how to change it.
An HBA’s queue depth can be set when you configure it during system setup. Most default to 32.
From the vSphere 4.1 iSCSI SAN Config Guide
Set Maximum Queue Depth for Software iSCSI
If you notice unsatisfactory performance for your software iSCSI LUNs, you can change their maximum queue depth by using the vicfg-module command.
Procedure
1 On the vSphere CLI, run the following command:
vicfg-module -s iscsivmk_LunQDepth=value iscsi_vmk
The iscsivmk_LunQDepth parameter sets the maximum number of outstanding commands, or queue depth, for each LUN accessed through the software iSCSI adapter. The default value is 128.
2 Reboot your system.
CAUTION Setting the queue depth to a value higher than the default can decrease the total number of LUNs supported.
Note that you can also set how many outstanding commands a VM can have to a LUN.
From the same doc:
Equalize Disk Access Between Virtual Machines
You can adjust the maximum number of outstanding disk requests with the Disk.SchedNumReqOutstanding parameter in the vSphere Client. When two or more virtual machines are accessing the same LUN, this parameter controls the number of outstanding requests that each virtual machine can issue to the LUN.
Adjusting the limit can help equalize disk access between virtual machines.
This limit does not apply when only one virtual machine is active on a LUN. In that case, the bandwidth is limited by the queue depth of the storage adapter.
Procedure
1 In the vSphere Client, select the host in the inventory panel.
2 Click the Configuration tab and click Advanced Settings under Software.
3 Click Disk in the left panel and scroll down to Disk.SchedNumReqOutstanding.
4 Change the parameter value to the number of your choice and click OK.
This change can impact disk bandwidth scheduling, but experiments have shown improvements for diskintensive workload
