4

I'm experimenting with the effects of different SCSI queue depth values on a Dell server running CentOS Linux 5.4 (x86_64).

The server has two QLogic QLE2560 FC HBAs connected via multipathing to a storage system. The storage system has allocated two LUNs to the server, each connected through four paths in an active-active-active-active round-robin configuration. All in all, the two LUNs exist as eight /dev/sdX devices, represented by two devices in /dev/mpath.

I currently adjust the queue depth values in /etc/modprobe.conf and check the result (after rebooting) by looking in the seventh column of /proc/scsi/sg/devices.

Two questions related to that:

  • Is there a way to adjust queue depths without rebooting or unloading the qla2xxx kernel module? E.g., can I echo a new queue depth value into some /proc or /sys-like file to update the queue depth?

  • If I set the queue depth to 128, is that

    • 128 in total for all devices handled by the qla2xxx module?, or
    • 128 for each HBA? (256 in total), or
    • 128 for each of the eight /dev/sdX devices (1024 in toal)?, or
    • 128 for each of the two /dev/mpath/... devices (256 in total)?

    This is important for me to know so that my server doesn't flood the storage system, affecting other servers connected to it.

2 Answers 2

2

Usually the queue depth is per LUN/path, in rr multipathing configurations. It's very dependent on the implementation of the multipathing driver and you would have to check with the documentation of your multipathing driver which in your case is the device mapper's multipathing wich AFAIR uses the same concept as ESX in which the queue depth for all paths it the smallest queue depth for any of the paths see my example from ESX here:

ESX's native multipathing configures the queue depth in rr multipathing configurations to be the smallest common depth of any of the involved queues. Ie with a depth of 32 for each LUN/path and 4 paths the total queue depth is only 32! That's the reason why some vendors offer their own driver (like EMC's powerpath). From a performance perspective the round robing load distribution ESX doesn't cycle over any given path for each IO, the default schedules the first 1000 IOs go over the first path, the next 1000 IOs over the next on a per VM basis which should smooth the overall load over all paths as an per single IO path switch would incur a latency penalty for each IO.

1

My understanding of this is from an ESX environment where it is PathsQueue DepthLuns which would be 1024. I'd be very surprised if it was different with CentOS.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .