Solaris iSCSI read speeds increase when we start writing

Question

Here's one I haven't seen before. We have an IBM Storwize V3700 storage array, 24 TB, RAID 5 connected to a Solaris 10 file server via an iSCSI link over Gigabit Ethernet (no MPIO at present).

We have two volumes on the RAID array:

c2t602d0 - ZFS
c2t603d0 - UFS

I've shortened the names for brevity.

Now, we were seeing very slow read speeds from the ZFS volume on the unit (~ 1-3 MB/s). I created the UFS volume as a test and ran Bonnie++ on it to do some benchmarking of the unit.

Observe the output of iostat prior to starting up Bonnie++:

$ iostat -Dnx c2t602d0 c2t603d0 rmt/1 5 1000

                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
   13.4    0.0 1491.0    0.0  0.0  1.5    0.0  112.6   0  99 c2t602d0 (ZFS)
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c2t603d0 (UFS)
    0.0   10.2    0.0 1305.9  0.0  0.0    0.0    1.2   0   1 rmt/1    (Tape backup)
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
   15.8    0.0 1807.4    0.0  0.0  1.3    0.0   84.8   0  99 c2t602d0 (ZFS)
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c2t603d0 (UFS)
    0.0   11.4    0.0 1459.2  0.0  0.0    0.0    1.2   0   1 rmt/1    (Tape backup)
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
   10.8    0.0 1233.7    0.0  0.0  1.2    0.0  110.1   0  99 c2t602d0 (ZFS)
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c2t603d0 (UFS)
    0.0    7.6    0.0  972.9  0.0  0.0    0.0    1.2   0   1 rmt/1    (Tape backup)
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
   18.0    0.0 2060.6    0.0  0.0  1.3    0.0   70.9   0  98 c2t602d0 (ZFS)
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c2t603d0 (UFS)
    0.0   12.6    0.0 1612.7  0.0  0.0    0.0    1.2   0   2 rmt/1    (Tape backup)

rmt/1 is our tape backup drive, which is currently doing a full backup of c2t602d0 -- the ZFS volume. Notice that we're reading from the volume at about 1.2 - 2.0 MB/s.

Next, watch what happens when I start up the Bonnie++ benchmark on the UFS disk (which is simply another volume on the same IBM storage unit):

$ iostat -Dnx c2t602d0 c2t603d0 rmt/1 5 1000

                    extended device statistics
    r/s    w/s   kr/s     kw/s wait actv wsvc_t asvc_t  %w  %b device
   81.0    0.0 10204.4     0.0  0.0  4.1    0.0   51.2   0  96 c2t602d0 (ZFS)
    0.2   75.6     1.6 58547.5  0.0 12.8    0.0  168.8   0  99 c2t603d0 (UFS)
    0.0   77.0     0.0  9859.1  0.0  0.1    0.0    1.2   0   9 rmt/1    (Tape backup)
                    extended device statistics
    r/s    w/s   kr/s     kw/s wait actv wsvc_t asvc_t  %w  %b device
   90.4    0.0 11465.3     0.0  0.0  4.3    0.0   48.0   0  97 c2t602d0 (ZFS)
    0.0   83.4     0.0 57903.1  0.0 13.7    0.0  164.9   0 100 c2t603d0 (UFS)
    0.0   86.0     0.0 11004.7  0.0  0.1    0.0    1.2   0  11 rmt/1    (Tape backup)
                    extended device statistics
    r/s    w/s   kr/s     kw/s wait actv wsvc_t asvc_t  %w  %b device
   79.8    0.0 10048.3     0.0  0.0  3.2    0.0   40.7   0  97 c2t602d0 (ZFS)
    0.0   86.0     0.0 60239.9  0.0 13.3    0.0  155.0   0  98 c2t603d0 (UFS)
    0.0   74.4     0.0  9527.7  0.0  0.1    0.0    1.2   0   9 rmt/1    (Tape backup)
                    extended device statistics
    r/s    w/s   kr/s     kw/s wait actv wsvc_t asvc_t  %w  %b device
   91.2    0.0 11587.8     0.0  0.0  5.2    0.0   56.7   0  97 c2t602d0 (ZFS)
    0.0   71.0     0.0 55932.3  0.0 13.3    0.0  186.5   0 100 c2t603d0 (UFS)
    0.0   89.2     0.0 11423.4  0.0  0.1    0.0    1.2   0  11 rmt/1    (Tape backup)

Bonnie++ is writing to the UFS disk on the storage unit at around 55 - 60 MB/s. The weird part is that the read speed on the ZFS volume has now jumped to ~10 MB/s. Still not great for Gigabit ethernet, but much better. It's not an anomaly, either. It sustains speeds above 10 MB/s as long as the heavy writes from Bonnie++ are taking place. If I kill Bonnie++, the read speeds on the ZFS volume drop back down to around 1-2 MB/s.

Any ideas on how I can explain this? If anything, I would have thought that the opposite would occur. We have both these volumes on the same storage unit connected via iSCSI to our file server. If I start heavily writing to one of them, I would have expected the performance of reads on the other to decrease rather than soaring to read 5 times as fast as it was.

Thanks for your insight.

Answer 1

I'd start by checking the server for compliance with the storage's configuration requirements. Check queue depth and driver levels, as well as switch configuration (jumbo frames, flow control, etc). Something is definitely wrong, but it doesn't sound like it's the storage. If you were seeing a performance bottleneck from storage hardware, it'd get worse when you increase the load.

A lot of the legwork for host configuration on storage is done for you if you install a host-kit. There should be an IBM solaris host kit for this storage- start by installing that and test again.