Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Here's one I haven't seen before. We have an IBM Storwize V3700 storage array, 24 TB, RAID 5 connected to a Solaris 10 file server via an iSCSI link over Gigabit Ethernet (no MPIO at present).

We have two volumes on the RAID array:

c2t602d0 - ZFS
c2t603d0 - UFS

I've shortened the names for brevity.

Now, we were seeing very slow read speeds from the ZFS volume on the unit (~ 1-3 MB/s). I created the UFS volume as a test and ran Bonnie++ on it to do some benchmarking of the unit.

Observe the output of iostat prior to starting up Bonnie++:

$ iostat -Dnx c2t602d0 c2t603d0 rmt/1 5 1000

                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
   13.4    0.0 1491.0    0.0  0.0  1.5    0.0  112.6   0  99 c2t602d0 (ZFS)
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c2t603d0 (UFS)
    0.0   10.2    0.0 1305.9  0.0  0.0    0.0    1.2   0   1 rmt/1    (Tape backup)
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
   15.8    0.0 1807.4    0.0  0.0  1.3    0.0   84.8   0  99 c2t602d0 (ZFS)
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c2t603d0 (UFS)
    0.0   11.4    0.0 1459.2  0.0  0.0    0.0    1.2   0   1 rmt/1    (Tape backup)
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
   10.8    0.0 1233.7    0.0  0.0  1.2    0.0  110.1   0  99 c2t602d0 (ZFS)
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c2t603d0 (UFS)
    0.0    7.6    0.0  972.9  0.0  0.0    0.0    1.2   0   1 rmt/1    (Tape backup)
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
   18.0    0.0 2060.6    0.0  0.0  1.3    0.0   70.9   0  98 c2t602d0 (ZFS)
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c2t603d0 (UFS)
    0.0   12.6    0.0 1612.7  0.0  0.0    0.0    1.2   0   2 rmt/1    (Tape backup)

rmt/1 is our tape backup drive, which is currently doing a full backup of c2t602d0 -- the ZFS volume. Notice that we're reading from the volume at about 1.2 - 2.0 MB/s.

Next, watch what happens when I start up the Bonnie++ benchmark on the UFS disk (which is simply another volume on the same IBM storage unit):

$ iostat -Dnx c2t602d0 c2t603d0 rmt/1 5 1000

                    extended device statistics
    r/s    w/s   kr/s     kw/s wait actv wsvc_t asvc_t  %w  %b device
   81.0    0.0 10204.4     0.0  0.0  4.1    0.0   51.2   0  96 c2t602d0 (ZFS)
    0.2   75.6     1.6 58547.5  0.0 12.8    0.0  168.8   0  99 c2t603d0 (UFS)
    0.0   77.0     0.0  9859.1  0.0  0.1    0.0    1.2   0   9 rmt/1    (Tape backup)
                    extended device statistics
    r/s    w/s   kr/s     kw/s wait actv wsvc_t asvc_t  %w  %b device
   90.4    0.0 11465.3     0.0  0.0  4.3    0.0   48.0   0  97 c2t602d0 (ZFS)
    0.0   83.4     0.0 57903.1  0.0 13.7    0.0  164.9   0 100 c2t603d0 (UFS)
    0.0   86.0     0.0 11004.7  0.0  0.1    0.0    1.2   0  11 rmt/1    (Tape backup)
                    extended device statistics
    r/s    w/s   kr/s     kw/s wait actv wsvc_t asvc_t  %w  %b device
   79.8    0.0 10048.3     0.0  0.0  3.2    0.0   40.7   0  97 c2t602d0 (ZFS)
    0.0   86.0     0.0 60239.9  0.0 13.3    0.0  155.0   0  98 c2t603d0 (UFS)
    0.0   74.4     0.0  9527.7  0.0  0.1    0.0    1.2   0   9 rmt/1    (Tape backup)
                    extended device statistics
    r/s    w/s   kr/s     kw/s wait actv wsvc_t asvc_t  %w  %b device
   91.2    0.0 11587.8     0.0  0.0  5.2    0.0   56.7   0  97 c2t602d0 (ZFS)
    0.0   71.0     0.0 55932.3  0.0 13.3    0.0  186.5   0 100 c2t603d0 (UFS)
    0.0   89.2     0.0 11423.4  0.0  0.1    0.0    1.2   0  11 rmt/1    (Tape backup)

Bonnie++ is writing to the UFS disk on the storage unit at around 55 - 60 MB/s. The weird part is that the read speed on the ZFS volume has now jumped to ~10 MB/s. Still not great for Gigabit ethernet, but much better. It's not an anomaly, either. It sustains speeds above 10 MB/s as long as the heavy writes from Bonnie++ are taking place. If I kill Bonnie++, the read speeds on the ZFS volume drop back down to around 1-2 MB/s.

Any ideas on how I can explain this? If anything, I would have thought that the opposite would occur. We have both these volumes on the same storage unit connected via iSCSI to our file server. If I start heavily writing to one of them, I would have expected the performance of reads on the other to decrease rather than soaring to read 5 times as fast as it was.

Thanks for your insight.

share|improve this question
    
I may be way off here, but does the switch handling the iSCSI traffic have Flow Control enabled by any chance? –  fukawi2 Oct 23 '13 at 22:28
add comment

1 Answer

I'd start by checking the server for compliance with the storage's configuration requirements. Check queue depth and driver levels, as well as switch configuration (jumbo frames, flow control, etc). Something is definitely wrong, but it doesn't sound like it's the storage. If you were seeing a performance bottleneck from storage hardware, it'd get worse when you increase the load.

A lot of the legwork for host configuration on storage is done for you if you install a host-kit. There should be an IBM solaris host kit for this storage- start by installing that and test again.

share|improve this answer
    
I had a look on IBM's site at the V3700 downloads, and I don't see a host kit available. Perhaps this is because it is a lower end storage unit? Nevertheless, I will look into your other suggestions. Thank you. –  Jeff Shantz Oct 23 '13 at 14:38
    
I found this link that has a bunch of configuration settings for solaris on iSCSI: pic.dhe.ibm.com/infocenter/storwize/v3700_ic/… –  Basil Oct 23 '13 at 20:21
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.