Here's one I haven't seen before. We have an IBM Storwize V3700 storage array, 24 TB, RAID 5 connected to a Solaris 10 file server via an iSCSI link over Gigabit Ethernet (no MPIO at present).
We have two volumes on the RAID array:
c2t602d0 - ZFS
c2t603d0 - UFS
I've shortened the names for brevity.
Now, we were seeing very slow read speeds from the ZFS volume on the unit (~ 1-3 MB/s). I created the UFS volume as a test and ran Bonnie++ on it to do some benchmarking of the unit.
Observe the output of iostat prior to starting up Bonnie++:
$ iostat -Dnx c2t602d0 c2t603d0 rmt/1 5 1000
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
13.4 0.0 1491.0 0.0 0.0 1.5 0.0 112.6 0 99 c2t602d0 (ZFS)
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t603d0 (UFS)
0.0 10.2 0.0 1305.9 0.0 0.0 0.0 1.2 0 1 rmt/1 (Tape backup)
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
15.8 0.0 1807.4 0.0 0.0 1.3 0.0 84.8 0 99 c2t602d0 (ZFS)
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t603d0 (UFS)
0.0 11.4 0.0 1459.2 0.0 0.0 0.0 1.2 0 1 rmt/1 (Tape backup)
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
10.8 0.0 1233.7 0.0 0.0 1.2 0.0 110.1 0 99 c2t602d0 (ZFS)
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t603d0 (UFS)
0.0 7.6 0.0 972.9 0.0 0.0 0.0 1.2 0 1 rmt/1 (Tape backup)
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
18.0 0.0 2060.6 0.0 0.0 1.3 0.0 70.9 0 98 c2t602d0 (ZFS)
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t603d0 (UFS)
0.0 12.6 0.0 1612.7 0.0 0.0 0.0 1.2 0 2 rmt/1 (Tape backup)
rmt/1
is our tape backup drive, which is currently doing a full backup of c2t602d0
-- the ZFS volume. Notice that we're reading from the volume at about 1.2 - 2.0 MB/s.
Next, watch what happens when I start up the Bonnie++ benchmark on the UFS disk (which is simply another volume on the same IBM storage unit):
$ iostat -Dnx c2t602d0 c2t603d0 rmt/1 5 1000
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
81.0 0.0 10204.4 0.0 0.0 4.1 0.0 51.2 0 96 c2t602d0 (ZFS)
0.2 75.6 1.6 58547.5 0.0 12.8 0.0 168.8 0 99 c2t603d0 (UFS)
0.0 77.0 0.0 9859.1 0.0 0.1 0.0 1.2 0 9 rmt/1 (Tape backup)
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
90.4 0.0 11465.3 0.0 0.0 4.3 0.0 48.0 0 97 c2t602d0 (ZFS)
0.0 83.4 0.0 57903.1 0.0 13.7 0.0 164.9 0 100 c2t603d0 (UFS)
0.0 86.0 0.0 11004.7 0.0 0.1 0.0 1.2 0 11 rmt/1 (Tape backup)
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
79.8 0.0 10048.3 0.0 0.0 3.2 0.0 40.7 0 97 c2t602d0 (ZFS)
0.0 86.0 0.0 60239.9 0.0 13.3 0.0 155.0 0 98 c2t603d0 (UFS)
0.0 74.4 0.0 9527.7 0.0 0.1 0.0 1.2 0 9 rmt/1 (Tape backup)
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
91.2 0.0 11587.8 0.0 0.0 5.2 0.0 56.7 0 97 c2t602d0 (ZFS)
0.0 71.0 0.0 55932.3 0.0 13.3 0.0 186.5 0 100 c2t603d0 (UFS)
0.0 89.2 0.0 11423.4 0.0 0.1 0.0 1.2 0 11 rmt/1 (Tape backup)
Bonnie++ is writing to the UFS disk on the storage unit at around 55 - 60 MB/s. The weird part is that the read speed on the ZFS volume has now jumped to ~10 MB/s. Still not great for Gigabit ethernet, but much better. It's not an anomaly, either. It sustains speeds above 10 MB/s as long as the heavy writes from Bonnie++ are taking place. If I kill Bonnie++, the read speeds on the ZFS volume drop back down to around 1-2 MB/s.
Any ideas on how I can explain this? If anything, I would have thought that the opposite would occur. We have both these volumes on the same storage unit connected via iSCSI to our file server. If I start heavily writing to one of them, I would have expected the performance of reads on the other to decrease rather than soaring to read 5 times as fast as it was.
Thanks for your insight.