Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I've just built a small NAS/SAN box and am doing some testing with it. Configuration is:

  • 2GB RAM
  • 2x Opteron 265s
  • Iwill DK8N
  • 5x500GB SATA drives (1 spare, rest mirrored) = 1TB usable space
  • Emulex LPC10000DC
  • NexentaCore 3.0.2

The first thing I'm testing is COMSTAR performance over fibre-channel. I created a 700g zvol and exposed it over COMSTAR. Write-caching is enabled. My FC initiator is a Windows 7 box with a Q-Logic QLA2340 card in it.

Everything works great, except for write speeds. For some reason, I'm only seeing ~30MB/s, whereas I can peak at about 90MB/s read speeds, using IOMeter. Doing some hunting around, I ran a zpool iostat command and saw the following pattern:

$ zpool iostat tank 1
tank        88.2G   840G      0  5.47K      0  43.4M
tank        88.2G   840G      0  16.1K      0   128M
tank        88.4G   840G      1  6.69K  8.93K  52.6M
tank        88.4G   840G     52  4.89K   110K  38.3M
tank        88.4G   840G    151      0   280K      0
tank        88.4G   840G    112      0   253K      0
tank        88.4G   840G      1  9.89K  16.0K  78.6M
tank        88.4G   840G      0  14.3K      0   113M
tank        88.6G   839G    150  2.99K   318K  20.2M
tank        88.6G   839G     15      0  45.0K      0
tank        88.6G   839G      3      0  17.8K      0
tank        88.6G   839G      0      0      0      0
tank        88.6G   839G      0      0      0      0
tank        88.6G   839G      0  11.3K      0  89.9M
tank        88.6G   839G      0  13.8K      0   110M
tank        88.8G   839G     90  7.91K   184K  61.0M
tank        88.9G   839G    105     91   220K   140K
tank        88.9G   839G     42      0  84.6K      0
tank        88.9G   839G      0      0      0      0
tank        88.9G   839G      0  6.52K  7.99K  51.8M
tank        88.9G   839G      0  15.6K   1011   124M
tank        88.9G   839G     91  4.30K   193K  31.9M
tank        89.1G   839G      1  9.48K  15.8K  74.7M
tank        89.1G   839G     41      0  99.9K      0
tank        89.1G   839G    166      0   374K      0
tank        89.1G   839G     87      0   162K      0
tank        89.1G   839G      0      0  7.92K      0

It appears to be writing a lot, then stopping.

What's going on here?

share|improve this question
up vote 1 down vote accepted

That's normal for ZFS. What's happening is it's coalescing the writes in memory and then committing them to disk at the same time. This a great wiki about what it's doing and how you can tune it:

share|improve this answer
Interesting. I guess my concern was around the speed seen by the COMSTAR initiator - Bonnie++ and these results indicate that 4 spindles can do ~125M/s quite easily, seeing only 30MB/s at the other end of a 2GB FC link made me question what I was doing wrong. Also, the fact that I see a similar bursting pattern with IOMeter on the client gave cause for concern. If it were caching properly, I'd expect to see a fairly constant IO level on the client. I'll read up on that link and see if I can get total throughput up anyhow. – growse Feb 11 '11 at 9:13

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.