Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm trying to build my own network attached storage based on FreeBSD+ZFS+standard components, but there are strange performance issues.

The hardware specs are:

  • AMD Athlon II X2 240e processor
  • ASUS M4A78LT-M LE mainboard
  • 2GiB Kingston ECC DDR3 (two sticks)
  • Intel Pro/1000 CT PCIe network adapter
  • 5x Western Digital Caviar Green 1.5TB

I created a RAID-Z2 zpool from all disks. I installed FreeBSD 8.1 on that zpool following the tutorial. The SATA controllers are running in AHCI mode.

Output of zpool status:

pool: zroot
state: ONLINE
scrub: none requested
config:
    NAME                                            STATE     READ WRITE CKSUM
    zroot                                           ONLINE       0     0     0
      raidz2                                        ONLINE       0     0     0
        gptid/7ef815fc-eab6-11df-8ea4-001b2163266d  ONLINE       0     0     0
        gptid/80344432-eab6-11df-8ea4-001b2163266d  ONLINE       0     0     0
        gptid/81741ad9-eab6-11df-8ea4-001b2163266d  ONLINE       0     0     0
        gptid/824af5cb-eab6-11df-8ea4-001b2163266d  ONLINE       0     0     0
        gptid/82f98a65-eab6-11df-8ea4-001b2163266d  ONLINE       0     0     0

The problem is that write performance on the pool is very very bad (<10 MB/s) and every application that is accessing the disk is unresponsive every few seconds when writing. It seems like writing is fine until the ZFS ark cache is full and then ZFS stalls the entire system I/O till it's finished writing that data.

Also I'm getting kmem_malloc to small kernel panics. I've already tried to put

vm.kmem_size="1500M"
vm.kmem_size_max="1500M"

into /boot/loader.conf, but it doesn't help.

Does anyone know what's going on here? Am I really not having enough memory for ZFS to handle this RAID-Z2?

share|improve this question
2  
Try running systat and enter :vm<enter> then watch that screen while doing a large write, see where the bottleneck is. ZFS's more advanced features aren't exactly optimized yet and you might be hitting a bug. –  Chris S Nov 11 '10 at 2:22

3 Answers 3

up vote 9 down vote accepted

I suspect you are probably running into the issue that I'm going to call the "512 byte sectors are not 4K sectors" issue. Anyway, google up on gnop, 4K sector, WD Green and I suspect you'll find the fix. I first learned of it on this site, and it was very informative about that issue, as well as a variety of other tunings for FreeBSD and ZFS. Good luck!

Edit: to quote from the linked site:

... Finally, I came across references to problems with Western Digital’s 1.5 TB (WE15EADS) Green drives that I am using.

The drives have a 4KB physical sector but report 512 Bytes to the BIOS. So performance drops off on really big writes because zfs on FreeBSD sends 4KB of data to the drive as 8 separate writes of 512 bytes, which requires the firmware in the drive to increase its work load by an estimated factor of 60 (1st 512 Bytes - write 4KB, 2nd 512 Bytes, read 4K, write 4K, ..., 8th 512 Bytes, read 4K, write 4K -- so 4KB of writes become 4KB write + (4KB read + 4KB write)X(4KB/512Bytes - 1) = 60. The drives built in 32 MB cache helps until it fills and the zfs arc kicks in and then the arc begins to fill. So all in all, no big deal right?

Actually it is a very big deal if you are writing files to zfs that are larger than your arc plus the size of the buffer on the drive. ...

share|improve this answer
    
I heard of that but I thought the greens don't have 4K sectors. Thank you for that hint. I will try to repartition the drives accordingly. –  Axel Gneiting Nov 11 '10 at 2:57
    
Right. The greens apparently have 512 byte sectors. But (some of them, at least) also apparently report 4K sectors erroneously. –  Jed Daniels Nov 11 '10 at 3:01
    
No I think you are right. WD15EARS do have 4096k physical sectors. I will try a reinstall tomorrow. I will accept your answer if the problems are gone :) –  Axel Gneiting Nov 11 '10 at 3:08
    
Your answer is correct, but the issue is more problematic than thought. The drives lie about their physical sector size and ZFS fragments write down to 512 bytes because of that even if the partitions are aligned. Thanks anyway. –  Axel Gneiting Nov 11 '10 at 15:13
    
Don't know about FreeBSD, but I've been using 1,5TB WD Green EARS 4k sector drives with ZFS on OpenSolaris for about 9 months without problems. 6 drives in raidz2 has been enough to saturate gigabyte network. –  Illotus Nov 20 '10 at 18:47

In addition to the 4K/512b issue, WD green disks are not the ideal choice for your use as they are parking their heads after 8 seconds of inactivity and it takes a couple of second for them to recover to a functional state. This is the write delay you are observing.

http://opensolaris.org/jive/thread.jspa?messageID=480641&tstart=0

share|improve this answer
    
The thing is, the idle timer "issue" is still basically imaginary, because there seems to be no data to indicate that it would cause the drives to fail prematurely. –  Illotus Nov 20 '10 at 18:49
    
The question is not about drive failing but about performance issues. How can you deny the obvious ? The idle timer issue is not imaginary but very real and I 'm definitely experiencing it too. Note also that it is possible to modify the idle timeout on some but not all of these drives depending on their firmware. –  jlliagre Nov 20 '10 at 21:46

Try disabling the timer using WDidle tool.

share|improve this answer
1  
The idle timer wasn't problem. I returned the drives in the meantime anyway. –  Axel Gneiting Feb 27 '11 at 14:41

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.