3

I'm starting to look at ZFS, doing tests with FreeNAS on an old HP MicroServer (N40L). Didn't get extra RAM yet so it's only running with 2 GB, so pretty much no cache.

I created a pool on a single 1 TB SATA disk, created a zvol, and exported it with iSCSI. I just ran a few tests with CrystalDiskMark on Windows 10 and I'm getting weird results when changing the pool bock size.

ZFS 16K and NTFS 4K

-----------------------------------------------------------------------
CrystalDiskMark 5.1.0 x64 (C) 2007-2015 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

   Sequential Read (Q= 32,T= 1) :   105.938 MB/s
  Sequential Write (Q= 32,T= 1) :    98.709 MB/s
  Random Read 4KiB (Q= 32,T= 1) :     8.330 MB/s [  2033.7 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :     6.243 MB/s [  1524.2 IOPS]
         Sequential Read (T= 1) :    93.946 MB/s
        Sequential Write (T= 1) :    72.780 MB/s
   Random Read 4KiB (Q= 1,T= 1) :     8.948 MB/s [  2184.6 IOPS]
  Random Write 4KiB (Q= 1,T= 1) :     6.898 MB/s [  1684.1 IOPS]

  Test : 1024 MiB [X: 0.0% (0.2/849.9 GiB)] (x1)  [Interval=5 sec]
  Date : 2016/01/12 18:54:05
    OS : Windows 10 Professional [10.0 Build 10586] (x64)

Everything is as I'd expect here compared to the same disk used as a SATA disk in the client PC.
- A little sequential performance is a lost because of the iSCSI overhead.
- Random reads are better, some caching there.
- Random writes are better because of the COW makes the writes not actually random.

ZFS 4K and NTFS 4K

-----------------------------------------------------------------------
CrystalDiskMark 5.1.0 x64 (C) 2007-2015 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

   Sequential Read (Q= 32,T= 1) :    24.405 MB/s
  Sequential Write (Q= 32,T= 1) :    81.038 MB/s
  Random Read 4KiB (Q= 32,T= 1) :     3.048 MB/s [   744.1 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :    55.971 MB/s [ 13664.8 IOPS]
         Sequential Read (T= 1) :    42.362 MB/s
        Sequential Write (T= 1) :    45.292 MB/s
   Random Read 4KiB (Q= 1,T= 1) :     0.961 MB/s [   234.6 IOPS]
  Random Write 4KiB (Q= 1,T= 1) :     9.576 MB/s [  2337.9 IOPS]

  Test : 1024 MiB [X: 0.0% (0.2/799.9 GiB)] (x1)  [Interval=5 sec]
  Date : 2016/01/12 19:10:22
    OS : Windows 10 Professional [10.0 Build 10586] (x64)

A few notes:
- The network usage doesn't go over 600 Mb (task manager on the Windows 10), so less than in the first test were it was fully used.
- The CPU usage is about double that of the first test, but still at least 20% idle (top on FreeNAS server)

Here I didn't expect that!
- Sequential performance is lower. Maybe more smaller packets transmitted, so network not fully used?
- Lower random reads. It's really only less caching, still over the disk performance. But why less cache?
- Random writes are much better with Q=32. I'd say ZFS is writing 16K (12K empty) when writing 4K in the previous test. Is that really the case?

1

Replying to your notes:

  1. network utilization should not be the culprit, as standard Ethernet frames are 1500 bytes large at most (excluding jumbo frames). The reduced read performance seems related to something wrong with FS and device alignment. Can you double check this? Anyway, it can be also due to more fragmentation
  2. smaller blocks are less cache friendly. However, this can be a side-effect of how Windows caches iSCSI disks
  3. with higher queue depth, both the Windows and FreeNAS IO schedulers have a chance to reorder and coalesce multiple writes. But this optimization seems to work only with small (4KB) IO requests. Maybe larger (>= 16KB) IO requests are treated in a different manner.
0

If you're using zvols in ZFS, the best practice for non-Solaris distributions is to use a large block size. Under Linux, we create zvols with 128K block size.

Can you try doing the same?

  • I remember reading about this, a bug report about Linux performance with small blocks on github maybe, but that could be Linux specific. Recommended block size on Solaris was 8K I think. – Sacha K Jan 13 '16 at 3:16
  • FreeNAS defaults on 16K. I'll try to find the FreeBSD recommendation, and test with other block sizes. – Sacha K Jan 13 '16 at 3:20
0

I knew I didn't have enough RAM so no data would be cached, but it looks like 2 GB wasn't even enough to store the metadata in RAM with a 4K block size - more blocks so more metadata. If the metadata is not in RAM when writing the metadata has to be read from disk to find free space and it kills performance.

I got 8 GB ECC RAM and did more tests, still single disk. The results now make sense with the random write performance being better when ZFS and NTFS block sizes are matching and the sequential write performance being the same.

Block size could have an effect on sequential speeds but it's limited by the network here. I'll do more tests if I get extra network cards to do multipath.

ZFS 4K and NTFS 4K

   Sequential Read (Q= 32,T= 1) :   118.472 MB/s
  Sequential Write (Q= 32,T= 1) :   101.721 MB/s
  Random Read 4KiB (Q= 32,T= 1) :   115.943 MB/s [ 28306.4 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :    88.872 MB/s [ 21697.3 IOPS]
         Sequential Read (T= 1) :   102.555 MB/s
        Sequential Write (T= 1) :    80.744 MB/s
   Random Read 4KiB (Q= 1,T= 1) :     9.668 MB/s [  2360.4 IOPS]
  Random Write 4KiB (Q= 1,T= 1) :     9.410 MB/s [  2297.4 IOPS]

ZFS 8K and NTFS 4K

   Sequential Read (Q= 32,T= 1) :   118.154 MB/s
  Sequential Write (Q= 32,T= 1) :   102.506 MB/s
  Random Read 4KiB (Q= 32,T= 1) :   113.806 MB/s [ 27784.7 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :    73.798 MB/s [ 18017.1 IOPS]
         Sequential Read (T= 1) :   106.324 MB/s
        Sequential Write (T= 1) :    91.004 MB/s
   Random Read 4KiB (Q= 1,T= 1) :    10.161 MB/s [  2480.7 IOPS]
  Random Write 4KiB (Q= 1,T= 1) :     9.628 MB/s [  2350.6 IOPS]

ZFS 8K and NTFS 8K

   Sequential Read (Q= 32,T= 1) :   118.051 MB/s
  Sequential Write (Q= 32,T= 1) :    99.229 MB/s
  Random Read 4KiB (Q= 32,T= 1) :   113.327 MB/s [ 27667.7 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :    75.365 MB/s [ 18399.7 IOPS]
         Sequential Read (T= 1) :   106.110 MB/s
        Sequential Write (T= 1) :    87.242 MB/s
   Random Read 4KiB (Q= 1,T= 1) :     9.880 MB/s [  2412.1 IOPS]
  Random Write 4KiB (Q= 1,T= 1) :     9.878 MB/s [  2411.6 IOPS]

ZFS 16K and NTFS 4K

   Sequential Read (Q= 32,T= 1) :   118.040 MB/s
  Sequential Write (Q= 32,T= 1) :   100.274 MB/s
  Random Read 4KiB (Q= 32,T= 1) :   114.780 MB/s [ 28022.5 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :     6.014 MB/s [  1468.3 IOPS]
         Sequential Read (T= 1) :   108.443 MB/s
        Sequential Write (T= 1) :    90.182 MB/s
   Random Read 4KiB (Q= 1,T= 1) :    11.281 MB/s [  2754.2 IOPS]
  Random Write 4KiB (Q= 1,T= 1) :     8.229 MB/s [  2009.0 IOPS]

ZFS 16K and NTFS 8K

   Sequential Read (Q= 32,T= 1) :   118.046 MB/s
  Sequential Write (Q= 32,T= 1) :    99.386 MB/s
  Random Read 4KiB (Q= 32,T= 1) :   115.745 MB/s [ 28258.1 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :    51.101 MB/s [ 12475.8 IOPS]
         Sequential Read (T= 1) :   105.913 MB/s
        Sequential Write (T= 1) :    91.225 MB/s
   Random Read 4KiB (Q= 1,T= 1) :    10.244 MB/s [  2501.0 IOPS]
  Random Write 4KiB (Q= 1,T= 1) :     9.373 MB/s [  2288.3 IOPS]

ZFS 16K and NTFS 16K

   Sequential Read (Q= 32,T= 1) :   118.046 MB/s
  Sequential Write (Q= 32,T= 1) :   101.422 MB/s
  Random Read 4KiB (Q= 32,T= 1) :   113.856 MB/s [ 27796.9 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :    40.821 MB/s [  9966.1 IOPS]
         Sequential Read (T= 1) :   107.572 MB/s
        Sequential Write (T= 1) :    89.345 MB/s
   Random Read 4KiB (Q= 1,T= 1) :    11.112 MB/s [  2712.9 IOPS]
  Random Write 4KiB (Q= 1,T= 1) :     9.691 MB/s [  2366.0 IOPS]

For info the same disk locally connected - NTFS 4K

   Sequential Read (Q= 32,T= 1) :   130.080 MB/s
  Sequential Write (Q= 32,T= 1) :   121.335 MB/s
  Random Read 4KiB (Q= 32,T= 1) :     1.005 MB/s [   245.4 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :     1.290 MB/s [   314.9 IOPS]
         Sequential Read (T= 1) :   129.997 MB/s
        Sequential Write (T= 1) :   128.324 MB/s
   Random Read 4KiB (Q= 1,T= 1) :     0.406 MB/s [    99.1 IOPS]
  Random Write 4KiB (Q= 1,T= 1) :     1.307 MB/s [   319.1 IOPS]

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.