2

I bring up, yet again, the ever-present question of how to best optimize disk structures. In my organization, we have a 14TB Linux software RAID array dedicated to storing backups made using Symantec Backup Exec. These are large files, 10GB - 100GB each, with some supporting metadata files a couple KB in size. Long story short, we have to recreate the array, and I would like to know the optimal array chunk size for this use case.

Details of our setup:

A Netgear ReadyNAS Pro, running a clean & updated install of CentOS 6.4.

6 x 3TB consumer (SATA II, 7200 RPM) hard drives from assorted vendors (identical in size).

Each drive has 3 identical partitions which form 3 software RAID devices:

  • /dev/md0: 6 x 32GB for / in a RAID6
  • /dev/md1: 6 x 4GB swap in a RAID10
  • /dev/md2: 6 x 2.7TB storage in a RAID5 for ~14TB total useful storage

Additionally, there is an integrated 128MB flash device set up as /boot

/dev/md2 is the array I'm focused on. It is made available as drive "R:" to a Windows Server 2008 R2 box running Symantec Backup Exec via multipath iSCSI over dual gigabit NICs on both machines (also running 9k jumbo frames).

On the Server 2008 box, R: is formatted as NTFS with a 64k cluster size, and is dedicated to storing backup files. The average file is generally between 40MB and 5GB, depending on the current proportion of full vs incrementals/differentials present. Disk usage is about a 50/50 split between read and write, as we mirror backups from this drive to tape as well.

Overall, given the hardware, I've think optimized this setup fairly well, however I'm not a storage expert, and the implications of the RAID chunk size are slightly beyond me. I know the default mdadm chunk size is 512KB. Is this optimal for my scenario? Should I adjust this to match NTFS cluster size? Or is there some magic formula I've missed?

Thanks for any help you can provide.

Edit: Benchmark results below. Not all combinations were tested.


##########   4K Chunk##########
-----------------------------------------------------------------------
CrystalDiskMark 3.0.2 x64 (C) 2007-2013 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :   111.551 MB/s
          Sequential Write :    96.759 MB/s
         Random Read 512KB :   107.033 MB/s
        Random Write 512KB :    56.770 MB/s
    Random Read 4KB (QD=1) :     9.500 MB/s [  2319.2 IOPS]
   Random Write 4KB (QD=1) :     5.042 MB/s [  1231.0 IOPS]
   Random Read 4KB (QD=32) :   101.717 MB/s [ 24833.3 IOPS]
  Random Write 4KB (QD=32) :     8.237 MB/s [  2010.9 IOPS]

  Test : 1000 MB [R: 0.0% (0.1/13791.8 GB)] (x5)
  Date : 2013/07/12 13:10:31
    OS : Windows Server 2008 R2 Enterprise Edition (Full installation) SP1 

[6.1 Build 7601] (x64)

##########  32K Chunk##########
-----------------------------------------------------------------------
CrystalDiskMark 3.0.2 x64 (C) 2007-2013 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :    91.276 MB/s
          Sequential Write :    11.119 MB/s
         Random Read 512KB :     0.000 MB/s
        Random Write 512KB :     0.000 MB/s
    Random Read 4KB (QD=1) :     0.000 MB/s [     0.0 IOPS]
   Random Write 4KB (QD=1) :     0.000 MB/s [     0.0 IOPS]
   Random Read 4KB (QD=32) :     0.000 MB/s [     0.0 IOPS]
  Random Write 4KB (QD=32) :     0.000 MB/s [     0.0 IOPS]

  Test : 1000 MB [R: 0.0% (0.1/13791.8 GB)] (x5)
  Date : 2013/07/12 14:37:05
    OS : Windows Server 2008 R2 Enterprise Edition (Full installation) SP1 

[6.1 Build 7601] (x64)

##########  64K Chunk##########
-----------------------------------------------------------------------
CrystalDiskMark 3.0.2 x64 (C) 2007-2013 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :   111.968 MB/s
          Sequential Write :   103.318 MB/s
         Random Read 512KB :   105.047 MB/s
        Random Write 512KB :    48.321 MB/s
    Random Read 4KB (QD=1) :    10.373 MB/s [  2532.5 IOPS]
   Random Write 4KB (QD=1) :     5.180 MB/s [  1264.5 IOPS]
   Random Read 4KB (QD=32) :    95.106 MB/s [ 23219.3 IOPS]
  Random Write 4KB (QD=32) :     9.108 MB/s [  2223.6 IOPS]

  Test : 1000 MB [R: 0.0% (0.1/13791.8 GB)] (x5)
  Date : 2013/07/12 12:47:37
    OS : Windows Server 2008 R2 Enterprise Edition (Full installation) SP1 

[6.1 Build 7601] (x64)

########## 128K Chunk##########
-----------------------------------------------------------------------
CrystalDiskMark 3.0.2 x64 (C) 2007-2013 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :   111.908 MB/s
          Sequential Write :    94.305 MB/s
         Random Read 512KB :   104.772 MB/s
        Random Write 512KB :    43.821 MB/s
    Random Read 4KB (QD=1) :     9.247 MB/s [  2257.6 IOPS]
   Random Write 4KB (QD=1) :     4.929 MB/s [  1203.3 IOPS]
   Random Read 4KB (QD=32) :   101.764 MB/s [ 24844.8 IOPS]
  Random Write 4KB (QD=32) :     7.949 MB/s [  1940.6 IOPS]

  Test : 1000 MB [R: 0.0% (0.1/13791.8 GB)] (x5)
  Date : 2013/07/12 13:52:01
    OS : Windows Server 2008 R2 Enterprise Edition (Full installation) SP1 

[6.1 Build 7601] (x64)

########## 512K Chunk##########
-----------------------------------------------------------------------
CrystalDiskMark 3.0.2 x64 (C) 2007-2013 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :   110.237 MB/s
          Sequential Write :    93.149 MB/s
         Random Read 512KB :   104.892 MB/s
        Random Write 512KB :    41.407 MB/s
    Random Read 4KB (QD=1) :     6.760 MB/s [  1650.3 IOPS]
   Random Write 4KB (QD=1) :     3.539 MB/s [   864.0 IOPS]
   Random Read 4KB (QD=32) :   101.139 MB/s [ 24692.3 IOPS]
  Random Write 4KB (QD=32) :     7.166 MB/s [  1749.6 IOPS]

  Test : 1000 MB [R: 0.0% (0.1/13791.8 GB)] (x5)
  Date : 2013/07/12 12:22:58
    OS : Windows Server 2008 R2 Enterprise Edition (Full installation) SP1 

[6.1 Build 7601] (x64)

##########1024K Chunk##########
-----------------------------------------------------------------------
CrystalDiskMark 3.0.2 x64 (C) 2007-2013 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :   112.327 MB/s
          Sequential Write :    92.353 MB/s
         Random Read 512KB :   107.015 MB/s
        Random Write 512KB :    39.793 MB/s
    Random Read 4KB (QD=1) :     9.536 MB/s [  2328.0 IOPS]
   Random Write 4KB (QD=1) :     3.671 MB/s [   896.3 IOPS]
   Random Read 4KB (QD=32) :   101.990 MB/s [ 24900.0 IOPS]
  Random Write 4KB (QD=32) :     0.000 MB/s [     0.0 IOPS]

  Test : 1000 MB [R: 0.0% (0.1/13791.8 GB)] (x5)
  Date : 2013/07/12 14:17:08
    OS : Windows Server 2008 R2 Enterprise Edition (Full installation) SP1 

[6.1 Build 7601] (x64)
5
  • With drives those size (and consumer especially) you need to be using RAID6 or RAID10, not RAID5.
    – longneck
    Jul 10 '13 at 3:45
  • If this was storing production data, I would definitely be using RAID6 or RAID10, but keep in mind this is only storing backup data, much of which gets written to tape eventually anyway. If the array fails and all the data is lost, I just shrug my shoulders and start a new set of full backups running. Jul 17 '13 at 14:43
  • backup data still deserves RAID6. nothing like a bad backup to ruin your day all over again when a critical server dies.
    – longneck
    Jul 17 '13 at 15:06
  • I don't disagree - believe me, if I had a larger budget and could afford more space and better drives, I would have a better solution. I inherited this setup however, and at the moment, can't afford the loss of space that RAID6 would cause. That aside, all of the full backups get written to tape anyway. This holds all the diffs/incs, and is a staging area for the fulls, so we can take several backups simultaneously, which we can't do directly to tape. The most we would lose is a day or two if this crashed at the same time as another server. Jul 17 '13 at 15:12
  • The myth of "you shouldn't use RAID-5" needs to die, as does the notion that "consumer drives" aren't as reliable as "enterprise drives." All primary data should have regular backups, even if on a RAID array. If that data is on a RAID-5 and the person using it has read up on the slow/long rebuild time after a failure, they obviously had budgetary concerns which led them to that choice. I just built a 3x3TB 7200RPM RAID-5 array; the resync took 6.2 hours. In "we lose $5K an hour" servers that's horrible; in "I have a $600 file server budget" terms it's totally acceptable. Jan 5 '17 at 0:22
4

At a minimum, you want the chunk size to be a multiple or divisor of the filesystem block size. You've got that.

Everything else is likely to be implementation dependent. Since you're starting from scratch, you should roll your own benchmarks. Instead of creating a 14 TB RAID set, test with just 500 GB from each drive in various chunk sizes. The smaller volume sizes will reduce the amount of time needed to create the volume.

When you find the optimal number for your setup, then create your 14 TB RAID set. Test again to make sure you haven't had a performance degradation.

1
  • D'oh! I really should have thought to do this earlier! I went ahead and did some benchmarks, finding that a 64K chunk size did indeed provide the best test results, primarily in terms of sequential reads/writes. I will post my results below as well for community reference. Thank you! Jul 12 '13 at 18:42
0

It depends on usage scenario. Suppose that you have very little number of small files < 512KB suppose it is 10000 files and the rest is very large files >10MB 5000 files. It is better to have a 1MB cluster size when formatting the drive and if you have 8 disk raid 10 setup it will read 128KB each at the same time.

You will loose at most 508KB * 15000 disk space if all of your small files and tail-trunk of large files are 4KB which translates to ~ 7500MB which is 7.5 GB disk space loss.

You will gain 4 times more speed if compared to the jinxed setup of chunk size to 512KB and cluster size to 64KB.

So as a final recipe :

  1. Select your file system cluster size according to file size distribution
  2. divide it to number of disks used at the same time determine chunk size
  3. If you choose larger chunk&cluster you will gain speed loose miniscule disk space for large files.
  4. For small files do not optimize mechanical setup too much, rather consider adding a nvme or ssd cache for files smaller than 1MB

Additionaly:

  • Consider bandwidth limitations of your network infrastructure
  • Check for bottlenecks before raid setup(dd dev/zero to disks watch iostat)
  • Consider hba or raid cards - sata cards (chips) and pci-e slots total bandwidth and per lane bandwidth
  • Check for damaged or loose cables (dmesg)
  • Use ext4 partitions for mdadm software raid and leave a little space for disk compatibility do not use them raw
  • btrfs and alike (zfs,nilfs, cloud - dedupe etc ) file systems are whole another issue

! VERY IMPORTANT:

  • Do not use raid 5 (nearly impossible to reconstruct and buggy)

  • Do not use btrfs with its own raid 5 or raid 6 setup (known bugs)

  • Do not use raid 0 for mission critical data, it may go for 10 years however its a gamble

  • Use ups with software raid setups

  • Backup rombios software of LSI chipset cards and buy a spare card make sure they have same rombios version and make sure if you replace one other can reconstruct raid

These are some of from my 2 years of file server setup experiences

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.