Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have about 8 TB worth of 'sample' data with the following characteristics:

each sample: 5-15 GB in one folder containing ~20k files and ~10k subfolders (2000 top-level, 5 sub-level containing a ~.5-2MB data files and small settings files).

I am setting up a Dell T710 server running Windows server 2008 R2 with 19 TB effective space (RAID5) in order to consolidate the data. I have previously seen significant slow-downs when opening/browsing/copying on a computer with about 1.5 TB of this type of data on a dedicated internal drive (NTFS).

Each sample will be copied to this server for storage, but analysis will occur elsewhere (data copied off of server). So no daily change in existing data, just new data.

What is the best drive configuration to handle this type of data? Drive is GPT and currently has EFI, MSR, 70 GB system partition, and empty 19 TB data partition.

  • one large 19 TB volume
  • several smaller volumes (less fragmentation?)

would it be advisable to create a per-sample zip archive and store this instead? I would hesitate about this because users understand folders intuitively, and corruption has worse effects on archives -- we could afford a few corrupted sub-folders (sample 'pixels', more or less) in the extreme case, but corrupting an entire sample archive would be bad.

share|improve this question
5  
With that much data, I'd personally reconsider using RAID 5... –  Bart Silverstrim Jan 5 '12 at 20:17
    
Is this in-house software? Is there a way to put the data into a database rather than relying on the filesystem? –  Bart Silverstrim Jan 5 '12 at 20:20
    
We are using RAID5 for the 19TB volume. The sample folders are created by a commercial instrument with a fixed directory structure expected by the analysis software. –  Isaiah Jan 5 '12 at 20:25
    
I misunderstood: why would you reconsider RAID5 - performance or something else? The data will be mirrored to a RAID6 GlusterFS off-site, run by a central group, so I'm not considering RAID5 as the backup! –  Isaiah Jan 5 '12 at 20:33
1  
Isaiah: Sure, RAID6 is better. The odds of an unrecoverable read error (URE) are 1bit in 11TB, so when you lose a drive and go to replace it, the rebuild will almost certainly cause another failure, this time probably irrevocably. RAID 6 adds another parity bit so that you can recover from a URE during rebuild. –  Matt Simmons Jan 5 '12 at 21:15

2 Answers 2

up vote 3 down vote accepted

19TB in a single RAID-5 volume is awfully big. You don't mention how many disks you have in that volume but, being in a Dell T710, I think it's highly likely that you've got more than 1TB per disk. I get antsy with RAID-5 members being that large. If that's a single RAID-5 span that's even more scary to me. (I don't like a span larger than 5 or 6 disks, especially with disks that large.)

Your choice of RAID-5 aside, in my experience that's a fairly large number of files to be asking NTFS to handle. Anything that you can do to reduce the number of files being stored is going to help performance. Compressing the "sample" as you describe would radically decrease the number of files you're asking NTFS to handle. Depending on how well your data compresses you could see significant performance increases in transferring the files over the network, as well.

In my opinion you shouldn't be worrying about "corruption" of the data. If you don't have enough faith that your backup system and primary storage will work w/o corrupting the files then you should concentrate on beefing those "foundation" components up. RAID-10 or RAID-50 would be a good first step toward beefing up the primary storage. Since you don't talk about how you're doing backup I can't really speak to that.

Edit:

I'm wary of RAID-5 for availability. The seminal article about this is Why RAID 5 Stops Working in 2009. The gist is that bit-error rates on larger disks make rebuilds of large RAID-5 volumes statistically improbable.

If you have another copy of the data off-site then it's probably less of a concern. You should think about what the ramification would be of a complete loss of the RAID-5 volume. Will you be able to spin up a new volume and continue working while you re-copy data from the off-site copy? Will you need to wait for some quantity of data to copy before work can begin again? If there is idle time what will the cost be?

share|improve this answer
    
Thanks, the T710 has 12x2TB disks. I would not be opposed to splitting it into two RAID5 volumes, I'll rebuild and do that. I would prefer to use a modern file system, but the system has to be supportable long-term by an all-windows in-house IT group. Backup will be to an off-site RAID6/GlusterFS system run by the central IT department (big conglomerate center..) –  Isaiah Jan 5 '12 at 20:51

You lost disk space if you have many small files. The reason is the block size of you file system. My first suggestion is to use a Linux system for long term support. And my second suggestion is to save the files without zipping on the file system because understanding the system is much more important the losing some bytes. I had the same problem with genomic data (shotgun analyzer). My third suggestion is to use a RAID10 or RAID50.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.