Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

In a large online store (1PB), if you have lots of small files(32Kb<), images (256KB-5MB) and videos (>100MB) is it possible to use multiple disks with different stripe size?

Lets say 10 disk with 16 KB stripe size, 20 disk with 256KB, 30 disk with 1MB. When you are uploading an video, it uploads to disks with 1MB stripes, or when you are uploading a small files ip uploads to disks with 16KB stripes.

small stripe size causes too many chucks of a large files, and large stripes are ineffective if you have small files <32kb, can it increase read/write performance if you use different stripe sizes.

share|improve this question
1  
Maybe this could help you a bit [facebook haystack for image storage]: facebook.com/note.php?note_id=76191543919 –  Martino Dino Jan 31 '13 at 11:08

2 Answers 2

up vote 1 down vote accepted

While the answer to your question is plain and simple:

Yes you can create different Virtual disks in your controller and set the different properties to each virtual disk like [stripe size, raid level and so on...] as long as the controller supports it.

I believe 1petabyte is a typo, otherwise your question is pointless as you should not have such amount of data in a single storage enclosure. You will need to look at different solutions, most probably distributed object storage to differentiate the back-end for each category of objects [videos,thumbnails,big pictures] something similar to the comment i posted originally.

share|improve this answer

is it possible to use multiple disks with different stripe size?

Yes, but not in the same array, they work by striping equal sizes across all array members, I imagine something as flexible as say ZFS may allow such fiddling just for the hell of it but I imagine it would be a very bad idea indeed.

Of course you could have multiple arrays, each tuned for different file sizes/requirements - but it'll be your job to ensure the appropriate files get written to the correct array/volume.

Don't forget that most filesystems you'll deal with will automatically have deferred writes, so it'll 'bunch up' a lot of writes, thus at least partially negating any positive impact splitting your files into different volumes would have.

Ultimately I'd suggest just getting a really good array controller with a decent amount of FBWC and trust the filesystem to do its job.

share|improve this answer
    
Didn't EVA allow this? Or was that just different raid levels with the same stripe sizes? –  pauska Jan 31 '13 at 11:50
    
The latter I'm afraid. –  Chopper3 Jan 31 '13 at 13:36

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.