Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

The web application I'm working on, will be used to upload/download large number/amounts of smaller size files - I'm looking at close to 1B files with total size of > 10Pb. I'm currently struggling with deciding the scalable architecture that would support such amounts. And here's my question - is there a way of building some sort of storage that would be seen by a windows server as one huge (10Pb and up) network storage drive, so I can write all the files to subfolders of that virtual drive? And how would it perform?

Right now I'm trying to understand if that's even possible, or if I have to implement software level sharding - writing files to different drives based on some key.

I'm a developer, not a sys admin, so I apologize if it's a naive question, and thanks in advance for patience in explaining me possibly trivial things.

Andrey

share|improve this question
    
I would research if SQL Server filestreams + NTFS can handle this. –  alex Sep 2 '10 at 20:11
    
You mean, store files "in" SQL Server? –  Andrey Sep 2 '10 at 20:40
    
the FileStream data type is a special kind of field. It does not store the data in the database (this would be bad) but it's kinda like holding the file on the disk and the metadata in the SQL server, but integrated. –  Mark Henderson Sep 3 '10 at 5:36
add comment

4 Answers

up vote 2 down vote accepted

as a 'normal but huge' fileserver:

with a file-like application level library:

  • amazon S3
  • rackspace cloudfiles
  • mogilefs

generic key-value:

share|improve this answer
add comment

Check out how Backblaze is storing its data. Very good read and they have a blog about the new 3TB drives. This probably will not answer the question about file system. I am not sure how Backblaze does there file structure. But good information nevertheless.

share|improve this answer
    
They have a nice design, but the key concern is performance - a single server with just one processor and 4Gb of memory can't be speedy serving 45 drives. But it's an interesting read, so +1 –  Andrey Sep 3 '10 at 0:09
    
Ah - it can. Sorry. One processor is 6 cores these days. 4gb memory for a simple FS + Linux is enough. discs is 25% of the capacity of a RAID card like from Adaptec's 5xxx series. You should be more concerned with bandwidth (go go infiniband, 10gb here) which is going to be the botttlenech. –  TomTom Sep 3 '10 at 3:31
add comment

Before you continue looking, you need to decide a bit more exactly what kind of semantics you need. For instance, you say they're files - do you need POSIX file semantics (mostly concerned with consistency and locking) on them on the storage? or is 'eventual consistency' of various distributed datastores enough? What are your I/O requirements: how much concurrent access? What are your redundancy requirements? Also: what kind of hardware are you going to use? 10Pb arrays don't grow on trees and just managing them is a full time job - that much hardware means failure is a normal event, so constant repair and replacement is needed.

From what you've said "web application... storing files..." I think an OpenStack or S3 kind of solution should do you. Since you're mostly a developer, I'd suggest you probably want to actually use amazon or Rackspace or whoever as your provider unless you really want to get into the hardware management biz.

share|improve this answer
add comment

These days you might consider HDFS and the general Hadoop ecosystem.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.