Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am trying to design an infrastucture for the project I am working on. It would be somehow a file-sharing/downloading project (like rapidshare) and I would need high storage sizes and good scability, and I would add new storage nodes after my project grows up.

I have come up with 3 solutions for my project which are using Luster, GlusterFS, HDFS, RDBD.

For start, i would have 2 servers, one server is for glusterfs client + webserver + db server+ a streaming server, and the other server is gluster storage node. (After sometime, i would be adding more node servers, and client servers (dont know how many new client new servers to add, will see later)

So, i am thinking to work with glusterfs. But i really wonder that if i have to use high performance servers with high sotrage sizes or avarage/slow servers with high storage sizes? Or nas/das/san solutions are better for glusterfs storage nodes? I might buy a nas and install glusterfs onto it. I would be happy to listen to your recommendations for the server properties (for each clients and nodes) . I really dont know if I really need high amount of ram and good cpus to for the nodes. I am sure i need it for client servers.

The files would be streamed as well, so the Automatic file replication is important, thus, my system should work like a cloud, when needed, according to high traffic, the storage nodes should copy the most demanded file to be streamed and would help me to get rid of scability problems and my visitors would able to stream/download those files.

Also, i am open to your experiences/thoughts about any good solution. Luster, hdfs, rbdb are the other options and i would be happy to listen to your thoughts here. I would be very very happy to hear back from anyone commented of any words I have used here.



I know the IOPS is the critical variable that i have to count on in every calculation if my network design, thats why i say random requests. But unfortunately, i dont have any statistics at all. Thats why i am here :)

My project is like that, you enter a download url to my website, my url downloads it, and you start download it from my own server, like a proxy downloader.

So i have a server 100mbit connection and 2TB hdd for now. I am thinking add nas servers. Really dont know if i have to add duplicated storage nodes in nas. And is there a limit that i can connect nas devices ? i mean i can connect max 2 nas servers to my main server?

share|improve this question

Your questions are non trivial and there is not enough info to give a good answer. I can give an answer (clustered filesystem over fibre channel SAN) - but it may well turn out to be more expensive and complex than it needs to be.

So I'll just throw out some comments/thoughts. Really stuff for you to consider. Perhaps after reading this brain dump, you'll be able to restate your app's intended behaviour and maybe then we can give you a better answer.

NAS devices export file systems (e.g. CIFS, NFS), so you don't really connect them to your servers - your servers mount file systems from them. That means reads and writes to them need to go over your a connection. So if you have a 100mbit network connection between your NAS and your server and your read/writes occur at a 1:1 ratio, then the best you'll get is 50mbit reads, because for every byte you read, you also write a byte. If your client and download traffic are on that same network then you can halve it again. Clearly if you want to use a NAS then you going to want multiple NICs in your servers and multipel networks/VLANs in your architecure.

Assuming there are 4 possible data locations in your app.

  • A) Orignal data source, e.g. internet.
  • B) Your server.
  • C) NAS.
  • D) client List item

Then there are 4 possible data vectors

  • AB i.e. the data download from A(the net) to B(your server).
  • BC i.e. writing data from your server to the NAS.
  • CB reading data from the NAS to your server
  • BD writing data from your server to the client

Depending on how your app works and ignoring protocol overhead you may (worst case) then need 4 100mbit networks to transport 100mbit per second to your clients.

So you'll need to consider the read and the write bandwidth to the NAS if you use a NAS. If you use a FC SAN you can reduce your network needs and you get other advatangges.

E.g. Depending on OS and the filesystem you end up using, a SAN will allow you to grow your LUNs dynamically and grow your filessyems live as well as share the LUNs wth more hosts, again potentially as a live operation.

You can reduce the cost of the SAN by not using fibre channel, e.g. you could use iSCSI. In which case you'll again want separate networks for your data and you'll want dedicated NICs, ideally with tcp/iSCSI offload hardware. That will give you most of the advatnges of a SAN with lower cost.

I have not really used iSCSI exxcept for the most basic single LUN to a single host, with simple linux LVM and ext3, so I am not 100% sure if it si really as good as FC SAN, but I gather it can be if well implemented.

SAN arrays are probably the better choice if you are going to use a clustered filesystem. The question is do you really need a clustered file system? That will depend on the characteristics of your app and your architecture.

Now if your app can guarantee that only node node will write to a given file at a given time, then you can probably go to a NAS. But you may have problems if you modify a file with one host while it is being read with another host, so your app would need to detect and deal with that scenario. If that is a scenario that you dont want to bother with then a clustered file system is probably better choice - they are designed to work with that sort of scenario.

So questions like some of these listed below might make a big difference to your architecture:

  • Does the file need to be reused again after it has been downloaded once and sent to the client - i.e. might it be re-read from storage and served to another client?
  • Does a file need to be written completely to storage before it is sent to a client?
  • Can a file be stored on local disk on the server and served to the client from local disk, and then be written to NAS/SAN after it has been served to the client?
  • Are multiple clients likely to be using the same file at once? E.g. is it likely that 50 clients will access one file or 50 clients will access 50 different files.
  • If 50 clients each request the same file, will it be downloaded once or 50 times?
  • If another client comes along 3 hours later and requests the same file will the file be downloaded again or will it come from disk?
  • Is the disk a cache or a slow buffer?
  • Will there be any other processing performed on the file before it is returned to eh clients, e.g. security scanned, have URLs re-written etc.

Given the limited info we have I'd say the safest architecture is the most expensive and complex architecture as that will deal with most of the worst case problems and be very scalable. I.e. Fibre channel SAN and clustered file system.

In all cases whatever your storage, DAS, SAN, NAS, all other things being equal more spindles are better.

share|improve this answer

I would go for a DAS based architecture. The problem is at one point the file system is irrelevant - the question is, given a specific IO requirement, how many GB you can put into a certain infrastructure cost (size, power) for the best price.

  • SuperMicro has special cases that take up to 48 hard discs. They are not cheap, but SAS based.
  • You probably have to go with a decent SAS controller for those.
  • Processing power may also be an issue, as does memory. WHen yo have like 40.000gb in a box, that caching may take 8 or so GB ram to be efficient ;)

So, at the end i would go with a pretty decent dual processor AMD server on a special casing setup that can handle a LOT of discs in a specialized cage.

THat being said, Cluster is probably as good as it gets - as long as you dont require super fast disc access that is typical for large databases. It should do most of what you ask for. But once you get going, keeping the price per gigabyte small may be the most important thing - without blowing your administrative overhead through the roof.

share|improve this answer
hello, i forgot to mention that, by meaning "design" i would rent dedicated servers from one of the providers, so i wont setup my own datacenter at all. sas is a special hardware or it is some kind of specialized server ? One more thing to add: I want to use all of the bandwidth capabilities I own (100mbit for the client nodes). Thus, IO speed should be at least 10MB per second. Considering multiple requests at the same time, 10.MB per sec is really seems hard. (isnt it?) – TORr0t May 12 '10 at 8:14
Well, seriously - that won't work. I dont have my own datacenter,but i am not renting my servers ;) THey provide a rack. Main problem you face is that - well - your requirements are not usual, so normal providers just don't have it. You are better off leasing your own server(s) and rent space / conenctivity from a data center. IO speed 10mbyte is not the issue - IOPS is. Considering multiple requests, I have seen databases going down to 1MB with the head trashing like mad. THis asks for MANY discs on a GOOD controller. – TomTom May 12 '10 at 8:38
Basically, forget MB - talk IOPS, like a database does. Full random read/write efficiently as many file requests are handled same time. – TomTom May 12 '10 at 8:38
If scaling is a factor I'd steer away from DAS. NAS/SAN allow you to make the same storage, either file systems or LUNS available to multiple hosts - and more importantly allow you to easily allow access to more hosts at a later date it required. I.e. SAN/NAS will let you scale horizontally if necessary. So DAS is fine while you have one server accessing the data, but what happens if your app gets popular and you need multiple servers accessing your data? Given that it looks like you want to use glusterfs I'd say SAN is probably a better choice than NAS. And as Tom says more spindles is best. – Jason Tan Aug 14 '10 at 14:25
if scaling is a factor AND MONEY IS - thn a NAS / SAN architecture is a lot more expensive than most DAS architectures, AND NAS / SAN is a lot more expensive to scale. Multiple servers accessing the data (SQL) get their own copy of the data. Guess what - you need multiple copies in a NAS, too, if you outscale the IOPS load of a single copy. DAS is fine and scales quite far.... and is a LOT cheaper than a comparable SAN. Money, for most people, is an issue. – TomTom Aug 14 '10 at 14:35

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.