1

I am using Windows platform to setup a web based file share system. In more details, individual users could upload and share documents from web interface (content may be big document, like video files), similar scenario to Google Docs.

My current issue is how to make storage scalable? In more details, I have 4-5 front end web servers (to make a web server farm) and I want to know how to setup storage system to store uploaded files.

I want the storage to be automatically grow -- i.e. each web server seems to use unlimited disk space (no need to handle disk full issue); another issue is I do not know how to store files efficiently and reliably (e.g. if each web server has its individual separate storage, suppose abc.wmv is stored in web server A's storage, then if web server A downs, no one can access abc.wmv). Another trouble I can think of is, if I increase the # of front end web server, for the new web server, how to decide which uploaded files should be stored (shall I migrate some files from other web servers to the new web server)?

I am consider to use SAN, but I am not sure whether SAN could resolve all of the issues. I want to learn some best practices to handle this issue.

thanks in advance, George

0

4 Answers 4

3

You probably would want a shared storage space, like a SAN, so you could centrally manage and back up the data (and it would all be synchronized). You don't mention what kind of load you expect...dozen users, hundreds?...so your front-end servers may be a little overkill to have four or five of them.

Using a back-end storage array would simplify your configuration and management, and allow you to add web servers if you find they're needed. Then you need to focus on setting up heavy duty storage and backup. I.e., RAID 10, plus backup storage schemes to get it back up if the RAID fails. Again you don't mention how heavy your usage is planned to be and what load it'll have so I don't know if you'd want to double up your hardware as a backup in case the SAN fails and/or set up data replication between SAN nodes (over a high speed link to different geographical areas, if need be).

This can get very expensive very fast and you may even need to have services brought in to consult with a company that specialized in high-volume data replication. If you're a small startup, you could look at setting up a dedicated NAS for storage and using something like DRBD for RAID over ethernet to different locations. Different avenues to consider but heavily impact budget and are affected, again, by your budget, expertise, and necessary uptime expectations (home projects are one thing, keeping average users in small business another, and hardware needed to keep customer credit card information flowing and where downtime of an hour can cost your company a few thousand dollars and your neck will be yet another)...

4
  • Thanks, two more comments, 1. There may be 10k concurrent connections and about 100T storage. About your solution of using SAN, if I find 100T is not enough and wants to add additional 10T, could I add such additional storage transparent to my application and without stopping my application? 2. About the solution of using SAN, there should be no separate individual storage for each front end web server, and all web server share the same storage?
    – George2
    Dec 17, 2009 at 14:57
  • 1
    1) yes, I'm pretty sure with forethought that it can be done, but you need to research your vendor and see what it supports. There are NAS systems you can set up so that using LVM (Linux) you can grow a volume while it's in use online, and specialized NAS/SAN solutions may have their own way to do it. 2) If you're mounting or pointing to a volume on a NAS/SAN, then they get the same storage capacity. But you can't split things like iSCSI (I think, someone correct me) among systems). Using it as a network storage volume should work. Dec 17, 2009 at 16:38
  • 1
    You'll want to investigate how RAID performance (and fast hard disks) affect your user experience. We have a NAS from Dell that had trouble due to slow drives (using RAID 5...a no-no nowadays) when a lot of users logged in and tried pulling their profiles all at once. Maxes out the spindles and drive throughput. Dec 17, 2009 at 16:42
  • 1
    It's also very possible that by the time you need to increase your storage, your hardware may be due for replacement too. Otherwise you're going to have to look at a way to add volumes to something like a RAID 10 array and add that new volume in, and the hardware will need to support the added power and cooling requirements, and your backup system will need to support that kind of added capacity. Dec 17, 2009 at 16:43
3

There are a number of ways to do it but I'd recommend a NAS that manages its own file system. That really depends on the NAS though. My experience is primarily with NetApp and it could easily accommodate what you're asking in addition to providing additional functionality (like Disk De-dupe and an advanced snapshot system). NetApp's hardware is extremely scalable, makes it easy to grow a volume, and supports a lot of different network protocols. Like any professional SAN/NAS you're going to pay for it though.

http://www.netapp.com

On the lower end, you might consider trying something like Synology's DJ509+ diskstation. I've only talked to people who have used it but many "corporate" IT have been pretty impressed by it's flexibility and low cost.

http://www.synology.com/us/products/DS509+/index.php

The absolute cheapest method would be to do something like using OpenSolaris and ZFS. You could build your own hardware and create your own RAID volume. I don't know of anyone personally who is using it in a situation like yours but anecdotally I've heard of many people being impressed by how robust it is.

2
  • Thanks, two more comments, 1. About your solution of using NAS, if I applied 100T at the beginning, but later I find 100T is not enough and wants to add additional 10T, could I add such additional storage transparent to my application and without stopping my application? 2. About the solution of using NAS, there should be no separate individual storage for each front end web server, and all web server share the same storage?
    – George2
    Dec 17, 2009 at 15:00
  • 1
    Are you talking about creating a single 100TB volume? While that much storage is definitely feasible with NetApp, you wouldn't normally have a single volume that large. I'd highly recommend chatting directly with them to see what they recommend. They work with a lot of large companies that have huge data requirements. As far as growing shares, it's a piece of cake with NetApp filers. It's completely transparent to the webservers, takes a few mouse clicks to do, and is ridiculously easy. Since you have some pretty product specific questions, I'd highly recommend contacting them direct.
    – Mr Furious
    Dec 17, 2009 at 16:50
1

What you want is MogileFS: http://www.danga.com/mogilefs/ We have many, many terabytes (a petabyte yet? haven't checked) of data in MogileFS and it keeps scaling up pretty well.

1
  • Thanks! Actually I am more interested in SAN/NAS (which is more popular industry solutions). Two more comments, 1. There may be 10k concurrent connections and about 100T storage. About the solution of using SAN, if I find 100T is not enough and wants to add additional 10T, could I add such additional storage transparent to my application and without stopping my application? 2. About the solution of using SAN, there should be no separate individual storage for each front end web server, and all web server share the same storage?
    – George2
    Dec 17, 2009 at 16:27
0

I'd consider a NetApp box, they're not the cheapest but they're pretty flexible and can offer you thin-provisioned NFS shares which seems to fit you requirement and can be scaled pretty well (about 1.2PB iirc).

Alternatively you could look at HP's "massive scale-out' technology, not all of it is on their site but if you speak to their storage sales people they have stuff that'll build out to exabytes.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .