6

We currently have a windows server that stores images for a web project. The project has been really successful and we are seeing performance issues and we will obviously run out of space in the future with the current 100k or so images we have stored (On the local disk).

The servers are dedicated machines hosted in a data centre in the UK, what options are available for scalable file storage (Ideally we can put in the same data centre as the images need to be accessed often).

2 Answers 2

3

In order to store your data properly, you have to use some additional software, since native File and Storage Services that come out of the box are not capable of consolidating the storage into single pool dispersing and replicating the data between hosts to achieve more redundancy and performance.

DFS has a lot of issues like not having the proper storage "locking" mechanics, potentially leading to split-brain in the case of network isolation, can't replicate open files and its mechanics is unable to distinguish which replica is "correct", this might lead to some issues in multi-server environments.

You can not reduce the amount of data since media data has high entropy and deduplication/compression will not work and scaling up the amount of storage on a single server is also a bad idea since should this server fail you will (temporary) lose all the data.

To transform your existing Windows-based servers into scalable storage servers that create a single shared storage pool best option would be using some software-defined storage solutions that can be installed and run directly on Windows OS:

1) StarWind VSAN https://www.starwindsoftware.com/starwind-virtual-san

Alternatively, there are good options that run as Virtual Machines on top of Hyper-V role installed:

2) HP VSA http://www8.hp.com/us/en/products/data-storage/free-vsa.html

3) EMC Unity VSA http://www.emc.com/products-solutions/trial-software-download/unity-vsa.htm

If you are OK with investing into Windows Server 2016 Datacenter edition you might be interested in

4) Storage Spaces Direct https://technet.microsoft.com/en-us/windows-server-docs/storage/storage-spaces/storage-spaces-direct-overview

If Windows is not crucial you should definitely take a look at:

5) Ceph http://ceph.com/

2
  • Yeah. Or use Windows. I mean, after installing a current version of windows AND reading the documentation you realize that - well - that is all that is needed.
    – TomTom
    Nov 13, 2016 at 14:45
  • 1
    Before Microsoft will bring in reliable super-Nano version of Windows Server with no need for notorious frequent mini-patches every week using Unix-based storage has LOTS of benefits. Few "Windows Inside" storage projects like EMC CLARiiON were abandoned in favor of a FreeBSD or Linux fork-outs for a very good reason. Nov 15, 2016 at 21:37
-4

THere is no easy solution besides what Windows offers out of the box (current one and you have to RTFM; really).

The problem is that you really do not WANT ONE store. What about you start distributing the images into different storage pools? Then you can addd additional pools as needed. You obviously need to keep a database of what is there.

The main advantage of this approach is that while you CAN keep a terribly large storage space synchronized and do backups on it - it is a lot easier if you can keep that one down in size.

Now, obviously for every one of those shards you can use something like storage spaces with either DFS replication or a lower level mirroring (storage spaces direct). This one pretty much makes any hardware SAN look like a really good choice (and licensing costs for a web projects are not that high given SPLA pricing).

But even then you will hit a problem at one point, thus sharding on the logical level.

Oh, and buy proper hardware. SuperMicro has some really good cases for storage intensive. You talk of internal hard disc? What about a 2RU case with space for 80 discs ;)

1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .