5

I want to build let's name it a lowcost Ra*san which would host for our social site the images (many millions) we have 5 sizes of every photo with 3 KB, 7 KB, 15 KB, 25 KB and 80 KB per Image.

My idea is to build a Server with 24x consumer 240 GB SSD's in Raid 6 which will give me some 5 TB Disk space for the photo storage. To have HA I can add a 2nd one and use drdb.

I'm looking to get above 150'000 IOPS (4K Random reads).

As we mostly have read access only and rarely delete photos i think to go with consumer MLC SSD. I read many endurance reviews and don't see there a problem as long we don't rewrite the cells.

What you think about my idea? - I'm not sure between Raid 6 or Raid 10 (more IOPS, cost SSD). - Is ext4 OK for the filesystem - Would you use 1 or 2 Raid controller, with Extender Backplane

If anyone has realized something similar i would be happy to get Real World numbers.


UPDATE

I have buy 12 (plus some spare) OCZ Talos 480GB SAS SSD Drive's they will be placed in a 12-bay DAS and attached to a PERC H800 (1GB NV Cache, manufactured by LSI with fastpath) Controller, I plan to setup Raid 50 with ext4. If someone is wondering about some benchmarks let me know what you would like to see.

1
  • You may be able to use high-quality MLC flash in certain enterprise applications. The Intel 320 series looks especially attractive, as it is capacitor-backed and tracks wear percentage against its rated life automatically. However, I would not recommend using consumer-grade 240GB SSDs, especially if they're the ones I think you're talking about. You will regret using bottom-of-the-barrel flash in an enterprise array.
    – Skyhawk
    Oct 14, 2011 at 22:24

7 Answers 7

6

I would consider a hybrid solution which could be achieved with OpenSolaris, SolarisExp 11, OpenIndiana, or Nexenta. Hybrid pool would be a lot less costly, and with a few thousand bucks worth of RAM, you will have your 150k+ IOPS with mostly normal spinning disks. At Nexenta we have many, many customers who do just exactly this. ZFS is a robust filesystem, and with enough RAM and/or SSDs for additional Read/Write caching you can have a very robust solution at a relatively low cost. With Nexenta Core, which is community, you get an 18TB at no cost at all. Of course, a new release of OpenIndiana would allow a lot of the same functionality. Add to this snapshots, cloning, replication usinf ZFS send/recv and you can build a SAN that will give any EMC a run for its money at a far lower cost. Lots of SSDs are nice, but there are other options, some not half-bad.

2
  • I suggested this initially but backed away from it when @Nenad mentioned 150K read IOPS (way beyond my real-world experience with Nexenta ZFS). Nice to hear that ZFS actually can meet these requirements.
    – Skyhawk
    Oct 13, 2011 at 2:12
  • ZFS can do far more than 150K IOPs. In fact, recently at VMWorld in Vegas we ran 60 or so percent of labs, and numbers were far greater than that. It is all possible, but hardware is always the limiting factor.
    – slashdot
    Oct 13, 2011 at 3:06
7

Use RAID6 over RAID10. For mainly read based I/O loads the throughput should be similar when the array is not degraded, you get better redundancy (any two drives can fail at the same time with R6, R10 can not survice if both failed drives are on the same leg (so can only survive four of the six two drive failure combinations in a 4-drive array, I'm not sure off the top of my head how that 4/6 figure scales for larger arrays)), and you get a larger usable array size unless you arrange the drives in 4-drive sub-arrays (see below).

Your space calculation is out, certainly for RAID10. 24*240Gb is 5760Gb with no redundancy (RAID0 or JBOD). With RAID10 you'll get only 2880Gb as there are (ussually) two exact copies of every block. If you use all the drives as one large RAID6 array you will get your 5Tb (5280Gb, two drives worth of parity info spread over the array) but I personally would be more paranoid and create smaller RAID6 arrays and join them with RAID0 or JBOD - that way you have shorter rebuild times when drives are replaced and you can survive more drives failing at once in many cases (two drives per leg can die, rather than two drives out of the total 24, without the array becoming useless). With four drives per leg you get the same amount of space as RAID10. Four 6-drive arrays may be a good compromise (4*4*240=3840Gb usable space) or three 8-drive arrays (3*6*240=4320Gb usable space).

With regard to controllers: these can be a single-point-of-failure for RAID. If the controller dies you lose all the drives attached to it at once. While such failures are pretty rare (random corruption is more common) there is no harm in taking care to reduce the impact should it happen to you. If you use RAID10 make sure that no pair of drives are both on the same controller (which means having at least two). If splitting into 4-drive RAID-6 arrays use four controllers and have one drive or a given array on each. This of course assumes you are using software RAID and simple controllers which might be unlikely (you are spending this much on drives, you may as well get some decent hardware RAID controllers to go with them!).

You should give a thought to a backup solution too if you have not already. RAID will protect your from certain hardware failures but not from many human errors and other potential problems.

3
  • Thanks, Yes, you're right. I think I will create two R6 arrays per server and a Raid0 between them or split by two Raid controllers.
    – Nenad
    Oct 12, 2011 at 19:52
  • 1
    Bad experience with software raid6 over large fast drives, CPU latency adds a large delay and in my case the PCIE bus became a bottleneck because of the 2 extra parity drives taking up bandwidth. If i had to make a choice between a good raid6 card/HDD and software Raid/SSD.. well i think you already know where im going with this.
    – Silverfire
    Oct 13, 2011 at 0:09
  • @Silverfire: It likely isn't CPU power that hit you, unless you are using a low-power CPU like an Atom. The bottleneck is the buses between the CPU and controller. With hardware RAID6 the OS just writes one block to the controller and it sends that block to a drive and updates the two relevant parity blocks. With software RAID6 the OS reads those two parity blocks then writes all three, multiplying the amount of info being transferred between main CPU+RAM and the I/O controller significantly. With fast enough drives this can be significant. Oct 13, 2011 at 9:17
4

Just buy two FusionIO Octal cards and mirror them - far simpler, far faster (might be a bit more expensive however).

4
  • Too expensive, not expandable, to less space. We use Fusio ioDrive Duo SLC for DB Servers and the Real World performance don't impress us.
    – Nenad
    Oct 12, 2011 at 19:44
  • 1
    But you wanted 5TB usable and >150kIOPS - one of those card will do that, two just lets you R1 them. I use these and they work very well.
    – Chopper3
    Oct 12, 2011 at 19:47
  • Sorry, I don't know that they have 5 TB :-) Our ioDrive duo SLC has 320 GB. But if i do the math, our 320 GB card cost $15K, then this card would be somewhere about $150K and this two times... That money we can not spend. 24x 240GB SSD would cost around $7K.
    – Nenad
    Oct 12, 2011 at 19:58
  • Please don't tell me you want to use cheap OCZ drives made with substandard components. There might be a reason why they retail for less per GB than the wholesale cost of decent flash memory.
    – Skyhawk
    Oct 13, 2011 at 6:13
4

Answering your key questions:

  1. RAID 6 vs. RAID 10: You almost certainly do not need to worry about IOPS if you are using SSDs as primary storage.

  2. SLC vs. MLC: There are subtler differences. If you are going to use MLC, I would suggest buying Intel. The Intel 320 series has a SMART counter that you can use to track the wear level percentage and replace the drive before it fails.

However, you may want to look at ZFS on the Nexenta OS (or possibly FreeBSD, unsure of development status) if you want to use SSDs to improve storage performance in a reliable way:

  1. ZFS allows you to build a "RAID-Z2" (somewhat like RAID-6) array of conventional disks that use SSDs as massive read (L2ARC) and write (ZIL) caches, allowing you to get the performance benefits that you're looking for without the cost of an all-Flash array.

  2. Blocks that are accessed often will be read from the SSDs, and blocks that are used less often will still be read from disk. All writes will go to SSD first and be committed to disk when it is convenient for the array.

  3. Because you will need fewer SSDs, you will buy higher-quality devices and you will not have the kind of catastrophic failure that is to be expected if you build a RAID array out of consumer-grade MLC devices from OCZ (or whatever).

  4. Even if you don't use high-quality devices, the consequences are less severe. If you use MLC devices for your ZFS L2ARC and they fail, you still have your data preserved on disk.

4
  • Thanks, I will take a look on ZFS it sounds similar as the SSD Cache solution of Adaptec, but it will not solve our problem as the requested files are really mostly different and we don't get any benefit of only store a few of them on SSD. Regarding IOPS I'm looking to get above 150'000 IOPS (random read) and I'm not sure if R6 can deliver that, this was the reason I ask for R6 or R10. The goal is to have a storage server that provide around 50% of the performance of a Ramsan solution.
    – Nenad
    Oct 12, 2011 at 17:07
  • Got it. I would strongly suggest adding your specific performance requirements in an edit to your original question.
    – Skyhawk
    Oct 12, 2011 at 17:12
  • 2
    These people are doing more or less precisely what you want in a commercial product that may or may not fit your budget: purestorage.com/products/pure-storage-flash-array.html Their base model apparently has 5.5TB raw capacity and can deliver 200,000 random read IOPS.
    – Skyhawk
    Oct 12, 2011 at 17:14
  • Thank you, this Purestorage solutions is exactly what I would like to have, but I'm not sure if it will fit our budget. I will send a Mail to the sales. I'm looking further how to build self such a solution.
    – Nenad
    Oct 12, 2011 at 17:21
3

150k IOPS with 4k blocks is 585 Mb/s throughput. Make sure your controller and backplane can handle that. As for raid, remember that protection against SSD failures is all it'll buy you. A controller failure (or memory fault, processor outage, or failure of any other single point of failure on the server) will render your data unusable. Keeping another identical server around in synch would be needed to avoid downtime and potentially having to go back to tape.

This second server, if filled with SSDs like the first one, might make it so it's almost cheaper to buy a centralized storage device that supports SSD, if it has no single points of failure. If, however, you keep your second server in synch using real hard drives, you can save a large chunk of change without affecting performance. Since most of the IO is reads, the load on the drives will be minimal except during times when the primary server is offline. This would allow you the financial flexibility to buy more than one replication target, and maybe even move some offsite in case of a site failure.

2
  • Thanks, I will check that the controller meet the requirements. And yes the idea is to have a synced 2nd server with same setup. What you mean with "centralized storage"?
    – Nenad
    Oct 12, 2011 at 19:48
  • I mean some sort of disk controller with no single point of failure that allows servers to access data stored on it via fibre channel or iSCSI (if you use block level scsi access) or NFS/CIFS (if you want the file system to reside on the storage controller). Many modular systems these days will do SSD, and since you're doing all reads, you can go for a cheap one with little cache.
    – Basil
    Oct 12, 2011 at 20:12
1

You can avoid the RAID controller issue entirely by using ZFS - it can detect AND CORRECT silent corruption (data errors that get past ECC checks), which virtually no raid controller is able to to (detect yes, but not fix) and on a large drives (2Tb+) you can expect 1-2 errors per year, per drive.

Unfortunately if you want this with vendor support you'll need to use Solaris. Some Linux vendors support it it but it's still a beta product (That said, I use it on linux and I've found it virtually impossible to kill, right up to pulling several drives out of their bays while hot. At worst the array shuts down - but there is no data corruption)

0

Doing all of this as a single server with expensive disks might not be the best answer. Given your budget and needs I would recommend looking at STF. It was designed as an image storage for one of the largest blogging services in Japan:

https://github.com/stf-storage/stf

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .