3

Thinking about setting-up as distributed RAID array over iSCSI. Has anyone else tried this, if so what was your experience.

To be more specific I am thinking of a couple of CentOS servers each with say four drives, all published as iSCSI targets. Then one Windows Server accessing all of these via iSCSI and connecting those targets together to create a software RAID array.

One of my concerns is the rebuild time if one of those servers is offline for a short amount of time. Would it then be necessary to rebuild the entire array from scratch or is software RAID clever enough to only rewrite sectors that have changed? My concern being that a small network glitch could otherwise potentially cause a long rebuild process.

Thanks, Nick

6
  • 3
    Why would you want to do this instead of creating RAID on the server and export this as one iSCSI volume? This sounds like a pretty delicate environment.
    – Sven
    Nov 13, 2012 at 23:26
  • What problem are you trying to solve with this?
    – MDMarra
    Nov 13, 2012 at 23:40
  • Are you wanting a highly available iSCSI target? that actually makes sense.
    – hookenz
    Nov 13, 2012 at 23:55
  • @Matt: HA is about eliminating possible points of failure, not creating them.
    – Sven
    Nov 14, 2012 at 0:35
  • If you want a distributed filesystem, build a distributed filesystem. There are plenty out there; some even work under Windows.
    – mfinni
    Nov 14, 2012 at 1:32

6 Answers 6

6

This has many drawbacks and not a single advantage I can see, so I don't understand why you would want to do this.

  • Any outage of the iSCSI connection will likely require a full RAID rebuild. The RAID subsystem doesn't know that the disc is the same if it sees it again and that it is more or less unaffected, but even if that would be the case, it doesn't keep a log of write operations since the failure that it could use to bring the drive up to date again.

  • The network connection will be a serious bottleneck, especially in case of a rebuild. You will have a small number (likely just one) 1 GBit/s network connections compared to multiple SATA/SAS connections with up to 6 GBit/s each, connected over the PCIe bus.

  • This whole setup is really delicate and easy to bring to a complete halt.

2
  • The whole idea is to create a robust distributed RAID environment using iSCSI targets so that there is not a singular point of failure. By the sound of the comments here iSCSI might not be the best way to do this. To me the advantage of iSCSI is that it lets me use Linux servers to store NTFS based data for our Windows users.
    – NickC
    Nov 14, 2012 at 15:51
  • This can be done via iSCSI, but doing it reliable will involve a lot more than your somewhat simplistic approach. If it needs good performance, it will be expensive as well (think multiple 10 GB ethernet connections/switches or even Infiniband to reduce latency). Other approaches might be more appropriate and cheaper to reach that goal.
    – Sven
    Nov 14, 2012 at 16:07
4

You cannot do what you're intended to do with a built-in software RAID from Microsoft. Reason is within Windows network stack comes earlier than storage stack so your RAID will always start in broken state and will do long and painful resyncs and rebalances and rechecks. You may however use some third-party software to aggregate iSCSI volumes into sort of a central storage. Can be iSCSI and you can connect to it in a loopback. Companies like FalconStor, DataCore and StarWind also do a business on building a unified storage pool from many many separate SAN and NAS boxes. DataCore and FalconStor are expensive as F and StarWind may do what you want even with a free version. Linux/FreeBSD with LIO is another alternative if you care. Good luck!

1

I wouldn't do it this way. There are lots of manufactured NAS devices with multiple disk arrays both for home use and business use that have ISCSI built in. These devices are designed with this purpose in mind and will outrun the software configuration that you are thinking about at a comparable or cheaper cost.

1

Just for your convenience here is a performance analysis report for Distributed iSCSI RAID architecture.

http://www.ele.uri.edu/tcca/camera_ready/Ben_iRAID-SNAPI-cr.pdf

0
-1

Wanting to glue together multiple iscsi targets is a perfectly valid idea. There are plenty of reasons you'd want to do it.

Here is a simple scenario:

1) I've got 2 servers, and each has 12 disks in a Raid6, giving me 36TB usable space on each server. All of that storage is already RAIDed, to provide redundancy and protect from disk failures etc.

2) I want to setup one server with an ISCSI initiator that connects to all the targets, and then use LVM to glue the ISCSI volumes together into a single 72TB volume, with a single 72TB filesystem. To do that, i should be able to use LVM to join the 2 iscsi volumes together into a 72TB volume and format with XFS.

3) When that 72TB filesystem is full, i want to be able to get a 3rd server and have it export another 36TB ISCSI disk. Then, on my master server, which is running the initiator/LVM, i connect to the new iscsi disk, setup an LVM physical volume on it, add that PV to the pool, expand my 72TB LV to 108TB LV, and then XFS allows me to easily expand the filesystem into the new space. Easy.

Basically, this should be an easy way to aggregate fragmented storage across a number of servers, to provide a very large filesystem.

The whole point of iSCSI is that it is supposed to present a remote SCSI device as a local one. If it is OK to use local disks in a RAID, it should be OK to use ISCSI disks in a RAID.

If i temporarily pull a disk out of a hardware RAID0 with no redundancy, i do not lose any data on that RAID. EVER (assuming a properly journalled filesystem). The RAID just goes offline. And if i put the disk back, then the RAID is back online. And it should be no different for a RAID of iSCSI targets.

2
  • 3
    No that is not valid. What you will have just done is make a really fragile system. The A in HA stands for availability. Something that fails often is not enough even if it fails safe.
    – JamesRyan
    Jan 24, 2016 at 2:43
  • for anyone reading, its theoretically valid in a lab setup (though no one experiments with 72 TB) ... however when experience comes in, I will safely assume that a 100 TB range storage is not a home setup, also is not exactly entry level enterprise storage and whoever will fund this must have something important to store. In that case, all technical opinions aside, GET THE RIGHT TOOL ... whether vendor or open source, just get something designed and sized correctly ... also crossing the 100 TB range, you might wanna drop iSCSI and take FC as the cost/GB will start to become relatively close
    – a.atlam
    Jun 1, 2017 at 17:53
-1

I have a valid reason for such a setup.

I have a few EOL servers lying around filled with 146GB HDDs. I created a VM setup for a CloneZilla server along with a few test VMs utilizing OVirt and GlusterFS. The CloneZilla image repository is distributed in three CentOS servers running GlusterFS with 6x146GB in RAID6 each. If I can use iSCSI aggregation using ESOS, 2 disks from each of those servers can be added to the RAID6 array, which is now being used for the OS. ESOS can run on a USB or SD Card. Also, the extra RAM can be moved to the VM host. A 2TB NAS can serve as the backup.

0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .