up vote 1 down vote favorite
share [g+] share [fb]

I'm just doing some testing on a NAS RAID array we've been lent to see if we want to deploy it. Just to see whats what, I decided to see what kind of network performance we get when it's re-building a RAID-5 array.

I'm noticing that network connectivity has dropped to about 7MB/s, which is about 60Mbps. A long way shy of the 1Gb connection it has to the network. When it's not re-building it's closer to 70Mb/s (with Jumbo Frames).

Would you expect this kind of a massive drop-off when re-building? At the moment it has three 750Gb 7200RPM SATA drives installed.

link|improve this question
93% accept rate
3  
Do you have an option to change rebuild priority? – Chopper3 Jun 19 '09 at 9:38
Hmm, thats a good question. I'll need to investigate... – Mark Henderson♦ Jun 19 '09 at 9:50
3  
Be careful when lowering rebuild priority. You don't want the rebuild to take too long as until it is complete your array is vulnerable. Many RAID implementations allow you to limit the speed of the rebuild to reduce the effect on other activity, but I'm usually of the opinion that the rebuild should get as much throughput as it wants to get the job done ASAP. A rebuild shouldn't be a regular occurrence anyway. – David Spillett Jun 19 '09 at 12:35
feedback

5 Answers

up vote 2 down vote accepted

YEs, that would be the expected throughput, rebuilding the array is as you probaly know a disk intensive activity as it has multiple simultaneous read and write requests to both the remaining disks in the array.

I would expect a higher throughput on a server with a dedicated raid card like a PERC 5 or 6 but for a NAS device that is likely using software RAID that would be a reasonable throughput.

link|improve this answer
That explains a lot. Yes, I'm used to devices with PERC and CERC cards and dedicated controllers... – Mark Henderson♦ Jun 19 '09 at 9:49
feedback
up vote 1 down vote

You're rebuilding RAID-5 and you're still getting data out? My hat's off to you!

When I've seen RAID arrays rebuilding, they've usually been considered off limits for any real work. Usually we don't put replacement drives in until after normal business hours if we can help it.

I'd just let the array rebuild and come back to it later. Are you sure it's RAID-5? Many of the newer RAID systems are actually RAID-6 / RAID-DP.

link|improve this answer
1  
you made me laugh! 90% sure R6 needs 4 disks though, so he's presumably using R5 – Chopper3 Jun 19 '09 at 10:14
Well, as mentioned, it's just for testing purposes. It's not actually being used for anything, except to see how it responds under pressure! And it's def. not RAID-6, it only has three bays. At an office I worked at once they had a 14-disk SCSI-320 array that was absolutally hammered 24/7 (callcenter logging). It was RAID-6 with two hot spares. When one of the disks failed, it took over a week to rebuild... – Mark Henderson♦ Jun 19 '09 at 10:25
feedback
up vote 1 down vote

i have found that it depends on whether your controller card is real raid or fake raid.

i have rebuilt raid 5 in the past with rocketraid (dedicated memory and card onboard) a raid 5 with users being able to continue doing work normally. the priority of the rebuild is set to high and this is on high end xeon processors with much memory (if it makes any difference).

i have also rebuilt raid 5 through the intel matrix raid (using the ich chipset etc for the sata connection) and have had no real issues either serving out the files. this was a vanilla supermicro type server for basic office (20 users or less) but with xeon cpu's and much ram.

i have always done raid rebuilds (raid 1, raid 5 raid6 and raid10) first thing in the morning, inline hot swap and started the rebuild right away. i only have 1 day a week dedeicated to each client, typically, so i have to do it when i am there and do it right away. i would monitor it while i was there, never any complaints. maybe just lucky but i normally do a raid rebuild from a failed drive at least 5 times a year with the amount of clients i have gained in the past 7 years.

link|improve this answer
also dedicated cards that i have used without issue in windows and linux enviro are 3ware stuff.gd – user8256 Jun 19 '09 at 12:22
feedback
up vote 0 down vote

A certain external disk array I've worked with in the past did the unthinkable (or so I thought) and disabled the on-controller cache during rebuild operations. If it was able to pump out 40-60MB/s normally, you could get I/O saturation with writes as low as 4MB/s depending on what I set the priorities at. This was with 500GB 7.2K RPM SATA drives, too. For a solution with a hardware RAID and advertised as being a multi-access fibre-channel device, it certainly fell flat on its face when recovering from faults.

We ended up largely ignoring this device. It's currently doing service as a dedicated backup-to-disk array for one backup server. I reformatted it with RAID10 to greatly simplify how long it takes to recover from bad disks. Something that can go to that low level of write I/O just simply could not be used in production.

Which is a long way of saying, your experience is perfectly in line with reality.

link|improve this answer
feedback
up vote 0 down vote

With all the RAID controllers I work with, performance hit is negligible when rebuilding. For example performance of a 5 drives RAID-6 750 GB on a 3Ware 9560 is :

  • 79614 MB/s writing
  • 85333 MB/s reading

NFS performance is about 80% of the disk througput ( 60 MB/s). CIFS performance is about half NFS throughput ( 35 MB/s).

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.