3

I have set up a RAID5 array using 4 disk partition. All disks are 2TB. When the RAID5 array was initially syncing it was too slow. So I did a

echo 32768 > /sys/block/md2/md/stripe_cache_size

and then it was quiet fast. But still it took a days time to finish syncing. Now I want am doing a

dd if=/dev/zero of=/dev/md2 bs=1024k

onto the RAID5 array and it is like a day over and yet it has not finished. Will this take too much time am I doing some thing wrong.

My raid details are here http://dpaste.com/749742/
Iostat details here http://dpaste.com/749761/ hdd partitioning details here http://dpaste.com/749765/

I have an identical machine with the similar details. But that's quiet faster.

UPDATE: I have added my hdd partitioning details.

1
  • Have you tried to play with dev.raid.speed_limit_min/dev.raid.speed_limit_max via sysctl ?
    – ALex_hha
    Mar 20, 2016 at 19:07

3 Answers 3

2

Normal time to build such array is 5.5-6 hours. There are a few things that can slow down array rebuild process - concurrent IO on the same disk, high cpu usage on the system, faulty HDD or cables.

dd if=/dev/zero of=/dev/md2 bs=1024k - will slow down rebuild. If you really need to rewrite your array with zeroes - do it after array has built.

Check if you have load on other partitions - using atop or iostat -nx 1

Check your load average in atop or top

Check for HDD errors in system logs and with smartctl.

6
  • RAID rebuild is over. I started dd only after rebuild
    – proy
    May 18, 2012 at 9:02
  • can you show dd's speed. use killall -USR1 `pgrep dd` to see it
    – DukeLion
    May 18, 2012 at 9:04
  • this would kill my dd process right ?
    – proy
    May 18, 2012 at 9:29
  • no, it will just make it to write current statistics to it's output
    – DukeLion
    May 18, 2012 at 9:44
  • NO HDD errors in log and smartctl also gives a clean chit
    – proy
    May 18, 2012 at 9:51
1

in raid 5 you should look about raid chunk size. run : cat /proc/mdstat to check yours.

to change it run :

mdadm --grow /dev/md0 --chunk=16

this will rebuild your array, it takes long times to complete. you can read this for more informations about it : https://raid.wiki.kernel.org/index.php/RAID_setup#Chunk_sizes

then you can test again your write performance with dd for exemple.

I've made a small script to test read ans write :

#!/bin/bash

MDDEV="`cat /proc/mdstat | grep md | head -1 | awk '{print $1}'`"
if [ -z "$MDDEV" ]
then
    echo "I can\'t find any md" 
    exit 1
fi

DEVS="`cat /proc/mdstat | grep $MDDEV | tr " " "\n" | grep '^sd' | grep -v \(S\)$ | awk '{print substr($0,3,1)}' | tr -d "\n"`"

NUMDEVS=${#DEVS}

# write dd performance
echo 3 > /proc/sys/vm/drop_caches
dd if=/dev/zero of=/dev/$MDDEV bs=1M count=1024 conv=fdatasync,notrunc

for RASIZE in 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 
do
    echo $RASIZE
    MDRASIZE=$[$RASIZE*$NUMDEVS]
    blockdev --setra $MDRASIZE /dev/$MDDEV
    blockdev --setra $RASIZE /dev/sd[$DEVS]
    blockdev --setra $RASIZE /dev/sd[$DEVS]1

    echo 3 > /proc/sys/vm/drop_caches
    hdparm -t /dev/$MDDEV | grep "Timing"
    #echo 3 > /proc/sys/vm/drop_caches
    #dd if=/dev/$MDDEV of=/dev/null bs=1M count=1024 | grep "bytes"
done

for read performance this script will give you the best read ahead value. then put the blocdev command in your /etc/rc.local file to make it persistent

0

Software RAID-5 is notoriously slow, and as DukeLion has pointed out, any other load on the system - CPU or IO - will slow it down even more. If you must do RAID-5, I would very strongly advise you to do it in hardware (and proper hardware, not one of these cards where the driver does it in software with the system CPU); or shrug, prepare to lose a third of your array, and do software RAID-1 (which is just fine, performance-wise).

Edit: I didn't say RAID-5 was slow; I said RAID-5 in software was slow. If you compare IO stats from your other (decent) system with those from a hardware RAID-5 I think you'll find that the difference is noticeable. I have a new system here with a decent hardware RAID card (Dell H700) and I'd be happy to do some timing tests with you, if you'd like.

I accept that it's frustrating that your two software RAID-5s perform differently, but without knowing a lot more about the physical and logical setup of each, it's difficult to make tuning recommendations, and I'm not sure I'd bother, since tuning software RAID-5 is, in my experience (and please pardon the expression), a bit like gilding a turd.

4
  • I do not agree that RAID5 is slow. I have another identical system which performs good enough. Bot of them are new. There is nothing running on the system other than dd currently. This machine is being used as a backup mirror so there seemed to be less need for an investment in a hardware RAID5 card
    – proy
    May 18, 2012 at 9:26
  • I have added more details of my 2 TB HDD partitioning
    – proy
    May 18, 2012 at 9:43
  • Hardware raid5 is faster just 'cause pricey controller can deal with write hole using a battery backed cache. Software RAID implementation need to live with write hole, so developers try to do it as short as possible, sacrificing performance.
    – DukeLion
    May 18, 2012 at 9:43
  • 1
    RAID-5 in software is plenty fast. RAID-5 writes are throttled by the fact that even if parity stripe updates are cached there is a hard limit at which all of those cached stripe updates are flushed to disk, but RAID-5 reads (particularly sequential reads) are often extremely fast EVEN FOR SOFTWARE RAID-5. The only advantages to a dedicated hardware RAID controller today are offloading parity calculations, huge on-card caches, and on-card battery backup. For all but the biggest iron imaginable this advantage is not worth the added cost in a modern computer with a remotely decent CPU. Oct 23, 2017 at 18:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .