I have set up a RAID5 array using 4 disk partition. All disks are 2TB. When the RAID5 array was initially syncing it was too slow. So I did a

echo 32768 > /sys/block/md2/md/stripe_cache_size

and then it was quiet fast. But still it took a days time to finish syncing. Now I want am doing a

dd if=/dev/zero of=/dev/md2 bs=1024k

onto the RAID5 array and it is like a day over and yet it has not finished. Will this take too much time am I doing some thing wrong.

My raid details are here http://dpaste.com/749742/
Iostat details here http://dpaste.com/749761/ hdd partitioning details here http://dpaste.com/749765/

I have an identical machine with the similar details. But that's quiet faster.

UPDATE: I have added my hdd partitioning details.

  • Have you tried to play with dev.raid.speed_limit_min/dev.raid.speed_limit_max via sysctl ? – ALex_hha Mar 20 '16 at 19:07

Normal time to build such array is 5.5-6 hours. There are a few things that can slow down array rebuild process - concurrent IO on the same disk, high cpu usage on the system, faulty HDD or cables.

dd if=/dev/zero of=/dev/md2 bs=1024k - will slow down rebuild. If you really need to rewrite your array with zeroes - do it after array has built.

Check if you have load on other partitions - using atop or iostat -nx 1

Check your load average in atop or top

Check for HDD errors in system logs and with smartctl.

| improve this answer | |
  • RAID rebuild is over. I started dd only after rebuild – proy May 18 '12 at 9:02
  • can you show dd's speed. use killall -USR1 `pgrep dd` to see it – DukeLion May 18 '12 at 9:04
  • this would kill my dd process right ? – proy May 18 '12 at 9:29
  • no, it will just make it to write current statistics to it's output – DukeLion May 18 '12 at 9:44
  • NO HDD errors in log and smartctl also gives a clean chit – proy May 18 '12 at 9:51

in raid 5 you should look about raid chunk size. run : cat /proc/mdstat to check yours.

to change it run :

mdadm --grow /dev/md0 --chunk=16

this will rebuild your array, it takes long times to complete. you can read this for more informations about it : https://raid.wiki.kernel.org/index.php/RAID_setup#Chunk_sizes

then you can test again your write performance with dd for exemple.

I've made a small script to test read ans write :


MDDEV="`cat /proc/mdstat | grep md | head -1 | awk '{print $1}'`"
if [ -z "$MDDEV" ]
    echo "I can\'t find any md" 
    exit 1

DEVS="`cat /proc/mdstat | grep $MDDEV | tr " " "\n" | grep '^sd' | grep -v \(S\)$ | awk '{print substr($0,3,1)}' | tr -d "\n"`"


# write dd performance
echo 3 > /proc/sys/vm/drop_caches
dd if=/dev/zero of=/dev/$MDDEV bs=1M count=1024 conv=fdatasync,notrunc

for RASIZE in 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 
    echo $RASIZE
    blockdev --setra $MDRASIZE /dev/$MDDEV
    blockdev --setra $RASIZE /dev/sd[$DEVS]
    blockdev --setra $RASIZE /dev/sd[$DEVS]1

    echo 3 > /proc/sys/vm/drop_caches
    hdparm -t /dev/$MDDEV | grep "Timing"
    #echo 3 > /proc/sys/vm/drop_caches
    #dd if=/dev/$MDDEV of=/dev/null bs=1M count=1024 | grep "bytes"

for read performance this script will give you the best read ahead value. then put the blocdev command in your /etc/rc.local file to make it persistent

| improve this answer | |

Software RAID-5 is notoriously slow, and as DukeLion has pointed out, any other load on the system - CPU or IO - will slow it down even more. If you must do RAID-5, I would very strongly advise you to do it in hardware (and proper hardware, not one of these cards where the driver does it in software with the system CPU); or shrug, prepare to lose a third of your array, and do software RAID-1 (which is just fine, performance-wise).

Edit: I didn't say RAID-5 was slow; I said RAID-5 in software was slow. If you compare IO stats from your other (decent) system with those from a hardware RAID-5 I think you'll find that the difference is noticeable. I have a new system here with a decent hardware RAID card (Dell H700) and I'd be happy to do some timing tests with you, if you'd like.

I accept that it's frustrating that your two software RAID-5s perform differently, but without knowing a lot more about the physical and logical setup of each, it's difficult to make tuning recommendations, and I'm not sure I'd bother, since tuning software RAID-5 is, in my experience (and please pardon the expression), a bit like gilding a turd.

| improve this answer | |
  • I do not agree that RAID5 is slow. I have another identical system which performs good enough. Bot of them are new. There is nothing running on the system other than dd currently. This machine is being used as a backup mirror so there seemed to be less need for an investment in a hardware RAID5 card – proy May 18 '12 at 9:26
  • I have added more details of my 2 TB HDD partitioning – proy May 18 '12 at 9:43
  • Hardware raid5 is faster just 'cause pricey controller can deal with write hole using a battery backed cache. Software RAID implementation need to live with write hole, so developers try to do it as short as possible, sacrificing performance. – DukeLion May 18 '12 at 9:43
  • 1
    RAID-5 in software is plenty fast. RAID-5 writes are throttled by the fact that even if parity stripe updates are cached there is a hard limit at which all of those cached stripe updates are flushed to disk, but RAID-5 reads (particularly sequential reads) are often extremely fast EVEN FOR SOFTWARE RAID-5. The only advantages to a dedicated hardware RAID controller today are offloading parity calculations, huge on-card caches, and on-card battery backup. For all but the biggest iron imaginable this advantage is not worth the added cost in a modern computer with a remotely decent CPU. – Jody Bruchon Oct 23 '17 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.