I can't figure out why I'm getting such terrible times out of my mdadm and in particular the lvm partitions in it.

I made the raid:

mdadm --create --verbose /dev/md0 --level=5 --chunk=1024 --raid-devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1

# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 sda1[0] sdd1[3] sdc1[2] sdb1[1]
2930279424 blocks level 5, 1024k chunk, algorithm 2 [4/4] [UUUU]

I then created the physical volume, volume group, and logical volumes, I then formatted the logical volumes to ext4 using the following commands I got from here: http://busybox.net/~aldot/mkfs_stride.html

mkfs.ext3 -b 4096 -E stride=256,stripe-width=768 /dev/datavg/blah

Now I'm confused, I had these lvs running real quick before in mdadm but now that I've 'optimized' everything it's slower, eg, before:

Timing buffered disk reads: 598 MB in 3.01 seconds = 198.85 MB/sec

but now after:

Timing buffered disk reads:  198 MB in  3.00 seconds =  65.96 MB/sec

That's pitiful! What's happened here? Did I not follow the instructions correctly? Can i reshape the ext4 partitons to default back to what they were? (I used defaults before and they were fine!)

  • Why not simply recreate the array? Don't you have backups? – Steven Monday Nov 6 '10 at 15:42
  • I can do that, I've got backups, but I'm first wondering where I went wrong using the above settings - not much point making the new array if I'm going to put bad settings in it again. It's a time consuming exercise so trying to avoid it – beatbreaker Nov 7 '10 at 21:41

I haven't performed many tests, but I noticed that RAID5 with even number of devices is slow - there are read-modify-writes even with simple dd to md.

  • I've performed lots of tests, and I'm prepared to perform more if necessary. I know that somehow things are not optimised anymore, even though I followed optimisation steps to the letter when creating the raid etc... It was in MDADM and LVM before and worked really well, now again after recreating and it's terrible. I'm thinking of pulling it out of lvm and just having partitions in mdadm that I can grow would be good if it's faster – beatbreaker Nov 6 '10 at 22:46
  • Have you tried doing a simple dd to array (e.g. write) and watching the drives in atop? With default mdadm settings no matter what parameters I passed to dd I could see reads on drives (indicating R-M-W cycles). What's more, if you don't have a really fancy setup (lvm-on-lvm-on-network-block-device), mkfs should be able to pick the best possible settings by itself. And finally: if you're using ext, you could be limited by its metadata performance. Are you sure you have extent based allocation enabled? – Hubert Kario Nov 6 '10 at 23:53
  • I wrote this script for testing: paste.pocoo.org/show/287092 I've got 4G ram on the system so this just goes over showing real speeds. That's right, last time i did mkfs.ext4 the defaults it set its self were much better. I tried to pull one partition down and make it again but it still seems slow. As for the ext4 settings: Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize – beatbreaker Nov 7 '10 at 0:24
  • Ok I just removed the LV then recreated it again, dfisk, mkfs, same results :( – beatbreaker Nov 7 '10 at 0:35
  • 1
    Check the size of physical extent in LVM (pvdisplay or lvdisplay), you can see what extents it uses by using lvdisplay --map /dev/vg/lv, then move the offending fragments by pvmove -n /dev/vg/lv /dev/sdx0:from_start-from_end /dev/sdx0:to_start-to_end. man pvmove could be of use... – Hubert Kario Nov 10 '10 at 9:30

blockdev --getra /dev/datavg/audio == ? blockdev --getra /dev/md0 == ?

Also, I'd prefer testing with time dd if=...device... of=/dev/null bs=128M count=NNN iflag=direct

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.