I have a 6 disk raid6 mdadm array I'd like to benchmark writes to:
root@ubuntu:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sda[0] sdf[5] sde[4] sdd[3] sdc[2] sdb[1]
1953545984 blocks level 6, 64k chunk, algorithm 2 [6/6] [UUUUUU]
Benchmarks can be inaccurate because of cache - for example, notice the write speed here is higher than it should be:
root@ubuntu:/mnt/raid6# dd if=/dev/zero of=delme bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.276026 s, 380 MB/s
Now we can disable each disk cache pretty easily:
root@ubuntu:~# hdparm -W0 /dev/sd*
/dev/sda:
setting drive write-caching to 0 (off)
write-caching = 0 (off)
/dev/sdb:
setting drive write-caching to 0 (off)
write-caching = 0 (off)
/dev/sdc:
setting drive write-caching to 0 (off)
write-caching = 0 (off)
/dev/sdd:
setting drive write-caching to 0 (off)
write-caching = 0 (off)
/dev/sde:
setting drive write-caching to 0 (off)
write-caching = 0 (off)
/dev/sdf:
setting drive write-caching to 0 (off)
write-caching = 0 (off)
But there is still Linux caching:
root@ubuntu:/mnt/raid6# dd if=/dev/zero of=delme bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.00566339 s, 1.9 GB/s
To disable Linux caching, we can mount the filesystem synchronously:
mount -o remount,sync /mnt/raid6
But after this writes become way slower than they should be:
root@ubuntu:/mnt/raid6# dd if=/dev/zero of=delme bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 23.3311 s, 449 kB/s
It's as if mdadm requires async mounts in order to function. What's going on here?