I'm benchmarking disk write performance using this command:
dd if=/dev/zero of=/mnt/test.dmp bs=1M count=256 conv=fdatasync
After a random time and random number of attempts, the file system performance drops to 1/4 of the initial value (unmout, reformat file system and mount to retry).
I'm using Linux guest Ubuntu 14.04 (kernel 3.13.0-65-generic x86_64 and ). I used several different file systems (ext4, ext2, vfat)
It's not clear for me what is happening. I initially thought that it was due to some file system cache getting filled up, but I can't find a clear pattern.
Update 1:
Hardware: Dell PowerEdge R710, 64GB RAM, PERC H200, 2 SATA disks
Update 2:
dd in VMWare ESXi (via SSH) gives a constant 20 MBytes/s write rate on our storage. Guest is Linux 3.13.0-65-generic x86_64.
watch "grep -i writeback /proc/meminfo"
to see if it's OS cache. It could also be the disk controller cache. Also use much greater file sizes and write directly on the block device. There is no point in formating to write zeros.