Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I'm checking the disk speed of a hdd on CentOS 6 using dd command. The setup is a DRBD setup - with this server being primary. To ensure that I get correct values, I execute the dd command 3 times with different output files and then take average of the read/write time. However the first reading of the dd command is order of magnitude slower than the next 2. Eg.

time -p dd if=/dev/zero of=/mailstore/testfile bs=16k count=16384
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 88.5175 s,
3.0 MB/s
real 90.12
user 0.00
sys 0.66

time -p dd if=/dev/zero of=/mailstore/testfile1 bs=16k count=16384
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 0.226015 s, 1.2 GB/s
real 0.30
user 0.00
sys 0.22


time -p dd if=/dev/zero of=/mailstore/testfile2 bs=16k count=16384
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 0.22094 s, 1.2 GB/s
real 0.22
user 0.00
sys 0.21

Is this normal ? Should I ignore the first reading and take 3 more after that ?

As suggested by poige, the dd command gives consistent output using the opflag=direct option. Eg.

time -p dd if=/dev/zero of=/mailstore/filetest33 bs=16k count=16384 oflag=direct
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 296.587 s, 905 kB/s
real 296.61
user 0.03
sys 1.07

time -p dd if=/dev/zero of=/mailstore/filetest44 bs=16k count=16384 oflag=direct
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 260.308 s, 1.0 MB/s
real 260.42
user 0.04
sys 1.13

time -p dd if=/dev/zero of=/mailstore/filetest56 bs=16k count=16384 oflag=direct
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 253.681 s, 1.1 MB/s
real 253.68
user 0.03
sys 1.06
share|improve this question
    
You may as well try other (bigger) bs'es, the results may vary. –  poige Jan 20 at 15:59

2 Answers 2

up vote 5 down vote accepted

dd has direct option which requires kernel to bypass any caching and send data directly to block device. If you need benchmarking your device, not RAM, you have to use it, for e. g.:

dd if=/dev/zero of=direct_output bs=1M count=100 oflag=direct

share|improve this answer
    
Does it mean that first time caching did not happen but second and third time it happened ? –  amolkul Jan 20 at 9:01
    
@amol, it means you're measuring incorrectly, I don't want to guess. Use direct then we'll see. –  poige Jan 20 at 9:55
    
Isn't the sync command worth it as well? –  SamK Jan 20 at 13:50
    
@SamK, not too much — why blow the caches anyway if you don't need'em. –  poige Jan 20 at 14:19
1  
dd with oflag=direct is safe to use on production servers right ? –  amolkul Jan 23 at 9:24

So, you said it was mounted over the network. Which filesystem, CIFS or NFS? Either way, I suspect you may've been delegated the file. When your client gets a file delegated, it's able to cache writes locally. However, when you use O_DIRECT (that's what oflag=direct means), the writes are sent to the server immediately instead of any caching.

Either way, there's something strange going on. You should be getting more than 1.1MB/s unless you're actually using just 10Mbps ethernet.

Also, you can get a mid-transfer update for speed by running, in another window, killall -USR1 dd If you check the dd man page, you'll see that the USR1 signal does not kill/stop dd but print out I/O statistics. I often do this when waiting for large disk transfers:

while sleep 10; do killall -USR1 dd; done

Oh, and if you suspect disk caching, use this command to flush the readcache: echo 1 > /proc/sys/vm/drop_caches

Good luck!

share|improve this answer
    
Actually its a DRBD setup. The server I'm testing is the primary. I'll edit the question too. –  amolkul Jan 21 at 11:31

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.