6

Just finished setting up a poor mans SAN with iSCSI and want to benchmark it. What are some good I/O performance benchmarks for Linux besides:

hdparm -tT /dev/sda1

How do I get a IOPS measurement?

Thanks.

1

7 Answers 7

5

I'd recommend using bonnie++ for disk performance testing. It's made specifically for doing that sort of thing.

2
  • Ok, ran a bonnie++, but what in gods name do I do with the output? Is there a single number or score?
    – Justin
    Jan 7, 2011 at 9:46
  • Bonnie doesn't provide an IOPS measurement. It uses 1B and 16KB block size and you can only compute an approximative value of IOPS.
    – Zulu
    Sep 20, 2014 at 10:09
13

On at least Linux, all synthetic benchmarking answers should mention fio - it really is a swiss army knife I/O generator.

A brief summary of its capabilities:

  • It can generate I/O to devices or files
  • Submitting I/O using a variety different methods
    • Sync, psync, vsync
    • Native/posix aio, mmap, splice
  • Queuing I/O up to a specified depth
  • Specifying the size I/O is submitted in
  • Specifying I/O type
    • Sequential/random
      • If I/O is random you can specify what distribution you want to skew it in to be more realistic
    • Reads/writes or some mixture of the two
    • I/O recorded with blktrace can be replayed

The statistics it gives you cover

  • Amount of I/O generated in MBytes
  • Average bandwidth
  • Submission/completion latency with minimum, maximum, average and standard deviation
  • IOPS
  • Average queue depth

The list of features and output goes on and on.

It won't generate one unified number that represents everything at the end but if you're serious about understanding storage performance you will know that single number cannot explain all of what you need to know. Even Linus Torvalds thinks fio is good:

[G]et Jens' FIO code. It does things right [...] Anything else is suspect - forget about bonnie or other traditional tools.

Brendan Gregg (a Netflix Performance Engineer) has also mentioned fio positively:

My other favorite benchmarks are fio by @axboe [...]

PS: Are you about to publish benchmarks that you did using fio on a website/in a paper etc? Don't forget to follow https://github.com/axboe/fio/blob/master/MORAL-LICENSE !

2
  • The google plus link no longer works, not sure if it's archived somewhere
    – pipe
    Jun 7, 2020 at 18:39
  • @pipe You're right, that link hasn't worked since Google shut Google+ down and sadly it isn't archived. However you can see a fuller version of the quote over on unix.stackexchange.com/a/138516/109111 ...
    – Anon
    Jun 7, 2020 at 18:54
2

I can suggest you to read these two posts/articles:

http://www.linuxinsight.com/how_fast_is_your_disk.html http://www.linuxforums.org/forum/red-hat-fedora-linux/153927-iscsi-raid1-lvm-setup-poor-write-performance.html

In particular:

First, I would suggest you using a more accurate and controllable tool to test performance. hdparm was designed to change IDE device parameters and the test it does is quite basic. You can't also tell what's is going on when using hdparm on compound devices on LVM and iSCSI. Also, hdparm does not test write speed, which is not related with read speed as there are different optimizations for both (write back caches, read ahead and prefetching algorithms, etc).

I prefer to use the old&good dd command which allows you to fine control block sizes, length of tests and use of the buffer-cache. It also gives you a nice and short report on transfer rate. You can also choose to test buffer-cache performance.

Also, do realize that there are several layers involved here, including the filesystem. hdparm only tests access to the RAW device.

TEST COMMANDS I suggest using the following commands for tests:

a) For raw devices, partitions, LVM volumes, software RAIDs, iSCSI LUNs (initiator side). Block size of 1M is OK to test bulk transfer speed for most modern devices. For TPS tests, please use small sizes like 4k. Change count to make a more realistic test (I suggest long test to test sustained rate against transitory interferences). "odirect" flag avoids using buffer-cache, so the test results should be repeatable.

Read test: dd if=/dev/zero of=/dev/ bs=1M count=1024 oflag=direct Write test: dd if=/dev/ of=/dev/null bs=1M count=1024 iflag=direct

Example output for dd with 512x1M blocks: 536870912 bytes (537 MB) copied, 10.1154 s, 53.1 MB/s

The WRITE test is DESTRUCTIVE!!!!!! You should do it BEFORE CREATING FILESYSTEM ON THE DEVICE!!!! On raw devices, beware that the partition table will be erased. You should force the kernel to reread the partition table on that case to avoid problems (with fdisk). However, performance on the whole device and on a single partition should be the same.

b) For filesystem, just change the device for a file name under the mount point. Read test: dd if=/dev/zero of=/mount-point/test.dat bs=1M count=1024 oflag=direct Write test: dd if=/mount-point/test.dat of=/dev/null bs=1M count=1024 iflag=direct

Note that even accessing a file, we are not using the buffer-cache.

c) For the network, just test raw TCP sockets on both directions between servers. Beware of the firewall blocking TCP port 5001.

server1# dd if=/dev/zero bs=1M count=1024 | netcat 5001 server2# netcat -l -p 5001 | dd of=/dev/null

TEST LAYERS Now you have a tool to test disk performance for each layer. Just follow this sequence:

a) Test local disk performance on iSCSI servers. b) Test network TCP performance between iSCSI targets and initiators. c) Test disk performance on iSCSI LUNs on iSCSI initiator (this is the final raw performance of iSCSI protocol). d) Test performance on LVM logical volume. e) Test performance on large files on top of filesystem.

There should be a large performance gap between the layer being responsible for the loss and the following layer. But I don't think this is LVM. I suspect of the filesystem layer.

Now some tips for possible problems:

a) You didn't describe if you defined a stripped LVM volume on iSCSI LUNs. Stripping could create a bottleneck if synchronous writing were used on iSCSI targets (see issue with atime below). Remember that default iSCSI target behaviour is synchronous write (no RAM caching). b) You didn't describe the kind of access pattern to your files: -Long sequential transfers of large amounts of data (100s of MB)? -Sequences of small block random accesses? -Many small files?

I may be wrong, but I suspect that your system could be suffering the effects of the "ATIME" issue. The "atime" issue is a consequence of "original ideas about Linux kernel design", which we suffer in the last years because of people eager to participate in the design of an OS which is not familiar with performance and implications of design decisions.

Just in a few words. For almost 40 years, UNIX has updated the "last access time" of an inode each time a single read/write operation is done on its file. The buffer cache holds data updates which don't propagate to disk for a while. However, in Linux design, each update to inode's ATIME has to be updated SYNCHRONOUSLY AND INMEDIATELY to disk. Just realize the implications of interleaving sync. transfers in a stream of operations on top of iSCSI protocol.

To check if this applies, just do this test: -Read a long file (at least 30 seconds) without using the cache. Of course with dd!!! -At the same time, monitor the I/O with "iostat -k 5".

If you observe a small, but continuous flow of write operations while reading data, it could be the inode updates.

Solution: The thing is becoming so weird with Linux that they have added a mount option to some filesystems (XFS, EXT3, etc) to disable the update of atime. Of course that makes filesystems semantics different from the POSIX standards. Some applications observing last access time of files could fail (mostly email readers and servers like pine, elm, Cyrus, etc). Just remount your file system with options "noatime,nodiratime". There is also a "norelatime" on recent distributions which reduces obsolescence in "atime" for inodes.

Please, drop a note about results of these tests and the result of your investigation.

1

It depends on the purpose of the disk. The fastest and the simplest way is "dd" as tmow mentioned, but I would additionaly recommend iozone and orion.

  1. IOzone in my opinion is more precise in filesystem benchmarking than bonnie++
  2. Orion ("ORacle IO Numbers" from Oracle) is very scalable and can benchmark properly even very large/powerful storage, and I find it very useful in scaling storage for databases. (I collect results of orion from different disk arrays, disk controllers and raid configurations, and then compere them)
1
  • With IOzone you can tune the test to match your environment. This way you don't get theoretical performance, but the performance for your use case.
    – Christian
    Jan 7, 2011 at 13:17
1

Disk benchmarking:

$ dd if=/dev/zero of=/tmp/zero bs=1k count=100k

Network benchmarking:

$ yes | pv | ssh $host "cat > /dev/null"
4
  • 3
    Using SSH add's a CPU-bound component, so you can develop a bottleneck when testing on slow CPUs. Use netcat/socat instead and then you have next to no overhead.
    – Marcin
    Jan 7, 2011 at 14:22
  • Like the nice simple dd command - thats all I was after!
    – Codek
    May 16, 2012 at 10:26
  • 2
    Two problems here: Unless your machine has less than 100MB of free memory, your dd command will be simply writing straight to cache held in RAM. fsync/fdatasync/dsync options will ensure the data is written to disk but it's still a very rough estimate and won't report IOPS as requested by the OP - bonnie++ will give better results. Secondly, As @Marcin stated, ssh will introduce a CPU bottleneck and an overhead that you're not measuring. Results will differ dramatically in each environment. Sep 28, 2013 at 20:21
  • SSH is really not a benchmark tool. Value returned is not really significative of something.
    – Zulu
    Sep 20, 2014 at 10:13
0

In addition to the others, you could also use postmark benchmark

0

You can also use the native Linux Disk utility to perform disk speed tests without entering commands into the Terminal. Here is my solution on ubuntu 21:

Step 1: Open the Disk Utility

Step 2: Select the disk for the test

Step 3: Run the Benchmark Feature

Step 4: Perform Tests

I also use this source that complains graphical method and fio.https://cloudzy.com/knowledge-base/test-disk-speed-in-linux/

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .