The partition is in RAID1 using software RAID.

    Command line used: iozone -s 190g -r 300k -O -b test_file
    OPS Mode. Output is in operations per second.
                                                            random  random    bkwd   record   stride                                   
          KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
   199229440     300     240     234      254      253      58     127     100     6539       91      234      234     254      254

so for 300K blocks random read using one thread was 58 ops/second of about 17MB/s.

But when I tried 5 threads each reading 300K blocks, unexpectedly for me it worked faster, although I'd expect more random access:

    OPS Mode. Output is in operations per second.
    Command line used: iozone -s 80g -r 300k -O -l5 -u 5 -F test_file1 test_file2 test_file3 test_file4 test_file5

    Children see throughput for 5 random readers    =      69.27 ops/sec
    Parent sees throughput for 5 random readers     =      69.27 ops/sec
    Min throughput per process                      =      13.81 ops/sec 
    Max throughput per process                      =      13.89 ops/sec
    Avg throughput per process                      =      13.85 ops/sec

Does anyone knows what reading/caching policies in Linux OS or hard drives makes 5 thread reading random 300K blocks faster than 1 thread reading random 300K blocks?

  • The subject says 300k files (unspecified size) and the question says 300k blocks. Did you do the reads to a single large file (which had at least 300k blocks)? Or something else? Can you provide more specific detail of the files that were created for the test, and the size of the entire potential surface? – Matthew Ife Apr 13 '14 at 15:50
  • it's reading blocks of 300K. iozone creates files and read blocks of specific size, it is a HD benchmark software. – Mladen Adamovic Apr 13 '14 at 17:50
  • How come you have different output for two tests? The first test suggests you were doing random writes yet the second test does not indicate that? – Matthew Ife Apr 13 '14 at 22:17
  • I've been able to duplicate this on my own machine using fio. What kernel version are you using? – Matthew Ife Apr 13 '14 at 23:15
  • 3.2.0-60-generic I think the underlying reason might be RAID1 with partition, with 2+ threads one thread reads from one drive, another from another drive (or something like that theoretically). – Mladen Adamovic Apr 14 '14 at 21:51

What is surprising?

1 thread means command, wait, read, loop. There is a wait element in - the latency.

Multiple threads eliminate that and - allow the discs to be more optimized thanks to queueing of multiple commands. That you get more IOPS with 3 threads is not surprising at all.

  • This might be correct. fread and read has a parameter which specifies file size, so hard drive controller can know for each block which clusters it has to read. When having multiple threads reading, it can benefit from optimizing the head movement between each blocks. – Mladen Adamovic Apr 16 '14 at 7:08
  • Even if not. There is a thing called NCQ (SATA) or TCQ (SAS) - drives get multiple commands. Multiple threads thus can be handled in a more optimal fashion. Stnadard Admin 101 knowledge. – TomTom Apr 16 '14 at 7:33

It's a raid1, so writes go to both disks, but reads can be satisfied with one, so with more threads, I'd expect better performance. But it might be best with threads equal to the number of disks in the raid 1 (or the raid 1 part of a raid 10).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.