Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I was sorting some large files (91GB across 27 files) with GNU sort when I noticed that iostat -dxk 3 showed very slow read speeds, between 5 MB/s and 10 MB/s, with 100% disk utilization. I tried cat large-file > /dev/null, and got similar performance, only slightly higher. The same for cp large-file /tmp/, with /tmp on a separate disk. vim experiences the same, as well as scripts I write in Ruby reading files, if that helps. Write speed is fine and fast though.

EDIT: It looks like these operations are only slow on files in a certain directory. The same operations on other files in a sibling directory (same disk partition), end up being fast, with above 90 MBPS read speed. This makes no sense to me. Could it possibly be due to the manner in which these files were constructed? I created them by reading in a lot of other files, and writing each line into an appropriate "bucket file", depending on the first character in the line (so a-z, and a single file for others). So I was pretty much simultaneously appending lines to 27 files, one at a time, through 8 processes while reading a couple thousand files. Could this cause the sequential order the blocks representing a file to be out of order instead? Hence the slow sequential reads afterwards?

However, I tried using fio to measure sequential read performance, and it clocked in at 73 MB/s. Also notable is that my boss got proper read speeds when downloading some files via FTP from the same machine.

So I'm guessing this is some configuration issue somewhere, but I have no idea where. What could the reason be and how can I try to fix it?

Edit: This machine is running under Citrix Xen virtualization.

Edit: Output of iostat -dxk while sort is loading a large file into its buffer (get similar output for cat/cp):

Device:         rrqm/s   wrqm/s   r/s   w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
xvdb              0.00     0.00 1000.00  0.00  6138.61     0.00    12.28    24.66   24.10   0.99  99.41
xvdb1             0.00     0.00 1000.00  0.00  6138.61     0.00    12.28    24.66   24.10   0.99  99.41
xvda              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
xvda1             0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

Edit: Further performance degradation after a few hours (with breaks for the disk when sort was processing). It almost looks like random IO, but there's only a single sort operation going on, with no other processes doing any IO, so reads should be sequential =/ :

Device:         rrqm/s   wrqm/s   r/s   w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
xvdb              0.00     0.00 638.00  0.00  2966.67     0.00     9.30    25.89   40.62   1.57 100.00

Device:         rrqm/s   wrqm/s   r/s   w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
xvdb              0.33     0.00 574.67  0.00  2613.33     0.00     9.10    27.82   47.55   1.74 100.00 

Device:         rrqm/s   wrqm/s   r/s   w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
xvdb              0.00     0.00 444.33  0.00  1801.33     0.00     8.11    28.41   65.27   2.25 100.00
share|improve this question
    
How much free space in the vdisk and the underlying storage? What's the underlying storage? –  Rob Olmos Nov 20 '10 at 22:58
    
Lots of free space, and just a single, regular spinning HDD for stoarge. Thanks for inquiring, but the problem is most definitely the file fragmentation, which mattdm suggested below. –  ehsanul Nov 20 '10 at 23:18

2 Answers 2

up vote 3 down vote accepted

Are your slow files highly fragmented? Run /usr/sbin/filefrag -vfilename to find out. You'll get output like:

Checking big.file
Filesystem type is: ef53
Filesystem cylinder groups is approximately 4272
Blocksize of file big.file is 4096
File size of big.file is 96780584 (23629 blocks)
First block: 88179714
Last block: 88261773
Discontinuity: Block 6 is at 88179742 (was 88179719)
Discontinuity: Block 12233 is at 88192008 (was 88191981)
Discontinuity: Block 17132 is at 88197127 (was 88196911)
Discontinuity: Block 17133 is at 88255271 (was 88197127)
big.file: 5 extents found, perfection would be 1 extent

or perhaps much worse.

You mention that the system is running under virtualization. Is this access with a virtual disk image file?

share|improve this answer
    
This is exactly it. Found 1188454 discontinuities in a 7GB file. The reason I'm guessing is what my latest edit in bold in my question says. I made the files by appending line by line on all 27 files simultaneously, while also reading other files from the same disk. Does that sound right? I guess I'll have to do a less naive scheduling of read/writes in my scripts, at least make them work in bulk. Thanks so much! –  ehsanul Nov 20 '10 at 22:57
    
Cool — glad I could help! –  mattdm Nov 20 '10 at 23:01

Soo, i believe this is a simple case of not enough RAM...

For a start when reading/writing a file smaller than your RAM ( every thing is fast - like your "fio" test..)

Once you start working with data larger than your OS can cache, your OS cache begins to swap ( sometimes even to disk) ( in fact you should check your ram usage when you have 4mb's read speed)

It sounds like something ive experience before, your getting slow speed for reading such a large file.. ( ive seen a DB do exactly the same when it was using large index's that didnt fit into RAM )

Given also the overhead of working on a VM ( this sounds very typical to me )

I would check that your disks arent swapping or your active memory isnt full ( and let me know ) :D

share|improve this answer
    
Good try, but this is not the case here. I've got 16GB of RAM, and all my "large" files are smaller, upto 11GB (which is on purpose btw). I've actually got plenty of free memory, and am not swapping at all. In fact, with the sort I'm doing now, I've specified with the -S option a buffer size of 12GB, so that sort itself doesn't make files in /tmp as it usually does given large files, and instead makes use of the free memory. Also, this slowness in cp/cat/sort occurs with small files as well. Also, look at the commands like cat file > /dev/null, which take no memory. –  ehsanul Nov 20 '10 at 6:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.