Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have an NFS3 server with multiple clients. Each client is sequentially reading a different large file, and performance is very poor.

Here is what I am observing in iostat on the server for the disk where the files reside:

Device: rrqm/s  wrqm/s    r/s   w/s  rMB/s  wMB/s avgrq-sz avgqu-sz  await svctm  %util
sdX      24.33    0.00  712.67 0.00  18.41   0.00    52.91    11.95  16.91  1.40 100.00

As you can see, %util is 100%. At the same time, the aggregate I/O throughput (rMB/s+wMB/s) is about 18MB/s, which is 10-20 times slower than what the disk is capable of.

This, and the ratio of rMB/s to r/s, lead me to conclude that instead of reading large chunks of each file at a time, NFS ends up reading the files in smallish chunks with lots of interleaving of chunks between different files. This in turns leads to lots of disk seeks, killing performance.

Would you say the conclusion is justified by the evidence?

What would you recommend as a way to address this? I can change the reading app, and can tweak NFS settings on both the server and the client. I am using RedHat 5.6 with kernel 2.6.18, which I believe limits rsize to 32KB (I'd be happy to be proved wrong on this).

edit: This is how things look when there's only a single client reading a single file:

Device: rrqm/s  wrqm/s     r/s   w/s    rMB/s  wMB/s avgrq-sz avgqu-sz  await svctm  %util
sdX     343.33    0.33 1803.33  0.67   105.78   0.00   120.09     0.91   0.50  0.31  56.47

As you can see, the throughput is a lot better, and %util is also much lower.

share|improve this question
    
You have provided the output of your local block device which shows a read load - could you describe what you are actually measuring here? –  pfo Nov 9 '11 at 13:57
    
Not sure I understand the question. The client apps are achieving very low I/O throughput, and the bottleneck turned out to be the sdX device in the NFS server (see my post). Specifically, "rMB/s+wMB/s" is extremely low yet "%util" is at 100%. Let me know if this doesn't clarify things. –  NPE Nov 9 '11 at 14:02
    
If you stop all but one of the clients, does the throughput increase? –  sciurus Nov 9 '11 at 14:43
    
@sciurus: Yes, a single client reading a single file easily saturates a gig-E link. In iostat, rMB/s is in excess of 100 MB/s, and %util is about 50%. I've edited the post to include this. –  NPE Nov 9 '11 at 15:27
    
Makes sense. I think NFS isn't the main factor here; you'd probably see a similar degradation in performance if this was all local i/o. –  sciurus Nov 10 '11 at 3:08

1 Answer 1

Faster disks, more memory in the box. I think your conclusion is right - you're seek bound.

How much memory does your NFS server have vs. your working set? Will your working set fit into cache?

What is the backend storage? You say it does ~180-360MB/sec throughput but how does it perform for random I/O? I'd suggest using something like fio to get an idea. seekwatcher is also fun to visualize the I/O. But if you can avoid hitting the disks so much the better.

share|improve this answer
    
Thanks for this. I'll conduct some experiments directly on the box to see whether the same thing happens if I read the files locally. The real point is that there's no reason for the reading to be seek-bound: I am sequentially reading a fairly small number of large files. –  NPE Nov 10 '11 at 8:30
    
How many clients, and how many files are they reading? In my experience with NFS servers it doesn't take many clients to turn sequential (from each clients perspective) into very random I/O for the server. –  James Nov 15 '11 at 1:43

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.