Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

NFS had been working great for me for the last years, but I'm now confronted to a performance issue to which I can't actually find a solution.

My problem is that, on the NFS server I have around 5Gb of small files and when, from a client, I do a "ls" or a "du" into the mounted directory, it can take more than 2 minutes to list all the files.

I think the problem is coming from the fact that for every single file, NFS will send a query for the file's stats, wait for the response and then send a new query for the following file. If this is the case, I'm pretty certain this is what's causing my bad performance issue.

Now, I tried looking for a solution to this, but I did not manage to find one, so I decided to open this thread.

Does any of you have any idea of how I could fix my performance issue?

Many thanks from a linux sys-admin padawan.

share|improve this question
    
How often do these files change? How many files are there per directory? NFS has to send stat requests and this can really eat up a lot of network overhead if you have a lot of files per directory. Also see: publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/… –  jeffatrackaid Nov 20 '12 at 15:24
    
NFSv4 adds extensive client caching capability which can alleviate the burden of subsequent runs. –  Steve-o Nov 20 '12 at 15:37
add comment

1 Answer

My feeling is that this problem isn't specific to NFS. Historically, UNIX filesystems in general have problems with flat directories containing large numbers of files. Certainly, the rule of thumb I was told many years ago is that performance degrades with the square of the size of the directory file. As you point out, doing an ls -la means stating each inode, and that takes a lot of time once the directory file starts to grow; the latency added by NFS will exacerbate this, but it's only bringing an underlying problem to your attention, not causing it.

The solution, as I constantly tell my developers, is not to store large numbers of files in shallow, wide structures, but in narrow, deep ones.

Look at how existing utilities store files when they have many to store: yum makes many files under /var/lib/yum/yumdb, so it stores them in subdirectories by leading initial:

drwxr-xr-x.   4 root root  4096 Sep  9  2011 C
drwxr-xr-x.   3 root root  4096 Sep  9  2011 M
drwxr-xr-x.   3 root root  4096 Jul 13 10:05 S
drwxr-xr-x.  24 root root  4096 Jul 13 10:05 a
drwxr-xr-x.  18 root root  4096 Nov  7 11:10 b
[c through y omitted to save space]
drwxr-xr-x.   5 root root  4096 Dec 28  2011 z

Squid cache, when initialised with squid -z, makes /var/spool/squid/0[0-F], and under each of these makes subdirectories ./[0-F][0-F]. innd pulls a similar trick, if memory serves, when it isn't using a ring-buffer-type file structure. All these daemons, and many other similar ones, know that if they need to store lots of small files, having a deep set of subdirectories to store them in is essential to efficient operation.

Edit: 1s is a very long time to take to do an ls on a single local directory. As I said, I think the NFS latency is exacerbating your issue; but it isn't responsible for the problem, only for making it big enough to cause you grief.

share|improve this answer
    
Very interesting indeed! I'll go see the devs and try to find a way to rearrange the files organisation. I'll keep you in touch. Thank you very much for the quick answer! –  Cocotton Nov 20 '12 at 15:35
    
Well actually, thinking about it, the problem doesn't really seem to come from the filesys, since if I do a "ls" on the server, it will take less than a second to get the output. To me, the problem seems to come from how NFS works (the cpu also get high on the client side). Would you agree with me on this one? –  Cocotton Nov 20 '12 at 15:40
    
Have your devs check out this: stackoverflow.com/questions/671260/… –  jeffatrackaid Nov 20 '12 at 15:40
    
Thank you for the edit Mad, very interesting! I'll go see the devs :) Jeff, I'll go take a loot at that right away –  Cocotton Nov 20 '12 at 15:55
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.