5

We are a computer science department at a small university, running a RHEL 7 server using NFS4 and Fedora 24 clients (~40 client machines, ~150 users--rarely concurrent). We are having latency problems, and we're having trouble troubleshooting/trying to figure out what the problem is. Examples of the symptoms:

  • Emacs takes about 2 minutes to start/be usable. The GUI pops up quickly, but then the application hangs if you tried to open a file on startup. If you just try to open emacs and then try to open a file, the application hangs for two minutes. After about two minutes, you can create files, read files, etc., without issue. EDIT: running emacs on a local file (e.g., /tmp/test.out) does not have the latency issue. Also, opening networked files using idle3 or gedit has no issues.
  • Checking out a project using svn+ssh on the command-line is very slow from one of the Linux client/desktop machines--on the order of 3 minutes. If you checkout the project, using svn+ssh from another machine, checkout takes 3 seconds.
  • You cannot configure Idle. When you click on the configure menu, the application hangs. UPDATED: This seems to be a bug in idle3 that wasn't fixed in Fedora 24, but we were able to apply the fix.
  • When you click "open/browse" in an application (e.g., emacs, Eclipse) or open the file manager, the application will hang for awhile, while it retrieves files. Using ls and cd from the command line is fast.

After the long delays, you can read/edit/create files without issue.

The only commonality I have found with these applications is that they are using hidden configuration files (.emacs.d, .idle, .eclipse, ...). I can't seem to find any documentation that hidden files would be handled differently.

Any advice is appreciated!

1
  • How are you mounting NFS? These sound more like automounter problems. Are you using autofs or the fstab? Also, have you looked into reviewing NFS performance metrics with sar or netstat?
    – Spooler
    Nov 4, 2016 at 8:01

2 Answers 2

4

How I would troubleshoot this:

  1. if ~/.emacs.d/ is provided by a NFS mount, and
  2. the target file is a NFS mount provided by a second NFS server, and
  3. copying all of that local to the client removes the delay,

I'd then move one of them at a time back to NFS and try to recreate the problem.

On re-reading your original post, I realize I assumed you have two or more NFS servers, because that's what I've seen at previous employers. One NFS server provided home directories and a second provided binaries. We found running the binaries locally on the client improved performance. :-)

If you have one NFS server, are you able to set up a second one for troubleshooting? Perhaps the single NFS server is overwhelmed at certain times; working with a second NFS server might help isolate this case.

If the problem shows up on only one or two clients, I'd try to find what makes those clients unique. If the problem shows up on every client, I'd look at the NFS server.

Looking at the logs on the RHEL7 NFS server will help in any case.

A Google search for "NFS Troubleshooting" provided many useful pages, including tldp. There's also Red Hat's nfs server config. You have probably looked at both of those already.

You said the NFS server is Red Hat EL 7. If I had a support contract with my copy of RHEL, I would open a ticket with Red Hat and ask them to help troubleshoot, too.

I hope this helps. Good luck.


1
  • Thank you! This is good advice; it gives me some places to look since I'm a bit out of my element. We only have one NFS server serving home directories. Most of our binaries are hosted locally, but Eclipse and a few home-grown scripts are exported by the server. And, the problems seem to be on all clients, so I should focus on the server. I wasn't sure which log files to look at. Those links are quite helpful too! Thank you! Oct 29, 2016 at 3:06
0

What mount options do you use for your nfs? Removing options like lookupcache=none (and allow the default instead) will allow the clients to cache a lot more aggressively, we noticed this can confuse users when a file in their home directory is added and it takes an hour before it is visible on the remote machine, but for binaries turned out to be fine.

We also set actimeo=60 and noactime on our applications mount.

Home folders: rw,noatime,nfsvers=4,minorversion=1,soft,tcp,sec=sys,lookupcache=none,sloppy

applications/binaries: rw,noatime,nfsvers=4,minorversion=1,soft,tcp,sec=sys,actimeo=60,sloppy

see also the DATA AND METADATA COHERENCE section in the man page https://linux.die.net/man/5/nfs

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .