Lots of nfs4_reclaim_open_state: Lock reclaim failed entries in syslog

Question

I have an Ubuntu 11.10 VM set up in VMSphere. I'm storing some data on a nfs mount. The VM has been going down frequently. I haven't been able to pin the reason why down, but I think it has to do with this error:

Jan 19 09:53:07 ws-test-moodlearchive-01 kernel: [  384.523617] nfs4_reclaim_open_state: Lock reclaim failed!

It shows up in /var/log/syslog thousands of time. Most often after cron starts running.

I originally was saving the output of one cron job to a text file stored on the NFS mount. Switching that to the local disk seems to have reduced the number of errors, but it's still showing up.

Google has been been very unhelpful, nothing I found seemed to apply. Didn't find anything even close on this site, or StackOverflow.

So, what does that error mean? And how can I keep it from occurring?

Answer 1

The NFS server I was connecting to was running version 3. I was connecting with version 4. Switching to version 3 seems to have fixed the problem. I no longer see the nfs4_reclaim_open_state error in my syslog.

To make NFS use version 3 when connecting, I added nfsvers=3 to my fstab file. So an entry like this:

nfsserverip:/export/homes   /home       nfs  rw      0 0

Changed to:

nfsserverip:/export/homes   /home       nfs  nfsvers=3,rw      0 0

I still have not found out exactly what the error message was telling me. But at least I fixed it.

Answer 2

Actually this will not show in NFS3 because this is a NFS4 only code, NFS3 do not have this feature :) NFS3 have a different error recovery and it might be just hiding the problem.

Thist may happen when the NFS4 client get a complete action with some error and try to recover from it. When recovering, this error will show if the NFS tried to reclaim the lock and fail.

There are many reasons for the lock reclaim to fail, since some bug or races in the nfs server, to network problems. If you think this is a issue, you will have to do a tcpdump to catch the NFS traffic (client side preferred) and try to understand the request flow before the error show up, to understand first why some unknown NFS4 action was failed and then what happen during the lock reclaim

So the first thing to check is probably the network, check the cables, switch and port errors, duplicate IPs, bad bounding/LACP, packet lost, etc

Answer 3

NFS 4 servers are allowed to clear a client's locks (and all state information) if that client has not renewed them within the "NFS4 lease time". Once they are cleared, they cannot be "reclaimed," although NEW locks could be requested by the application once it learns of the error (which usually comes to it as EIO -- IO error). How well the client application reacts to such an event is up to the application code and it's skill at error recovery.

The lower that the NFS4 lease time is set, the more likely you are to have temporary communication gaps that are long enough to let an NFS Server clear the client's state information and locks. Linux NFS Servers usually default to 90 seconds, which is pretty good protection. However, people who don't fully understand the impact of lease time often drastically reduce this value. Plus, some non-linux NFS Servers default to much lower values. For example, NetApp NFS servers default to 30 seconds, which doesn't offer as much protection against short network outages.

So one cause / solution is: NFS Server's lease time is too low and needs to be raised.

Another cause: If the network between the client and server were highly reliable, there would be fewer chances of this kind of thing happening. So checking all network hardware / firmware for anything that degrades reliability can be important.

If a Cluster is in use, sometimes these errors correspond to events where a NFS Server cluster had to fail NFS export resources over to another node. NFS4 Server clusters that are very carefully designed can preserve lock status from one node to another, but most people's NFS4 clusters are not designed that well. So it is fairly common that EVERY cluster resource migration of NFS4 exports can cause locks (and other states) to be lost, causing clients to log these kinds of errors. Hence cluster design may need to be examined, to make sure that the necessary design and configuration to preserve locks and states (during failovers and migrations) is accomplished.

And yes, NFS v3 doesn't allow NFS Servers to clear locks just because the client didn't renew them (that concept doesn't even exist in v3), so this exact type of situation doesn't come up there.