3

Since this afternoon something is wrong with the server. On the server side I see messages in dmesg as follows:

statd: server rpc.statd not responding, timed out
lockd: cannot unmonitor <client>
statd: server rpc.statd not responding, timed out
lockd: cannot monitor <client>

On the client side I see in dmesg:

lockd: server <server> not responding, still trying
lockd: server <server> OK

This is paralysing the entire network! I have tried this solution suggested by Xian, but it makes no difference.

Server, Debian Linux, Squeeze 64-bit:

>> uname -a
Linux <server> 2.6.32-5-amd64 #1 SMP Fri May 10 08:43:19 UTC 2013 x86_64 GNU/Linux

Clients, Linux Mint 13-64bit:

>> uname -a
Linux <client> 3.2.0-49-generic #75-Ubuntu SMP Tue Jun 18 17:39:32 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

I have not run an update on the server, so I don't know what could have changed. I did upgrade one of our client machines, but can't see why that would mess with the server, since all machines seem affected. Any ideas on how to fix this?

UPDATE 1

The server stalls for a while at

Starting portmap deamon
Starting NFS common utilities: statd idmapd

This takes about 2 minutes until boot continues...

UPDATE 2

It is indeed the client machine that was upgraded that caused this. It seems it somehow stalled statd on the server, causing all other machines to have issues. I rebooted the entire network, leaving that one machine off and I did not encountered any problems. Not really a fix, but I have since downgraded that machine again, and everything seems to be stable.

12
  • 1
    Are you sure the physical network is okay? Switches? Cables? Jul 31, 2013 at 0:03
  • What does the server use for storage, is it mounting an external nfs mount, or is it serving nfs from local storage? check your available devices on the server with df, if it hangs your storage is probably offline / dead / unplugged. Jul 31, 2013 at 0:16
  • @user16081 The storage is internal raid drive. df comes back normal.
    – Markus
    Jul 31, 2013 at 0:23
  • @DavidSchwartz I believe the network is ok. I can still log into the server remotely, without disconnects. Not sure how else to test the network, as it is all locked away somewhere.
    – Markus
    Jul 31, 2013 at 0:24
  • noticing you upgraded 1 client, check this, last comment, to see if it might apply to your case? forums.gentoo.org/viewtopic-t-850532-start-0.html Jul 31, 2013 at 0:26

4 Answers 4

2

Here comes couple of suggestions:

I once managed to break the loopback interface (lo) and thanks to it several services, such as NFS, stopped working properly. See with ifconfig if you still have your beloved lo interface up and running. If it's not, go see /etc/network/interfaces and see what's going on.

Also as some people already mentioned, check the commands pgrep -v statd and netstat -tlnpu to see if statd is running.

Or perhaps someone has changed something under /etc at the server side? If you do not have /etc under version control, see if any files have been recently modified: find /etc -mtime -14 would show files changed during last 14 days, for example.

1

Have a look here : http://sophiedogg.com/lockd-and-statd-nfs-errors/

Try :

# /etc/init.d/nfs-common stop
# /etc/init.d/nfs-kernel-server stop
# rm -rf /var/lib/nfs/statd/sm/*
# rm -rf /var/lib/nfs/statd/sm.bak/*
# /etc/init.d/nfs-common start
# /etc/init.d/nfs-kernel-server start

I had the same problem, and this solved it... but for only one month. I don't know why for now. I had to delete the files again today.

0

I had the same issue on a nfs debian squeeze server, and it seemed to be triggered by some new clients too (Fedora 20). Downgrading the clients was not an option for me, after some long, painful and unsuccessful debugging I ended up discovering a (different and probably unrelated) readdir loop bug on nfs exported ext4 filesystem with lots of file in it similar to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1240143

(I could be wrong, from the little I understood this was fixed on recent kernels, so debian squeeze may be affected)

long story short, to get rid at least of THAT bug I upgraded my nfs server to debian wheezy (forcing the nfs version to 3) and now (with the same filesystem and the same clients) it's been a week without the "cannot monitor"/"not responding" issue (before the upgrade it was a daily thing)

0

This worked for my case :

https://lists.debian.org/debian-user/2004/10/msg00932.html

Just edit the /etc/init.d/halt script, at the end there should be the line

halt -d -f -i $poweroff $hddown

The "-i" option makes all network interfaces to be shutdown, but this >seems to be too early for diskless clients, just try to remove this option, so

halt -d -f $poweroff $hddown

Note that my problem was with NFS on client with disk.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .