Of course I need to solve a particular problem, but I'm hoping some more general answers on "how to troubleshoot automounting nfs" come up that will help more people with similar symptoms.
We've got a file server 'nas1' and three client servers, 'server1', 'server2', 'server3' with identical /etc/auto.master files including this line:
/net -hosts acregmax=10,acdirmax=10
All servers run CentOS 5.
'nas1' has the following in /etc/exports:
/proj/ptn *(sync,rw,no_root_squash)
One of the servers has been up and running without any issues, but on two occasions, separated by several weeks, we've had issues with the mount points appearing empty on the client servers.
For the case where 'server3' is unable to use '/net/nas1/proj/ptn', here are a few things that may help with troubleshooting:
ssh nas1
Works as expected, so it's apparent that the connection is not lost between. It didn't occur while the nfs mount was inaccessible, but it's worth mentioning that we have had intermittent issues (averaging maybe once a week) with a server becoming unresponsive and having to power-cycle it to bring it back up. This becomes apparent since a load balancing server is checking the servers constantly for connectivity to keep track of its available pool of servers.
lsof | grep /proj/ptn
automount 3414 root 5r DIR 0,19 0 13390 /net/nas1/proj/ptn
cat /proc/mounts
included /etc/auto.misc /misc autofs rw,fd=7,pgrp=3414,timeout=300,minproto=5,maxproto=5,indirect 0 0 -hosts /net autofs rw,fd=13,pgrp=3414,timeout=300,minproto=5,maxproto=5,indirect 0 0 -hosts /net/nas1/proj/ptn autofs rw,fd=13,pgrp=3414,timeout=300,minproto=5,maxproto=5,offset 0 0
service autofs restart
Stopping automount: [ OK ]
Starting automount: [ OK ]
This didn't fix the problem, but allowed for unmounting:
umount /net/nas1/proj/ptn
mount /net/nas1/proj/ptn
mount: can't find /net/nas1/proj/ptn in /etc/fstab or /etc/mtab
The mount behaviour is expected, but it was tried, so I'm including it.
service network restart
Shutting down interface eth0: [ OK ]
Shutting down loopback interface: [ OK ]
Bringing up loopback interface: [ OK ]
Bringing up interface eth0: [ OK ]
Didn't help.
/etc/init.d/autofs restart
Stopping automount: [ OK ]
Starting automount: [ OK ]
This brought it back up.
Since 'service autofs restart' had been run before, it's unknown why '/etc/init.d/autofs restart' worked, unless it has to do with the fact the 'umount' and 'service network restart' were done in between.
Considering it may be weeks before we see this again, what's the best thing to try next time it goes down or right now to make sure that it never does go down again? One would hope AUTO mount would automatically fix the mount when it goes down :-(
