4

i've got a problem removing mounts created with mount -o bind from a locally mounted NFS folder. Assume the following mount structure:

NFS mounted directory:

$ mount -o rw,soft,tcp,intr,timeo=10,retrans=2,retry=1 \
 10.20.0.1:/srv/source /srv/nfs-source

Bound directory:

$ mount -o bind /srv/nfs-source/sub1 /srv/bind-target/sub1

Which results in this mount map

$ mount
/dev/sda1 on / type ext3 (rw,errors=remount-ro)
# ...
10.20.0.1:/srv/source on /srv/nfs-source type nfs (rw,soft,tcp,intr,timeo=10,retrans=2,retry=1,addr=10.20.0.100)
/srv/nfs-source/sub1 on /srv/bind-target/sub1 type none (rw,bind)

If the server (10.20.0.1) goes down (eg ifdown eth0), the handles become stale, which is expected.

I can now un-mount the NFS mount with force

$ umount -f /srv/nfs-source

This takes some seconds, but works without any problems. However, i cannot un-mount the bound directory in /srv/bind-target/sub1. The forced umount results in:

$ umount -f /srv/bind-target/sub1
umount2: Stale NFS file handle
umount: /srv/bind-target/sub1: Stale NFS file handle
umount2: Stale NFS file handle

Here is a trace http://pastebin.com/ipvvrVmB

I've tried umounting the sub-directories beforehand, find any processes accessing anything within the NFS or bind mounts (there are none).

lsof also complains:

$ lsof -n
lsof: WARNING: can't stat() nfs file system /srv/nfs-source
      Output information may be incomplete.
lsof: WARNING: can't stat() nfs file system /srv/bind-target/sub1 (deleted)
      Output information may be incomplete.
lsof: WARNING: can't stat() nfs file system /srv/bind-target/
      Output information may be incomplete.

I've tried with recent stable Linux kernels 3.2.17, 3.2.19 and 3.3.8 (cannot use 3.4.x, cause need the grsecurity patch, which is not, yet, supported - grsecurity is not patched in in the tests above!).

My nfs-utils are version 1.2.2 (debian stable).

Does anybody have an idea how i can either:

  • force the un-mount some other way? (any dirty trick is welcome, data loss or damage neglible at this point)
  • use something else instead of mount -o bind? (cannot use soft links, cause mounted directories will be used in chroot; bindfs via FUSE is far to slow to be an option)

Thanks, Paul

Update 1

  • With 2.6.32.59 the umount of the (stale) sub-mounts work just fine. It seems to be a kernel regression bug.
  • The above tests where with NFSv3. Additional tests with NFSv4 showed no change.

Update 2

  • We have tested now multiple 2.6 and 3.x kernels and are now sure, that this was introduced in 3.0.x. We will fille a bug report, hopefully they figure it out.
6
  • 2
    Try umount -l Jun 7, 2012 at 13:24
  • @Dimitri: Sorry, did not mention this. I've tried, does not work. Neither does umount -lf. Jun 7, 2012 at 15:26
  • What happens if you unmount the /srv/bind-target/sub1 first and then is /srv/nfs-source?
    – quanta
    Jun 8, 2012 at 4:58
  • @quanta: Same thing. Also the trace looks exactly the same. Jun 8, 2012 at 9:22
  • Is the question on how to handle the current situation without rebooting, or on how to make the system resilient to that problem in the future? Jun 8, 2012 at 11:41

3 Answers 3

2

Something that worked for me in my particular setup, to get cleints working, was this:

I had an autofs tree with a mounted nfs fs on /fs/doom and another mounted on /fs/doom/localvol5. After server reboot it was possible to access /fs/doom and /fs/doom/localvol5/sub, but /fs/doom/localvol5 itself gave ESTALE on everything, including umount -f, -l, -fl.

What I did to get the client running without reboot was to move the whole /fs/doom hierarchy to another tree:

    mkdir /dev/shm/garbage-mount
    mount --move /fs/doom /dev/shm/garbage-mount

That moved the whole tree, and apparently works only because /fs/doom was accessible and a mountpoint. I sitll cannot unmount any of these filesystems, but I was able to restart autofs and get a fresh and working tree.

This should work with any autofs tree that has malfunctioning nfs subdirs.

Hope that helps.

0
1

You could simply mount the remote filesystem on /srv/bind-target/sub1.

If you're expecting this level of unavailability you should also specify the sync (although maybe defailt) option to NFS to lower the chance of having unwritten changes on your client

1
$ umount -l

Worked for me, before none of them were working:

$ mount -f ...
2
  • 5
    Posting blind commands is generally a poor way to answer questions - it encourages voodoo system administration. Please update your post to clarify what the commands do and why they're useful so the OP can actually learn something...
    – voretaq7
    Jun 17, 2013 at 19:08
  • 1
    He also mention over a year ago that he's tried this. Jun 17, 2013 at 19:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .