4

I have a ZFS dataset that exists in /proc/mounts and in /etc/mtab, but the folder doesn't exist in the filesystem, zfs umount reports the dataset doesn't exist, and zfs mount reports that the dataset is already mounted.

This is on RHEL 8.8 (fully updated) running zfs dkms 2.1.11-1

Is it possible to umount this dataset without rebooting the server? How would I even begin to troubleshoot this sort of problem?

I believe this occurred when it had the sharenfs setting enabled on the dataset and then a zfs umount was attempted while the nfs daemon was sharing. I have tried setting sharenfs=off (which succeeds, but doesn't fix the problem). The nfs services have been shut down and the problem still occurs. I can zfs send the datasets to another system and verify they're intact, but I just need them mounted on this server.

I am not using containers or additional mount namespaces (that I'm aware of).

The mounts are listed in /etc/mtab and /proc/mounts

topher@nzxt:/$ grep Backups -H /etc/mtab /proc/mounts
/etc/mtab:array/Backups /array/Backups zfs rw,seclabel,nosuid,nodev,noexec,noatime,xattr,noacl 0 0
/etc/mtab:array/Backups/gwen /array/Backups/gwen zfs rw,seclabel,nosuid,nodev,noexec,noatime,xattr,noacl 0 0
/proc/mounts:array/Backups /array/Backups zfs rw,seclabel,nosuid,nodev,noexec,noatime,xattr,noacl 0 0
/proc/mounts:array/Backups/gwen /array/Backups/gwen zfs rw,seclabel,nosuid,nodev,noexec,noatime,xattr,noacl 0 0

The directory doesn't exist in the filesystem

topher@nzxt:/$ ls /array/Backups
ls: cannot access '/array/Backups': No such file or directory

The dataset do exist in zfs (pool is array)

topher@nzxt:/$ zfs list array/Backups array/Backups/gwen
NAME                 USED  AVAIL     REFER  MOUNTPOINT
array/Backups       2.62T  1.59T     1.71T  /array/Backups
array/Backups/gwen   364G  1.59T      364G  /array/Backups/gwen

Umounting (lazy, forced, or otherwise) fails.

topher@nzxt:/$ sudo zfs umount array/Backups
cannot unmount '/array/Backups/gwen': no such pool or dataset

topher@nzxt:/$ sudo zfs umount array/Backups/gwen
cannot unmount '/array/Backups/gwen': no such pool or dataset

topher@nzxt:/$ sudo umount /array/Backups/gwen
umount: /array/Backups/gwen: no mount point specified.

topher@nzxt:/$ sudo umount /array/Backups
umount: /array/Backups: no mount point specified.

I'm unable to export the zfs pool either.

topher@nzxt:~$ sudo zpool export array
cannot unmount '/array/Backups/gwen': unmount failed

Attempts to remount the directory report that it's already mounted

topher@nzxt:/$ sudo zfs mount array/Backups
cannot mount 'array/Backups': filesystem already mounted

topher@nzxt:/$ sudo zfs mount array/Backups/gwen
cannot mount 'array/Backups/gwen': filesystem already mounted

lsof throws errors attempting to walk the affected directory and does not return any open filehandles in those paths

topher@nzxt:/$ sudo lsof | grep Backups
lsof: WARNING: can't stat() zfs file system /array/Backups
      Output information may be incomplete.
lsof: WARNING: can't stat() zfs file system /array/Backups/gwen
      Output information may be incomplete.

fuser doesn't think the mount exists

topher@nzxt:~$ sudo fuser -vm /array/Backups
Specified filename /array/Backups does not exist.
topher@nzxt:~$ sudo fuser -vm /array/Backups/gwen
Specified filename /array/Backups/gwen does not exist.
6
  • 1
    Can you export the zpool? Or are there are other, still-mounted, filesystems on it that you need to leave mounted? Jun 3, 2023 at 18:32
  • Yes, there are other datasets in use. Let me see if I can move those services elsewhere to test exporting. Jun 3, 2023 at 21:47
  • Export fails with umount error. :/ Updating question. Jun 3, 2023 at 22:03
  • What happens if you try to mount the filesystem? If it succeeds, can you then unmount it successfully? Jun 4, 2023 at 3:20
  • Attempts to mount say that it's already mounted... D: Jun 4, 2023 at 6:16

1 Answer 1

3

Try a lazy or forced umount and see what happens?

umount -l or umount -f

See: https://github.com/openzfs/zfs/issues/2435 and Linux mountpoint can not be unmounted no matter what?

6
  • 1
    Sorry I didn't mention that neither of these worked. I'll amend my question. Jun 3, 2023 at 16:23
  • Can you just reboot?
    – ewwhite
    Jun 3, 2023 at 20:02
  • 1
    If there are stale handles, what does showmount or netstat -anp | grep 2049 show?
    – ewwhite
    Jun 3, 2023 at 20:03
  • samba and nfsd are off, nothing listening on those ports. showmount predictably times out after a minute with the message clnt_create: RPC: Timed out I might be able to reboot, but this is a production server, I'd like to know how to troubleshoot this issue (so that I can prevent/fix it) as rebooting is not always convenient. I suspect the mount is busy, but I don't know how to find out what is keeping open file handles and/or why zfs and the kernel can't agree on the current state. Jun 3, 2023 at 22:00
  • 1
    Sometimes it's not worth troubleshooting... This is not a normal condition, so efforts to prevent it may not be fruitful. You can examine zpool history and zpool events for your own records. But take the downtime hit if this is important.
    – ewwhite
    Jun 4, 2023 at 2:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .