Synology DS1513+ with DSM 6.2.2-24922
This is a Synology NFS issue. Everything else on and off the NAS works. Maybe some NFS gurus know what's going on.
Edit: Is there a way to reinstall or reset NFS on the NAS, sort of like a package in Debian or whatever?
The power went out a while ago and suddenly NFS shares on the NAS are timing out. Pretty much convinced it's the NAS since this is from three different Linux machines with different OS versions (Debian 9, Debian 10 upgrade, Linux Mint Whatever). Debian 10 Live also fails. CIFS mounts work fine from a client with this problem. Also working on a Windows 7 machine. Literally everything else related, and not related to the NAS, other than NFS on the NAS, is working.
- interfaces haven't changed, on everything
- firewall rules haven't changed, on everything
- I can scan it with nmap from my computer and all the appropriate ports are open
- I can log in to the NAS via DSM and SSH; everything else works
- no dropped packets
- all share and file service settings are unchanged on the NAS
- tried disabling/enabling NFS and recreating NFS permissions but nope
- no recent updates on anything...
On the client...
The fstab entries look like this:
NAS:/volume1/Share1 /mnt/nfs/Share1 nfs noauto,_netdev,x-systemd.automount,x-systemd.device-timeout=3,rw,noexec,nosuid,soft,timeo=20 0 0
NAS:/volume1/Share2 /mnt/nfs/Share2 nfs noauto,_netdev,x-systemd.automount,x-systemd.device-timeout=3,rw,noexec,nosuid,soft,timeo=20 0 0
etc...
Trying to mount a share results in repeated timeouts. This occurs on systemd automounts, or trying to mount manually:
$ mount -v Share1
mount.nfs: timeout set for Mon Jul 1 14:03:35 2019
mount.nfs: trying text-based options 'soft,timeo=20,nfsvers=4.2,addr=10.2.3.4,clientaddr=10.2.3.123'
mount.nfs: mount(2): Connection timed out
mount.nfs: trying text-based options 'soft,timeo=20,nfsvers=4.2,addr=10.2.3.4,clientaddr=10.2.3.123'
mount.nfs: mount(2): Connection timed out
etc...
I noticed it never tries any other NFS version, but then it's timing out, not failing a connection. The option for NFS 4.1 is enabled on the NAS, so I tried forcing version 4.1 anyway, and also disabling 4.1, but same result.
On the server...
nfsd appears to be running. Changing settings causes them to restart as expected. There are something like 50+ nfsd processes. ps says:
$ ps -el | grep nfs
1 S 0 479 2 0 60 -20 - 0 rescue ? 00:00:00 nfsiod
1 S 0 19310 2 0 60 -20 - 0 rescue ? 00:00:00 nfsd4
1 S 0 19311 2 0 60 -20 - 0 rescue ? 00:00:00 nfsd4_callbacks
1 S 0 19315 2 0 80 0 - 0 svc_ge ? 00:00:00 nfsd
1 S 0 19316 2 0 80 0 - 0 svc_ge ? 00:00:00 nfsd
... etc x 50+ ...
1 S 0 19382 2 0 80 0 - 0 svc_ge ? 00:00:00 nfsd
I can't seem to find much in any logs on the NAS, except these older messages (note there are no messages on mount attempts):
$ dmesg | grep -i nfs
[ 1.634369] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 4.457531] NFS: Registering the id_resolver key type
[ 64.183851] Installing knfsd (copyright (C) 1996 [email protected]).
[ 65.032600] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[ 65.032650] NFSD: starting 90-second grace period (net ffffffff81853a80)
[ 732.887033] nfsd: peername failed (err 107)!
[ 3747.212750] nfsd: last server has exited, flushing export cache
[ 3747.857575] Installing knfsd (copyright (C) 1996 [email protected]).
[ 3748.106390] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[ 3748.114294] NFSD: starting 90-second grace period (net ffffffff81853a80)
[ 3766.547251] nfsd: last server has exited, flushing export cache
[ 3767.179133] Installing knfsd (copyright (C) 1996 [email protected]).
[ 3767.417179] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[ 3767.425074] NFSD: starting 90-second grace period (net ffffffff81853a80)
[64250.094837] nfsd: peername failed (err 107)!
[67058.897383] nfsd: last server has exited, flushing export cache
[67071.416444] Installing knfsd (copyright (C) 1996 [email protected]).
[67071.686715] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[67071.694600] NFSD: starting 90-second grace period (net ffffffff81853a80)
[67762.742233] nfsd: last server has exited, flushing export cache
[67783.915610] Installing knfsd (copyright (C) 1996 [email protected]).
[67784.287170] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[67784.295081] NFSD: starting 90-second grace period (net ffffffff81853a80)
[67826.140602] nfsd: last server has exited, flushing export cache
[67827.558978] Installing knfsd (copyright (C) 1996 [email protected]).
[67828.044110] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[67828.052009] NFSD: starting 90-second grace period (net ffffffff81853a80)
[67865.118316] nfsd: last server has exited, flushing export cache
[67866.194700] Installing knfsd (copyright (C) 1996 [email protected]).
[67866.444936] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[67866.452861] NFSD: starting 90-second grace period (net ffffffff81853a80)
Found some more info in the Arch Wiki (which is well known to be awesome)...
$ cat /proc/fs/nfs/exports
# Version 1.1
# Path Client(Flags) # IPs
That's it. I find this suspicious. Should this list exports?
$ cat /etc/exports
/volume1/Share1 Client1(rw,async,no_wdelay,insecure,all_squash,insecure_locks,sec=sys,anonuid=1024,anongid=100)
etc...
They're all the same. There are three exports files: exports, exports_map, and exports_syno, all the same.
$ cat /proc/net/rpc/nfsd
rc 0 0 0
fh 0 0 0 0 0
io 0 0
th 63 0 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
ra 128 0 0 0 0 0 0 0 0 0 0 0
net 0 0 0 2
rpc 0 0 0 0 0
proc2 33 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
proc3 22 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
proc4 2 0 0
proc4ops 59 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
I dunno.
$ cat /etc/nfs/syno_nfs_conf
udp_read_size=32768
udp_write_size=32768
nfsv4_enable=yes
nfs_unix_pri_enable=0
statd_port=0
nlm_port=0
nfs_custom_port_enable=no
kerberos_principal=nfs/nas.lan@LAN
Tried debug flags:
$ sysctl -w sunrpc.rpc_debug=1023
$ sysctl -w sunrpc.nfsd_debug=1023
But I can't find a log that's being updated with anything. Tried /var/log/messages, dmesg, kern.log, syslog.log, sysnotify.log.
This is getting seriously annoying... Any ideas? I posted in the Synology community forums, but it's been over a week and it's pretty unresponsive.
Edit:
Just noticed that dmesg is filled with:
[46814.703044] NFSD: laundromat service - starting
[46814.707587] NFSD: laundromat_main - sleeping for 90 seconds
repeat like 40 times...
[46846.668090] svc: server ffff88006d210000, no data yet
[46846.668165] svc: server ffff880077f60000, no data yet
[46846.668167] svc: server ffff880077f60000 waiting for data (to = 3600000)
[46846.668175] svc: server ffff88007d57e000, no data yet
[46846.668178] svc: server ffff88007d57e000 waiting for data (to = 3600000)
[46846.668183] svc: server ffff88006ff0a000, no data yet
repeat like 50 times...
I don't remember seeing this before. Not sure if the svc stuff is related...
