0

Synology DS1513+ with DSM 6.2.2-24922

This is a Synology NFS issue. Everything else on and off the NAS works. Maybe some NFS gurus know what's going on.

Edit: Is there a way to reinstall or reset NFS on the NAS, sort of like a package in Debian or whatever?

The power went out a while ago and suddenly NFS shares on the NAS are timing out. Pretty much convinced it's the NAS since this is from three different Linux machines with different OS versions (Debian 9, Debian 10 upgrade, Linux Mint Whatever). Debian 10 Live also fails. CIFS mounts work fine from a client with this problem. Also working on a Windows 7 machine. Literally everything else related, and not related to the NAS, other than NFS on the NAS, is working.

  • interfaces haven't changed, on everything
  • firewall rules haven't changed, on everything
  • I can scan it with nmap from my computer and all the appropriate ports are open
  • I can log in to the NAS via DSM and SSH; everything else works
  • no dropped packets
  • all share and file service settings are unchanged on the NAS
  • tried disabling/enabling NFS and recreating NFS permissions but nope
  • no recent updates on anything...

On the client...

The fstab entries look like this:

NAS:/volume1/Share1 /mnt/nfs/Share1 nfs noauto,_netdev,x-systemd.automount,x-systemd.device-timeout=3,rw,noexec,nosuid,soft,timeo=20 0 0
NAS:/volume1/Share2 /mnt/nfs/Share2 nfs noauto,_netdev,x-systemd.automount,x-systemd.device-timeout=3,rw,noexec,nosuid,soft,timeo=20 0 0
etc...

Trying to mount a share results in repeated timeouts. This occurs on systemd automounts, or trying to mount manually:

$ mount -v Share1

mount.nfs: timeout set for Mon Jul 1 14:03:35 2019
mount.nfs: trying text-based options 'soft,timeo=20,nfsvers=4.2,addr=10.2.3.4,clientaddr=10.2.3.123'
mount.nfs: mount(2): Connection timed out
mount.nfs: trying text-based options 'soft,timeo=20,nfsvers=4.2,addr=10.2.3.4,clientaddr=10.2.3.123'
mount.nfs: mount(2): Connection timed out
etc...

I noticed it never tries any other NFS version, but then it's timing out, not failing a connection. The option for NFS 4.1 is enabled on the NAS, so I tried forcing version 4.1 anyway, and also disabling 4.1, but same result.

On the server...

nfsd appears to be running. Changing settings causes them to restart as expected. There are something like 50+ nfsd processes. ps says:

$ ps -el | grep nfs

1 S 0 479 2 0 60 -20 - 0 rescue ? 00:00:00 nfsiod
1 S 0 19310 2 0 60 -20 - 0 rescue ? 00:00:00 nfsd4
1 S 0 19311 2 0 60 -20 - 0 rescue ? 00:00:00 nfsd4_callbacks
1 S 0 19315 2 0 80 0 - 0 svc_ge ? 00:00:00 nfsd
1 S 0 19316 2 0 80 0 - 0 svc_ge ? 00:00:00 nfsd
... etc x 50+ ...
1 S 0 19382 2 0 80 0 - 0 svc_ge ? 00:00:00 nfsd

I can't seem to find much in any logs on the NAS, except these older messages (note there are no messages on mount attempts):

$ dmesg | grep -i nfs

[ 1.634369] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 4.457531] NFS: Registering the id_resolver key type
[ 64.183851] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 65.032600] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[ 65.032650] NFSD: starting 90-second grace period (net ffffffff81853a80)
[ 732.887033] nfsd: peername failed (err 107)!
[ 3747.212750] nfsd: last server has exited, flushing export cache
[ 3747.857575] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 3748.106390] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[ 3748.114294] NFSD: starting 90-second grace period (net ffffffff81853a80)
[ 3766.547251] nfsd: last server has exited, flushing export cache
[ 3767.179133] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 3767.417179] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[ 3767.425074] NFSD: starting 90-second grace period (net ffffffff81853a80)
[64250.094837] nfsd: peername failed (err 107)!
[67058.897383] nfsd: last server has exited, flushing export cache
[67071.416444] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[67071.686715] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[67071.694600] NFSD: starting 90-second grace period (net ffffffff81853a80)
[67762.742233] nfsd: last server has exited, flushing export cache
[67783.915610] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[67784.287170] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[67784.295081] NFSD: starting 90-second grace period (net ffffffff81853a80)
[67826.140602] nfsd: last server has exited, flushing export cache
[67827.558978] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[67828.044110] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[67828.052009] NFSD: starting 90-second grace period (net ffffffff81853a80)
[67865.118316] nfsd: last server has exited, flushing export cache
[67866.194700] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[67866.444936] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[67866.452861] NFSD: starting 90-second grace period (net ffffffff81853a80)

Found some more info in the Arch Wiki (which is well known to be awesome)...

$ cat /proc/fs/nfs/exports

# Version 1.1
# Path Client(Flags) # IPs

That's it. I find this suspicious. Should this list exports?

$ cat /etc/exports

/volume1/Share1   Client1(rw,async,no_wdelay,insecure,all_squash,insecure_locks,sec=sys,anonuid=1024,anongid=100)
etc...

They're all the same. There are three exports files: exports, exports_map, and exports_syno, all the same.

$ cat /proc/net/rpc/nfsd

rc 0 0 0
fh 0 0 0 0 0
io 0 0
th 63 0 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
ra 128 0 0 0 0 0 0 0 0 0 0 0
net 0 0 0 2
rpc 0 0 0 0 0
proc2 33 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
proc3 22 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
proc4 2 0 0
proc4ops 59 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

I dunno.

$ cat /etc/nfs/syno_nfs_conf

udp_read_size=32768
udp_write_size=32768
nfsv4_enable=yes
nfs_unix_pri_enable=0
statd_port=0
nlm_port=0
nfs_custom_port_enable=no
kerberos_principal=nfs/nas.lan@LAN

Tried debug flags:

$ sysctl -w sunrpc.rpc_debug=1023
$ sysctl -w sunrpc.nfsd_debug=1023

But I can't find a log that's being updated with anything. Tried /var/log/messages, dmesg, kern.log, syslog.log, sysnotify.log.

This is getting seriously annoying... Any ideas? I posted in the Synology community forums, but it's been over a week and it's pretty unresponsive.

Edit:

Just noticed that dmesg is filled with:

[46814.703044] NFSD: laundromat service - starting
[46814.707587] NFSD: laundromat_main - sleeping for 90 seconds
repeat like 40 times...
[46846.668090] svc: server ffff88006d210000, no data yet
[46846.668165] svc: server ffff880077f60000, no data yet
[46846.668167] svc: server ffff880077f60000 waiting for data (to = 3600000)
[46846.668175] svc: server ffff88007d57e000, no data yet
[46846.668178] svc: server ffff88007d57e000 waiting for data (to = 3600000)
[46846.668183] svc: server ffff88006ff0a000, no data yet
repeat like 50 times...

I don't remember seeing this before. Not sure if the svc stuff is related...

0

If the power went down maybe some equipment rebooted and went back to the configuration it had stored that differs from what it was running in volatile memory.

When running network storage it is a common practice to optimize the network, enterprise switches don't store the configuration right away, you have to save it or it is lost after reboot.

  • Review your NAS server, network switches and all the intermediate equipment to see if maybe:
    • You had enabled jumbo frames on all but now it is only available in some ports on the NAS or the switches.
    • VLANS are not properly configured.
    • A network bond is failing.
  • Test network using ping from NAS to clients and clients to NAS
  • Connect to the NAS using a direct network cable and a laptop and test NFS
  • The switch does require saving settings, but I know I did that last time. I can port scan and ping the NAS, and everything else on the NAS works. I'm currently using CIFS from a client with the NFS problem. I may try connecting a laptop directly to the NAS as a sanity check, but literally everything else related and not related to the NAS is working. Pretty confident the network is fine. I never set up jumbo frames on anything. I guess it could magically get enabled on the NAS, but everything else on it is working... – g4rb4g3_ju1c3 Jul 8 at 21:31
  • It is a mystery. Check DNS on the NAS and on the clients. – Javier Ruiz Jul 9 at 12:57
  • The DNS server as handled by the router is working. I can ping the NAS via hostname, and I can ping the client from the NAS via hostname. DNS clients are working. It very much is a mystery, but I'm 99% sure it's an NFS problem on the NAS, and I'm not an NFS or RPC expert. I do wonder if I won't become one by the end of this. Thanks for your input though.. – g4rb4g3_ju1c3 Jul 10 at 2:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.