0

I am running FreeBSD 12-RELEASE on our local NFS server. Previously it was working fine, but as of a few days ago some of our tools that require file locking stopped working. Locking does work on our NFS clients for other servers.

Here is some information I think might be relevant:

On our FreeBSD server, san6:

    root@san6:~ # lsof | grep rpc.lockd
    lsof: WARNING: compiled for FreeBSD release 12.0-RELEASE-p5; this is 12.0-RELEASE.
    rpc.lockd 1063  root  cwd   VDIR                 2,16               1024        2 /
    rpc.lockd 1063  root  rtd   VDIR                 2,16               1024        2 /
    rpc.lockd 1063  root  txt   VREG                 2,16              73176 31059299 /usr/sbin/rpc.lockd
    rpc.lockd 1063  root  txt   VREG                 2,16             148320 12279169 /libexec/ld-elf.so.1
    rpc.lockd 1063  root  txt   VREG                 2,16              49560 31070185 /usr/lib/librpcsvc.so.5
    rpc.lockd 1063  root  txt   VREG                 2,16            1949672 46066959 /lib/libc.so.7
    rpc.lockd 1063  root    0u  VCHR                 2,18             0t1091      530 /dev/pts/0
    rpc.lockd 1063  root    1u  VCHR                 2,18             0t1091      530 /dev/pts/0
    rpc.lockd 1063  root    2u  VCHR                 2,18             0t1091      530 /dev/pts/0
    rpc.lockd 1063  root    3u  unix   0xfffff8013dc176d0                0t0          ->0xfffff8013dec6a38

    root@san6:~ # ps aux | grep lockd
    root  1063    0.0  0.0  48016 13692  -  Ds   11:54       0:00.43 rpc.lockd -d 2
    root  3102    0.0  0.0  11256  2736  1  S+   05:53       0:00.00 grep lockd

    root@san6:/usr/ports # rpcinfo -p canada
       program vers proto   port  service
        100000    4   tcp    111  rpcbind
        100000    3   tcp    111  rpcbind
        100000    2   tcp    111  rpcbind
        100000    4   udp    111  rpcbind
        100000    3   udp    111  rpcbind
        100000    2   udp    111  rpcbind
        100024    1   udp  58683  status
        100024    1   tcp  36401  status
        100021    1   udp  33875  nlockmgr
        100021    3   udp  33875  nlockmgr
        100021    4   udp  33875  nlockmgr
        100021    1   tcp  33610  nlockmgr
        100021    3   tcp  33610  nlockmgr
        100021    4   tcp  33610  nlockmgr

    root@san6:~ # dmesg | tail -n 1
    NLM: failed to contact remote rpcbind, stat = 5, port = 28416

    root@san6:~ # mount | grep nfs
    data0 on /data0 (zfs, NFS exported, local, noatime, nfsv4acls)

And on one of our clients, "canada", running Centos 7.6.1810:

[root@canada ~]# dmesg | tail -n 10
[   24.205026] virbr0: port 1(virbr0-nic) entered disabled state
[  588.552140] FS-Cache: Loaded
[  588.594282] FS-Cache: Netfs 'nfs' registered for caching
[  888.710226] lockd: server san6 not responding, still trying
[ 1265.871541] lockd: server san6 OK
[ 1265.871551] lockd: unexpected unlock status: 7
[ 1415.519506] nfs: server san6 not responding, still trying
[ 1439.592184] nfs: server san6 OK
[ 2137.130503] nfs: server san6 not responding, still trying
[ 2622.383586] nfs: server san6 OK

[root@canada ~]# mount | grep /projects
auto.direct on /projects type autofs (rw,relatime,fd=29,pgrp=15535,timeout=300,minproto=5,maxproto=5,direct,pipe_ino=87674)
san6:/data0/projects on /projects type nfs (rw,nosuid,noatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.252.48.106,mountvers=3,mountport=816,mountproto=udp,local_lock=none,addr=10.252.48.106)

Note that lockd is in "Ds" state, I'm not sure which file it is waiting for disk io from. It seems to enter this state when I try to kill/restart it and also after running for awhile. I've tried restarting the server with debug flags turned on (the -d2 in the ps log) but didn't see any more information in the syslog.

I've also tried adding the server to the trusted zone of one of our Centos 7 clients to see if that helped the "NLM: failed to contact remote rpcbind, stat = 5, port = 28416" but it did not seem to, no change to whether that client could lock files or not.

I greatly appreciate any help with this issue!

UPDATE: I've reproduced the issue on another server, still the same symptoms. I'm still not sure what is going on here.

0

I figured it out. I thought that adding the server to a single client's trusted zone in the firewall would fix the client. Instead, it turns out that all the client's need the server's IP/ports whitelisted in every client's firewall. If even one client blocks, it seems to spoil locking for everyone.

On CentOS 7, the command is

firewall-cmd --permanent --zone=trusted --add-source=<server's ip>

if you don't mind trusting all ports from the server. Don't forget to reload the firewall's configuration:

firewall-cmd --reload

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.