13

I have an NFSv3 export from a Linux fileserver which used to mount fine. The fileserver had to go down for hardware maintenance. After bringing the server back up, the Linux client can no longer mount the nfs export.

No configuration has changed on the server or the client. I did do a software update and reboot the client after the first mount failed, but that did not help.

[root@client ~]# showmount -e ark
Export list for ark:
/mnt/bigraid *

[root@client ~]# mount -t nfs ark:/mnt/bigraid raid

It just hangs at this point. In another terminal...

[root@client ~]# dmesg | tail
[ 2526.676437] nfs: server ark not responding, timed out
[ 2529.183107] nfs: server ark not responding, timed out
[ 2531.689778] nfs: server ark not responding, timed out
[ 2538.196432] nfs: server ark not responding, timed out
[ 2540.703107] nfs: server ark not responding, timed out
[ 2543.209767] nfs: server ark not responding, timed out
[ 2545.716436] nfs: server ark not responding, timed out
[ 2548.223098] nfs: server ark not responding, timed out
[ 2550.729775] nfs: server ark not responding, timed out
[ 2557.236435] nfs: server ark not responding, timed out

...OK, but I could see the exports with showmount...

[root@client ~]# ping ark
PING ark.homebase (10.10.10.2) 56(84) bytes of data.
64 bytes from ark.homebase (10.10.10.2): icmp_seq=1 ttl=64 time=0.067 ms
64 bytes from ark.homebase (10.10.10.2): icmp_seq=2 ttl=64 time=0.043 ms
64 bytes from ark.homebase (10.10.10.2): icmp_seq=3 ttl=64 time=0.048 ms
64 bytes from ark.homebase (10.10.10.2): icmp_seq=4 ttl=64 time=0.042 ms
^C
--- ark.homebase ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2999ms

So I don't get it.

The server is running OpenSUSE. I ensured that the firewall was off (not that it was ever on) and that network connectivity seems fine.

ark:/etc # cat exports
/mnt/bigraid    *(rw,root_squash,insecure,no_subtree_check,sync)

Edit: here's the list of RPC ports in use

ark:/etc/init.d # rpcinfo -p
program vers proto   port
100000    2   tcp    111  portmapper
100005    1   udp  37599  mountd
100005    1   tcp  33880  mountd
100005    2   udp  37599  mountd
100005    2   tcp  33880  mountd
100005    3   udp  37599  mountd
100005    3   tcp  33880  mountd
100024    1   udp  49522  status
100024    1   tcp  41314  status
100003    2   udp   2049  nfs
100003    3   udp   2049  nfs
100003    4   udp   2049  nfs
100021    1   udp  51887  nlockmgr
100021    3   udp  51887  nlockmgr
100021    4   udp  51887  nlockmgr
100003    2   tcp   2049  nfs
100003    3   tcp   2049  nfs
100003    4   tcp   2049  nfs
100021    1   tcp  49804  nlockmgr
100021    3   tcp  49804  nlockmgr
100021    4   tcp  49804  nlockmgr
100000    2   udp    111  portmapper

Edit 2: Got some tcpdump information

(Edit 3: Removed tcpdump output, since it may not be relevant.)

I'm not at all familiar with what a proper nfs negotiation looks like. I also dumped a pcap file if you want to look at the data segments. It's at filedropper

Edit 3: I may be hitting this issue

I was following @CIA's advice below, and did this:

ark:/etc/init.d #  ./nfsserver stop
Shutting down kernel based NFS server: nfsd statd mountd idmapd       done
ark:/etc/init.d # ./portmap stop
Shutting down RPC portmap daemon                                      done
ark:/etc/init.d # ./portmap start
Starting RPC portmap daemon                                           done
ark:/etc/init.d # ./nfsserver start
Starting kernel based NFS server: idmapdexportfs: Warning: /mnt/bigraid does not support NFS export.
 mountd statd nfsd sm-notify                                          done

Despite the warning, the export now seems mountable.

10
  • Is selinux enabled/disabled?
    – CIA
    Feb 26, 2013 at 5:34
  • @CIA to the best of my knowledge, selinux is neither installed nor supported on OpenSUSE.
    – Nathan
    Feb 26, 2013 at 12:32
  • Are you sure? As of version 11.1, openSUSE contains SELinux “basic enablement”. news.opensuse.org/2008/08/20/…
    – CIA
    Feb 26, 2013 at 17:22
  • The fileserver is running OpenSuSE 11.0
    – Nathan
    Feb 26, 2013 at 18:13
  • I can see you are able to ping to your nfs box. Are you able to ping your server from the nfs box? Also, were you trying to mount to ~/root/raid or did you mean to mount to /raid? And for your /etc/exports, have you tried explicitly defining the IP instead of using "*"?
    – CIA
    Feb 26, 2013 at 18:34

7 Answers 7

4

So, NFS is weird in the it relys on portmapper to be running, so it can map a specific port to an RPC port. (I guess it's not weird. It's just the way it works.) If NFS is up before portmapper, NFS doesn't know how to route requests, because it checks portmapper for this at the start of the process. If portmapper is not up before NFS, NFS doesn't know how to map port to rpc.

Here's more documentation about the process (even though its for CentOS, it's still relevant): http://www.centos.org/docs/5/html/Deployment_Guide-en-US/s2-nfs-methodology-portmap.html

As for your new error message, reboot the box you're mounting with and remount to see if the error comes back.

1
  • I think I've seen the "new" error message before, and it doesn't appear to actually matter when it comes to being able to mount the export. It's quite possibly related to the ancient version of OpenSUSE we're running. One day maybe I'll get an intern to upgrade it :). Anyway, thanks for the help.
    – Nathan
    Feb 27, 2013 at 23:22
4

to summarize the solution from given answers, following steps below directed me to the right direction to fixing NFS mount issue without restating the box.

  1. run a tcpdump in client side to the NFS server IP address (assuming it's 1.2.3.4)

    tcpdump -i <replace-with-correct-INTERFACE_name -n host 1.2.3.4
    
  2. keep running tcpdump and try mounting the NFS share path.

  3. lookup for the ports NFS is communicating with the server to mounting the path (in your case [tcpdump result in edit revision], shows those are ports: 880, 2049 only).

  4. do telnet on NFS server IP and all the ports you got from tcpdump output in step3 and make sure you have telnet and ports are open from server side to client (for your case, only below two ports found).

    telnet 1.2.3.4 880
    telnet 1.2.3.4 2049
    
  5. if you don't have telnet on any of those ports captured in step3, you need to open those ports at netwrok level (server side and/or firewall if you have)

  6. try to mount the NFS again now.

2
tcpdump -i $LAN_IF -n host 10.10.10.2

should show you which of the NFS components fails.

4
  • I have added tcpdump information to the question. Let me know if you find it helpful.
    – Nathan
    Feb 26, 2013 at 12:59
  • I don't understand the entries like 10.10.10.10.316725468. But it's obvious that you use NFS4 and that communication with the server is possible in general. I didn't know that tcpdump understands NFS. I just read that you get more NFS related information by giving -vv as an argument. It seems that your client calls the NFS "command" "176 getattr fh 0,0/35" and shortly after that finishes the connection. Even more strange: The server continues sending data after that and gets a connection reset. There's no need to print lines after the first "Flags [R.]". Feb 26, 2013 at 15:36
  • I don't think I'm actually using NFSv4. I believe I have it set up with v3. Could this be the source of the problem?
    – Nathan
    Feb 26, 2013 at 18:12
  • OK, I am not familiar enough with NFS. TCP on port 2049 was possible with NFSv3. I thought that nonetheless connections to portmapper and mountd were necessary (which are not necessary in NFSv4). Feb 26, 2013 at 18:20
1

Well i used to get the same error. I realized the only reason for there to be a timeout is because the connection has not been established properly. Digging deeper into the problem, I checked my firewall, and damnn NFS4 service was blocked !!

Solutiion:- Use the below command to configure your firewall settings and add * next to NFS4 service to enable it.

$ sudo system-config-firewall-tui

0

Enable the following TCP and UDP ports for clients to access nfs share on server.

For NFS3 tcp: 111,662,875,892,2020,2049,32803
udp: 111,2049,32769

For NFS4

tcp: 111,2049 udp: 111,2049

Edit: try to telnet the above ports from nfs client

2
  • I can connect to 111 and 2049. I think the other ports may be randomly assigned, since they're different on my server (check my edit to the post above).
    – Nathan
    Feb 26, 2013 at 12:29
  • For NFSv4 you only need port 2049. rpcbind runs on port 111 and isn't needed by NFSv4. In fact, because rpcbind can be used to amplify DDOS attacks, having an open 111 port could get you quarantined by your information security office.
    – pgoetz
    Mar 18, 2022 at 12:32
0

If you you have host based firewall and using nfs, check out:

http://wiki.debian.org/SecuringNFS

You may have specify which ports your daemons are using, so they aren't being randomly assigned.

0

In my case the problem was that the server had been renumbered but the client was still trying to connect to the old IP address.

This helped to identify the issue on the NFS client host:

mount | grep addr=

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .