0

We have a round-robin DNS setup that map the same name ("gsx") to six different IP addresses. This is verified as working from the client by using things such as 'nslookup' and 'ping'. However when we attempt to NFS mount using the name ("gsx") it will only mount from a single system.

If we specifically mount via IP address, we can mount from any of the 6 servers, so all 6 servers are correctly exporting and servicing NFS requests. The problem seems to lie somewhere in the mount.nfs addressing, or whatever it relies upon.

The client in question is a Linux system running CentOS 6.8.

To illustrate the issue please see the following commands which illustrate [1] nslookup correctly rotates among the addresses on the client; [2] ping correctly rotates among the addresses on the client; [3] mount -t nfs does NOT correctly rotate among the addresses on the client, but instead will always use the same address:

[1] nslookup test

for i in 1 2 3 4 5 6 ; do nslookup gsx; done

Server:         192.168.1.19
Address:        192.168.1.19#53

Name:   gsx.backbone.lan
Address: 192.168.10.16
Name:   gsx.backbone.lan
Address: 192.168.10.11
Name:   gsx.backbone.lan
Address: 192.168.10.12
Name:   gsx.backbone.lan
Address: 192.168.10.13
Name:   gsx.backbone.lan
Address: 192.168.10.14
Name:   gsx.backbone.lan
Address: 192.168.10.15

Server:         192.168.1.19
Address:        192.168.1.19#53
Name:   gsx.backbone.lan
Address: 192.168.10.11
Name:   gsx.backbone.lan
Address: 192.168.10.12
Name:   gsx.backbone.lan
Address: 192.168.10.13
Name:   gsx.backbone.lan
Address: 192.168.10.14
Name:   gsx.backbone.lan
Address: 192.168.10.15
Name:   gsx.backbone.lan
Address: 192.168.10.16

Server:         192.168.1.19
Address:        192.168.1.19#53

Name:   gsx.backbone.lan
Address: 192.168.10.12
Name:   gsx.backbone.lan
Address: 192.168.10.13
Name:   gsx.backbone.lan
Address: 192.168.10.14
Name:   gsx.backbone.lan
Address: 192.168.10.15
Name:   gsx.backbone.lan
Address: 192.168.10.16
Name:   gsx.backbone.lan
Address: 192.168.10.11

Server:         192.168.1.19
Address:        192.168.1.19#53
Name:   gsx.backbone.lan
Address: 192.168.10.13
Name:   gsx.backbone.lan
Address: 192.168.10.14
Name:   gsx.backbone.lan
Address: 192.168.10.15
Name:   gsx.backbone.lan
Address: 192.168.10.16
Name:   gsx.backbone.lan
Address: 192.168.10.11
Name:   gsx.backbone.lan
Address: 192.168.10.12

Server:         192.168.1.19
Address:        192.168.1.19#53

Name:   gsx.backbone.lan
Address: 192.168.10.14
Name:   gsx.backbone.lan
Address: 192.168.10.15
Name:   gsx.backbone.lan
Address: 192.168.10.16
Name:   gsx.backbone.lan
Address: 192.168.10.11
Name:   gsx.backbone.lan
Address: 192.168.10.12
Name:   gsx.backbone.lan
Address: 192.168.10.13

Server:         192.168.1.19
Address:        192.168.1.19#53
Name:   gsx.backbone.lan
Address: 192.168.10.15
Name:   gsx.backbone.lan
Address: 192.168.10.16
Name:   gsx.backbone.lan
Address: 192.168.10.11
Name:   gsx.backbone.lan
Address: 192.168.10.12
Name:   gsx.backbone.lan
Address: 192.168.10.13
Name:   gsx.backbone.lan
Address: 192.168.10.14

[2] ping example

for i in 1 2 3 4 5 6 ; do ping -c3 gsx; done

PING gsx.backbone.lan (192.168.10.16) 56(84) bytes of data.
64 bytes from gsx.backbone.lan (192.168.10.16): icmp_seq=1 ttl=64 time=0.065 ms
64 bytes from gsx.backbone.lan (192.168.10.16): icmp_seq=2 ttl=64 time=0.093 ms
64 bytes from gsx.backbone.lan (192.168.10.16): icmp_seq=3 ttl=64 time=0.063 ms

--- gsx.backbone.lan ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.063/0.073/0.093/0.016 ms
PING gsx.backbone.lan (192.168.10.11) 56(84) bytes of data.
64 bytes from gsx.backbone.lan (192.168.10.11): icmp_seq=1 ttl=64 time=0.064 ms
64 bytes from gsx.backbone.lan (192.168.10.11): icmp_seq=2 ttl=64 time=0.089 ms
64 bytes from gsx.backbone.lan (192.168.10.11): icmp_seq=3 ttl=64 time=0.061 ms

--- gsx.backbone.lan ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.061/0.071/0.089/0.014 ms
PING gsx.backbone.lan (192.168.10.12) 56(84) bytes of data.
64 bytes from gsx.backbone.lan (192.168.10.12): icmp_seq=1 ttl=64 time=0.133 ms
64 bytes from gsx.backbone.lan (192.168.10.12): icmp_seq=2 ttl=64 time=0.124 ms
64 bytes from gsx.backbone.lan (192.168.10.12): icmp_seq=3 ttl=64 time=0.061 ms

--- gsx.backbone.lan ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.061/0.106/0.133/0.032 ms
PING gsx.backbone.lan (192.168.10.13) 56(84) bytes of data.
64 bytes from gsx.backbone.lan (192.168.10.13): icmp_seq=1 ttl=64 time=0.080 ms
64 bytes from gsx.backbone.lan (192.168.10.13): icmp_seq=2 ttl=64 time=0.090 ms
64 bytes from gsx.backbone.lan (192.168.10.13): icmp_seq=3 ttl=64 time=0.060 ms
--- gsx.backbone.lan ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.060/0.076/0.090/0.016 ms
PING gsx.backbone.lan (192.168.10.14) 56(84) bytes of data.
64 bytes from gsx.backbone.lan (192.168.10.14): icmp_seq=1 ttl=64 time=0.106 ms
64 bytes from gsx.backbone.lan (192.168.10.14): icmp_seq=2 ttl=64 time=0.154 ms
64 bytes from gsx.backbone.lan (192.168.10.14): icmp_seq=3 ttl=64 time=0.114 ms

--- gsx.backbone.lan ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.106/0.124/0.154/0.024 ms
PING gsx.backbone.lan (192.168.10.15) 56(84) bytes of data.
64 bytes from gsx.backbone.lan (192.168.10.15): icmp_seq=1 ttl=64 time=0.072 ms
64 bytes from gsx.backbone.lan (192.168.10.15): icmp_seq=2 ttl=64 time=0.097 ms
64 bytes from gsx.backbone.lan (192.168.10.15): icmp_seq=3 ttl=64 time=0.081 ms

--- gsx.backbone.lan ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.072/0.083/0.097/0.012 ms

[3] mount -t nfs examples of incorrect behavior

for i in 3 4 5 6 ; do mount --verbose -t nfs -o ro gsx:/gpm604/80${i} /tmp/tony/80${i} ; done

mount.nfs: timeout set for Tue Mar  7 17:46:51 2017
mount.nfs: trying text-based options 'vers=4,addr=192.168.10.16,clientaddr=192.168.10.21'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'addr=192.168.10.16'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.10.16 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.10.16 prog 100005 vers 3 prot UDP port 597
gsx:/gpm604/803 on /tmp/tony/803 type nfs (ro)
mount.nfs: timeout set for Tue Mar  7 17:46:51 2017
mount.nfs: trying text-based options 'vers=4,addr=192.168.10.16,clientaddr=192.168.10.21'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'addr=192.168.10.16'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.10.16 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.10.16 prog 100005 vers 3 prot UDP port 597
gsx:/gpm604/804 on /tmp/tony/804 type nfs (ro)
mount.nfs: timeout set for Tue Mar  7 17:46:51 2017
mount.nfs: trying text-based options 'vers=4,addr=192.168.10.16,clientaddr=192.168.10.21'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'addr=192.168.10.16'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.10.16 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.10.16 prog 100005 vers 3 prot UDP port 597
gsx:/gpm604/805 on /tmp/tony/805 type nfs (ro)
mount.nfs: timeout set for Tue Mar  7 17:46:52 2017
mount.nfs: trying text-based options 'vers=4,addr=192.168.10.16,clientaddr=192.168.10.21'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'addr=192.168.10.16'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.10.16 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.10.16 prog 100005 vers 3 prot UDP port 597
gsx:/gpm604/806 on /tmp/tony/806 type nfs (ro)

[Addition] We discovered that when using 1 Gb/s ethernet connections/LAN's we do NOT have this problem. This problem only seems to occur when we attempt to mount things on our 10 Gb/s ethernet LAN's. We mounted the same volumes from the same NFS servers, on the same clients, with the same client OS and system binaries, utilizing the same DNS server, and the mounts rotated as expected (and as mentioned they should in many online how-to's). The problem now seems to be confined to utilizing 10 Gb/s ethernet for some reason.

  • I don't know about others, but my experience is to use link aggregation for NFS and Round-Robin for iSCSI. – Jacob Evans Mar 24 '17 at 17:07
  • Not posting as a real answer, cause I'm guessing and taking a wild stab in the dark, but have you checked DNS caching? Try "lookupcache=none" as an NFS option when mounting. Good luck! – Jed Daniels Mar 24 '17 at 17:21
  • JacobEvans - only single 10 Gb/s connection, so link aggregation not possible JedDaniels - 'lookupcache' is not a valid mount.nfs option in Linux, thanks for the suggestion though – AkosPrime Mar 24 '17 at 17:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.