3

I have set up an NFSv4 server. I am sharing files with Linux machines in a VM farm and Windows 7, 8, and 10 machines, so I need to share files with both NFS and Samba1. In the past, I have done this by configuring an NFS machine with direct access to the underlying ext4 file system and configuring a separate Samba server that mounts the NFS exports and shares them2.

I would like to unify the file sharing onto one server. Critically, I need to be sure that NFS and Samba are not stepping on each other's files, blissfully ignoring locks. As a result, I must continue using Samba on top of NFS and not in parallel with NFS.

This seems trivial. The NFS server should simply mount its own share, just as the Samba server would have done, and then that same machine must run Samba and export that mounted share. The only different would seem to be that traffic need not travel over the network between two machines. I would expect better performance.

Instead, the NFS server saturates the CPU on the machine while giving only 2.5 MB/s performance!

top - 11:34:57 up 15 days,  1:18,  1 user,  load average: 32.09, 8.07, 2.74
Tasks: 101 total,  13 running,  88 sleeping,   0 stopped,   0 zombie
%Cpu(s):  5.6 us, 93.7 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0 st
KiB Mem :   899776 total,   124840 free,   184624 used,   590312 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   261184 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
11058 root      20   0       0      0      0 R 11.5  0.0   5:53.84 nfsd
11061 root      20   0       0      0      0 R 11.5  0.0   5:53.88 nfsd
11057 root      20   0       0      0      0 R 11.2  0.0   5:53.72 nfsd
11059 root      20   0       0      0      0 R 11.2  0.0   5:53.78 nfsd
11060 root      20   0       0      0      0 R 11.2  0.0   5:53.80 nfsd
11062 root      20   0       0      0      0 R 11.2  0.0   5:53.86 nfsd
11064 root      20   0       0      0      0 R 11.2  0.0   9:23.40 nfsd
11063 root      20   0       0      0      0 R 10.9  0.0   6:00.00 nfsd

The only other sign I know of that something is wrong is that countless lines get written to /var/log/messages that look something more or less like that.

Apr 25 11:35:01 nas kernel: nfsv4 compound op #4/4: 9 (OP_GETATTR)
Apr 25 11:35:01 nas kernel: nfsd: fh_verify(36: 01070001 00060001 00000000 9509b6fe 7a49ad56 5a4d5dbc)
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88dbf606a080 opcnt 4 #4: 9: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound returned 0
Apr 25 11:35:01 nas kernel: --> nfsd4_store_cache_entry slot ffff88db7bbae000
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88dac5072080 opcnt 4 #2: 22: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound op #3/4: 38 (OP_WRITE)
Apr 25 11:35:01 nas kernel: nfsd: write complete host_err=131072
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88dac5072080 opcnt 4 #3: 38: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound op #4/4: 9 (OP_GETATTR)
Apr 25 11:35:01 nas kernel: nfsd: fh_verify(36: 01070001 00060001 00000000 9509b6fe 7a49ad56 5a4d5dbc)
Apr 25 11:35:01 nas kernel: nfsd_dispatch: vers 4 proc 1
Apr 25 11:35:01 nas kernel: nfsv4 compound op #1/4: 53 (OP_SEQUENCE)
Apr 25 11:35:01 nas kernel: __find_in_sessionid_hashtbl: 1555688119:2835816624:106:0
Apr 25 11:35:01 nas kernel: nfsd4_sequence: slotid 5
Apr 25 11:35:01 nas kernel: check_slot_seqid enter. seqid 9615 slot_seqid 9614
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88dbf606a080 opcnt 4 #1: 53: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound op #2/4: 22 (OP_PUTFH)
Apr 25 11:35:01 nas kernel: nfsd: fh_verify(36: 01070001 00060001 00000000 9509b6fe 7a49ad56 5a4d5dbc)
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88db7b529080 opcnt 4 #4: 9: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound returned 0
Apr 25 11:35:01 nas kernel: --> nfsd4_store_cache_entry slot ffff88db7bbaf000
Apr 25 11:35:01 nas kernel: nfsd_dispatch: vers 4 proc 1
Apr 25 11:35:01 nas kernel: nfsv4 compound op #1/4: 53 (OP_SEQUENCE)
Apr 25 11:35:01 nas kernel: __find_in_sessionid_hashtbl: 1555688119:2835816624:106:0
Apr 25 11:35:01 nas kernel: nfsd4_sequence: slotid 1
Apr 25 11:35:01 nas kernel: check_slot_seqid enter. seqid 9626 slot_seqid 9625
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88db7b529080 opcnt 4 #1: 53: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound op #2/4: 22 (OP_PUTFH)
Apr 25 11:35:01 nas kernel: nfsd: fh_verify(36: 01070001 00060001 00000000 9509b6fe 7a49ad56 5a4d5dbc)
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88dbf606f080 opcnt 4 #4: 9: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound returned 0
Apr 25 11:35:01 nas kernel: --> nfsd4_store_cache_entry slot ffff88db291eb000
Apr 25 11:35:01 nas kernel: nfsd_dispatch: vers 4 proc 1
Apr 25 11:35:01 nas kernel: nfsv4 compound op #1/4: 53 (OP_SEQUENCE)
Apr 25 11:35:01 nas kernel: __find_in_sessionid_hashtbl: 1555688119:2835816624:106:0
Apr 25 11:35:01 nas kernel: nfsd4_sequence: slotid 2
Apr 25 11:35:01 nas kernel: check_slot_seqid enter. seqid 9614 slot_seqid 9613
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88dbf606f080 opcnt 4 #1: 53: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound op #2/4: 22 (OP_PUTFH)
Apr 25 11:35:01 nas kernel: nfsd: fh_verify(36: 01070001 00060001 00000000 9509b6fe 7a49ad56 5a4d5dbc)
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88dac507e080 opcnt 4 #2: 22: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound op #3/4: 38 (OP_WRITE)
Apr 25 11:35:01 nas kernel: nfsd: write complete host_err=131072
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88dac507e080 opcnt 4 #3: 38: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound op #4/4: 9 (OP_GETATTR)
Apr 25 11:35:01 nas kernel: nfsd: fh_verify(36: 01070001 00060001 00000000 9509b6fe 7a49ad56 5a4d5dbc)
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88dac5072080 opcnt 4 #4: 9: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound returned 0
Apr 25 11:35:01 nas kernel: --> nfsd4_store_cache_entry slot ffff88dbf65cc000
Apr 25 11:35:01 nas kernel: nfsd_dispatch: vers 4 proc 1
Apr 25 11:35:01 nas kernel: nfsv4 compound op #1/4: 53 (OP_SEQUENCE)
Apr 25 11:35:01 nas kernel: __find_in_sessionid_hashtbl: 1555688119:2835816624:106:0
Apr 25 11:35:01 nas kernel: nfsd4_sequence: slotid 4
Apr 25 11:35:01 nas kernel: check_slot_seqid enter. seqid 9584 slot_seqid 9583
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88dac5072080 opcnt 4 #1: 53: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound op #2/4: 22 (OP_PUTFH)
Apr 25 11:35:01 nas kernel: nfsd: fh_verify(36: 01070001 00060001 00000000 9509b6fe 7a49ad56 5a4d5dbc)
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88dbf606a080 opcnt 4 #2: 22: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound op #3/4: 38 (OP_WRITE)
Apr 25 11:35:01 nas kernel: nfsd: write complete host_err=131072
Apr 25 11:35:01 nas kernel: nfsv4 compound op ffff88dbf606a080 opcnt 4 #3: 38: status 0
Apr 25 11:35:01 nas kernel: nfsv4 compound op #4/4: 9 (OP_GETATTR)
Apr 25 11:35:01 nas kernel: nfsd: fh_verify(36: 01070001 00060001 00000000 9509b6fe 7a49ad56 5a4d5dbc)

I am not mounting the NFS shares locally via 127.0.0.1. The local NFS mount uses the private IP address of the machine itself. I could change that, but it would require additional complexity in the NFS exports definition, and I would actually prefer that this machine sees itself exactly the same way as the other machines that will be mounting the NFS shares3.

Footnotes

  1. I know of ways of using NFS clients on Windows and Samba clients on Linux. Such options are not workable in this case.
  2. I am aware of admonishments about re-exporting NFS shares with Samba, but I and many other sysadmins I know regularly do this without problems. The issue seems to have arisen here rather because the NFS server is mounting its own share. Nonetheless, I am curious about the exact reasons for such warnings, so if you know please share.
  3. If anybody has some reason why this entire architecture is fundamentally flawed, I am open to hearing that. I have years of experience serving Samba on top of NFS using two machines, so I supposed that I could cut out the Samba machine and move Samba to the NFS server machine with an NFS self-mount.
Improve this question

1 Answer 1

Reset to default
3

It's very likely that your workload is dying a scheduling death. You're asking for a ton on context switches on the same system when exporting a filesystem back to itself and then exporting that export again.

You may see that splitting this workload out across more than one kernel (kernels that do not share the same CPU core) results in a performance increase, despite the involvement of a network link between those systems.

This could easily be accomplished with a collocated VM set; one running NFS, one running Samba. Collocating these on the same hypervisor should eliminate most network latencies, and should work well provided that these VMs do not share the same CPU cores.

There are also userspace implementations for NFS - NFS Ganesha being a very common one. However, Samba is also going to be responsible for a non-trivial amount of this repetitive userspace-kernel switching. NFS Ganesha may be worth testing if these workloads absolutely must share a kernel for some reason.

In short, reducing context switching within the same system should help significantly with this.

Improve this answer
5
  • Huh! Context switching as an explanation is consistent with the high "sys" CPU usage in top, right? Could I alleviate the problem just by passing another core into the existing VM. The entire idea of moving from two VMs for these services to one was to reduce resource usage, so I would like to avoid moving back to two if possible. Is mounting the NFS share through 127.0.0.1 likely to improve the situation at all? I don't know what additional context switching might be necessary for the NFS traffic to potentially involve traffic outside the machine.
    – rg6
    Apr 25, 2019 at 19:52
  • I don't think serving on localhost will help with this, since the traffic in question doesn't leave even a virtual network phy already. The main reason I brought up using separate machines entirely was to keep these schedule-heavy workloads from stomping on each other, resource utilization aside. You might have some luck with adding more CPUs to this one machine, and that's probably the most reasonable first step.
    – Spooler
    Apr 25, 2019 at 19:59
  • 1
    Brilliant. Adding a second core to the VM increased the transfer rate to 35 MB/s with 5% CPU usage by nfsd (only 2 nfsd processes working) and 25% CPU usage by smbd. I'll keep testing various sides of this, but I'm one happy customer. Haha! Thanks!
    – rg6
    Apr 25, 2019 at 20:59
  • By the way, do you think adjusting NFS parameters to higher-than-usual transfer chunks could decrease context switching significantly in this case?
    – rg6
    Apr 25, 2019 at 21:31
  • 1
    In general, increasing the message size sent to NFS should accomplish that, at the cost of more memory utilization. I would encourage playing with rsize and wsize parameters of the client connection to find a "sweet spot" of performance anyways, since the use case of this mount is so specific and narrow. You could also increase the MTU size of the interface you're using for this, and that should help to alleviate context switching on TCP's behalf. This could potentially be a good argument for using a loopback interface, which has a very high MTU "out of the box".
    – Spooler
    Apr 25, 2019 at 21:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .