7

I'm trying to understand networking settings related to NFS and various buffer sizes (and there are quite a few).

I'm running wireshark and inspecting the TCP packets arriving at the NFS server. Wireshark is showing a packet size max 32626 during an extended write operation (client->server), assuming I am interpreting correctly ("bytes on the wire" which I suppose includes all network-layer headers, etc.)

The "rsize" and "wsize" NFS settings for exported storage is set to 32k on both C/S, so I figured the above results were a result of this setting. However, increasing these values does NOT increase the packet size shown by Wireshark.

So my question is, what other constraints could be in place? I've done a fair amount of research, and this is what I've come across so far. It seems to me that none of the network constraints below would limit transmission size to 32k:

From sysctl:

net.ipv4.tcp_mem          = 4096 87380 4194304
net.ipv4.tcp_{r,w}mem     = 4096 87380 4194304
net.core.{r,w}mem_max     = 131071
net.core.rmem_default     = 229376

My MTU is currently 8K

Improve this question

1 Answer 1

Reset to default
3

The NFS {r,w}size defined by client mount option and/or server capabilities. IOW, you can define them on command line like:

# mount -o rsize=1048576 .....

Linux client have different default values for v3 and v4 - 32k and 1MB. The nfs server may request a smaller or can support bigger sizes. You should be able to see that with wireshark as FSINFO call for v3 or FATTR4_MAXREAD/FATTR4_MAXWRITE file attributes, which requested with the very first GETATTR call.

The RPC layer may split single read or write requests into multiple RPC fragments. The TCP layer may split single RPC fragment into multiple TCP packets. On an other hand, TCP layer may marge multiple RPC requests into single TCP packet, if they fit.

There is a quite outdated document Optimizing NFS Performance, but will get an idea how to tweak the numbers.

Improve this answer
4
  • Thanks for the response - Great point about decoding the NFS packets! - I was seeing "Continuation of Data..." for the NFS packets. For other's reference, I had to right click -> Decode As -> RPC. Now that I can see the actual RPC values, I see no FATTR4_MAX{READ,WRITE}. I do see several others, including NUMLINKS=20, OWNER, SPACE_USED, etc... I did a search for MAX{READ,WRITE} and no results. This is definitely NFS4. Any suggestions?
    – Jmoney38
    Feb 6, 2014 at 15:58
  • Then it's probably NFSv4.1. Check response to CREATE_SESSION->csr_fore_chan_attrs->max resp size
    – kofemann
    Feb 6, 2014 at 16:49
  • Ah! Your initial comment was correct! - I had to re-mount on the client... Apparently the client only requests that additional information for a GETATTR call when needed. The value I'm finding is 1048576 - which, if this is in units of bits, equals 128KB, which corresponds to the client's net.core.{r,w}mem_max values. Additionally, if this is true, then I still wonder where the max packets size of 32K is coming in.
    – Jmoney38
    Feb 6, 2014 at 18:18
  • @Jmoney38 in case of nfsv4.0 it see at second GETATTR call after PUTROOTFH: reco_attr: MAXREAD (30)
    – kofemann
    Feb 6, 2014 at 18:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .