Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm trying to understand networking settings related to NFS and various buffer sizes (and there are quite a few).

I'm running wireshark and inspecting the TCP packets arriving at the NFS server. Wireshark is showing a packet size max 32626 during an extended write operation (client->server), assuming I am interpreting correctly ("bytes on the wire" which I suppose includes all network-layer headers, etc.)

The "rsize" and "wsize" NFS settings for exported storage is set to 32k on both C/S, so I figured the above results were a result of this setting. However, increasing these values does NOT increase the packet size shown by Wireshark.

So my question is, what other constraints could be in place? I've done a fair amount of research, and this is what I've come across so far. It seems to me that none of the network constraints below would limit transmission size to 32k:

From sysctl:

net.ipv4.tcp_mem          = 4096 87380 4194304
net.ipv4.tcp_{r,w}mem     = 4096 87380 4194304
net.core.{r,w}mem_max     = 131071
net.core.rmem_default     = 229376

My MTU is currently 8K

share|improve this question

1 Answer 1

The NFS {r,w}size defined by client mount option and/or server capabilities. IOW, you can define them on command line like:

# mount -o rsize=1048576 .....

Linux client have different default values for v3 and v4 - 32k and 1MB. The nfs server may request a smaller or can support bigger sizes. You should be able to see that with wireshark as FSINFO call for v3 or FATTR4_MAXREAD/FATTR4_MAXWRITE file attributes, which requested with the very first GETATTR call.

The RPC layer may split single read or write requests into multiple RPC fragments. The TCP layer may split single RPC fragment into multiple TCP packets. On an other hand, TCP layer may marge multiple RPC requests into single TCP packet, if they fit.

There is a quite outdated document Optimizing NFS Performance, but will get an idea how to tweak the numbers.

share|improve this answer
    
Thanks for the response - Great point about decoding the NFS packets! - I was seeing "Continuation of Data..." for the NFS packets. For other's reference, I had to right click -> Decode As -> RPC. Now that I can see the actual RPC values, I see no FATTR4_MAX{READ,WRITE}. I do see several others, including NUMLINKS=20, OWNER, SPACE_USED, etc... I did a search for MAX{READ,WRITE} and no results. This is definitely NFS4. Any suggestions? –  Jmoney38 Feb 6 at 15:58
    
Then it's probably NFSv4.1. Check response to CREATE_SESSION->csr_fore_chan_attrs->max resp size –  tigran Feb 6 at 16:49
    
Ah! Your initial comment was correct! - I had to re-mount on the client... Apparently the client only requests that additional information for a GETATTR call when needed. The value I'm finding is 1048576 - which, if this is in units of bits, equals 128KB, which corresponds to the client's net.core.{r,w}mem_max values. Additionally, if this is true, then I still wonder where the max packets size of 32K is coming in. –  Jmoney38 Feb 6 at 18:18
    
@Jmoney38 in case of nfsv4.0 it see at second GETATTR call after PUTROOTFH: reco_attr: MAXREAD (30) –  tigran Feb 6 at 18:18

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.