Trouble enabling Jumbo Frames?

Question

I'm trying to enable jumbo frames. I have a linux client & a windows server. I've standardized across the board on intel nics. On the windows server the only choice for jumbo frames is 9014 & 4088. I've researched on whether I should set the MTU speed in linux to 9014 or 9000. There has been conflicting posts googling around but it seems like probably 9000 is the correct number. I, of course, have tried both though.

Going off of this site i tried to test if everything was working correctly. From my linux client: ping -M do -s 8972 [destinationIP] everything seems to be working correctly:

8980 bytes from file ([destinationIP]): icmp_seq=1 ttl=128 time=0.946 ms
8980 bytes from file ([destinationIP]): icmp_seq=2 ttl=128 time=1.16 ms
8980 bytes from file ([destinationIP]): icmp_seq=3 ttl=128 time=1.02 ms
8980 bytes from file ([destinationIP]): icmp_seq=5 ttl=128 time=0.935 ms

but when it tried testing from the windows side: ping -f -l 9000 [destinationIP]

Pinging <linux client> with 9000 bytes of data:
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.

After googling around i seen conflicting post on what size you should use when pinging from windows. So i tried 8972 (9000-28) but it times out and i get no response. I even tried 8986 (9014-28) but i get the fragmented error. I thought it was an issue between windows/linux but when i tried pinging other windows server (jumbo frames is on for every server) i get the same issue. I found out that when i hit 8972 it started timing out and from 9014 to 8973 i get the fragmented error. So my questions are:

1. In linux is 9000 the correct MTU speed to use (because of the difference between linux & windows) or is 9014?
2. Anyone know why i'm getting these fragmented errors from the windows side & it appears from the linux side everything is ok?

Here is my switch which clearly supports jumbo frames up to 9216

Answer 1

You need to distinguish:

• Ethernet maximum frame size: needs to take Ethernet L2 overhead into account, normally 18 bytes - standard is 1518 bytes untagged
• IP MTU: this is what you usually aim for - standard over Ethernet is 1500 bytes, jumbos often use 9000
• maximum ping payload for testing: ping uses ICMP (8 bytes overhead) over IPv4 (20 bytes overhead) - so the maximum unfragmented ping payload length is 1472 bytes (MTU 1500) or 8972 bytes (MTU 9000)

For testing path MTU you need to send echo requests with the DF bit set. This is accomplished in Windows by -f and in Linux by -M prohibit. Your -M do *explicitly enables fragmentation. When testing the "Packet needs to be fragmented but DF set" shows the source machine to lack the correct setup. When the ping times out, the destination or something in between isn't working. Switches usually have a counter for oversized frames.

Setting the switch to a somewhat larger maximum frame size than necessary isn't hurtful. However, do make sure your end nodes and routers are all set to the exact same size: 9000 when setting the MTU or 9018 when setting the max L2 PDU/frame size.

Intel's Jumbo Packet setting is somewhat misleading. The help text shows that the FCS (4 bytes) isn't counted, so 9014 is the correct setting for MTU 9000 frames.