Update: Should I only set jumbo frame to server and file server, not client?

If so, is there any impact on communication between server and client?

I am running some performance test for our product.

Currently all the testing related machines(servers, file servers, clients, db) are on a 10G network connected by a powerful Dell OpenManage Switch.

We are using iscsi for the file server. We have a cluster server that contains several nodes.

The performance test I am running basically is to simulate the following scenario: 1. client machine will create a large number of threads to send http request to the server. 2. Based on the different type of requests, server needs to get some data from file server which is shared by all the other server nodes.

The test results is: Without jumbo frames, MTU 1500, server CPU 70%, and avg response time for the http request is 1 second.

With jumbo frames, MTU 9000, server CPU 20%, and avg response time for the http request is 5 seconds.

We have configured jumbo frames on all machines, and changed TCP settings.

Any ideas?

  • 3
    Jumbo frames should only be used for hosts accessing the iSCSI storage device, not from a client to the server fronting said storage. – SpacemanSpiff Apr 15 '11 at 3:51
  • But if I set client to 1500, and server to 9000, then server will still send 9000 to the client. Will that be a problem? – performanceuser Apr 15 '11 at 17:44
  • Only turn on jumbo frames for the iSCSI interfaces. You have dedicated interfaces for that, correct? – Doug Luxem Apr 15 '11 at 18:37
  • @Doug Did you mean: file server machines enable jumbo frames, all the other machines use standard MTU? – performanceuser Apr 15 '11 at 18:42


  • Bigger frames = more data on each package = your CPU works less to send data (it has a smaller number of packages per second), but takes longer to assemble each payload (more latency).
  • Smaller frames = less data on each package = your cpu works more to send data (more packages per second), but takes less time to assemble each payload (less latency).
  • 2
    Add to that apossible windows firewall.... can have significant impact on CPU usage... per packet. – TomTom Apr 15 '11 at 4:14
  • And you need to add CPU load on the switches as well. Also less interupts need to be serviced for a the volume of data - I guess that really comes down to less CPU required to do the work though, depending on how you look at it. – Jason Tan Apr 15 '11 at 6:40
  • 1
    I expect the CPU to be lower, but I don't think response time of http request should be 5 times longer – performanceuser Apr 15 '11 at 17:45
  • Just imagine that a lot of data must be put together to assemble a Jumbo package before it leaving the interface. – coredump Apr 15 '11 at 17:48
  • But if you think one level upper, for application level, they still need to get a complete http request. Maybe one request can be encapsulated into one 9000 frame, and into several 1500 frams, but the total time should be the same from sending a request to receiving the request. – performanceuser Apr 15 '11 at 17:51

I have been trying to read up and understand more about the impact of utilising Jumbo frames, and why it still hasn't become mainstream after more than a decade. This paper hints on the real world problems faced by Jumbo frame sizes, preventing it from achieving more than the bulk-file-transfer scenario.


Summary of issues contributing to high-latency delays

  1. Delayed pipelining across transmission mediums
  2. Small transmit/receive buffers causing dropped packets = retransmit
  3. Larger packet size = higher chance of collision = retransmit
  4. Lower CRC quality at greater payload lengths = corrupt packets = retransmit
  5. End-to-end path MTU discovery, both ways

Your "server" should have one interface for iSCSI (this interface is the disk bus!), and one interface for clients to access the server on. The iSCSI interface should be MTU9000, the client access interface should be have a default MTU. Also, your switch needs to be configured to support both jumbo frames and flow control, or performance will suck.

  • It is confusing. You said one interface for iscsi, which is a disk bus, and this interface should be MTU9000. Two questions: 1) If I understand correctly, the iscsi interface should connect to the NIC. Why it is a disk card? 2) How to set MTU9000 to a disk card? – performanceuser Apr 18 '11 at 18:45
  • iSCSI is block level storage over TCP/IP. The users do not talk to the iSCSI storage device the server does, they talk to your server, which in turn talks to the device. Would you run really long SATA cables from your server to every desktop? NO. – SpacemanSpiff Apr 18 '11 at 18:59
  • But the server has one NIC. It need to use this NIC to talk to file server and client. I can only set one MTU value for this NIC – performanceuser Apr 18 '11 at 19:05
  • Then I suggest you add another one if this is meant to be a permanent solution and if you want ideal performance. – SpacemanSpiff Apr 18 '11 at 19:08
  • 2
    This isn't to say jumbo frames may not help your clients as well, but right now your disk bus is competing for bandwidth with the client communication. You may end up using jumbo frames on both NICs, but you definitely need two. – SpacemanSpiff Apr 18 '11 at 19:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.