TCP buffer sizes and ethernet bonding

Question

(This question was originally posted on stack overflow; Andy told me that I could get faster help here, so it is re-posted here now.)

I am experimenting with TCP buffer size tuning on Linux, but various results make me confused.

The test programs include a server and a client. The server simply listens on a port, waiting for the client to send data from an mmaped file. The received data is copied into an application buffer using recv and then dropped. While sending data, the client uses send with the full size of the mmapped buffer as the initial argument.

The programs run on two nodes from two different datacenters, the ping response time between them is about 9 msecs. Both nodes are installed two Gigabit ethernet controllers. The maximum throughput is 256 MB/s, and a proper setting of send/recv buffer sizes should be about 256 MB/s * 0.09 s ~ 2415919 bytes.

I did several experiments.

In the first run, I ran one instance of server and one instance of client. I did not set the size of either the send buffer or the receive buffer, letting the kernel autotune them. The purpose of this case is to establish a baseline of other experiments.

The actual throughput in this setting was about 117 MB/s. A single pair of server and client made use of only one eithernet controller in this case. Checking with ifconfig, I saw that most packets went through a single interface between eth0 and eth1.

Then I tried two servers and two clients, this time the throughput moved up to about 225 MB/s, much nearer to the ideal maximum throughput.

This is the first issue confusing me:

1. Why do I need more than one process to use up the bandwidth? FWIW, below is part of /proc/net/bonding/bond0:

   Bonding Mode: IEEE 802.3ad Dynamic link aggregation
   Transmit Hash Policy: layer3+4 (1)
   MII Status: up
   MII Polling Interval (ms): 100
   Up Delay (ms): 0
   Down Delay (ms): 0
   

Then I tried several combinations of send/recv buffer sizes for a single pair of server and client. The following table summarizes the results:

| send buf size | recv buf size | throughput | comment                   |
|      (client) |      (server) |     (MB/s) |                           |
|       1048576 |             - |       51.5 |                           |
|       2621400 |             - |       48.6 | server uses autotuning    |
|        524288 |             - |       43.3 |                           |
|       8388608 |             - |       36.3 |                           |
|       2621400 |       2621400 |       33.0 | somewhat the theory value |
|             - |       2621400 |       30.4 | client uses autotuning    |
|       4194304 |             - |       30.3 |                           |
|        262144 |             - |       29.1 |                           |
|             - |       1048576 |       27.9 |                           |
|       6291456 |       6291456 |       26.5 |                           |
|       8388608 |       8388608 |       23.9 |                           |
|       6291456 |             - |       22.2 |                           |
|             - |       4194304 |       20.8 |                           |
|       1048576 |       1048576 |       19.8 |                           |
|       4194304 |       4194304 |       19.3 |                           |
|             - |       8388608 |       19.3 |                           |
|             - |       6291456 |       13.8 |                           |

Here are several other questions raised up from the above table:

2. Why does the theory value not result in the best throughput (117 MB/s)?
3. Why is the best result (51.5 MB/s) still not as good as the result of kernel autotuning (117 MB/s)?
4. Why does bigger buffer result in poor throughput?

Thanks in advance.