I just installed a fresh FreeBSD 9.0 on a new server. The machine is using the onboard gbit nic (Atheros AR8151). It runs Samba and FTP and has a ZFS pool of ~3TB. When I download files from some hosts I get really good read performance, close to 90mb/s over FTP. But some other hosts have really bad read performance, around 2mb/s. At first I though it was samba, but that turns out not to be true. I tried downloading some files from 2 separate hosts behind the same switch, one was an Ubuntu 10.10 machine the other was running Mac OS 10.4. The Ubuntu machine did 2mb/s while the Mac OS machine got 10mb/s.
As requested some more details on the network and the test setup. The network in questions is my universities network, it's all at least 1gbit.
I have a NetGear WNDR3700 router which is running OpenWRT. The WNDR is connected to a HP Procurve switch (which is out of my control). The Procurve is further connected to the university network via fiber optics.
The server is a running on an Intel Celeron G540, with the aforementioned NIC. It is connected directly to the WNDR3700 with gbit ethernet. To test and diagnose the problem I downloaded and uploaded some 1.5GB files from and to the server using several protocols. The following machines and protocols where used (download is traffic going out from the server):
A Windows 7 machine connected to the Procurve via gbit ethernet. I could only perform a download test on this one, it did around 2mb/s. This is a laptop which is less then a month old.
Secondly there is a Linux Mint machine connected to the WNDR, this computer is roughly 6 months old. I did a download and upload test on this one, both with excellent performance. I used FTP, SMB and scp.
Then there is another linux machine, running Ubuntu 10.10, again a laptop, but this one was roughly 3 years old. It performed really poorly transferring in both directions over all 3 protocols. This one was also connected to the Procurve.
Then there is another linux machine, don't know what distro it's running as it's not mine. This computer downloaded the file with 10MB/s both over SMB and FTP. Again connected to the Procurve (via 100mbit ethernet).
Then there is a 3 month old Macbook pro, connected via WIFI to the WNDR, this machine also performed excellently in all tests.
And lastly there is a VPS hosted a couple hundred kilometers away, I could only test FTP and scp on this one, but its performance was good.
All these tests where done a couple times, spread throughout the day, and all tests are consistent. At first I thought it might be the Procurve, which it's brand new, but as you can see from the list there are several computers performance excellently.
After running some more tests I've now concluded that it's not the server at all. I tried transferring some files from the Mint to the Ubuntu box, with varying results: Mint -> Ubuntu: ~2mb/s Ubuntu -> Mint: ~20mb/s
Both the server and the Mint box are really new Sandy Bridge machines, The Mint box has an Realtek 8111E NIC.