I have a CentOS 5.6 server with the base Samba install. It has an 8 drive Raid 5 attached to an Areca 1880, hdparm tests give me back ~450MBps and I can confirm close to that with a dd to /dev/null. Serving out files to the network is a 10gbe Myricom card, and the clients are all 1gbe broadcom/intel/etc, on the same switch.
The problem I have seems to be fairly consistent throughout, single stream file copy performance is poor when the files have not been cached into memory on the server. I am talking about 9-12MB files if I copy a 1GB file, performance is through the roof, usually 90%+ on the 1gbe clients. When I copy 9-12MB files, say 100 of them, it will get roughly 40-50% usage, once they are cached on the server it will do MAYBE 55-65% usage on the 1gbe client NIC, but never more. Why?!?
I can copy multiple cached sequential file sequences and get to maybe 85%+ usage on the client NIC, why can't a single file sequence hit 95%+? All the clients are setup to offload tx/rx packets to the adapter, unfortunately they aren't using Jumbo Frames but I have tested with Jumbos turned on and still saw the same performance.