I'm running Ubuntu 12.04 on an a dual processor Intel E5-2620-HexCore, 64GB of RAM, 1Gb network link.
When downloading a single test file from a location in the same datacenter (Public network) I get d/l rate of 80-100 MB/s (Bytes/second).
When crawling web pages using multi-process PHP design, no matter how many processes I run, the d/l rate stays steady on 200Mb/s (bits/second). rate of packets per second is around 300K inbound.
I've tried some of the settings recommended here http://http-kit.org/600k-concurrent-connection-http-kit.html but that didn't make any difference.