I'm running Ubuntu 12.04 on an a dual processor Intel E5-2620-HexCore, 64GB of RAM, 1Gb network link.

When downloading a single test file from a location in the same datacenter (Public network) I get d/l rate of 80-100 MB/s (Bytes/second).

When crawling web pages using multi-process PHP design, no matter how many processes I run, the d/l rate stays steady on 200Mb/s (bits/second). rate of packets per second is around 300K inbound.

I've tried some of the settings recommended here http://http-kit.org/600k-concurrent-connection-http-kit.html but that didn't make any difference.

Any ideas?

  • The large file download is apparantly limited by bandwidth and the other task runs into for instance a CPU or IO limit before reaching the bandwidth limit instead. – HBruijn Jul 18 '14 at 22:29
  • @Hbrujin hdparm shows my HD speed can get to 150MB/s write, where iotop shows current load of only 2M/s. cpu and memory load are not even close to their limit. so that's probably not the issue. – Roey B Jul 18 '14 at 22:54
  • hdparm is a horrible tool for testing hard drive performance. You really should be using something like bonnie++, or iometer. hdparm -t is a sequential raw-read operation, it doesn't tell you anything about the writes, or random access. It also completely ignores the filesystem overhead. – Zoredache Jul 18 '14 at 23:57
  • @Zoredache bonnie++ shows a not so different rate of 137MB/s. how does this rate limit d/l rate in any way? – Roey B Jul 20 '14 at 2:56

Have you looked at the firewall that is between your server and the internet? I know most firewalls will have default limits on packets and concurrent connections for any single host inside or outside the network. Especially if the firewall comes with IDS.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.