I'm handling large numbers of concurrent downloads (approx. 500 per server) using Java.
All the files are being downloaded from Amazon S3, and the downloading server is an EC2 m1.large instance.
Occasionally, 2 or more of the streams will simultaneously be broken, resulting in a java.net.SocketException. Occasionally up to 10 of the streams can be simultaneously broken.
I am having the same results downloading from both Amazon S3 and Akamai servers. It only happens when the load starts to be quite high (200 or more concurrent downloads).
I'm well within normal CPU, network load and memory bounds.
I strongly suspect the problem is on my server, and not S3 and Akamai's. How could I debug this and track down the cause?