I'm writing a script to consume an XML feed provided by a third party. I'm having problems downloading the files listed inside it but the provider swears there's no problem.

The files are usually around 700kb, but between 20% and 80% of the downloads just stall mid-way, and then after 10 minutes of no packets being received, I receive a RESET packet (according to wireshark).

In order to test this I have a very simple shell script:


curl $FILE > $OUTPUT

I've run this many times on two machines in my office, as well as from an Amazon EC2 instance, and in all cases I see a mixture of very quick successes and also progress just halting at some mid-way point (different each time) and then never completing.

The success rate varies hugely - I might get 10 successes and then 10 stalls in a row.

I'm hoping someone can tell me if this is a common symptom for an overloaded .NET server (IIS I presume?) or whether you think there's anything I can do to prove the problem more strongly. Or indeed if there's something different I can do at my end.

  • You may want to attempt doing some of the downloads from an intermediate server-like from a VPS or other server. If you can show that you have the same problem on different networks/installations sometimes that gives you a little leverage. – Josh Budde Jul 15 '10 at 16:01
  • Yes, I've tried this and I can replicate the problem with this script on two office desktops, an amazon EC2 virtual server, and another VPS in a different datacentre so I'm pretty confident the problem's at the other end. – Jamie Jul 15 '10 at 18:04

Have you tried different using a different user agent?

  • I haven't tried changing the user agent in cURL, but I see the same problems when downloading manually with Firefox so I'm pretty sure the user agent isn't affecting it. – Jamie Jul 15 '10 at 15:14

Watch it with Fiddler. You should see whats going on at the HTTP level, alongside wireshark which is showing you the traffic.

  • You may find an HTTP request/response that is causing the behavior.

  • Look at the headers, check content-length, etc -- looking for anything that could cause the behavior

  • See if the response looks right, etc

That should help you narrow down and at least understand the full scope of the issue. Then I'd bet you'll find something on your end or have the evidence to prove its them

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.