3

When I try to download a file via HTTP, the downloads sometimes stop after around 30 MB. The download rates goes down to 0 B/s and no data keeps coming. When I stop the download and resume again, the download still hangs. But when I redownload it from byte 0 again, everything works fine up to 30 MB when it stops again. Sometimes, after some hours, it just works again without problems. The position in the file when the download stops is variable, but most of the time it is around 30–35 MB.

As a download manager I use wget. The same behaviour happens though using curl and other download managers. The error occurs independently of the server I download from. I have also observed this error on other Linux computers in my network. All computers on my network run Gentoo Linux on x86.

All internet connections on my network go through a server on my network which runs a transparent Squid proxy on port 80. That server is connected to a router, which is a Speedport W 700V by Deutsche Telekom AG. That router is connected to the internet using ADSL, with 448 kbit/s down speed and 96 kbit/s up speed.

I am almost sure that my transparent proxy is not the problem. I turned that off without resolving the issue. I also connected to the router directly via WLAN without resolving the issue. I also tried to download over another port via HTTP. Furthermore, I tried to download the file using IPv6 with a gateway6 tunnel from my computer, which resulted in exactly the same problem.

Now the strange thing is that everything works fine using FTP and HTTPS (also with wget on the same computer). Even more strange: when I resume the download that hanged over HTTP using FTP or HTTPS, download a few bytes that way, stop wget and then resume again using HTTP, it loads data again! But after a few MB, it may stop again. Unfortunately, files downloaded that way are always broken (the MD5 sum is not correct), so at some point, there must have been bogus data. I tried searching for HTML error messages in the downloaded file, but grep -i html does not find anything. (I cannot think of a way to search for GZIP-compressed HTML error messages in the file, so I did not try that.)

I tried using strace on wget when it failed to resume a download, you can find the entire output on pastebin. The important lines are repeated every second:

clock_gettime(CLOCK_MONOTONIC, {326102, 62176435}) = 0
)                    = 1
write(2, "78% [++++++++++++++++++++++++++++"..., 19578% [+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++                                  ] 110,683,685 --.-K/s              ) = 195
select(4, [3], NULL, NULL, {0, 949999}) = 0 (Timeout) 

I have absolutely no idea what could be the reason of this problem. It seems like whatever causes the issue speaks HTTP. It seems to speak HTTP that intelligently that it even regognises it in an IPv6-over-IPv4 tunnel. But what could that be and why does it only happen sometimes? The other possibility would be that there is a problem on my computer that is the same on other Gentoo Linux computers as well.

Has anyone ever had such a problem? What could be the reason and where do I have to continue investigating to find out more about the issue?

Update:

I have just run into the problem again and tried to resume the download over the router’s WLAN, and this time it worked. Maybe I did something wrong during my last tests with the WLAN. Now maybe my transparent proxy server is in fact the problem. It is a very basic Squid proxy server that does not cache anything. Maybe the fact is interesting that a second Squid proxy runs on the same computer on another port.

Update:

A download hung again and this time I turned off all firewall settings and stopped all proxy servers. I failed to resume the download from my network server, which is directly connected to the router. So my proxy server definitely is not the cause the problem. I will try to upgrade the firmware of my router now, although I do not have admin access to it. I will see what I can do.

Update:

Upgrading to the newest firmware of my router didn’t help. I can see no other possibility than this being my ISP’s fault. My “solution” is now to tunnel all traffic through an SSH server somewhere else.

4
  • sounds very much like your proxy server -and if not yours, then your ISPs upstream. The fact it works fine for https&http is interesting as neither of those two would get proxied. Also, where are you downloading FROM? -or is it anywhere. Jan 2, 2010 at 18:21
  • Forgot to mention that in my question, the problem occurs independently from the server I am downloading from. I notice this problem often when downloading Gentoo distfiles. When a download from one Gentoo mirror hangs, resuming from another mirror does not help. I also have an own server on the internet, resuming from that does not help either.
    – cdauth
    Jan 2, 2010 at 18:34
  • Sounds more and more that its a problem between your internet connection and the internet itself; ergo: your ISP :( -actually, what about a) multiple PCs at your house, or b) from a liveCD on the same computer? Jan 2, 2010 at 19:05
  • I also very often get broken files while downloading through Squid with aria2c splitting to large (over 50) number of parts (specially patched aria2 for this). I've got used to always doing rsync after such downloading to "fix the breakage" after downloading). And sometimes Squid just can't parse that complex request (with many ranges, some open-ended).
    – Vi.
    Jun 14, 2010 at 14:20

2 Answers 2

2

Even more strange: when I resume the download that hanged over HTTP using FTP or HTTPS, download a few bytes that way, stop wget and then resume again using HTTP, it loads data again! But after a few MB, it may stop again. Unfortunately, files downloaded that way are always broken (the MD5 sum is not correct)

This screams "broken proxy" The protocol for resuming an HTTP download is not in any way complicated(it's just an extra header), but this is exactly the kind of thing that a broken proxy would be messing up.

I bet if you try to download a big file using wget, wait for it to fail, and then run wget -c changing http to https, it will resume just fine.

5
  • That’s true, resuming over https works, but the problem is that only a few servers support https, so this is not a permanent solution. So the question about the exact cause of the problem remains.
    – cdauth
    Jan 2, 2010 at 23:19
  • load whatismyip.com and whatismyip.com Do you get the same address? (edit: the links look the same, but one is http and one is https)
    – Justin
    Jan 2, 2010 at 23:27
  • IP address is the same. Although the HTTP version displays “Possible Proxy Detected” and the HTTPS doesn’t (of course, because Squid cannot append the Via header to the SSL stream). (Edit: Actually, the HTTPS stream does not run through my proxy.)
    – cdauth
    Jan 2, 2010 at 23:35
  • Yes, that is why I suggested trying to resume using the https url. I think your proxy is broken.
    – Justin
    Jan 2, 2010 at 23:38
  • Are both of your squid servers sharing the same cache directory? Maybe they are corrupting each other?
    – Justin
    Jan 3, 2010 at 0:53
1

This post is really old, and all the network infrastructure is long replaced, but I still wanted to post the solution, which I didn’t post back in the day.

The problem was caused by the network card that connected my network server with the router. Replacing the card solved the issue. I don’t know what exactly was the problem, it must have been a firmware bug that was caused by some particular byte sequences or certain other conditions.

1
  • actually, I have the same problem. I've installed debian on MacBook and there are problems for me to find appropriate drivers to maintain a wireless connection (and for video card also). So, probably, the cause also can be not only in the card, the hardware, but in the driver it uses too, like, I believe in my case, because another OS works fine with WiFi.
    – hsrv
    Oct 23, 2018 at 20:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .