Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

When I try to download a file via HTTP, the downloads sometimes stop after around 30 MB. The download rates goes down to 0 B/s and no data keeps coming. When I stop the download and resume again, the download still hangs. But when I redownload it from byte 0 again, everything works fine up to 30 MB when it stops again. Sometimes, after some hours, it just works again without problems. The position in the file when the download stops is variable, but most of the time it is around 30–35 MB.

As a download manager I use wget. The same behaviour happens though using curl and other download managers. The error occurs independently of the server I download from. I have also observed this error on other Linux computers in my network. All computers on my network run Gentoo Linux on x86.

All internet connections on my network go through a server on my network which runs a transparent Squid proxy on port 80. That server is connected to a router, which is a Speedport W 700V by Deutsche Telekom AG. That router is connected to the internet using ADSL, with 448 kbit/s down speed and 96 kbit/s up speed.

I am almost sure that my transparent proxy is not the problem. I turned that off without resolving the issue. I also connected to the router directly via WLAN without resolving the issue. I also tried to download over another port via HTTP. Furthermore, I tried to download the file using IPv6 with a gateway6 tunnel from my computer, which resulted in exactly the same problem.

Now the strange thing is that everything works fine using FTP and HTTPS (also with wget on the same computer). Even more strange: when I resume the download that hanged over HTTP using FTP or HTTPS, download a few bytes that way, stop wget and then resume again using HTTP, it loads data again! But after a few MB, it may stop again. Unfortunately, files downloaded that way are always broken (the MD5 sum is not correct), so at some point, there must have been bogus data. I tried searching for HTML error messages in the downloaded file, but grep -i html does not find anything. (I cannot think of a way to search for GZIP-compressed HTML error messages in the file, so I did not try that.)

I tried using strace on wget when it failed to resume a download, you can find the entire output on pastebin. The important lines are repeated every second:

clock_gettime(CLOCK_MONOTONIC, {326102, 62176435}) = 0
)                    = 1
write(2, "78% [++++++++++++++++++++++++++++"..., 19578% [+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++                                  ] 110,683,685 --.-K/s              ) = 195
select(4, [3], NULL, NULL, {0, 949999}) = 0 (Timeout) 

I have absolutely no idea what could be the reason of this problem. It seems like whatever causes the issue speaks HTTP. It seems to speak HTTP that intelligently that it even regognises it in an IPv6-over-IPv4 tunnel. But what could that be and why does it only happen sometimes? The other possibility would be that there is a problem on my computer that is the same on other Gentoo Linux computers as well.

Has anyone ever had such a problem? What could be the reason and where do I have to continue investigating to find out more about the issue?

Update:

I have just run into the problem again and tried to resume the download over the router’s WLAN, and this time it worked. Maybe I did something wrong during my last tests with the WLAN. Now maybe my transparent proxy server is in fact the problem. It is a very basic Squid proxy server that does not cache anything. Maybe the fact is interesting that a second Squid proxy runs on the same computer on another port.

Update:

A download hung again and this time I turned off all firewall settings and stopped all proxy servers. I failed to resume the download from my network server, which is directly connected to the router. So my proxy server definitely is not the cause the problem. I will try to upgrade the firmware of my router now, although I do not have admin access to it. I will see what I can do.

Update:

Upgrading to the newest firmware of my router didn’t help. I can see no other possibility than this being my ISP’s fault. My “solution” is now to tunnel all traffic through an SSH server somewhere else.

share|improve this question
    
sounds very much like your proxy server -and if not yours, then your ISPs upstream. The fact it works fine for https&http is interesting as neither of those two would get proxied. Also, where are you downloading FROM? -or is it anywhere. –  BuildTheRobots Jan 2 '10 at 18:21
    
Forgot to mention that in my question, the problem occurs independently from the server I am downloading from. I notice this problem often when downloading Gentoo distfiles. When a download from one Gentoo mirror hangs, resuming from another mirror does not help. I also have an own server on the internet, resuming from that does not help either. –  cdauth Jan 2 '10 at 18:34
    
Sounds more and more that its a problem between your internet connection and the internet itself; ergo: your ISP :( -actually, what about a) multiple PCs at your house, or b) from a liveCD on the same computer? –  BuildTheRobots Jan 2 '10 at 19:05
    
I also very often get broken files while downloading through Squid with aria2c splitting to large (over 50) number of parts (specially patched aria2 for this). I've got used to always doing rsync after such downloading to "fix the breakage" after downloading). And sometimes Squid just can't parse that complex request (with many ranges, some open-ended). –  Vi. Jun 14 '10 at 14:20
add comment

1 Answer

Even more strange: when I resume the download that hanged over HTTP using FTP or HTTPS, download a few bytes that way, stop wget and then resume again using HTTP, it loads data again! But after a few MB, it may stop again. Unfortunately, files downloaded that way are always broken (the MD5 sum is not correct)

This screams "broken proxy" The protocol for resuming an HTTP download is not in any way complicated(it's just an extra header), but this is exactly the kind of thing that a broken proxy would be messing up.

I bet if you try to download a big file using wget, wait for it to fail, and then run wget -c changing http to https, it will resume just fine.

share|improve this answer
    
That’s true, resuming over https works, but the problem is that only a few servers support https, so this is not a permanent solution. So the question about the exact cause of the problem remains. –  cdauth Jan 2 '10 at 23:19
    
load whatismyip.com and whatismyip.com Do you get the same address? (edit: the links look the same, but one is http and one is https) –  Justin Jan 2 '10 at 23:27
    
IP address is the same. Although the HTTP version displays “Possible Proxy Detected” and the HTTPS doesn’t (of course, because Squid cannot append the Via header to the SSL stream). (Edit: Actually, the HTTPS stream does not run through my proxy.) –  cdauth Jan 2 '10 at 23:35
    
Yes, that is why I suggested trying to resume using the https url. I think your proxy is broken. –  Justin Jan 2 '10 at 23:38
    
Are both of your squid servers sharing the same cache directory? Maybe they are corrupting each other? –  Justin Jan 3 '10 at 0:53
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.