Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm trying to wget an entire site to transfer from Server A to Server B. It works, but incredibly slowly.

I tested this by uploading a 100mb psd file to my server, and grabbing it with wget ftp. Transfer speeds were 26.76mb/sec.

But when I try and grab the whole site (minus the psd file) it takes 5 minutes to transfer 92mb of html files. It's a wordpress install with a theme and images basically.

Is this because it opens a new connection for each file? I tried doing this with php's ftp library, but that took just as long if not longer.

scp, ssh are not an option because 98% of the time I don't own the server, so I have to work with some sort of ftp. I'm using:

wget -r ftp://user:pass@domain.com/dir is what I'm using.
share|improve this question

4 Answers 4

up vote 0 down vote accepted

It is always slower to transfer many files than it is to transfer one big file. Zip the contents of your website and the transfer will go much faster. You can use tar to do this.

share|improve this answer
    
Can't zip, as I don't know if Server A has the library for it, and I don't have access other than ftp. I supposed I could log into the server, transfer a php file that would zip the contents, transfer that zip file, then delete with php file with unlink(SELF); or something. –  David May 3 '13 at 15:28

wget will try to keep the connection open, but even with a single connection there is a significant overhead at the server end to serve each request, which is most probably what's causing the slowdown you're seeing.

share|improve this answer

It's not as simple as -r I admit but you could try this: Make a list of the files, separate this list in e.g. six parts and start wget six times simultaneously each with a part of the list. A short look at the cumulated transfer speed should tell you whether the number of files is the problem.

share|improve this answer

you can't speed up the transfer since it does a connection for each file, but try ftpcopy which allows you to mirror an ftp

from ftpcopy site:

Speed

ftpcopy needed about 110 seconds to copy my site. Subsequent runs which don't need to copy anything take about 1.04 seconds. mirror needed about 250 seconds to copy my site. Subsequent runs which don't need to copy anything take about 2.6 seconds.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.