0

I'm trying to wget an entire site to transfer from Server A to Server B. It works, but incredibly slowly.

I tested this by uploading a 100mb psd file to my server, and grabbing it with wget ftp. Transfer speeds were 26.76mb/sec.

But when I try and grab the whole site (minus the psd file) it takes 5 minutes to transfer 92mb of html files. It's a wordpress install with a theme and images basically.

Is this because it opens a new connection for each file? I tried doing this with php's ftp library, but that took just as long if not longer.

scp, ssh are not an option because 98% of the time I don't own the server, so I have to work with some sort of ftp. I'm using:

wget -r ftp://user:pass@domain.com/dir is what I'm using.
0

4 Answers 4

0

It is always slower to transfer many files than it is to transfer one big file. Zip the contents of your website and the transfer will go much faster. You can use tar to do this.

1
  • Can't zip, as I don't know if Server A has the library for it, and I don't have access other than ftp. I supposed I could log into the server, transfer a php file that would zip the contents, transfer that zip file, then delete with php file with unlink(SELF); or something.
    – David
    May 3, 2013 at 15:28
0

wget will try to keep the connection open, but even with a single connection there is a significant overhead at the server end to serve each request, which is most probably what's causing the slowdown you're seeing.

0

It's not as simple as -r I admit but you could try this: Make a list of the files, separate this list in e.g. six parts and start wget six times simultaneously each with a part of the list. A short look at the cumulated transfer speed should tell you whether the number of files is the problem.

0

you can't speed up the transfer since it does a connection for each file, but try ftpcopy which allows you to mirror an ftp

from ftpcopy site:

Speed

ftpcopy needed about 110 seconds to copy my site. Subsequent runs which don't need to copy anything take about 1.04 seconds. mirror needed about 250 seconds to copy my site. Subsequent runs which don't need to copy anything take about 2.6 seconds.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.