up vote 0 down vote favorite
share [g+] share [fb]

I was asked by a client to move a website to a new server. Normally I would make a local copy by downloading it and then uploading it to the new server after making any necessary configuration changes of course.

This particular client has like 1 million images ranging up to 1MB and I still haven't found a solution as to how to move these properly. I have tried zipping them on the server first but the file gets too big and it seems to just stop.

Anyway, the client told me he can set up a "portal" so I don't have to download and re-upload the files. I have never heard of this, nor can I find what this is. Do any of you know something that fits this description or a solution to my image problem?

Thanks in advance

link|improve this question
100% accept rate
feedback

2 Answers

up vote 3 down vote accepted

You might be able to do this a little bit faster by piping tar through netcat on the old server and piping from netcat through tar on the new machine. So, on the old machine:

tar c /my/img/dir/ | nc newhost 8888

And on the new machine, from the directory you want these images to show up in:

nc -l 8888 | tar x

Make sure to execute the command on the new machine first.

link|improve this answer
is tar more capable than zip of handling a large amount of files? – KThompson Mar 11 '10 at 18:54
1  
I haven't seen any benchmarks, tar is just the conventional tool for these kinds of tasks. If you'd like to use zip make sure not to use compression. I assumed that since these million images were on a web server they'd already be in compressed formats, making any extra compression needless overhead. – Brandon Adams Mar 11 '10 at 19:07
TAR is not a compression algorithm, and as Brandon says, the images are most likely already compressed, so ZIPping will just add more overhead than it's worth. TAR def. sounds like the way to go here. – Mark Henderson♦ Mar 11 '10 at 21:37
feedback
up vote 1 down vote

Closest I've come to migrating something like that was using VMWare's cloning tool to move the physical server to a virtual server; once it comes back up, I just altered the network settings and it worked rather well afterwards.

If you're using Unix servers, could you mount the remote server's directory over SSH (sshfs) and do a copy/rsync from the original server to the new one?

I have no idea what the client means by a portal to help move the files.

link|improve this answer
2  
+1. Rsync is really the only way to go when you're dealing with this many files. – ErikA Mar 11 '10 at 18:24
Might want to shut down any services that will be touching the files while doing the copy also...side note I take for granted sometimes :-) He doesn't mention if this uses a database or anything like that. Also, depending on client needs, this might be a good time to look at decoupling parts of the website if that may be called for (i.e., store data on a SAN/NAS and have the web serving engine separate, for occasions like upgrading/migrating, if the site is big enough to warrant it...) – Bart Silverstrim Mar 11 '10 at 18:42
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.