Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote 1 down vote favorite

I have a rather big directory on one server (over 4000 files), which I'd like to copy to another server (which contains a previous version of this directory). rsync is the first option, but it will put the destination folder into waiting status for a rather long period of time (more than a minute).

I'd like to do it a bit differently:

  1. gzip the source folder
  2. scp the archive to the destination server
  3. gunzip the file there
  4. delete the archive at the source and the destination

What is the best way to accomplish all this?

share|improve this question
What does "put the destination folder into waiting status" mean? – David Schwartz Jun 8 '12 at 8:55
It means that some files are in new version already, while others are still in the previous one – yegor256 Jun 8 '12 at 8:56
3  
What are the requirements you're not telling us about? What's using the directory, how long can you live with them being different, what is it about rsync that your setup can't cope with, etc? – EightBitTony Jun 8 '12 at 8:58
It's a production system, with thousands of PHP files. Every second it receives a number of HTTP hits. I can't afford to keep this directory in mixed versions for a minute. I can afford a few seconds though.. – yegor256 Jun 8 '12 at 8:59
can you paste more informations about directory size and number of files ? – m0ntassar Jun 8 '12 at 8:59
show 4 more comments

5 Answers

up vote 6 down vote accepted

rsync has --delay-updates which seems to be what you need:

«…

This option puts the temporary file from each updated file into a holding directory until the end of the transfer, at which time all the files are renamed into place in rapid succession. This attempts to make the updating of the files a little more atomic. By default the files are placed into a directory named ".~tmp~" in each

…»

share|improve this answer
Thanks, that's exactly what I was looking for – yegor256 Jun 8 '12 at 9:11
As an addition to poige's answer, I'll recommend the OP using the lockfile command that is part of the procmail set of tools to lock access to the directory from readers. lockfile or a similar tool will need to be used by the writer and the reader. That way the data presented to the reader is consistent and not in the middle of an update. – nearora Jun 8 '12 at 9:12
up vote 3 down vote

The fastest way, if you have the space, is to rsync twice. Keep two copies of the files on the destination machine. First do a remote rsync to update the inactive copy. Then do a local rsync to update the active copy from the inactive copy.

share|improve this answer
up vote 2 down vote

rsync to a cold copy, then just change a symlink and delete the former active copy

share|improve this answer
up vote 0 down vote

Stop the web service on the target server for 10 minutes, do the update in any way you feel is sensible (rsync is fine), then start the web service again.

share|improve this answer
1  
Right, and I am guessing, that if you where in an operating room doing an organ transplant, you would suggest killing the patient, while you do the transplant and then shocking him back after the operation is complete? – Zoredache Jun 8 '12 at 9:25
Sorry, I didn't realise the web site was a life or death service. If you can't tolerate 10 minutes of downtime on the website, then you needed to have a better plan in the first place. – EightBitTony Jun 8 '12 at 9:44
This could also be a regular deployment scenario. And if you have a website that actually makes MONEY or where operation has been sold to a third party under SLA 10 minutes downtime per deployment can be dire. – rackandboneman Jun 8 '12 at 11:32
Indeed, none of which is mentioned in the original question, which is why I specifically asked for what we weren't being told. As you can see, the OP finally gave the actual requirement under prompting, after I'd written this answer. This answer is valid for plenty of websites, and not valid for others, but since the question didn't stipulate any significant downtime limitations, I felt it was reasonable. Jumping on it because you feel it doesn't work in business critical or life threatening situations is fine, but the OP didn't state that as a requirement. – EightBitTony Jun 8 '12 at 11:35
up vote -2 down vote

1-2-3: tar -c -O <source folder> | gzip -c | ssh <destination server> "gzip -d | tar -x "

4: rm <source folder>

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.