Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have to copy a relatively big file (6GB) to another server. To enforce consistency, I have to 'lock' this file and not use it during the transfer. I obviously want to minimize this downtime, but before this, I also want to estimate the time it will take to do so.

I can see 2 solutions :

  • copy the file locally and then transfer over the network (local). I'm thinking this will lead to the smallest downtime, but then, maybe I'm wrong as the disk will have to write and read the same file at the same time. Even with this, I have no idea how to estimate the time it will take to copy the file locally. Maybe creating a 6GB file and copying it would be meaningful, but then, how can I do this?

  • copy the file over the network directly (scp). This seams slow as network access is orders of magnitude slower than disk writing, but again, maybe I'm wrong.

Generally, how can I benchmark both approaches and decide which one is best for me?

share|improve this question
    
Super fast way to pull this off with minimal down time would be a mirrored disk that you can break off, and rebuild elsewhere, no need to ever copy the file. –  Tim Jan 9 '12 at 16:32
    
True! But this is not something I can do :( –  Julien Genestoux Jan 9 '12 at 16:49
add comment

3 Answers

Honestly... there's several ways to attack this. I have a few servers where I need to copy files that are in constant use... so I schedule a "maintenance window" of a few minutes where I can lock the files... create a snapshot (using dm-snapshot), bring the server back online and then do whatever with the snapshot'ed copy of the file. This allows me to have a stable version of the file... and bring the service back online with minimal down-time. Afterwards... I can remove the snapshot without affecting the live server. In your case... you can copy the 6gb snapshot'd file at your own pace as-needed.... without worry about the live-version of the file messing up the copy process.

http://tldp.org/HOWTO/LVM-HOWTO/snapshots_backup.html

share|improve this answer
    
+1 for copy on write snapshots. –  Jeff Ferland Jan 9 '12 at 16:51
add comment

Create a dummy file to transfer:

time dd if=/dev/urandom of=~/testfile bs=1M count=512
time cp testfile testfile2 #How long it takes to copy

time scp testfile user@otherserver:/path

/dev/urandom is selected to avoid any boost from compression. SCP should tell you interactively what the speed is. Basic math should allow extrapolation of files sizes to 6 gig.

share|improve this answer
add comment

You can use the *nix utility "Pipe Viewer" to watch these processes. It will provide you with a progress bar and estimated time of transfer.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.