Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

We have a new file server to handle the obscene amount of files generated by the company (PDFs, XLS, DOCs and JPGs). Files being moved to the new server total about 3tb. The problem is we can't take the company down for days to move the files. The other problem is the applications creating all these files have to reference previous files, so we can't simply point them to the new server. Also, there isn't an option to have the applications create files on the new server, but reference the old server for existing files.

The servers are x64 win2003 r2. Both servers are on the same subnet.

DFS doesn't work.

Is there an application that can handle this amount of data to copy the files over, throttle bandwidth, and do a 'merge'? By merge I mean constantly copying over newly created files until the two servers are synched.

share|improve this question
    
Why doesn't DFS work, i.e. what are the symptoms? –  DutchUncle Feb 3 '11 at 23:15
    
The number of files and folders is too much for win2003. You can't even open folders. Talk about bad software design... –  smackaysmith Mar 24 '11 at 2:02

3 Answers 3

up vote 7 down vote accepted

Use robocopy to do the copy.

Do an initial first push. Once that is done start the copy again. This will catch all the new/updated files and move them across. Keep doing this until the night of the actual cut over. On the cut over night tell all employees to be sure that all software is shutdown. Then run the robocopy script one last time, then power down the machine when it is done. Redirect the users to the new machine and you are done.

share|improve this answer
    
After the initial copy, which will take about four days, the next robocopy will take about the same time because it still has to traverse millions of folders and files (yes, really horrible design at play). Or am I missing something with robocopy. The app would have to be smart like DFS. –  smackaysmith Feb 3 '11 at 20:56
1  
It does have to traverse the structure, but that's pretty quick compared to actually moving files. Unless you've got millions of files in a single folder, no single folder should take more than a second. robocopy just compares meta information. If the meta information matches the file is skilled. Meta information is very quick to read which is why it doesn't take very long. –  mrdenny Feb 3 '11 at 21:40

This is pretty much the canonical use of rsync. The amount of data isn't really a problem (for anything, really). Rsync has the ability to limit the amount of network bandwidth consumed (via the --bwlimit=n command line option), and rsync's raison d'etre is it's ability to transfer only the differences between the source and the destination, rather than re-transmitting lots of duplicate data -- which makes it perfect for this sort of staged synchronization that you are attempting.

You have various options for running Rsync under Windows, including DeltaCopy or installing the command line tools under Cygwin.

This isn't a particularly Windows-specific solution. Other folks may be able to offer better solutions in that arena.

share|improve this answer

For this amount of sync load, you will benefit from having a dedicated link between the 2 file servers, and a dedicated switch when adding more file servers. Hopefully this network segregation will also allow you to make the anti-virus back off.

Also, take this opportunity to move old data into a read-only share.

And to max out network/CPU/harddisk bandwidth, run more than one synchronisation job at once. You will have to monitor/measure/figure out how many jobs is optimal and how to group your folders.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.