Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a NTFS partition that has a folder that holds over 5 million directories. Each directory contains more directories and then files.

I am moving to SSD and I need to copy all these data over to the new drive, but I am running into issues.

Directories like this will crash Windows Explorer, so I didn't even try it.

My first attempt is robocopy, which has worked well in the past. But when I tried to do the copy it just never started, even after leaving it for days. I think it was trying to index everything before starting the copy.

How do you copy directories of this size?

share|improve this question

14 Answers 14

I have never seen robocopy fail like this, but if it is failing because of size/pre-indexing, then why not write your own script to do the copying? I would write the thing in PERL and just do the copy on a per-file basis and recursively go through it. You could add in checks in to see if the file already exists with the same time stamp, etc.

You could also check out RichCopy which is multi-threaded.

share|improve this answer
2  
yeah, code will solve all problems, but is there any solutions that are "out of the box"? –  Pyrolistical Jul 22 '09 at 23:48

As some other answers have said, I've never seen RoboCopy fail like this - AFAIK, it doesn't do any indexing up front, it just starts with the first directory and gets on with the job.

Do you have the latest version of RoboCopy? There are some old copies floating around that date back to old NT days that can have problems with larger copies.

FWIW, here's the version header from my installation for comparison:

-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows     ::     Version XP010
-------------------------------------------------------------------------------
share|improve this answer

Not sure what else is on your partition, but you could use a Linux live CD (like GPartEd) and just copy the entire partition en-masse to the new drive. As a bonus, if you wanted to change the partition size while doing it, GPartEd will do that for you as well. Then delete whatever you didn't want to copy over. :)

share|improve this answer

It is possible you may be running into path-length problems with your directory structure. NTFS allows longer paths than both Explorer or CMD will access. This requires a bit of creativity to handle. Generally this is caused by file writes that don't use either method, or a share is there a directory or three down that'll allow it to work.

for /D %%D in (*.*) do call copydir.cmd %%D

[CopyDir.CMD]
subst m: %1
m:
xcopy *.* F:\newlocation\ /s/e/h/q/y
subst m: /del

Run that at the top of your directory structure to chop off the top of said directory structure and hopefully make your paths small enough to be visible to everything. Use whatever copy tool you need to use instead of xcopy.

share|improve this answer

You probably need to use something multithreaded like richcopy it allows you to specify how many threads and how deep to dig per thread. I haven't tried it on 5 million dirs but I've used it on thousands and it worked far better than robocopy. Another copy utility in my bag o tricks is XXcopy. It's not as fast but has some pretty cool directory cloning options. Since it's command line I don't think it's' even going to try to enumerate the structure, just pump it thru.

share|improve this answer

Can you back it up using Windows NTbackup or a similar utility? Typically when I'm moving large amounts of data, I will backup from one device and then restore to another.

share|improve this answer

Maybe try the good old xcopy

share|improve this answer
    
if robocopy didn't work, xcopy certainly will not –  Pyrolistical Jul 22 '09 at 23:34
1  
While you're probably right, that's a bit of flawed logic right there. I'd have at least TRIED it before dismissing it outright. :-) –  KPWINC Jul 23 '09 at 1:41

I've never used it and can't vouch for it, but people say good things about TeraCopy.

share|improve this answer

Not a real answer, more a suggested line of attack: The *nix way to do this would be using tar cf - | tar xvf -, essentially asking tar to create a archive and directly pipe it into tar to extract it (somewhere else). Maybe this can be done on windows in a similar fashion aswell.

share|improve this answer

This may sound like sort of a bad hack, but on several past occasions when copying huge directory structures, I used WinRAR to archive (with compression) to a file and then extract it to the destination drive.

Not the most graceful solution, I know, but it worked in a pinch.

These days, I'd probably try to use rsync or tar from the cygwin utilities.

share|improve this answer

I use FastCopy ( http://www.ipmsg.org/tools/fastcopy.html.en) . Its been the industry standard open source copier for 10+ years now. I wouldnt use anything else.

share|improve this answer

First thing I'd try is RichCopy:

http://technet.microsoft.com/en-us/magazine/2009.04.utilityspotlight.aspx?pr=blog

The next thing I'd try is booting up with an Ubuntu Live CD or a Knoppix CD and doing an RSYNC.

The big thing to remember here is not only do you want to copy the files over... you want to make sure they're copied over INTACT and there's no corruption.

I've copied over millions of files (over 4 million and counting) files using rsync and its proven to be extremely reliable.

share|improve this answer

If you don't care about preserving security ACLs you could use rsync via cygwin. Otherwise finding a way to break up the copy using one of the other mentioned utilities (robocopy, richcopy, xcopy) is probably the way to go.

share|improve this answer

I've been impressed with Beyond Compare for these types of file operations. It has the bonus of if the copy fails or you have to continue it later, it knows right where to start because it supports copy-only-diffs.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.