Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have to get a 20GB directory from my server to my desktop. It has probably millions of files, and I can't tgz them because for some reason it won't write the file past 7.5GB. What's the best way to get the files down? I need something better than FTP because that will take ages with so many files. Will I not have the 7.5GB file limit if I skip gz?

share|improve this question

closed as not a real question by Sirex, John Gardeniers, Ward, mdpc, Lucas Kauffman Dec 15 '12 at 23:01

It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.If this question can be reworded to fit the rules in the help center, please edit the question.

3  
What operating systems? –  Michael Hampton Dec 15 '12 at 5:05

3 Answers 3

up vote 2 down vote accepted

It has probably millions of files, and I can't tgz them because for some reason it won't write the file past 7.5GB.

You can tgz them, just not write them down on the server:

ssh you@your.server "tar zcvf - /directory/with/millions/of/files" > millions-of-files.tar.gz

I'm assuming you have ssh access if you were considering tgz in the first place.

share|improve this answer
    
I ended up using zip and it worked fine, but I would have gone with this had I seen it. –  Zonedabone Dec 19 '12 at 22:59

Something better than FTP? Using a different protocol is not going to reduce the number of files you have. rsync over ssh using tar gz seems like it would work best.

share|improve this answer

The previous answers are on the right track, but I'd recommend creating a multiple part compressed file, and then transfer the directory using your file transfer method of choice.

This way you can transfer many of these files in parellel, giving for a much faster completion, depending on your network connection of course.

See the question:

How can I create multipart tar file in Linux?

http://superuser.com/questions/198857/how-can-i-create-multipart-tar-file-in-linux

Also, I was assuming you were using Linux. But if you are not, WinRAR can easily do this on Windows. Then transfer using NFS or whatever you prefer.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.