I have to get a 20GB directory from my server to my desktop. It has probably millions of files, and I can't tgz them because for some reason it won't write the file past 7.5GB. What's the best way to get the files down? I need something better than FTP because that will take ages with so many files. Will I not have the 7.5GB file limit if I skip gz?
1
-
3What operating systems?– Michael HamptonDec 15, 2012 at 5:05
Add a comment
|
3 Answers
It has probably millions of files, and I can't tgz them because for some reason it won't write the file past 7.5GB.
You can tgz them, just not write them down on the server:
ssh [email protected] "tar zcvf - /directory/with/millions/of/files" > millions-of-files.tar.gz
I'm assuming you have ssh access if you were considering tgz in the first place.
-
I ended up using zip and it worked fine, but I would have gone with this had I seen it. Dec 19, 2012 at 22:59
Something better than FTP? Using a different protocol is not going to reduce the number of files you have. rsync over ssh using tar gz seems like it would work best.
The previous answers are on the right track, but I'd recommend creating a multiple part compressed file, and then transfer the directory using your file transfer method of choice.
This way you can transfer many of these files in parellel, giving for a much faster completion, depending on your network connection of course.
See the question:
How can I create multipart tar file in Linux?
https://superuser.com/questions/198857/how-can-i-create-multipart-tar-file-in-linux
Also, I was assuming you were using Linux. But if you are not, WinRAR can easily do this on Windows. Then transfer using NFS or whatever you prefer.
