Now I know you shouldn't ever put 10 million files into a single directory to begin with. Blame it on the developers, but as it stands that’s where I am at. We will be fixing it and moving them into folder groups, but first we gotta get them copied off of the production box.
I first tried rsync but it would fail out miserably. I assume it was because storing the name and path of the files in memory was greater than the ram and swap space.
Then I tried to compress it all into a tar.gz but it couldn't unzip it, file too large error (it was 60gigs).
I tried to just do a tar to tar exaction, but I got a "cannot open: file too large"
tar c images/ | tar x –C /mnt/coverimages/
/mnt/coverimages/ is an nfs share where we want to move the images to.
All files are images