2

i'm working on a server migration and run into the following problem:

I'm running rsync to migrate all the data but for some reason there is a folder depth limit... This is done via rsync over ssh command:

rsync -raz --verbose --update --delete --exclude-from 'rsync-exclude.txt' /home 192.168.70.50:/

70.50 is the new server, and i'm running the command on the old server. Folder structures are extremely deep, up to 20 folders deep.

So, most files and folders are copied over but all from folder depth 10 and over are not. If I specify deeper folders

rsync -raz --verbose --update --delete --exclude-from 'rsync-exclude.txt'/home/user/blablabla/domain/web to 192.168.70.50:/home/user/blablabla/domain

it starts copying the remaining folders recursively...

Anyone who might have a fix for this?

2
  • Some versions of rsync have problems with compression on large files. If the transfer is local try dropping compression rsync -ra
    – Brian
    Jun 13, 2015 at 23:40
  • 1
    What is the contents of rsync-exclude.txt?
    – TomOnTime
    Jun 14, 2015 at 18:24

1 Answer 1

1

rsync has no limit on the depth of subdirectories it will traverse. More specifically, it exceeds the limit of the Linux file system. I just copied something that was about 500 levels deep and rsync copied it just fine.

If you are experiencing this problem, check the following:

  • is the destination file system incapable of handling that many subdirectories?
  • is the destination full?
  • does the --exclude-from file specifies something that restricts files?
  • is the total length of the path ("d/d/d/d" is 4 levels but 7 chars) beyond what the filesystem permits?
  • does turning off incremental recursion make it work? --no-inc-recursive (this is a newer algorithm and you may have discovered a bug)
  • does the copy work using some other technique?
  • does the problem go away with (or without) compression? -z

Here's another technique to try. If it works, but rsync doesn't, then you may have found a bug in rsync:

This will tar the files on one side, send the tar file to stdout, ssh to the other side and untar the files there.

SOURCEDIR=/path/to/source
DESTHOST=machinename
DESTDIR=/path/to/destination
cd "$SOURCEDIR" && tar -c -v -f - . | ssh "$DESTHOST" "cd $DESTDIR && tar -x -p -f -"

Warning: If $DESTDIR has spaces or special chars in it, the above will need to be more carefully quoted.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .