Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

silly one,

do you have any problems with rsync'ing large [ >4GB ] files under modern linux? [ 32bit, 64bit, large file support turned on ]? i've done some tests on my own between 2 64bit boxes and didn't have any problems transferring 6-10GB files. to make test thorough i altered files, run rsync again, checked md5... - all seems ok.

but after i saw this bug report i got a bit worried. i some searching but have not found any confirmation of the problem.

thanks for your thoughts!

edit: file system: ext3, reiserfs

share|improve this question

5 Answers 5

up vote 2 down vote accepted

The error report that you've linked to does not seem to be a 4GB+ filesize related error. 429796854 bytes is just shy of 410Mb, and it seems to be a transport error rather than an rsync one. If

I would suspect that the transport connection (presumably SSH) has dropped, perhaps due to an inactivity timeout because the CPU spent so long between sends because it had to do something like checksum a very large file, and this is the reason that rsync reports a broken pipe.

I'm sure I've used rsync successfully on files over 4Gb in the past with 32bit clients and servers, and at least once where more then 4Gb was actually transferred rather than just considered for transfer.

share|improve this answer
    
lol - a very good point. i actually skipped the number and got quite unpleasantly surprised just by the title. –  pQd Jul 15 '09 at 13:48

Nope, I throw around 5-10GB VM images using rsync all the time, never seen a problem.

share|improve this answer

The largest file I have recently rsynced was 180GB which was in a a set of directories containing 15TB ( I have written a set of scripts which can do the sync in parallel, I can move the data at about 3TB an hour ... )

share|improve this answer

Depends on the filesystem that you're using. I've run into trouble with FAT32 filesystems. I had a 200GB portable harddrive (formatted as FAT32) and was trying to copy a DVD .iso onto it. It didn't work cause you can't have files greater than 4.somethingGB in FAT32

share|improve this answer

No, we synchronize two 30TB data sets (made of files ranging from 4 to 20GB) daily for months with rsync, no problem.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.