I have two servers: our production web server(freebsd) and a backup server(centos). The production server does mysql database dumps every 3 hours and compresses the dump file using bzip2. So we have a folder at /backups/dumps/ and the files are like database_dump-20120119_152100.bz2.

On our backup server, we have a /BackupRaid/webserver/ folder. On this server, an rsync script is ran every 3 hours sync the /backups/dumps folder on the production server to the /BackupRaid/webserver folder on the backup server. One of these backup files is around 500MB. If you uncompress it, it's 3.2GB.

This processes has worked fine for years now. But just recently I checked the backup server and saw that for the last week, there is a 500MB bz2 file for every 3 hour time period as expected, but then there's also a 3.2GB uncompressed version of each of those files. So now there is double the files, one uncompressed and one compressed for each time period.

I find out when the next sync is going to be and I sit and monitor it. On the backup server I see a ".database_dump-201201119_182100.tlv1d4" file, which means the rsync is in progress. I watch the file and it grows and grows passed the 500MB that it should be. It proceeds to download a 3.2GB uncompress dump file. I log in to the web server and check the folder that it's syncing with and there are only 500MB bz2 files in the folder. Once it's finished with the 3.2GB file, it proceeds to download the 500MB bz2 file.

So where is it getting this 3.2GB uncompressed dump file from? It's acting as if the remote server is uncompressing the file before transferring, transferring the 3.2GB dump file to the backup server and then the backup server compresses it again back to the 500MB size.

I used lsof, grepped for the filename, found the PID of rsync process. Checked the PID from ps aux and sure enough it was coming from my script. My script is a very simple command:

#Rsync files, timeout set to 5 seconds
until rsync -avv --progress --partial --timeout=5 "${SOURCE_USER}"@"${SOURCE_SERVER}":"${SOURCE_PATH}" "${TARGET_PATH}";
do echo "Didn't quite get the whole file before an error/timeout occurred. Restarting where it left off..." >&2;
sleep 1;

What is going on??


Looks like you are using the -a switch with rsync, which equates to -rlptgoD, which includes recursive, links, file times, etc.

On top of that, from the command you posted, it looks like you are calling out a path on the source side.

So when using a path, with the -a or -r switch, rsync will synchronize all files in the path. I assume the large file also exists in the source path, thus it is being copied over as well.

  • 2
    To be more explicit on this answer, the way you do your database dumps may create that 3.2 GB file. It then compresses the file. But your rsync process already has grabbed the 3.2 GB file too. If update the question with how you do the mysql backups, we may be able to give advice. – becomingwisest Jan 19 '12 at 19:17
  • @Tim - No, they do not exist in the path. – Safado Jan 19 '12 at 19:29
  • @Christopher That's what I'm guessing is happening. We do the database dump and then bzip the the dump. Since the file is 3.2GB, I'm guessing it takes a minute or two to compress it and it's during that time that the rsync starts. I've changed the cron to dump STOUT to a text file so I can see if rsync is actually seeing two files that it needs to sync instead of one. – Safado Jan 19 '12 at 19:32
  • @ChristopherEvans Thanks for the helpful input! +1/2 for each of us ;) – Tim Jan 19 '12 at 19:42
  • By default, crons that have output email the user that it is running as. If you want to make sure you get the output, be sure to add --progress to the rsync command. – becomingwisest Jan 19 '12 at 19:59

Sounds to me like the bzip2 is not finished before rsync starts. BTW - do you ever delete those dumps on either side?

  • Yup, it's the next line down on the script. – Safado Jan 19 '12 at 21:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.