0

I'm using rsync -z to backup a large amount of XML files. The source is a huge machine with thousands of GBs available on disk, but the destination is not that powerful.

What I want to do is to compress XML files during transfer and keep them compressed at the destination. The option -z does the compression only during the transit.

Is it possible to do it using just one crontab command?

This is the command I am currently using:

*/30 * * * * rsync -rz -e 'sshpass -p "MY_PASSWORD" ssh -p 5022' --ignore-existing --quiet --delete REMOTE_USER@REMOTE_HOST:/path/to/remote/ /path/to/local/

3
  • Why aren't they gzipped on the source? If you don't gzip them at the source then rsync will try to transfer them again and again each time you run it, which will waste even more bandwidth and time. – Michael Hampton Jul 22 '20 at 19:54
  • The permission of REMOTE_USER is read only. And the REMOTE_HOST belongs to a large company that has very strict security policies – optimus_prime Jul 22 '20 at 19:58
  • Better get more storage, then. – Michael Hampton Jul 22 '20 at 20:44
0

I managed to solve this with a workaround. On local host, I just compress the syncronized file and overwrite the original file with the string "0". It prevents rsync from synchronizing the same file again and significantly reduces disk load.

exec('find /path/to/local/ -name "*.xml" -size +10k',$output);
foreach ($output as $key => $filepath) {`
        
        //gzip the xml file
        $gz = gzopen("/path/to/gz/".basename($filepath).".gz", "wb9");
        gzwrite($gz, file_get_contents($filepath));
        gzclose($gz);
        
        //overwrite xml file
        $xml = fopen($filepath,"w");
        fwrite($xml, "0");
        fclose($xml);
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.