1

I know I can go like:

rsync -zaP uploads/ myserver:/path/to/

If I understand right, that will compress each file in that directory and sync one by one to the server. But if it's thousands of files, then it's going to take some time. Much faster would be to compress the whole directory and upload that.

So, is there a porcelain way for me to do that with rsync?

Helper function

I wrote a little bash function to compress and move move a whole directory with rsync. How would you simplify it or make it better?

function zipsync() {
    # Arguments
    local source=$1
    local target=$2

    # Get the host and path from the $target string
    IFS=':' read -a array <<< "$target"
    local host=${array[0]}
    local remote_path=${array[1]}

    # The archive file locations
    local remote_archive=${remote_path}${source}.tar.gz
    local local_archive=${source}.tar.gz

    # Colors
    cya='\033[0;36m'; gre='\033[0;32m'; rcol='\033[0m'

    echo -e "$cya Compressing files $rcol"
    tar -zcvf $local_archive $source

    echo -e "$cya Syncing files $rcol"
    rsync -avP $local_archive $target

    echo -e "$cya Extracting file in remote $remote_archive $rcol"
    ssh $host "cd ${remote_path}; tar zxvf ${remote_archive}"

    echo -e "$cya Removing the archives $rcol"
    ssh $host "rm $remote_archive"
    rm $local_archive

    echo -e "$gre All done :) $rcol"
}

Syntax:

zipsync source target

Example:

$ zipsync uploads my_server:/var/www/example.com/public_html

Problems with that function:

  1. Can't tab complete in local machine.
  2. Can't tab complete in remote server.
  3. Can't specify port in the target path, this won't work: zipsync uploads -p 5555 bob@xmpl.com:/path/ cause the -p is read as the second parameter.

My goal was to make a command that is really simple to use and reassembles rsync.

8

If I understand right, rsync -zaP uploads/ myserver:/path/to/ will compress each file in that directory and sync one by one to the server.

This is incorrect. The rsync command looks at the local files, compares them to the remote files (if any), and then synchronises the differences to the server. If there are no matching remote files then there will be no speed increase. However, for subsequent uploads where only some of the files have been changed, the speed increase can become dramatic. The -z flag attempts to apply compression to the data being transferred over the link.

But if it's thousands of files, then it's going to take some time. Much faster would be to compress the whole directory and upload that. So, is there a porcelain way for me to do that with rsync?

Your understanding was flawed so I think this question becomes moot. The rest of your post doesn't appear to be a question so I'm not sure what answers you're expecting. Please update the question if I've got that wrong.

| improve this answer | |
  • say the differences are all the files, e.g. the files do not exist at the destination, will rsync zip all the files individually and transport them one-by-one, or will it zip them together, and transfer them as one file (unzipping at the destination)? – James Owers Nov 19 '19 at 18:02
  • @JamesOwers I think you're misunderstanding rsync in the same way that the OP was. My answer applies to your comment too. – roaima Nov 19 '19 at 19:02
  • I understand it's going to send differences not whole files, is that what you mean (in my example above it will send whole files though)? The line in your answer of interest to me is "The -z flag attempts to apply compression to the data being transferred over the link" - does it great a single zip file internally and send that, or does it zip individual changes? – James Owers Nov 20 '19 at 7:50
  • @JamesOwers there is no zip involved. The data blocks are compressed (using the zlib library). More detail is in the man page for rsync. – roaima Nov 20 '19 at 8:09
  • 1
    @JamesOwers oh I see. I think. As far as I know the transmission blocks are treated by the compression layer as a continuous stream - including blocks that contain the list of files, if appropriate - so if you were to send two short identical files the second one might be compressed using the dictionary derived from the first. However, since I don't know this for sure I've asked the question over on Unix & Linux at Is the rsync block compression dictionary reset for each file. – roaima Nov 20 '19 at 10:17
5

If you want to run this only a single time where the destination is empty then it can be faster yes.
But your function is over complicated.
You can just run:

 tar zcvf - /source | ssh destination.example.com "cd /destination; tar xvzf -"

If you want to run the sync to synchronize changes, then see the answer from roaima.

| improve this answer | |
  • @faker Could you please clarify for what the dash - actually does there? – james_blunt Mar 26 '15 at 12:46
  • The first dash tells tar to ouput to stdout, the last dash tells tar to read from stdin – faker Mar 26 '15 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.