Using rsync from msysgit for binary files

Question

I have three windows boxes that are connected to a linux box on my network in various ways (one's local to the network, one is a laptop and may or may not be local, and the other is my work desktop). I'm currently making use of git to syncronize most of my stuff between the boxes (using the Linux box as the server) and it works like a champ.

Now, however, I'd also like to add my MP3 collection (12 gigs) to be efficiently synchronized between boxes. I initially thought I could use git for this, but it looks like rsync would be a better choice since I don't really need all the historical stuff. Since all my existing management scripts run under git bash (instead of cygwin), I'd like to be able to use it from git bash (MINGW32 is what it says in the title bar of the window). Can I just download it in Cygwin and copy it over or is there a better way to approach it? The Linux box is running the latest version of Ubuntu Server edition - do I need to do anything to it to enable rsync to work correctly on it?

I realize I'm probably doing this in a slightly harder way than is needed, but I'm also kind of using this as an exercise in improving my commandline skills. Any suggestions?

---

I can use rsync from my cygwin shell, but not from msysgit. I tried copying rsync.exe from cygwin's bin folder over to msysgit's bin folder, but it still isn't finding it. Is there some other step I need to do to install it there? It's strange that I'm getting an error that it can't find the file, instead of giving me an error that one of the file's dependencies is missing.

Answer 1

My original instructions to get rsync working with Git for Windows were quite a bit outdated, so I've revamped this answer for mid-2023 to cover MSYS2's switch to zstd and the additional dependencies required for rsync. The primary issue is that MSYS2 now compresses most packages (including zstd, rsync, and dependencies) using zstd, but zstd isn't included in the base environment. Because of this we can't get zstd from the MSYS2 repository--but we can get it directly from the zstd Github releases where it is a zip archive rather than zst. The basic idea for getting rsync working that I originally came across in this post is still the same, but due to the above mentioned issues it requires a bit more work. The whole process is still easily doable once we know everything required.

Below are commands to:

• download the zstd binary to use in extracting further packages
• download, extract, and place the rsync binary and it's dependencies in the correct bin folder

These steps have been tested on a clean install of Windows 11 with only the latest Git for Windows installed (should work for Windows 10 as well). No other dependencies should be required to complete the steps.

As always, examine what you paste into a terminal and use at your own risk

Step 1: zstd requirement

If you have an environment that doesn't already support extracting archives compressed with zstd (such as with Git for Windows), the easiest way is probably to download the latest zstd Windows binary directly from the ztsd release page: https://github.com/facebook/zstd/releases. If you already have zstd available and in your sourced path then skip to step 2 (and you may need to adjust the zstd binary location in the script).

Copy the below lines into a Git Bash terminal after adjusting below variables for latest source locations, versions, and your preferred download location:

working="/c/temp-downloads"
zstd_url="https://github.com/facebook/zstd/releases/download/v1.5.5/zstd-v1.5.5-win64.zip"
mkdir -p "$working"
curl -L -o "$working/zstd.zip" "$zstd_url"
unzip -j "$working/zstd.zip" "*/zstd.exe" -d "$working"

Step 2: Downloading compatible rsync and dependency binaries, extracting, and placing in proper location

Copy the below lines into an admin Git Bash terminal after adjusting below variables for latest source locations, versions, and your preferred download location. The terminal must be running as admin if the destination bin folder is in a location requiring privileged access.

# If you downloaded zstd above, make sure your working folder is the same as where the zstd binary is located
working="/c/temp-downloads"
# Set the location of the Git usr/bin folder (under normal installations we can simply use /usr/bin because the MSYS2 root is the Git installation folder)
git_usr_bin_folder_location="/usr/bin"
rsync_url="https://repo.msys2.org/msys/x86_64/rsync-3.2.7-2-x86_64.pkg.tar.zst"
libxxhash_url="https://repo.msys2.org/msys/x86_64/libxxhash-0.8.1-1-x86_64.pkg.tar.zst"
mkdir -p "$working"
curl -L -o "$working/rsync.tar.zst" "$rsync_url"
curl -L -o "$working/libxxhash.tar.zst" "$libxxhash_url"
tar -I "$working/zstd.exe" -xvf "$working/rsync.tar.zst" --directory="$working"
tar -I "$working/zstd.exe" -xvf "$working/libxxhash.tar.zst" --directory="$working"
cp -a "$working/usr/bin/." "$git_usr_bin_folder_location"

That should be all you need to get rsync working as everything else necessary should already be included in the base Git for Windows MSYS2 environment. You can now clean up the contents of the working folder. Hopefully at some point zstd will be included in the base environment and the extra step of acquiring it won't be needed, but it is what it is for now.

If you want to also use Pageant authentication with rsync over ssh, the ssh-pageant binary is already installed in Git for Windows and all you need to do is add a small bit to your ~/.bashrc or ~/.bash_profile file (see ssh-pageant usage section and Git for Windows wiki). I've modified it specifically for use with the Git for Windows environment and enhanced it with some detection for a missing socket file:

# ssh-pageant allows use of the PuTTY authentication agent (Pageant)
SSH_PAGEANT="$(command -v ssh-pageant)"
if [ -S "$SSH_AUTH_SOCK" ]; then
  PAGEANT_SOCK=$SSH_AUTH_SOCK
else
  PAGEANT_SOCK=${TEMP:-/tmp}/.ssh-pageant-$USERNAME
fi
if [ -x "$SSH_PAGEANT" ]; then
  eval $("$SSH_PAGEANT" -qra "$PAGEANT_SOCK")
fi
unset SSH_PAGEANT
unset PAGEANT_SOCK

Answer 2

I know the question isn't "Using rsync for MSYS", but I think mentioning this is relevant:

"MinGW Shell" (formerly best known as "MSYS (Basic System)") does indeed have rsync. I'm using that shell/system as often as possible since it's much smaller than Cygwin. (I'm probably the only one that cares about file sizes these days.)

"Git for Windows" / "Git Bash" could easily add rsync since it based on MinGW/MSYS and it has been mentioned on the mailing list earlier.

PS! If your MinGW Shell doesn't have rsync, just run "mingw-get install msys-rsync" to add it.

Updated 25th of March 2016:

Git for Windows now uses MSYS2. And MSYS2 uses "pacman", not "mingw-get", to install packages. Unfortunately, "Git for Windows" doesn't include pacman ... There are two options:

1. Install the Git for Windows SDK (which includes pacman).
2. Install Git inside MSYS2.

The last option is my favorite, but it's not streamlined yet so the first option is probably easier. Hopefully, it will be fixed.

PS! I haven't played much with MSYS2 / pacman, but it seems very promising.

Updated 7th of August 2019:

If you are using Git for Windows (Git Bash) it's much better to just grab the rsync package from the MSYS2 Package repository, as is described in the answer above.

Answer 3

I assume by git bash you really mean msysgit, aka "git for windows." The Start Menu option is git bash, which explains the name you used..

First, you are correct in dismissing git as an option here. One of the things git does not do well is handling binary files--and you'd end up with a huge, bloated repository. Additionally, since you won't be handling changes, using the overhead of git strikes me as overkill.

Secondly, it sounds like the question you're actually asking is, "Can I use rsync from msysgit/Windows?" The short answer is, not out of the box. msysgit is really just the Windows command line in terms of form and function. That being said, you have two options: download a third-party tool to enable rsync on your Windows machine (Stack Overflow: Rsync for Windows), or just use cygwin. Really, they have the same outcome--so its up to you to decide which ones work better.

Answer 4

Update January 2021:

Since mysys2 is now using zstd you need to download more than just rsync to get this working. As well as rsync-3.2.3-1-x86_64.pkg you also need to download and copy libzstd-1.4.8-1-x86_64.pkg libxxhash-0.8.0-1-x86_64.pkg and for completeness I also downloaded zstd-1.4.8-1-x86_64.pkg

and 7-zip doesn't do the job of extracting the .zst compressed files wither, so I had to get PeaZip.

Answer 5

For what it's worth, I was having similar problems getting cwRsync to work in msysgit when dropped into the bin path, but after trying lots of different things eventually found that simply updating to the latest version (2.10.2) of Git for Windows fixed the issue.

Answer 6

rsync is a great for unidirectional sync. You can either run rsync under cygwin, or for Windows there is also Deltacopy

If you wanted to have bidirectional sync, then take a look at unison

For a cross platform directory synchronisation tool, there is syncthing which runs on most platforms.