31

I have a home file server running FreeNAS 8. A few days ago I used rsync to upload my entire iTunes library from Mac so that I could load my library over the network instead of off a slow USB drive. This mostly worked, and iTunes runs much better now, but I'm running into issues accessing any songs that have non-ascii characters in it (I first noticed the problem when loading Queensrÿche tracks). The files would show up in the Finder, but any attempt to access them made them vanish until I reconnected to the server.

After some research I found out this is because OSX uses a different UTF character order from Linux. OSX filesystems use Unicode Normalization Form D (NFD), where linux uses Form C (NFC). Rsync doesn't convert these forms when it performs the copy from my mac to the server, now when iTunes tries to access a file with a special character over the network, the files on the server have the wrong encoding and afpd reports they don't exist.

What is the best way to address this problem? Is it possible to make rsync perform the unicode conversion while uploading the base library to the server? Can I configure afpd to transmit/receive filenames in NFD format? Is there an easy solution to change the filenames on the server? I found some stuff about a program named convmv, but I don't know if I can run that on FreeNAS.

2
  • 1
    Sounds like a bug with the OS X version of rsync. Jun 10, 2012 at 23:44
  • I know this is a real old thread but would like to add comments on my recent (Catalina) experience regarding this. It appears to me that Apple's new APFS file system is NFD/NFC agnostic. i.e. the files system accepts both for filenames (unlike HFS+ which forces NFD). My Synology NAS also accepts both NFC and NFD encoded files. Futhermore, The Mac iTunes app writes files with NFC encoding. So I've decided to convert my data on my APFS disk to NFC. rsync (via ssh) works great without any --iconv parameter. However, It appears that when mounting the NAS file system (either SMB or AFP) all the fil Feb 5, 2020 at 22:07

5 Answers 5

4

Note: If you are using version 3.0.0 or newer of rsync, the --iconv option as mentioned in the other answers is clearly the superior solution.

Something that should work is rsyncing between the source directory and the mounted remote file system (SMB, NFS, AFP), which rsync will just treat as local file system.

However, I do not know how well this works in practice, and you have to work around different issues, for example the delta-transfer algorithm won’t be used by default (since source and destination are “local”) (maybe --no-whole-file will work?), you have to check,e.g., that SMB effectively preserves modification times, etc.

1
  • This is ultimately what I ended up doing. I deleted the entire collection from the NAS and ran rsync again, using the locally mounted CIFS connection instead of the rsync daemon on the NAS. Now I'm just fixing itunes issues from filename capitalization. :/
    – Twipped
    Jul 15, 2012 at 23:52
59

You can use rsync's --iconv option to convert between UTF-8 NFC & NFD, at least if you're on a Mac. There is a special utf-8-mac character set that stands for UTF-8 NFD. So to copy files from your Mac to your NAS, you'd need to run something like:

rsync -a --iconv=utf-8-mac,utf-8 localdir/ mynas:remotedir/

This will convert all the local filenames from UTF-8 NFD to UTF-8 NFC on the remote server. The files' contents won't be affected.

6
  • 2
    I'm not the Original Poster so it's not up to me, but this is a much much cleaner and efficient solution than the one flagged as accepted. Starring this for sure, will be so useful.
    – ItsGC
    Sep 13, 2012 at 21:20
  • 1
    Great answer; I had no idea that UTF8-MAC stands for NFD; when used with iconv itself, this provides a generic mechanism for translating back and forth between NFC and NFD.
    – mklement
    May 7, 2014 at 15:00
  • great answer, this solves a long standing problem in sync a mac with a linux server!
    – meduz
    Nov 1, 2015 at 12:53
  • 2
    On mac you may also need brew tap homebrew/dupes && brew install homebrew/dupes/rsync && rehash Apr 20, 2016 at 6:05
  • 1
    I get rsync: --iconv: unknown option
    – KMC
    Dec 26, 2018 at 20:57
7

Currently I'm using rsync --iconv like this:

Copying files from Linux server to OS X machine

You should execute this command from OS X machine:

rsync -a --delete --iconv=UTF-8-MAC,UTF-8 '[email protected]:/home/username/path/on/server/' /Users/username/path/on/machine/

Copying files from OS X machine to Linux server

You should execute this command from OS X machine:

rsync -a --delete --iconv=UTF-8-MAC,UTF-8 /Users/username/path/on/machine/ '[email protected]:/home/username/path/on/server/'
1

Don't use rsync to copy the files to your NAS. When you use rsync to copy the files the filenames will be stored on your NAS in UTF NFD format (i.e. the OSX format) but Samba server running on your NAS only understands UTF NFC format filenames. Use the CIFS/SMB (Samba) interface to copy the files and the everything will be fine.

0

From my experience I recommend using SMB instead of ssh. Iconv solves problem with encoding, but there is still problem with allowed characters on diffrent systems:

Original file name on Mac:

https-//img-9gag-fun.9cache.com/photo/adK9jzN_460s

After copy by rsync over SMB:

-as seen by Mac (over SMB):     https-//img-9gag-fun.9cache.com/photo/adK9jzN_460s
-as seen by Ubuntu (over SMB):  https-img-9gag-fun.9cache.comphotoadK9jzN_460s
-as seen by Windows10 (over SMB):   https-∀∀img-9gag-fun.9cache.com∀photo∀adK9jzN_460s
-as seen by Ubuntu server locally:  https-img-9gag-fun.9cache.comphotoadK9jzN_460s

After copy by rsync over ssh (with ant without iconv flag):

-as seen by Mac (over SMB):     H0INHQ~6
-as seen by Ubuntu (over SMB):  H0INHQ~6
-as seen by Windows10 (over SMB):   H0INHQ~6
-as seen by Ubuntu server locally:  https-::img-9gag-fun.9cache.com:photo:adK9jzN_460s

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .