I have files with invalid characters like these


It is a Æ where something have gone wrong in the filename.

Is there a way to just remove all invalid characters?

or could tr be used somehow?

echo "009_-_�%86ndringshåndtering.html" | tr ???
  • 5
    The characters probably aren't "invalid", else the filesystem wouldn't store them (unless you did something really nasty to the FS). Have you tried changing your locale (e.g. to UTF8) to display the names correctly? – James O'Gorman Jan 10 '12 at 14:29

One way would be with sed:

mv 'file' $(echo 'file' | sed -e 's/[^A-Za-z0-9._-]/_/g')

Replace file with your filename, of course. This will replace anything that isn't a letter, number, period, underscore, or dash with an underscore. You can add or remove characters to keep as you like, and/or change the replacement character to anything else, or nothing at all.

| improve this answer | |
  • 5
    I used: f='file'; mv 'file' ${f//[^A-Za-z0-9._-]/_} – Louis Oct 7 '15 at 15:05
  • 1
    Look for the best solution by H. Hess below... (and my funny comment alongside :) ) – Jan Sila Feb 14 '19 at 15:20

I assume you are on Linux box and the files were made on a Windows box. Linux uses UTF-8 as the character encoding for filenames, while Windows uses something else. I think this is the cause of the problem.

I would use "convmv". This is a tool that can convert filenames from one character encoding to another. For Western Europe one of these normally works:

convmv -r -f windows-1252 -t UTF-8 .
convmv -r -f ISO-8859-1 -t UTF-8 .
convmv -r -f cp-850 -t UTF-8 .

If you need to install it on a Debian based Linux you can do so by running:

sudo apt-get install convmv

It works for me every time and it does recover the original filename.

Source: LeaseWebLabs

| improve this answer | |
  • 1
    this looks promising, but any idea how to tell what the encoding is? I have a directory called Save the current file in Word 97-2004 format\sco.workflow that got created on my Mac (via Microsoft Office) and the above encodings don't have any effect. – Sridhar Sarnobat Dec 7 '16 at 6:49
  • 1
    It's worth pointing out that by default convmv runs in "test" mode, where it just performs a dry run and tells you which files it would move. It will then tell you to run it again with the --notest option to actually rename the files. – Kenny Rasschaert Jan 28 '19 at 10:47

I had some japanese files with broken filenames recovered from a broken usb stick and the solutions above didn't work for me.

I recommend the detox package:

The detox utility renames files to make them easier to work with. It removes spaces and other such annoyances. It'll also translate or cleanup Latin-1 (ISO 8859-1) characters encoded in 8-bit ASCII, Unicode characters encoded in UTF-8, and CGI escaped characters.

Example usage:

detox -r -v /path/to/your/files
-r Recurse into subdirectories
-v Be verbose about which files are being renamed 
-n Can be used for a dry run (only show what would be changed)
| improve this answer | |
  • 4
    This should be much higher, I urge everyone to have a look at detox before essentially reinventing the wheel. If you look at the man page, you will see that it covers all the other proposed solutions here because of its flexibility. – emk2203 Apr 10 '18 at 8:04
  • 1
    Ezekiel 25:17 - Blessed is he who, in the name of charity and good will upvotes this solution, for he is truly his brother's keeper and the finder of lost children. – Jan Sila Feb 14 '19 at 15:18
  • 1
    Unintuitively, the path can not be '.' in debian. If you use a '.' it finds nothing. – isaaclw Sep 10 '19 at 18:19
  • I wonder if it really works, it seems remove/replace Chinese characters, e.g. 的节奏啊, but those characters are valid filename. – Fruit Sep 11 '19 at 19:50
  • Use absolute path! /home/you/some - relative "." doesnt work with it, otherwise great tool! :) – jave.web Mar 26 at 18:48

I assume you mean you want to traverse the filesystem and fix all such files?

Here's the way I'd do it

find /path/to/files -type f -print0 | \
perl -n0e '$new = $_; if($new =~ s/[^[:ascii:]]/_/g) {
  print("Renaming $_ to $new\n"); rename($_, $new);

That would find all files with non-ascii characters and replace those characters with underscores (_). Use caution though, if a file with the new name already exists, it'll overwrite it. The script can be modified to check for such a case, but I didnt put that in to keep it simple.

| improve this answer | |

Following answers at https://stackoverflow.com/questions/2124010/grep-regex-to-match-non-ascii-characters, You can use:

rename 's/[^\x00-\x7F]//g' *

where * matches the files you want to rename. If you want to do it over multiple directories, you can do something like:

find . -exec rename 's/[^\x00-\x7F]//g' "{}" \;

You can use the -n argument to rename to do a dry run, and see what would be changed, without changing it.

| improve this answer | |
  • Is there a way to modify this to keep foreign characters such as ü and ä for example? – Elder Geek Feb 6 '16 at 22:51
  • Only the second one worked for me. Everything was in the same directory so I'm not sure what's the difference..? – Shautieh Mar 9 '17 at 14:51
  • 1
    @Shautieh: the -n stops it from actually running. I'll clarify the answer. – naught101 Mar 13 '17 at 5:38
  • rename can be slow when dealing with lots of files. If you want to speed this up, push the check into find. I'm not sure how to do that though. – isaaclw Sep 10 '19 at 18:13

This shell script sanitizes a directory recursively, to make files portable between Linux/Windows and FAT/NTFS/exFAT. It removes control characters, /:*?"<>\| and some reserved Windows names like COM0.

sanitize() {
  shopt -s extglob;

  filename=$(basename "$1")
  directory=$(dirname "$1")

  filename_clean=$(echo "$filename" | sed -e 's/[\\/:\*\?"<>\|\x01-\x1F\x7F]//g' -e 's/^\(nul\|prn\|con\|lpt[0-9]\|com[0-9]\|aux\)\(\.\|$\)//i' -e 's/^\.*$//' -e 's/^$/NONAME/')

  if (test "$filename" != "$filename_clean")
    mv -v "$1" "$directory/$filename_clean"

export -f sanitize

sanitize_dir() {
  find "$1" -depth -exec bash -c 'sanitize "$0"' {} \;

sanitize_dir '/path/to/somewhere'

Linux is less restrictive in theory (/ and \0 are strictly forbidden in filenames) but in practice several characters interfere with bash commands (like *...) so they should also be avoided in filenames.

Great sources for file naming restrictions:

| improve this answer | |
  • 1
    It what I search! but add quotes to support dirs with spaces find "$1" -depth -exec bash -c 'sanitize "$0"' {} \; – mmv-ru May 22 '17 at 14:02

If you want to handle embedded newlines, multibyte characters, spaces, leading dashes, backslashes and spaces you are going to need something more robust, see this answer:

I put the script up on code.google.com if anyone is interested: r-n-f-bash-rename-script

| improve this answer | |
  • The script linked here solved the problem for me – Jeremiah Rose Nov 18 '18 at 23:36

I use this one-liner to remove invalid characters in subtitle files:

for f in *.srt; do nf=$(echo "$f" |sed -e 's/[^A-Za-z0-9.-]/./g;s/\.\.\././g;s/\.\././g'); test "$f" != "$nf" && mv "$f" "$nf" && echo "$nf"; done
  1. Only process *.srt files( * could be used in place of *.srt to process every file)
  2. Removes all other characters except for letters A-Za-z, numbers 0-9, periods ".", and dash's "-"
  3. Removes possible double or triple periods
  4. Checks to see if the file name needs changing
  5. If true, it renames the file with the mv command, then outputs the changes it made with the echo command

It works to normalize directory names of movies:

for f in */; do nf=$(echo "$f" |sed -e 's/[^A-Za-z0-9.]/./g' -e 's/\.\.\././g' -e 's/\.\././g' -e 's/\.*$//'); test "$f" != "$nf" && mv "$f" "$nf" && echo "$nf"; done

Same steps as above but I added one more sed command to remove a period at the end of the directory

X-Men Days of Future Past (2014) [1080p]
Modified to:

| improve this answer | |
  • That one liner worked perfectly ! Thanks. – himanshuxd Feb 14 at 14:50

for file in *; do mv "$file" $(echo "$file" | sed -e 's/[^A-Za-z0-9.-]//g'); done &

| improve this answer | |
  • 2
    You should explain what your code does and use proper formatting. Your code can cause files to be deleted by introducing collisions in the names. And running the entire thing in the background is kind of silly. – kasperd Jul 4 '17 at 23:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.