find and remove duplicates filenames in directory hierarchy

Question

#!/bin/sh
LASTBASE=""  
find $1 -type f -print | rev | sort | rev | while read FILE
do
    BASE=$(basename "$FILE")
    if [ "$BASE" = "$LASTBASE" ]; then
        rm "$FILE"
    LASTBASE="$BASE"
done

Answer 1

If you pipe the output of find into a while read loop you can deal with them line by line:

find nnn/ -type f -print | rev | sort | rev | while read FILE; do
    ...
done

Edit: So this method does break if filenames contain double (consecutive) spaces, because read actually splits the line up according to $IFS and then joins it again when storing the last variable. To address this you could temporarily change $IFS to disable splitting:

OIFS="$IFS"
IFS=""
find | while read...
IFS="$OIFS"

Edit: test (which is the same as [) doesn't have a == operator, you just want =.

Answer 2

I just found this "gem" in an old bash history and it, well, actually works without stumbling over whitespaces in filenames.

Content-wise Comparison

for hash in `find . -exec md5sum {} \; 2>/dev/null | sort | awk '{ print $1 }' | uniq -d`; do 
     find . -exec md5sum {} \; 2>/dev/null | grep $hash | awk '{print $2 }'; 
done;

informal:

• First line: traverse the directory tree and calculate the md5sum of all files below, sort this output (format: hash filename), grab the hash column, reduce it to doubled values. (means there are duplicates)
• Second line: for every one of the double-occuring hashes, traverse again and print the filename if the current file has the current hash (means the file is one of multiple)

example output:

./aFile
./aFolder/aFile
./1000digitsOfPI
./a/b/c/thousanddigitsofPI
./b File
./bFolder/cFolder/b File

Removing is not implemented here because it might be hard to decide which version of the doubled files you want to keep.

---

Filename-wise Comparison

If you just want to look at filenames and not at contents, it gets even easier:

for name in `find . -type f -printf "%f\n" | sort | uniq -d`; do 
    find . -name $name; 
done;

Update: Unfortunately this version is breaking with whitespaces in filenames again.

Answer 3

The problem lies in this line of code for FILE in $FILES; do - the for loop is assigning the FILE variable based on the white space separator. So if a file has one or more whitespaces then it won't work. Simply change the default IFS from space to new line or tab. If I remember correctly you can set IFS in bash using something like this -

IFS=$'\n'