I am looking for a bash script which is compatible with Mac, to find duplicate files in a directory.
24 Answers
Don't know about Mac compatibility, but this Works For Me(TM):
#!/bin/bash
[ -n "$1" ] || exit 1
exec 9< <( find "$1" -type f -print0 )
while IFS= read -r -d '' -u 9
do file_path="$(readlink -fn -- "$REPLY"; echo x)" file_path="${file_path%x}" exec 8< <( find "$1" -type f -not -path "$file_path" -print0 ) while IFS= read -r -d '' -u 8 OTHER do cmp --quiet -- "$REPLY" "$OTHER" case $? in 0) echo -n "cmp -- " printf %q "${REPLY}" echo -n ' ' printf %q "${OTHER}" echo "" break ;; 2) echo "\`cmp\` failed!" exit 2 ;; *) : ;; esac done
doneThe result is a set of commands you can run to check that the result is correct :)
Edit: The last version works with really weird filenames like:
$'/tmp/--$`\\! *@ \a\b\E\E\f\r\t\v\\"\' \n'it works for me on my mac, you will catch the duplicate file by their md5 value:
find ./ -type f -exec md5 {} \; | awk -F '=' '{print $2 "\t" $1}' | sortThis will find files under a dir with dupes. It's pretty raw, but it works.
#!/bin/bash
CKSUMPROG=md5sum
TMPFILE=${TMPDIR:-/tmp}/duplicate.$$
trap "rm -f $TMPFILE" EXIT INT
if [ ! -d "$1" ]
then echo "usage $0 directory" >2 exit 1
fi
PRINTBLANK=
# dump fingerprints from all target files into a tmpfile
find "$1" -type f 2> /dev/null | xargs $CKSUMPROG > $TMPFILE 2> /dev/null
# get fingerprints from tmpfile, get the ones with duplicates which means multiple files with same contents
for DUPEMD5 in $(cut -d ' ' -f 1 $TMPFILE | sort | uniq -c | sort -rn | grep -v '^ *1 ' | sed 's/^ *[1-9][0-9]* //')
do if [ -z "$PRINTBLANK" ] then PRINTBLANK=1 else echo echo fi grep "^${DUPEMD5} " $TMPFILE | gawk '{print $2}'
doneIf only interested in files in the current directory (as indicated by OP), then this is the simpliest. For linux and Windows (msys - tested, or MinGW or Cygwin with any with GnuWin32). This will list all duplicates.
md5sum * | sort | uniq -D -w 32For BSD/Mac OS X (it will list only the first duplicate)
md5 -r * | sort | uniq -d -w 32
