35

I have two directory trees with similar layouts, i.e.

.
 |-- dir1
 |   |-- a
 |   |   |-- file1.txt
 |   |   `-- file2.txt
 |   |-- b
 |   |   `-- file3.txt
 |   `-- c
 |       `-- file4.txt
 `-- dir2
     |-- a
     |   |-- file5.txt
     |   `-- file6.txt
     |-- b
     |   |-- file7.txt
     |   `-- file8.txt
     `-- c
         |-- file10.txt
         `-- file9.txt

I would like to merge the the dir1 and dir2 directory trees to create:

 merged/
 |-- a
 |   |-- file1.txt
 |   |-- file2.txt
 |   |-- file5.txt
 |   `-- file6.txt
 |-- b
 |   |-- file3.txt
 |   |-- file7.txt
 |   `-- file8.txt
 `-- c
     |-- file10.txt
     |-- file4.txt
     `-- file9.txt

I know that I can do this using the "cp" command, but I want to move the files instead of copying, because the actual directories I want to merge are really large and contain lots of files (millions). If I use "mv" I get the "File exists" error because of conflicting directory names.

UPDATE: You can assume that there are no duplicate files between the two directory trees.

2
  • Are you sure there is no duplication of filenames between the two folders? what do you want to happen if there is duplicates?
    – Zoredache
    Jun 29, 2010 at 18:13
  • If you literally have millions of files in a single directory then you should look into splitting the files up into separate sub directories for performance reasons - although this is irrelevant to the actual question asked.
    – DrStalker
    Jun 30, 2010 at 1:51

6 Answers 6

30
rsync -ax --link-dest=dir1/ dir1/ merged/
rsync -ax --link-dest=dir2/ dir2/ merged/

This would create hardlinks rather than moving them, you can verify that they were moved correctly, then, remove dir1/ and dir2/.

12
  • 9
    Kind of. It doesn't actually duplicate any disk usage, it simply creates another pointer to the same hunk of disk, and doesn't actually 'copy' any data. (See en.wikipedia.org/wiki/Hard_links) However, it does have to do that operation once per file. But that's essentially what all of these answers wind up doing, since you can't just move a single directory. Jun 29, 2010 at 19:15
  • 1
    Since it does not have the io overhead of copying files, this is a perfectly acceptable solution.
    – Tobu
    Jun 29, 2010 at 19:25
  • 2
    This only works if they are on the same file system though. Would rsync with the delete option do a move if they were on the same file system? (that is to say, just change the directory info, but not move the file). Jun 29, 2010 at 19:34
  • 1
    rsync will copy, then delete if it traverses filesystems.
    – karmawhore
    Jun 29, 2010 at 20:22
  • 5
    One caveat: make the --link-dest path absolute, or relative to merged/; or it will copy.
    – Tobu
    Jul 3, 2010 at 21:19
22

It's strange nobody noted that cp has option -l:

-l, --link
       hard link files instead of copying

You can do something like

% mkdir merge
% cp -rl dir1/* dir2/* merge
% rm -r dir*
% tree merge 
merge
├── a
│   ├── file1.txt
│   ├── file2.txt
│   ├── file5.txt
│   └── file6.txt
├── b
│   ├── file3.txt
│   ├── file7.txt
│   └── file8.txt
└── c
    ├── file10.txt
    ├── file4.txt
    └── file9.txt

13 directories, 0 files
3
  • This doesn't work across different hard drives...
    – Alex Leach
    Apr 18, 2012 at 11:18
  • 4
    It is more correct to say that it does not works across filesystems, because filesystems can span across multiple hard drives. Also, if what op wants is to avoid copying the files, it is a good thing that cp -l does not work across filesystems.
    – lvella
    May 22, 2012 at 11:59
  • 2
    You may want to use cp -a (synonym to cp -RPp) to keep all attributes of the files and avoid following symlinks: here the command becomes cp -al dir1/* dir2/* merge.
    – tricasse
    Dec 27, 2012 at 15:24
5

You can use rename (aka prename, from the perl package) for that. Beware that the name doesn't necessarily refer to the command I describe outside of debian/ubuntu (though it's a single portable perl file if you need it).

mv -T dir1 merged
rename 's:^dir2/:merged/:' dir2/* dir2/*/*
find dir2 -maxdepth 1 -type d -empty -delete

You also have the option of using vidir (from moreutils), and editing the file paths from your preferred text editor.

3

I like the rsync and prename solutions, but if you really want to make mv do the work and

  • your find knows -print0 and -depth,
  • your xargs knows -0,
  • you have printf,

then it is possible to handle a large number of files that might have random whitespace in their names, all with a Bourne-style shell script:

#!/bin/sh

die() {
    printf '%s: %s\n' "${0##*/}" "$*"
    exit 127
}
maybe=''
maybe() {
    if test -z "$maybe"; then
        "$@"
    else
        printf '%s\n' "$*"
    fi
}

case "$1" in
    -h|--help)
        printf "usage: %s [-n] merge-dir src-dir [src-dir [...]]\n" "${0##*/}"
        printf "\n    Merge the <src-dir> trees into <merge-dir>.\n"
        exit 127
    ;;
    -n|--dry-run)
        maybe=NotRightNow,Thanks.; shift
    ;;
esac

test "$#" -lt 2 && die 'not enough arguments'

mergeDir="$1"; shift

if ! test -e "$mergeDir"; then
    maybe mv "$1" "$mergeDir"
    shift
else
    if ! test -d "$mergeDir"; then
        die "not a directory: $mergeDir"
    fi
fi

xtrace=''
case "$-" in *x*) xtrace=yes; esac
for srcDir; do
    (cd "$srcDir" && find . -print0) |
    xargs -0 sh -c '

        maybe() {
            if test -z "$maybe"; then
                "$@"
            else
                printf "%s\n" "$*"
            fi
        }
        xtrace="$1"; shift
        maybe="$1"; shift
        mergeDir="$1"; shift
        srcDir="$1"; shift
        test -n "$xtrace" && set -x

        for entry; do
            if test -d "$srcDir/$entry"; then
                maybe false >/dev/null && continue
                test -d "$mergeDir/$entry" || mkdir -p "$mergeDir/$entry"
                continue
            else
                maybe mv "$srcDir/$entry" "$mergeDir/$entry"
            fi
        done

    ' - "$xtrace" "$maybe" "$mergeDir" "$srcDir"
    maybe false >/dev/null ||
    find "$srcDir" -depth -type d -print0 | xargs -0 rmdir
done
1
  • You can tell xargs to delimit its input to newline and skip the translation. for example the following would find and delete all your torrent files under the current directory, even ones with unicode characters or some other tomfoolery. find . -name '*.torrent' | xargs -d '\n' rm
    – PRS
    Jul 21, 2018 at 5:07
2

Brute force bash

#! /bin/bash

for f in $(find dir2 -type f)
do
  old=$(dirname $f)
  new=dir1${old##dir2}
  [ -e $new ] || mkdir $new
  mv $f $new
done

test does this

# setup 
for d in dir1/{a,b,c} dir2/{a,b,c,d} ; do mkdir -p $d ;done
touch dir1/a/file{1,2} dir1/b/file{3,4} dir2/a/file{5,6} dir2/b/file{7,8} dir2/c/file{9,10} dir2/d/file11

# do it and look
$ find dir{1,2} -type f
dir1/a/file1
dir1/a/file2
dir1/a/file5
dir1/a/file6
dir1/b/file3
dir1/b/file7
dir1/b/file8
dir1/c/file4
dir1/c/file9
dir1/c/file10
dir1/d/file11
1
  • 2
    The OP specified millions of files, which is likely to break this construction. Also, it will not properly handle filenames with spaces, newlines, etc.. Jun 29, 2010 at 20:50
0

I have had to do this several times for source code trees at different stages of development. My solution was to use Git in the following way:

  1. Create a git repository and add all of the files from dir1.
  2. Commit
  3. Remove all files and copy in files from dir2
  4. Commit
  5. View differences between the two commit points and make careful decisions about how I want to merge the results.

You can finesse it with branching and so forth but this is the general idea. And you have less fear about stuffing it up because you have complete snapshot of each state.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .