Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have two directory trees with similar layouts, i.e.

.
 |-- dir1
 |   |-- a
 |   |   |-- file1.txt
 |   |   `-- file2.txt
 |   |-- b
 |   |   `-- file3.txt
 |   `-- c
 |       `-- file4.txt
 `-- dir2
  |-- a
  |   |-- file5.txt
  |   `-- file6.txt
  |-- b
  |   |-- file7.txt
  |   `-- file8.txt
  `-- c
   |-- file10.txt
   `-- file9.txt

I would like to merge the the dir1 and dir2 directory trees to create:

 merged/
 |-- a
 |   |-- file1.txt
 |   |-- file2.txt
 |   |-- file5.txt
 |   `-- file6.txt
 |-- b
 |   |-- file3.txt
 |   |-- file7.txt
 |   `-- file8.txt
 `-- c
     |-- file10.txt
     |-- file4.txt
     `-- file9.txt

I know that I can do this using the "cp" command, but I want to move the files instead of copying, because the actual directories I want to merge are really large and contain lots of files (millions). If I use "mv" I get the "File exists" error because of conflicting directory names.

UPDATE: You can assume that there are no duplicate files between the two directory trees.

share|improve this question
    
Are you sure there is no duplication of filenames between the two folders? what do you want to happen if there is duplicates? –  Zoredache Jun 29 '10 at 18:13
    
If you literally have millions of files in a single directory then you should look into splitting the files up into separate sub directories for performance reasons - although this is irrelevant to the actual question asked. –  DrStalker Jun 30 '10 at 1:51

6 Answers 6

up vote 23 down vote accepted
rsync -aplx --link-dest=dir1/ dir1/ merged/
rsync -aplx --link-dest=dir2/ dir2/ merged/

This would create hardlinks rather than moving them, you can verify that they were moved correctly, then, remove dir1/ and dir2/

share|improve this answer
8  
Kind of. It doesn't actually duplicate any disk usage, it simply creates another pointer to the same hunk of disk, and doesn't actually 'copy' any data. (See en.wikipedia.org/wiki/Hard_links) However, it does have to do that operation once per file. But that's essentially what all of these answers wind up doing, since you can't just move a single directory. –  Christopher Karel Jun 29 '10 at 19:15
1  
Since it does not have the io overhead of copying files, this is a perfectly acceptable solution. –  Tobu Jun 29 '10 at 19:25
2  
This only works if they are on the same file system though. Would rsync with the delete option do a move if they were on the same file system? (that is to say, just change the directory info, but not move the file). –  Ronald Pottol Jun 29 '10 at 19:34
1  
rsync will copy, then delete if it traverses filesystems. –  karmawhore Jun 29 '10 at 20:22
2  
One caveat: make the --link-dest path absolute, or relative to merged/; or it will copy. –  Tobu Jul 3 '10 at 21:19

You can use rename (aka prename, from the perl package) for that. Beware that the name doesn't necessarily refer to the command I describe outside of debian/ubuntu (though it's a single portable perl file if you need it).

mv -T dir1 merged
rename 's:^dir2/:merged/:' dir2/* dir2/*/*
find dir2 -maxdepth 1 -type d -empty -delete

You also have the option of using vidir (from moreutils), and editing the file paths from your preferred text editor.

share|improve this answer

Brute force bash

#! /bin/bash

for f in $(find dir2 -type f)
do
  old=$(dirname $f)
  new=dir1${old##dir2}
  [ -e $new ] || mkdir $new
  mv $f $new
done

test does this

# setup 
for d in dir1/{a,b,c} dir2/{a,b,c,d} ; do mkdir -p $d ;done
touch dir1/a/file{1,2} dir1/b/file{3,4} dir2/a/file{5,6} dir2/b/file{7,8} dir2/c/file{9,10} dir2/d/file11

# do it and look
$ find dir{1,2} -type f
dir1/a/file1
dir1/a/file2
dir1/a/file5
dir1/a/file6
dir1/b/file3
dir1/b/file7
dir1/b/file8
dir1/c/file4
dir1/c/file9
dir1/c/file10
dir1/d/file11
share|improve this answer
2  
The OP specified millions of files, which is likely to break this construction. Also, it will not properly handle filenames with spaces, newlines, etc.. –  Chris Johnsen Jun 29 '10 at 20:50

I like the rsync and prename solutions, but if you really want to make mv do the work and

  • your find knows -print0 and -depth,
  • your xargs knows -0,
  • you have printf,

then it is possible to handle a large number of files that might have random whitespace in their names, all with a Bourne-style shell script:

#!/bin/sh

die() {
    printf '%s: %s\n' "${0##*/}" "$*"
    exit 127
}
maybe=''
maybe() {
    if test -z "$maybe"; then
        "$@"
    else
        printf '%s\n' "$*"
    fi
}

case "$1" in
    -h|--help)
        printf "usage: %s [-n] merge-dir src-dir [src-dir [...]]\n" "${0##*/}"
        printf "\n    Merge the <src-dir> trees into <merge-dir>.\n"
        exit 127
    ;;
    -n|--dry-run)
        maybe=NotRightNow,Thanks.; shift
    ;;
esac

test "$#" -lt 2 && die 'not enough arguments'

mergeDir="$1"; shift

if ! test -e "$mergeDir"; then
    maybe mv "$1" "$mergeDir"
    shift
else
    if ! test -d "$mergeDir"; then
        die "not a directory: $mergeDir"
    fi
fi

xtrace=''
case "$-" in *x*) xtrace=yes; esac
for srcDir; do
    (cd "$srcDir" && find . -print0) |
    xargs -0 sh -c '

        maybe() {
            if test -z "$maybe"; then
                "$@"
            else
                printf "%s\n" "$*"
            fi
        }
        xtrace="$1"; shift
        maybe="$1"; shift
        mergeDir="$1"; shift
        srcDir="$1"; shift
        test -n "$xtrace" && set -x

        for entry; do
            if test -d "$srcDir/$entry"; then
                maybe false >/dev/null && continue
                test -d "$mergeDir/$entry" || mkdir -p "$mergeDir/$entry"
                continue
            else
                maybe mv "$srcDir/$entry" "$mergeDir/$entry"
            fi
        done

    ' - "$xtrace" "$maybe" "$mergeDir" "$srcDir"
    maybe false >/dev/null ||
    find "$srcDir" -depth -type d -print0 | xargs -0 rmdir
done
share|improve this answer

I have had to do this several times for source code trees at different stages of development. My solution was to use Git in the following way:

  1. Create a git repository and add all of the files from dir1.
  2. Commit
  3. Remove all files and copy in files from dir2
  4. Commit
  5. View differences between the two commit points and make careful decisions about how I want to merge the results.

You can finesse it with branching and so forth but this is the general idea. And you have less fear about stuffing it up because you have complete snapshot of each state.

share|improve this answer

It's strange nobody noted that cp has option "-l":

-l, --link
       hard link files instead of copying

You can do something like

% mkdir merge
% cp -rl dir1/* dir2/* merge
% rm -r dir*
% tree merge 
merge
├── a
│   ├── file1.txt
│   ├── file2.txt
│   ├── file5.txt
│   └── file6.txt
├── b
│   ├── file3.txt
│   ├── file7.txt
│   └── file8.txt
└── c
    ├── file10.txt
    ├── file4.txt
    └── file9.txt

13 directories, 0 files
share|improve this answer
    
This doesn't work across different hard drives... –  Alex Leach Apr 18 '12 at 11:18
    
It is more correct to say that it does not works across filesystems, because filesystems can span across multiple hard drives. Also, if what op wants is to avoid copying the files, it is a good thing that cp -l does not work across filesystems. –  lvella May 22 '12 at 11:59
1  
You may want to use cp -a (synonym to cp -RPp) to keep all attributes of the files and avoid following symlinks: here the command becomes cp -al dir1/* dir2/* merge. –  tricasse Dec 27 '12 at 15:24

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.