Migrate multiple svn repositories into single git repository

Question

We want to migrate from svn to git permanently to be able to use git's better features in terms of branching and collaboration.

Our current svn repository looks like this

svnrepo/
   frontend/
      trunk
      branches/
         ng/
         ...
      tags/
         1.x
         ...
   backend/
      trunk
      branches/
         ng/
         ...
      tags/
         1.x
         ...

The working layout is that we check out the frontend project and inside this, we create a backend folder and checkout the backend project.

We now want to migrate to git, and give up the splitting between frontend and backend (in terms of being separate projects) because it gives us more problems than advantages. We want them both to be in a single git repository.

I wanted to use svn2git for the conversion. Unfortunatly the latest development all happened in a branch, and not in trunk, but I think this should not be a problem for svn2git. So the new git repository layout should look like this:

/            => svnrepo/frontend/branches/ng
/backend     => svnrepo/backend/branches/ng

Where => means "migrated/converted from".

For the conversion it is not necessary for us to convert all the tags and branches from the svn repository over to git. This is not important for us. What is important however is, that we have the full history of all commits to all files in the branches/ng directory, going back to the branching from trunk and all commits that happened in trunk before that. And we want all these commits to be with the mentioned layout in a single git repository. Is this even possible? And how would we do this?

I already searched with google and also in stackoverflow 1,2 but could not find an exact solution for our problem.

Answer 1

One solution would be to generate each of the repositories separately with svn2git or just git svn (it's a nice little tool already built into git), and then wire them together with git filter-branch.

1. Clone each svn repository individually.
2. In the repository you want to be root, add the other repositories as remotes, and fetch their branches you want to merge to that repo (you'll get warnings since the branches have no common history; that is expected).
3. Execute git filter-branch on those new branches, using an index filter to generate a new subdirectory for them.
4. Merge the filtered branches into master (or whatever branch you wanted) on the root repository. Full history would be preserved.

---

The command for step 3 would look something like this:

git filter-branch --index-filter '
    git ls-files -s |
    perl -pe "s{\t\"?}{$&newsubdir/}" |
    GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info &&
    mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE
' HEAD

The magic, and every time I have to do this it does feel a little like magic, is the perl statement. git filter-branch is filtering the index at each commit and prepending all blob paths (i.e. changing the working tree's file paths) with 'newsubdir'. You might have to experiment around to get the paths exactly right. A couple of lessons learned from someone who's walked this path before:

• Back everything up. git filter-branch is history destructive. Once you change it, you cannot easily change it back. Be sure to back up all the repository copies you're using. Nothing's worse then finishing a complex operation and discovering you missed a / in the path.
• Script everything. Unless you've got some serious skill; you won't get this right the first time. Script each individual step as you complete it, so that rerunning any of them is easy. Also if you discover a week later you screwed up a flag, you can replicate in moments.
• Spend $20 on a cluster compute instance in EC2. git filter-branch is enormously CPU intensive. An index-filter on a deep history could take hours to run on your local environment, but a fraction of that time on an AWS cluster compute instance. Sure, they cost a little more than $2 an hour, but you're only going to need one for a few hours. Save yourself pain and use those scripts you wrote on hardware that makes the operation trivial. It costs the price of a nice lunch.

Answer 2

One of the solution is to convert both SVN projects repositories to 2 Git repositories and then add one Git repository as a Git submodule of another one.

To convert your SVN repository to Git repositories you may use any git-svn-based script or SubGit. With the latest tool you run a single command

$ subgit install path/to/svn/repository

The converted git repositories will be at path/to/svn/repository/git.

Then you set up an access to both Git repositories and add one as a submodule of another one:

$ git clone <frontend_GitURL> frontend
$ git co
$ cd frontend
$ git submodule add -b ng <backend_GitURL> backend

Answer 3

All I can think of is that this is going to require some extreme hackery unless svn2git (of which I am not an expert) natively supports this somehow.

The problem is that a commit of the frontend is completely independent of a commit in the backend. Theres no real way to tell which commit will map up to which commit in a single repository. This leaves us with only one real option: The history will consist of two branches being merged together, which represents the history of the original project, and then once they are merged, the new branch is the "better model".

From now on, I'm going to assume that you have frontend in the svn-frontend branch imported and backend in the svn-backend branch imported, and both contain their own history.

The first issue is to fix svn-backend to be in the backend/ directory:

git checkout svn-backend
git filter-branch --index-filter '
  git ls-files -s |
  perl -pe "s{\t\"?}{$&newsubdir/}" |
  GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info &&
  mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD

(See this documentation, and the and the answer by @Christopher)

Now, unless these somehow contain the same commit as a base (unlikely unless svn2git creates some predefined base commit or something...), we have to make one. It shouldn't matter which branch you are on to start.

git symbolic-ref HEAD refs/heads/svn-base
rm .git/index
git clean -dxf

Git can't track empty directories. I've never tested to see if this applies to the root directory, but my assumption is not, so less create an empty git ignore file and commit:

touch .gitignore
git add .gitignore
git commit -m "Base for SVN branches"

Lets rewrite the history:

git rebase svn-base svn-frontend
git rebase svn-base svn-backend

We are almost done. Lets create the master branch now. If it already exists:

git update-ref master "$head"

Otherwise:

git branch master

Lets check it out:

git checkout master

Finally, the merge:

git merge svn-backend

Its a good idea to tag the old branches and then delete them:

git checkout svn-frontend
git tag svn-frontend
git branch -d svn-frontend
git checkout svn-backend
git tag svn-backend
git branch -d svn-backend
git checkout master
git branch -d svn-base