Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

We want to migrate from svn to git permanently to be able to use git's better features in terms of branching and collaboration.

Our current svn repository looks like this

svnrepo/
   frontend/
      trunk
      branches/
         ng/
         ...
      tags/
         1.x
         ...
   backend/
      trunk
      branches/
         ng/
         ...
      tags/
         1.x
         ...

The working layout is that we check out the frontend project and inside this, we create a backend folder and checkout the backend project.

We now want to migrate to git, and give up the splitting between frontend and backend (in terms of being separate projects) because it gives us more problems than advantages. We want them both to be in a single git repository.

I wanted to use svn2git for the conversion. Unfortunatly the latest development all happened in a branch, and not in trunk, but I think this should not be a problem for svn2git. So the new git repository layout should look like this:

/            => svnrepo/frontend/branches/ng
/backend     => svnrepo/backend/branches/ng

Where => means "migrated/converted from".

For the conversion it is not necessary for us to convert all the tags and branches from the svn repository over to git. This is not important for us. What is important however is, that we have the full history of all commits to all files in the branches/ng directory, going back to the branching from trunk and all commits that happened in trunk before that. And we want all these commits to be with the mentioned layout in a single git repository. Is this even possible? And how would we do this?

I already searched with google and also in stackoverflow 1,2 but could not find an exact solution for our problem.

share|improve this question

migrated from stackoverflow.com Aug 24 '12 at 14:15

This question came from our site for professional and enthusiast programmers.

3 Answers 3

up vote 2 down vote accepted

One solution would be to generate each of the repositories separately with svn2git or just git svn (it's a nice little tool already built into git), and then wire them together with git filter-branch.

  1. Clone each svn repository individually.
  2. In the repository you want to be root, add the other repositories as remotes, and fetch their branches you want to merge to that repo (you'll get warnings since the branches have no common history; that is expected).
  3. Execute git filter-branch on those new branches, using an index filter to generate a new subdirectory for them.
  4. Merge the filtered branches into master (or whatever branch you wanted) on the root repository. Full history would be preserved.

The command for step 3 would look something like this:

git filter-branch --index-filter '
    git ls-files -s |
    perl -pe "s{\t\"?}{$&newsubdir/}" |
    GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info &&
    mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE
' HEAD

The magic, and every time I have to do this it does feel a little like magic, is the perl statement. git filter-branch is filtering the index at each commit and prepending all blob paths (i.e. changing the working tree's file paths) with 'newsubdir'. You might have to experiment around to get the paths exactly right. A couple of lessons learned from someone who's walked this path before:

  • Back everything up. git filter-branch is history destructive. Once you change it, you cannot easily change it back. Be sure to back up all the repository copies you're using. Nothing's worse then finishing a complex operation and discovering you missed a / in the path.
  • Script everything. Unless you've got some serious skill; you won't get this right the first time. Script each individual step as you complete it, so that rerunning any of them is easy. Also if you discover a week later you screwed up a flag, you can replicate in moments.
  • Spend $20 on a cluster compute instance in EC2. git filter-branch is enormously CPU intensive. An index-filter on a deep history could take hours to run on your local environment, but a fraction of that time on an AWS cluster compute instance. Sure, they cost a little more than $2 an hour, but you're only going to need one for a few hours. Save yourself pain and use those scripts you wrote on hardware that makes the operation trivial. It costs the price of a nice lunch.
share|improve this answer
    
Thanks for the detailed answer! I will try this out, however I'm not sure if my very basic git-fu is sufficient to do this. :-) You mentioned that filter-branch is very cpu-intensive. We have a relatively small repository with currently just over 5000 commits. Would this still be very slow on a quad core cpu? (Can git filter-branch even take advantage of multiple cpu's?) –  Shyru Aug 20 '12 at 13:05
    
5000 commits is not a very deep history at all. You're probably fine on any of your available machines. It'll still take a while, but not a crippling amount of time. For what it's worth, when this procedure is done (no matter which solution you pursue), you'll have considerably more git-fu. –  Christopher Aug 20 '12 at 13:14
    
Thanks again for the help. I ended up using svn2git from the kde project (techbase.kde.org/Projects/MoveToGit/UsingSvn2Git) because it could deal better with our moves of files inside the svn-repository and still kept the file history intact which I did not get to work properly with the ruby svn2git. I then used git filter-branch as you proposed and it worked nicely! Thanks alot again! –  Shyru Dec 10 '12 at 11:00

One of the solution is to convert both SVN projects repositories to 2 Git repositories and then add one Git repository as a Git submodule of another one.

To convert your SVN repository to Git repositories you may use any git-svn-based script or SubGit. With the latest tool you run a single command

$ subgit install path/to/svn/repository

The converted git repositories will be at path/to/svn/repository/git.

Then you set up an access to both Git repositories and add one as a submodule of another one:

$ git clone <frontend_GitURL> frontend
$ git co
$ cd frontend
$ git submodule add -b ng <backend_GitURL> backend
share|improve this answer
    
The problem with this is that he doesn't want separate repositories like submodules would be essentially giving. –  alternative Aug 20 '12 at 12:41
    
Exactly, we would like to use submodules only if there is no other way. –  Shyru Aug 20 '12 at 12:47

All I can think of is that this is going to require some extreme hackery unless svn2git (of which I am not an expert) natively supports this somehow.

The problem is that a commit of the frontend is completely independent of a commit in the backend. Theres no real way to tell which commit will map up to which commit in a single repository. This leaves us with only one real option: The history will consist of two branches being merged together, which represents the history of the original project, and then once they are merged, the new branch is the "better model".

From now on, I'm going to assume that you have frontend in the svn-frontend branch imported and backend in the svn-backend branch imported, and both contain their own history.

The first issue is to fix svn-backend to be in the backend/ directory:

git checkout svn-backend
git filter-branch --index-filter '
  git ls-files -s |
  perl -pe "s{\t\"?}{$&newsubdir/}" |
  GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info &&
  mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD

(See this documentation, and the and the answer by @Christopher)

Now, unless these somehow contain the same commit as a base (unlikely unless svn2git creates some predefined base commit or something...), we have to make one. It shouldn't matter which branch you are on to start.

git symbolic-ref HEAD refs/heads/svn-base
rm .git/index
git clean -dxf

Git can't track empty directories. I've never tested to see if this applies to the root directory, but my assumption is not, so less create an empty git ignore file and commit:

touch .gitignore
git add .gitignore
git commit -m "Base for SVN branches"

Lets rewrite the history:

git rebase svn-base svn-frontend
git rebase svn-base svn-backend

We are almost done. Lets create the master branch now. If it already exists:

git update-ref master "$head"

Otherwise:

git branch master

Lets check it out:

git checkout master

Finally, the merge:

git merge svn-backend

Its a good idea to tag the old branches and then delete them:

git checkout svn-frontend
git tag svn-frontend
git branch -d svn-frontend
git checkout svn-backend
git tag svn-backend
git branch -d svn-backend
git checkout master
git branch -d svn-base
share|improve this answer
    
Careful with that sed call in filter-branch. Different implementations of sed treat that tab character differently. You probably want the perl modification in my answer, which was suggested on the git list. –  Christopher Aug 20 '12 at 12:43
    
Also there's no reason you couldn't remove file path specifications with git filter-branch, too. There's no need create a trash commit. Just clone 'frontend', index-filter frontend/ out of the path, and there you go. frontend/ is now the root level directory... if he needs this at all. If 'frontend' is its own repository, it should already clone at root level. –  Christopher Aug 20 '12 at 12:48
    
@Christopher frontend/ never existed in the first place and it shouldn't exist. –  alternative Aug 20 '12 at 12:49

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.