Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Are there any version control systems out there that are particularly good (or bad) at dealing with large files? Nothing too crazy, but from several hundred megabytes to a gigabyte, let's say.

We currently have subversion in place, but there are some mutterings about it not being ideal for this purpose. I'm not a developer myself, and I don't know how objective they are being, so I thought I'd do a quick community survey for extra info.

I'm interested in the behaviour or suitability of these VCS solutions from a systems point of view as well as the user point of view.

TIA.

share|improve this question
2  
i'd love to know an answer. for now, just rdiff-backup and scripts... –  Javier Oct 6 '09 at 15:10
    
@Javier, This is the solution that we use as well, so far its worked great! –  faultyserver Oct 6 '09 at 15:46
    
/svn/REPO/2012/ - 811Gb, subversion-1.6.23 (r1485506). Don't have any problem –  ALex_hha Aug 25 '13 at 10:29
add comment

11 Answers

up vote 4 down vote accepted

Another option made for multimedia and creative workflows is AlienBrain which is now owned by Avid. It's used by a lot of game studio's to version control the game assets and code.

http://www.alienbrain.com/

It may not be the best solution though if you're not dealing with media assets.

share|improve this answer
    
That's actually the one they used to use at a certain large game studio I worked at. Couldn't remember the name. Thanks. :) –  Some Guy Oct 6 '09 at 18:53
add comment

You'll find that they are very much of a muchness when it comes to binary files.

The mutterings you have heard are most likely originating from the notion that version controlling binaries is a little bit at odds with the power of version control. Binary files can't be diffed or merged, so they are treated as dumb copies. The whole file is replaced on every small change.

This isn't to say that you can't version control binary files or indeed that it isn't useful for you to do so. If you have the requirement to roll-back a file to the version that you committed yesterday. In which case it has served a use to you.

Although you might find a storage solution with snapshots serves you better and more efficiently.

share|improve this answer
2  
A storage system with snapshots is what I would recommend as well. I use ZFS to version my virtual machine hard drives an it works well. The snapshots are almost instant and they only take up the space necessary to store the changed blocks. –  Amuck Oct 6 '09 at 17:03
    
Snapshots might be a good option. I guess it depends on whether or not I can set it up so they can be (mostly) independent. –  Some Guy Oct 6 '09 at 18:54
add comment

The vast majority of those having to deal with loads of binary files (e.g. games) tend to use perforce, sometimes with a layer over it.

share|improve this answer
add comment

It sounds like Boar would satisfy your requirements. It is version control for large binary files such as videos or pictures.

share|improve this answer
    
This project appeals to me, because it doesn't have a working directory copy of the current repository state. Meaning only the backed up data (and revisions) and the original exist, not a 'backup' of the last updated state of the repo. Or at least that's how it seems to me. –  MrSnowflake Mar 10 '11 at 14:24
add comment

git-annex "allows managing files with git, without checking the file contents into git. While that may seem paradoxical, it is useful when dealing with files larger than git can currently easily handle, whether due to limitations in memory, time, or disk space."

share|improve this answer
add comment

Adobe offers Version Cue CS4, which was made for multimedia projects. You might want to check that out.

share|improve this answer
add comment

I believe Bazaar handles binary files quite well this seems to be documented here (4.1). I suppose this depends on if you want to spend money or not though as the documentation does state there are better tools out there (it doesn't name them however).

share|improve this answer
2  
Quoting from that site: That said, bzr is primarily a source code control system, not a media archive system. So it is not a priority to support enormous (hundred-megabyte) binaries or multi-gigabyte trees. There are other tools better suited to that. –  Cristian Ciupitu Oct 6 '09 at 20:13
add comment

The Wikipedia page for Subversion also states that it supports binary files although I don't have personal experience of this so can't comment on how well it works.

Native support for binary files, with space-efficient binary-diff storage.

share|improve this answer
add comment

If, and it's a big if, you're using Autodesk software (Maya, Autocad, Inventor, etc... ) Then there's Autodesk Vault.

I recently discovered this at work. It's not free, but it is seemingly the only VCS that works on Autodesk media asset files.

However, it's only really suitable for tracking changes in files it can inspect, so drawing files it's fine with, but not possible for 'rendered assets'.

I'd probably go with git.

share|improve this answer
add comment

An entirely lateral method is to use the union filesystem AUFS which is used by Docker to allow users to create diffs against entire filesystem nodes and publish them. They talk about it on their blog.

This isn't version control with all of the tools of git but it does allow one to add and modify files in a large tree with no real size limit.

This would be a very robust solution just for the media files but I don't think it gives granular control so it would be best for projects where the need is similar to Docker's.

share|improve this answer
add comment

Git will be able to deal with "several hundred megabytes to a gigabyte" binary files. It's very fast.

share|improve this answer
1  
It also is not server based. Now, while I really really like the idea of a distributed VCS where your local machine has a copy of the whole repository, this may be a LITTLE slow and cumbersome when your repository blows 1000gb - which is kind of trivial when you deal with files in the gigabyte size. Then a central repository + local workfolder make sure the local machines can stay sane. –  TomTom May 23 '12 at 13:00
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.