1

I have a storage dilemma on my network at work.

Currently we use version control (SVN) to backup our raw files from our workstations to a repository on a centralized Server.

After reading up, it seems that I may be running into a problem with some of our larger files, which sometimes are as large as 1 gig. We have 100s of these files per repository, making it quite a large repository overtime.

Is there better software to use out there? I would like the repository functionality, but with a more stable/workable environment. We really need to do a backup system overhaul for our centralized server.

Update:

As requested, OS for both Server and Workstation is XP. Could update Server to Server 2003 if need be.

1
  • Let us know what OS(es) are on the workstations and servers.
    – mfinni
    Feb 17, 2010 at 23:29

3 Answers 3

1

I am not sure the Subversion is really designed as a backup tool. You may want to look at Acronis. While the price tag is certainly more than home spun solutions, there's nothing more satisfying the restoring from bare-metal after a critical failure.

I mainly work in Linux and have switched to using R1Soft's CDP server, but prior to that I used Acronis in our office with good results.

1
  • I ended up going with Acronis, UI is clean, and for what it accomplishes and how, it is fairly cheap.
    – jsmith
    Feb 18, 2010 at 16:18
1

Subversion actually has the some of the best large-file handling of any modern revision control system. Git, Mercurial, Bazaar, etc. all have architectural or platform-specific limits (usually 2 or 4 GB, because they mem-map files to do various things). Git could probably be a good fit for your use case as a "stupid content tracker", if you can deal with pushing files to a git repo on a server and then doing a commit on the server via script. See this question for more (and its links about the memory-mapping issues on different platforms).

A bigger issue is that the "binary diff" and compression functionality provided by revision control systems is generally ineffective on most large media files, since they are already compressed and change so much. For example, if you have two movie files, and edit one to remove 4 seconds from the middle, you would think that it should be easy to store only the changes. But in fact, when you edit that file, even if you don't re-encode the video, the time-codes for each frame get shuffled around, resulting storing almost the whole file again. You might want to just turn off the compression and diff features in your tool if possible to save on CPU and backup time.

The only large files I've seen Subversion or Git do well with are SQL database backups, which uaually have a page-like structure and change very little from one backup to the next. And of course log files, which just compress like crazy even with gzip.

1

If you are using SVN (or any other version control software) as a backup system you may want to consider an actual backup system (Bacula, or perhaps something commercial).

You can continue to check the files in to SVN if they're something that warrants revision control, but in my experience SVN is a pretty lousy backup system as you scale up...

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .