Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm generating archives which are larger than my current physical media ( DVD ). I'd like to split those archives:

  • automatically - instead of generating mini-archives by hand;
  • consistently - so that an archive can be extracted independently of another.

For instance for a tree of 24GB which would be archived into 10GB I would get 3 archives, all of them < 4.7 GB and each of them being able to be extracted without the other 2.

I'm using dirvish so I'm archiving a filesystem tree.


Update: I'm using Linux.

share|improve this question

8 Answers 8

I'd be really interested to see how you're combining dirvish with a DVD backup system considering the main point of dirvish is to do backups incrementally, and you're losing that benefit with DVDs.

Anyway, the options for this problem that I can see:

  • Split your tree into smaller trees so that each one fits on a disk. This is probably easiest, but eventually they'll probably get too big again.
  • Script some archiving/compression into post_server, I'd use rar for this, it supports split archives where the maximum segment size is given as a parameter. Getting all the data back if one disk is lost could be difficult. (Using tar as previously suggested may get around this problem)
  • Split the branch without an archiver. It'd be possible to use find to write a shell script that would split out the branch into chunks less than 4.7gb. This is probably the best long term approach, but requires lots more effort

Personally, I'd try to avoid any solutions that involve building large 4.7gb archives If I thought there was a chance I'd have to extract them to get small files back out

share|improve this answer

PKZip and WinZIP both support the option of split a large archive into multiple, smaller sized files that can be user specified.

I'm unfamiliar with dirvish, but if you are looking just to backup and compress the files, with a user defined file size limit, it might be worth considering.

share|improve this answer
    
I'm using Linux - so all I get is zip. Are zipsplit files able to be operated independently? –  Robert Munteanu Jun 26 '09 at 14:54
    
No, they have to be opened together so that would be a problem since you are using DVD media. I did not consider that. Sorry. –  Richard West Jun 26 '09 at 20:19

You can tell tar to split an archive at a given size.

It was originally implemented for splitting archives across multiple-tapes, but equally applicable for other types of media.

You won't be able to initially compress the archive. But you should be able to gzip and later extract the split archives individually. Be sure to specify new names for each split archive though.

tar cvf archive.tgz -ML 4928307 items_to_archive
share|improve this answer

as far as I know you can't create a split archive so that one of its parts can be extracted independently.

You can tell tar, zip, 7zip, rar, ... to split the archive into multiple files specified by size. (rar -v, 7z -v, etc)

I know that RAR can create archives in such a way that you can extract only from one volume from the whole volume set.

share|improve this answer

What you want is unlikely to exist, if only because when doing a restore, the software has to figure out which media the backup is on, so it has to have some kind of directory to look such things up in, and such a directory has to be stored in the archive - which is going to require you to have both the 'directory' disk and the 'storage' disk be read. Even if you were to write it yourself, you'd end up having to at least temporarily (for the length of the backup session) track which files had been backed up so they didn't get backed up twice.

share|improve this answer

Periodically archive (tar/zip/...) your stable (not while running a dirvish job!) dirvish bank/vault then use the "PAR" (currently v2) archive format to split the archive.

It is a sort of "software RAIF" (RAID on files!)

See for example http://parchive.sourceforge.net/

share|improve this answer

GNU tar has multivolume support. Useful for when the archive created is larger than a single tape's capacity. I hope that all of you understand that tape can also mean file, or DVD, or CD, etc.

From section 9.6 of the GNU tar manual:

Use --multi-volume (-M) on the command line, and then tar will, when it reaches the end of the tape, prompt for another tape, and continue the archive. Each tape will have an independent archive, and can be read without needing the other. (As an exception to this, the file that tar was archiving when it ran out of tape will usually be split between the two archives; in this case you need to extract from the first archive, using --multi-volume (-M), and then put in the second tape when prompted, so tar can restore both halves of the file.)

emphasis mine

But do be careful about the exception! Maybe write out to files which you can check for size before writing the archive files to DVD. This could be done by a script. Or make the file size something smaller that 4.7G so that an archive that spans two DVDs never occurs.

share|improve this answer

On linux/unix you can use split, a command that will split a file in different parts that are all a certain size (10 bytes, 100 MB etc)

share|improve this answer
    
Unfortunately split does not satisfy the "and each of them being able to be extracted without the other 2" criteria. –  jmtd Jun 29 '09 at 13:08

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.