Tag Info

Hot answers tagged

19

Unfortunately, there is no best way for you. 30 year archival of digital media is a very hard problem and takes routine investment. About the only formats guaranteed to be readable in 30 years are ASCII and UTF8, which are not video formats. Storage formats change, the 8 track reel-to-reel tapes we were using 30 years ago are nigh impossible to read these ...


15

Compress, Gzip, Bzip, Bzip2 are not for archiving multiple files. They only compress single file. For archiving they are usually used with TAR. The problem with TAR is that it has no index table. It's only good if you're planning to restore the whole thing. If you're expecting that you ever need to restore only limited number of selected files, forget about ...


12

Tape. Simple like that. Quantum has a SuperSTore system that can handle way more than that and I have seen them for less than your 5000 price point - new. The good thing is that you can pull tapes out for storage so scaling this is going to be quite cost efficient, and tapes last.


11

I totally agree with sysadmin1138's post in every way bar one caveat - I don't think you're going to have the budget to really achieve what you want. There are 5 main functions you need to create; a standardised content and catalogue policy - I know you want to store everything in one format but you really should consider two - PDF for images and H.264 ...


10

Recommended reading: File Compression in the Multi-Core Era (Jeff Atwood a.k.a. CodingHorror, february 2009) I've been playing around a bit with file compression again, as we generate some very large backup files daily on Stack Overflow. We're using the latest 64-bit version of 7zip (4.64) on our database server. I'm not a big fan of more ...


10

Well, ZFS uses a storage layering called Hybrid Storage Pool (HSP): Layer: memory Layer: SSD-based read-caches (L2ARC) and write-caches (separate intent log aka slog). Layer: (cheap) harddisks With HSP its easy to automatically benefit from the advantages of SSDs compared to a harddisk-only solution. A system using HSP can be both faster and cheaper than ...


8

If anything ever achieved the status with respect to floppy images that ISO has with respect to CDs, I never heard of it. It's pretty much down to however particular imaging software works, though naturally a straight dd-style image is popular. I'd recommend using the fdimage utility that FreeBSD uses for its install .flp files; at minimum you can be ...


7

tar (and cpio and afio and pax and similar programs) are stream-oriented formats - they are intended to be streamed direct to a tape or piped into another process. while, in theory, it would be possible to add an index at the end of the file/stream, i don't know of any version that does (it would be a useful enhancement though) it won't help with your ...


7

Any reason for tar: Can't specify both -x and -t?* Yes: Because in the designer's eyes this would be useless. If you have a tar file you either want to extract its contents (-x), or you want to list them (-t). Much like you can't specify -c -t (You're creating an archive, why would you want to list the things you're adding to it, which you've presumably ...


6

I needed to add the f flag to tell tar I'm specifying the filename, i.e. $ tar -tf archive.tar file1.txt file2.txt $


6

Before going ahead and rewriting tar, you may want to profile the quick-and-easy method of reading the data twice, as it may not be much slower than doing it in one pass. The two pass method is implented here: http://www.g-loaded.eu/2007/12/01/veritar-verify-checksums-of-files-within-a-tar-archive/ with the one-liner: tar -cvpf mybackup.tar myfiles/| ...


6

I would recommend ext3/4 for use with Linux as my personal preference. For the file structure I would recommend option number 3 (a balance of directory depth and files per directory). This is really just about choosing a tree data structure. To achieve this for the files I would do a md5sum hash of each file and use the first x characters of each file as ...


6

use find ... -print0 | xargs -0 tar... these two options exist in order to deal with filenames with spaces.


6

As you probably know, the biggest player is Symantec Enterprise Vault. Here's a comparison between it and Mimosa. There's a comparative review here of ten email archiving products which might give you some leads.


6

I wouldn't try it. The floating charges that are used to store the bits can degrade within a few years. The more the flash memory has been used, the more likely this will happen. (The over-voltage used to erase the cells can be hard on the components.) Electrical storage media (these days) generally do not have as long a lifetime as their magnetic ...


5

I always recommend GFI MailArchiver (http://www.gfi.com/mailarchiver/). I have been using it since 2006. Couldn't be happier. To be honest, it's a dream to whoever needs to manage the exchange server for having an archive solution. It makes so much easier managing on the server side. The MailArrchiver is a web-based system built on ASP.net with using SQL at ...


5

There's an open source product to do what you're looking for, though it may be a bit heavy-weight for a single mailbox. It's got a normalized relational database schema and uses Postgres on the back-end for data storage (such that you can "connect" to the mail store with ODBC drivers, etc). Have a look: http://www.archiveopteryx.org/


5

Rule #1: Ignore (politely) anyone that isn't paying for the archive. All of the above answers are excellent but they leave out one thing: The important thing is "who pays for this?" The department that pays for it has requirements: how fast data can be restored, length of time to hold the data, security requirements, etc. Other people might have "helpful ...


5

Backups are there to restore things to the way they were at a previous point in time. We do backups to restore service, and usually don't think much about the underlying data. Archives are very data focused, so you really need to think about things like classification and the information life-cycle. Much like software development, deep understanding of the ...


5

Actually I'd use Laconica if you want control over what is microblogged. A live implementation can be seen on identi.ca. Actually just a clone of twitter, but at least you are in control.


5

Here's an off-the-wall answer (it's a good thing that I can't lose too many rep points for posting this, eh?): If you want to play with an open source project, have a look at Archiveopteryx (http://www.archiveopteryx.org/). It's got a really, really sweet normalized SQL-based data storage schema for email (using PostgreSQL). You could combine it with, say, ...


5

I do all of my backups with Rsync. Even if the server is Windows (which mine are) you can backup with a Linux computer or even use a Windows version of Rsync. Rsync will check which files have been modified since the last time they were backed up and only sync the difference. It will even do binary diffs and only transfer the changes of the file instead ...


5

There are many ways to do this with many pieces of software. I think most MTAs are capable of some kind of archiving based on configuration. With Postfix, you can almost one-liner it: add an always_bcc: your.address@archive.yourco.com to the configuration file. If all mail goes through it, you'll be set. Dealing with the files after they've been delivered ...


4

Will -xv not give you the equivalent of -xt?


4

First, I would advise avoiding Glacier. It sounds good, until you crunch the costs on actually restoring a large amount of data. This is an unofficial calculator you can use to calculate Glacier storage and retrieval costs, and judge for yourself. Restoring terabytes of data from Glacier is a pretty unattractive prospect. Second, I would advise that for ...


4

What you're seeing is valid. The VM machine is presenting windows with new hardware and a new CPU ID. Windows activation is doing its job because more than 5 pieces of hardware have changed. Are you moving the machines off a computer made by Dell, HP, Toshiba, etc? If so, their OEM product key is normally tied to their specific hardware, normally by an ...


4

you need to use an actual implementation of a zip program. take a look at gzip, bzip or 7zip. Each program will have different command line switches


4

I contacted RARLabs support and recieved a response. It turns out that RAR.EXE can handle streaming input similar to how gzip works. You just need to specify the -si option: -si[name] Read data from stdin (standard input), when creating an archive. Optional 'name' parameter allows to specify a file name of compressed stdin data in the ...


4

This is an extremely generic overview: CPIO does a better job of duplication a file system, including taking backups. It preserves things like hardlinks, fifos, and other not-a-standard-file features. Most implementations of CPIO do everything TAR does, including reading and writing .tar files. CPIO usually takes a list of files from standard input to ...


4

Amanda and BackupPC are both widely-used and well-respected F/OSS backup utilities. Either one of them could be configured to maintain the backup aging/rotation that you specified. If you wanted something more simple, you could look into rdiff-backup or rsnapshot.



Only top voted, non community-wiki answers of a minimum length are eligible