Tag Info

Hot answers tagged

20

Unfortunately, there is no best way for you. 30 year archival of digital media is a very hard problem and takes routine investment. About the only formats guaranteed to be readable in 30 years are ASCII and UTF8, which are not video formats. Storage formats change, the 8 track reel-to-reel tapes we were using 30 years ago are nigh impossible to read these ...


14

Compress, Gzip, Bzip, Bzip2 are not for archiving multiple files. They only compress single file. For archiving they are usually used with TAR. The problem with TAR is that it has no index table. It's only good if you're planning to restore the whole thing. If you're expecting that you ever need to restore only limited number of selected files, forget about ...


12

Tape. Simple like that. Quantum has a SuperSTore system that can handle way more than that and I have seen them for less than your 5000 price point - new. The good thing is that you can pull tapes out for storage so scaling this is going to be quite cost efficient, and tapes last.


11

I totally agree with sysadmin1138's post in every way bar one caveat - I don't think you're going to have the budget to really achieve what you want. There are 5 main functions you need to create; a standardised content and catalogue policy - I know you want to store everything in one format but you really should consider two - PDF for images and H.264 ...


10

Recommended reading: File Compression in the Multi-Core Era (Jeff Atwood a.k.a. CodingHorror, february 2009) I've been playing around a bit with file compression again, as we generate some very large backup files daily on Stack Overflow. We're using the latest 64-bit version of 7zip (4.64) on our database server. I'm not a big fan of more ...


10

Well, ZFS uses a storage layering called Hybrid Storage Pool (HSP): Layer: memory Layer: SSD-based read-caches (L2ARC) and write-caches (separate intent log aka slog). Layer: (cheap) harddisks With HSP its easy to automatically benefit from the advantages of SSDs compared to a harddisk-only solution. A system using HSP can be both faster and cheaper than ...


8

If anything ever achieved the status with respect to floppy images that ISO has with respect to CDs, I never heard of it. It's pretty much down to however particular imaging software works, though naturally a straight dd-style image is popular. I'd recommend using the fdimage utility that FreeBSD uses for its install .flp files; at minimum you can be ...


8

Before going ahead and rewriting tar, you may want to profile the quick-and-easy method of reading the data twice, as it may not be much slower than doing it in one pass. The two pass method is implented here: http://www.g-loaded.eu/2007/12/01/veritar-verify-checksums-of-files-within-a-tar-archive/ with the one-liner: tar -cvpf mybackup.tar myfiles/| ...


7

tar (and cpio and afio and pax and similar programs) are stream-oriented formats - they are intended to be streamed direct to a tape or piped into another process. while, in theory, it would be possible to add an index at the end of the file/stream, i don't know of any version that does (it would be a useful enhancement though) it won't help with your ...


7

Any reason for tar: Can't specify both -x and -t?* Yes: Because in the designer's eyes this would be useless. If you have a tar file you either want to extract its contents (-x), or you want to list them (-t). Much like you can't specify -c -t (You're creating an archive, why would you want to list the things you're adding to it, which you've presumably ...


6

I wouldn't try it. The floating charges that are used to store the bits can degrade within a few years. The more the flash memory has been used, the more likely this will happen. (The over-voltage used to erase the cells can be hard on the components.) Electrical storage media (these days) generally do not have as long a lifetime as their magnetic ...


6

As you probably know, the biggest player is Symantec Enterprise Vault. Here's a comparison between it and Mimosa. There's a comparative review here of ten email archiving products which might give you some leads.


6

I needed to add the f flag to tell tar I'm specifying the filename, i.e. $ tar -tf archive.tar file1.txt file2.txt $


6

I would recommend ext3/4 for use with Linux as my personal preference. For the file structure I would recommend option number 3 (a balance of directory depth and files per directory). This is really just about choosing a tree data structure. To achieve this for the files I would do a md5sum hash of each file and use the first x characters of each file as ...


6

use find ... -print0 | xargs -0 tar... these two options exist in order to deal with filenames with spaces.


5

I always recommend GFI MailArchiver (http://www.gfi.com/mailarchiver/). I have been using it since 2006. Couldn't be happier. To be honest, it's a dream to whoever needs to manage the exchange server for having an archive solution. It makes so much easier managing on the server side. The MailArrchiver is a web-based system built on ASP.net with using SQL at ...


5

Rule #1: Ignore (politely) anyone that isn't paying for the archive. All of the above answers are excellent but they leave out one thing: The important thing is "who pays for this?" The department that pays for it has requirements: how fast data can be restored, length of time to hold the data, security requirements, etc. Other people might have "helpful ...


5

Backups are there to restore things to the way they were at a previous point in time. We do backups to restore service, and usually don't think much about the underlying data. Archives are very data focused, so you really need to think about things like classification and the information life-cycle. Much like software development, deep understanding of the ...


5

Actually I'd use Laconica if you want control over what is microblogged. A live implementation can be seen on identi.ca. Actually just a clone of twitter, but at least you are in control.


5

Here's an off-the-wall answer (it's a good thing that I can't lose too many rep points for posting this, eh?): If you want to play with an open source project, have a look at Archiveopteryx (http://www.archiveopteryx.org/). It's got a really, really sweet normalized SQL-based data storage schema for email (using PostgreSQL). You could combine it with, say, ...


5

There's an open source product to do what you're looking for, though it may be a bit heavy-weight for a single mailbox. It's got a normalized relational database schema and uses Postgres on the back-end for data storage (such that you can "connect" to the mail store with ODBC drivers, etc). Have a look: http://www.archiveopteryx.org/


5

I do all of my backups with Rsync. Even if the server is Windows (which mine are) you can backup with a Linux computer or even use a Windows version of Rsync. Rsync will check which files have been modified since the last time they were backed up and only sync the difference. It will even do binary diffs and only transfer the changes of the file instead ...


5

I contacted RARLabs support and recieved a response. It turns out that RAR.EXE can handle streaming input similar to how gzip works. You just need to specify the -si option: -si[name] Read data from stdin (standard input), when creating an archive. Optional 'name' parameter allows to specify a file name of compressed stdin data in the ...


5

Simpler than you think: $ tar cf small-archive.tar /big/tree --exclude-from <(find /big/tree -size +3M) On a semi-related note (relating to your statement that you can't use find) to get a listing of all files (including directories) under a path minus files larger than 3MiB, use: $ find . -size -3M -o -type d You could then do: $ tar cf ...


5

There are many ways to do this with many pieces of software. I think most MTAs are capable of some kind of archiving based on configuration. With Postfix, you can almost one-liner it: add an always_bcc: your.address@archive.yourco.com to the configuration file. If all mail goes through it, you'll be set. Dealing with the files after they've been delivered ...


4

It isn't all about efficiency and speed. Sure they are important and you can look at the benchmarks for those and choose wisely from the options (though I'd recommend some simple benchmarking of your own with your own data on your own server). But archiving inevitably leads at some point to accessing your data again (otherwise why not just delete it?). Or ...


4

You might find this link to be helpful: Archiving Twitter data with Python


4

What you're seeing is valid. The VM machine is presenting windows with new hardware and a new CPU ID. Windows activation is doing its job because more than 5 pieces of hardware have changed. Are you moving the machines off a computer made by Dell, HP, Toshiba, etc? If so, their OEM product key is normally tied to their specific hardware, normally by an ...


4

you need to use an actual implementation of a zip program. take a look at gzip, bzip or 7zip. Each program will have different command line switches


4

This is an extremely generic overview: CPIO does a better job of duplication a file system, including taking backups. It preserves things like hardlinks, fifos, and other not-a-standard-file features. Most implementations of CPIO do everything TAR does, including reading and writing .tar files. CPIO usually takes a list of files from standard input to ...



Only top voted, non community-wiki answers of a minimum length are eligible