Tag Info

Hot answers tagged

15

lzma (aka xz) should do notably better than bzip2, but will take a bit longer. paq (aka zp) will do quite a bit better yet, but will take ages to compress and just as long to decompress. Both are available for Windows and *nix environments (most *nix systems have packages available) A quick test on a smartd log: Original 3900K GZip 208K ...


6

find is your friend. I reckon the following ought to do it: find <target_dir> -not -name \*.bz2 -exec bzip2 \{\} \; i.e. if the dir where the files you want to bzip are is /var/log/blah it would be: find /var/log/blah -not -name \*.bz2 -exec bzip2 \{\} \;


5

How about something like: tar -ztf file.tar.gz | egrep '^[^/]+/?$'


4

Why not use the industry standard for this kind of stuff: "gzip"? mysqldump --all-databases | gzip > mybackup.gz Size Comparison: 720K mybackup.gz (compressed) 2.6M mybackup.sql (same data, but uncompressed for comparison)


3

Your command mysqldump --single-transaction -u $MUSER -h $MHOST -p$MPASS DB | bzip2 -k -v > $FILE says dump a database and pipe the output to bzip2 which should write it's stdout to $FILE. The contents of stdin will be compressed and written to for example a.tar.gz.bz2. This doesn't create either a tar archive or a gzip compressed tar archive ...


3

tar --exclude='*/*' -tf yourarchive.tar should do it. That's almost certainly a GNU tar-ism. But who doesn't use GNU tar, right? (Another fun fact: in recent versions of GNU tar, you don't need the 'z' or 'j' to list or uncompress .gz or .bz files -- it autodetects those and it just works.)


2

Use xz: mysqldump --all-databases | xz -9 -c > mybackup.xz The compression ratio is much higher than zip, just make sure to watch the memory usage. If you are running xz in a memory-restricted environment you could use the following table to tune the command and avoid paging: Preset DictSize CompCPU CompMem DecMem -0 256 KiB 0 ...


2

It is not so simple. Find can only call a binary with an "exec" argument, but you now try to call a bunzip|gzip pipe. It is not a simple binary, it is a chain of binaries, and to start a such thing you need something, which can handle pipes. The best thing for this were a shell. You have to call a shell to call this pipe: find . -name "*.bz2" -type f -exec ...


2

Can you untar it on Linux? If you don't have a convenient Linux machine, perhaps you could set up one on a virtual machine to try. I'm sort of assuming this backup is worth the effort. Some older tools don't necessarily work properly with files greater than 4 GB in size. You may be running into this problem; it might be worth trying recent tools to see ...


1

Either upgrade zip to at least version 3.0 to support Zip64: $ zip -v This is Zip 3.0 (July 5th 2008), by Info-ZIP. Zip special compilation options: LARGE_FILE_SUPPORT (can read and write large files on file system) ZIP64_SUPPORT (use Zip64 to store large files in archives) or use the different tools such as: 7z, pbzip2, ... You ...


1

I needed this exact same thing. I came up with this surprisingly fast python code (it joined two 2GB files with an 800MB overlap in 30 seconds.) Adjust overlap_size as necessary for your chunks. It should be as long as possible but less than the real overlap size. #!/usr/bin/env python import sys overlap_size = 100000000 # 100MB a = ...


1

I think you're just going to have to write such a tool yourself. Start out with the largest file and copy it into memory as your image. Then run through all the files, one by one, looking for an overlap with either the first or last chunk of the current memory image. If you find an overlap, extend the memory image. Repeat until you take a pass through all ...


1

like a linux command checking for the longest overlap between the end of one file and the start of another Traditionally, this would be diff. It will produce the "difference" of two given text files as the output, along with some control information (what has been added, what has been removed, which lines to check). The patch command is able to reverse ...


1

I don't have a tool for you to do the job completely, but you can use tools like: cmp -l dump1 dump2 This will give you the list of different bytes and their offsets. The overlapping is where there is no offset printed by cmp. Also, you can use dd command to copy part of a dump and append it to another dump. You can try writing your own script that use ...


1

Why do you have the -w option in your tar command? It means "Interactive mode, ask for confirmation for everything". This likely causes the error.


1

Another thought is that the tar file format is always aligned on a 512 byte boundary, it pads it out with NUL characters if it's shorter (per file). Now granted, the tar should be being done before the bz2, so it should still be varying in size (theoretically). But perhaps it's compressing first and then putting it into the tar, causing it to be aligned to ...


1

In the directory containing your bz2 files paste this command: for file in *.bz2;do echo "checksum for ${file/.bz2/}: $(bunzip2 -c $file|md5sum)";done If the checksums all differ then the uncompressed files are different.


1

Perhaps there is a bug in the script archive. Compare files: cmp mysql_full_export.Wed.bz2 mysql_full_export.Tue.bz2 Compare the contents of archives(use diff or cmp).


1

With zsh (setopt extended_glob must be on): bzip2 **/^*.bz2(.) ** recurses in subdirectories; ^*.bz2 matches everything except *.bz2; (.) restricts to regular files. With bash 4, if you're ok with ignoring bzip2's complains about being invoked on directories: bzip2 **/!(*.bz2)


1

Off the top of my head (sorry, don't have a shell handy to test quoting etc.): for _t in `find . -print |grep -v -E "\.bz$"`; do bzip2 -9 $_t && echo OK $_t || echo FAIL $_t; done This uses find to find all files, grep to strip out the ones with a .bz2 extension and then feed them one at a time into bzip2. I expect some of the quoting is wrong, ...


1

bzip2 apparently works the same way as gzip for that operation, even using the same -c option. From the bzip2 man page: bunzip2 will correctly decompress a file which is the concatenation of two or more compressed files. The result is the concatenation of the corresponding uncompressed files. Integrity testing (-t) of concateā€ ...


1

Try to untar it on filesystem which can support size > 4 GB. Run 'file x.tar' to see if it is acutally tarred or tar bzipped or something else. You wrote bzip2 deleted original file. That means you should have .tar.bz2 and not .tar. Am I missing something here? Uncompress on latest Linux, I have faced problems when files dont get uncompressed on windows ...



Only top voted, non community-wiki answers of a minimum length are eligible