0

What is the best way to compress a file if it is more than 4 GB in size?

I am using the following for last several months.

mysqldump --all-databases | zip > mybackup.zip

Today I am got an error:

zip error: Entry too big to split, read, or write (file exceeds Zip's 4GB uncompressed size limit)

I am considering using bzip2. Is my choice correct?

0

3 Answers 3

4

Why not use the industry standard for this kind of stuff: "gzip"?

mysqldump --all-databases | gzip > mybackup.gz

Size Comparison:

720K    mybackup.gz     (compressed)
2.6M    mybackup.sql    (same data, but uncompressed for comparison)
2

Use xz:

mysqldump --all-databases | xz -9 -c > mybackup.xz

The compression ratio is much higher than zip, just make sure to watch the memory usage. If you are running xz in a memory-restricted environment you could use the following table to tune the command and avoid paging:

Preset DictSize   CompCPU   CompMem   DecMem
-0     256 KiB       0        3 MiB    1 MiB
-1       1 MiB       1        9 MiB    2 MiB
-2       2 MiB       2       17 MiB    3 MiB
-3       4 MiB       3       32 MiB    5 MiB
-4       4 MiB       4       48 MiB    5 MiB
-5       8 MiB       5       94 MiB    9 MiB
-6       8 MiB       6       94 MiB    9 MiB
-7      16 MiB       6      186 MiB   17 MiB
-8      32 MiB       6      370 MiB   33 MiB
-9      64 MiB       6      674 MiB   65 MiB

xz file size is unlimited (filesystem dependant).

1

Either upgrade zip to at least version 3.0 to support Zip64:

$ zip -v

This is Zip 3.0 (July 5th 2008), by Info-ZIP.

Zip special compilation options:
    LARGE_FILE_SUPPORT   (can read and write large files on file system)
    ZIP64_SUPPORT        (use Zip64 to store large files in archives)

or use the different tools such as: 7z, pbzip2, ...

You should be able to compress files larger than 4GB with pbzip2.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .