Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

What is the best way to compress a file if it is more than 4 GB in size?

I am using the following for last several months.

mysqldump --all-databases | zip > mybackup.zip

Today I am got an error:

zip error: Entry too big to split, read, or write (file exceeds Zip's 4GB uncompressed size limit)

I am considering using bzip2. Is my choice correct?

share|improve this question

3 Answers 3

up vote 4 down vote accepted

Why not use the industry standard for this kind of stuff: "gzip"?

mysqldump --all-databases | gzip > mybackup.gz

Size Comparison:

720K    mybackup.gz     (compressed)
2.6M    mybackup.sql    (same data, but uncompressed for comparison)
share|improve this answer

Use xz:

mysqldump --all-databases | xz -9 -c > mybackup.xz

The compression ratio is much higher than zip, just make sure to watch the memory usage. If you are running xz in a memory-restricted environment you could use the following table to tune the command and avoid paging:

Preset DictSize   CompCPU   CompMem   DecMem
-0     256 KiB       0        3 MiB    1 MiB
-1       1 MiB       1        9 MiB    2 MiB
-2       2 MiB       2       17 MiB    3 MiB
-3       4 MiB       3       32 MiB    5 MiB
-4       4 MiB       4       48 MiB    5 MiB
-5       8 MiB       5       94 MiB    9 MiB
-6       8 MiB       6       94 MiB    9 MiB
-7      16 MiB       6      186 MiB   17 MiB
-8      32 MiB       6      370 MiB   33 MiB
-9      64 MiB       6      674 MiB   65 MiB

xz file size is unlimited (filesystem dependant).

share|improve this answer

Either upgrade zip to at least version 3.0 to support Zip64:

$ zip -v

This is Zip 3.0 (July 5th 2008), by Info-ZIP.

Zip special compilation options:
    LARGE_FILE_SUPPORT   (can read and write large files on file system)
    ZIP64_SUPPORT        (use Zip64 to store large files in archives)

or use the different tools such as: 7z, pbzip2, ...

You should be able to compress files larger than 4GB with pbzip2.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.