7Zip extraction on linux is so slow

Question

Extracting 7zip with many small files on linux xfs, extraction speed is so slow.

My Test System Environment.

• System1 : 2vCore/4GB CentOS 6.5 64bit guest on XenServer.
• System2 : 2vCore/4GB Windows 2012 guest on XenServer.

Large small files Creation Script

#!/bin/bash

for i in {1..100000}
do
    dd if=/dev/zero of=./dmp/dd$i.dmp bs=1K count=4
done

ok. I got 4KB files of 100,000 in dmp directory. and then I formatted /testfs with xfs.

Test

Compression Time on System1 :
   # time 7za a dmp.7z ./dmp     --> real   0m27.518s
Extraction Time on System1: 
   # time 7za x dmp.7z -o/testfs --> real   9m5.420s
   - 7za use only single core.
   - Dirent system call use most of the CPU time.
Extraction Time on System2:
   - around 25s
   - 7za use multi-core.

My Question

1. Although system2 use multi-core and system1 use single-core, such a big different result is unreliable. I think. How can I explain this ?

2. If I use xfs on linux for extracting 7zip file, How can I speed up ? any idea ?

Thanks in advance.

Answer 1

I had the same issue, and the fix was to upgrade 7z. On Ubuntu 20.04 LTS, the 7z you get from apt install is from 2016. Instead, go download the latest version from the official website.

For the same archive with 60k small files on the same server, I went from 8 minutes to 6 seconds(!).

Answer 2

When this question was originally asked, 9 years ago in July 2015, 7zip sorted files to be compressed by extension type, alphabetically. Thus, *.avi then *.bmp then *.doc, etc.

On file systems that sorted by file name (particularly ext4 and ntfs), this lead to many, many additional seeks. The problem is confounded on extant file systems where small files occupied less space than the cluster size (again, particularly problematic on both ext4 and ntfs).

This was fixed in 7z v15.06, coincidentally, on 6 August 2015, a mere 21 days after this question was asked.

Citation: https://www.7-zip.org/faq.html

Why 7z archives created by new version of 7-Zip can be larger than archives created by old version of 7-Zip?

New versions of 7-Zip (starting from version 15.06) use another file sorting order by default for solid 7z archives.

Old version of 7-Zip (before version 15.06) used file sorting “by type” (“by extension”).

New version of 7-Zip supports two sorting orders:

• sorting by name - default order.
• sorting by type, if ‘qs’ is specified in Parameters field in “Add to archive” window, (or -mqs switch for command line version).

You can get big difference in compression ratio for different sorting methods, if dictionary size is smaller than total size of files. If there are similar files in different folders, the sorting “by type” can provide better compression ratio in some cases.

Note that sorting “by type” has some drawbacks. For example, NTFS volumes use sorting order “by name”, so if an archive uses another sorting, then the speed of some operations for files with unusual order can fall on HDD devices (HDDs have low speed for “seek” operations).

Answer 3

Here's what I did to get Nautilus and Archive Manager (file-roller) in Ubuntu 22.04 to update to use 7-Zip 22.01:

Download https://7-zip.org/a/7z2201-linux-x64.tar.xz

In the Terminal (Ctrl-Alt-T) copy and paste the following commands:

sudo apt-get install p7zip-full
sudo 7z e 7z2201-linux-x64.tar.xz 7z2201-linux-x64.tar
sudo 7z e -o/usr/lib/p7zip -y 7z2201-linux-x64.tar 7zz 7zzs
sudo echo exec /usr/lib/p7zip/7zz '"$@"' > 7z
sudo mv 7z /bin/7z
sudo chmod ugo+rx /usr/lib/p7zip/7zz /usr/lib/p7zip/7zzs /bin/7z