5

Extracting 7zip with many small files on linux xfs, extraction speed is so slow.

My Test System Environment.

  • System1 : 2vCore/4GB CentOS 6.5 64bit guest on XenServer.
  • System2 : 2vCore/4GB Windows 2012 guest on XenServer.

Large small files Creation Script

#!/bin/bash

for i in {1..100000}
do
    dd if=/dev/zero of=./dmp/dd$i.dmp bs=1K count=4
done

ok. I got 4KB files of 100,000 in dmp directory. and then I formatted /testfs with xfs.

Test

Compression Time on System1 :
   # time 7za a dmp.7z ./dmp     --> real   0m27.518s
Extraction Time on System1: 
   # time 7za x dmp.7z -o/testfs --> real   9m5.420s
   - 7za use only single core.
   - Dirent system call use most of the CPU time.
Extraction Time on System2:
   - around 25s
   - 7za use multi-core.

My Question

  1. Although system2 use multi-core and system1 use single-core, such a big different result is unreliable. I think. How can I explain this ?

  2. If I use xfs on linux for extracting 7zip file, How can I speed up ? any idea ?

Thanks in advance.

Improve this question
4
  • 1
    You're missing a huge variable here: IO. What do the storage systems of the two systems look like?
    – EEAA
    Jul 14, 2015 at 2:01
  • 1
    ok. Two systems share the same nfs storage repository that is abstracted by hypervisor. thanks.
    – fromcloud
    Jul 14, 2015 at 2:41
  • I have absolutely the same problem. 60 MB archive with small files takes minutes on Fedora and less than 1s on Windows. Some shit happens with p7zip on Linux.
    – Maxim
    Jan 19, 2020 at 23:33
  • 1
    And it is not related to file system. I have XFS, EXT4, with luks and without. Resut is the same. Considering that question is 4 years old, I have to switch to another archiving tool.
    – Maxim
    Jan 19, 2020 at 23:36

2 Answers 2

Reset to default
4

I had the same issue, and the fix was to upgrade 7z. On Ubuntu 20.04 LTS, the 7z you get from apt install is from 2016. Instead, go download the latest version from the official website.

For the same archive with 60k small files on the same server, I went from 8 minutes to 6 seconds(!).

Improve this answer
6
  • 1
    The OP specifically specified CentOS vs. Windows. Giving an Ubuntu solution may or may not work.
    – doneal24
    Jan 20, 2023 at 21:36
  • @doneal24 From the question title and its tags I understand it to be about Linux in general and that CentOS was only used as an example.
    – bfontaine
    Jan 23, 2023 at 15:05
  • 1
    This gives a 20x speed up in Ubuntu 18.
    – Kais Hasan
    Aug 16, 2023 at 7:13
  • Ops are using centos, why are u posting a ubuntu solution? DOes this works on centos??
    – Nicholas Jela
    Mar 25 at 6:47
  • @NicholasJela I already answered that question in another comment. I don’t know if it works on CentOS, but it’s quite simple to test.
    – bfontaine
    Mar 25 at 20:32
0

Here's what I did to get Nautilus and Archive Manager (file-roller) in Ubuntu 22.04 to update to use 7-Zip 22.01:

Download https://7-zip.org/a/7z2201-linux-x64.tar.xz

In the Terminal (Ctrl-Alt-T) copy and paste the following commands:

sudo apt-get install p7zip-full
sudo 7z e 7z2201-linux-x64.tar.xz 7z2201-linux-x64.tar
sudo 7z e -o/usr/lib/p7zip -y 7z2201-linux-x64.tar 7zz 7zzs
sudo echo exec /usr/lib/p7zip/7zz '"$@"' > 7z
sudo mv 7z /bin/7z
sudo chmod ugo+rx /usr/lib/p7zip/7zz /usr/lib/p7zip/7zzs /bin/7z
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .