Hot answers tagged

65 votes
Accepted

bzip2 too slow. Multiple cores are avaible

There are many compression algorithms around, and bzip2 is one of the slower ones. Plain gzip tends to be significantly faster, at usually not much worse compression. When speed is the most important, ...
user avatar
  • 994
41 votes

bzip2 too slow. Multiple cores are avaible

Use pbzip2. The manual says: pbzip2 is a parallel implementation of the bzip2 block-sorting file compressor that uses pthreads and achieves near-linear speedup on SMP machines. The output of ...
user avatar
  • 5,242
40 votes
Accepted

How to zip files with a size limit?

You can use the "split archive" functionality of "zip" itself using the "--split-size" option. From "zip" manpage ("man zip"): (...) One use of ...
user avatar
  • 1,112
39 votes

How to split a file and compress directly?

Use "--filter": split --bytes=1024M --filter='gzip > $FILE.gz' /path/to/input /path/to/output
user avatar
  • 391
18 votes

How do I load a sql.gz file to my database? (importing)

If you get an error from zcat, in which the error message contains the file name with an extra suffix .Z, then try using gzcat instead, as described at https://stackoverflow.com/questions/296717/zcat-...
user avatar
11 votes

How to zip files with a size limit?

tar -czvf - /path/to/files | split -b 10M - archive.tar.gz Will give you a number of files: archive.tar.gzaa archive.tar.gzab ... Which then can be uncompressed with: cat archive.tar.* | tar -...
user avatar
10 votes

Break up a dd image into multiple files

It is my command line: dd if=/dev/sda bs=4M | gzip -c | split -b 2G - /mnt/backup_sda.img.gz It will create 2GB files in this fashion: backup_sda.img.gz.aa backup_sda.img.gz.ab backup_sda.img.gz.ac ...
user avatar
  • 101
9 votes

Fastest way to extract tar.gz

If you want to see progress use something like pv. Here is an Example: pigz -dc mysql-binary-backup.tar.gz | pv | tar xf -
user avatar
9 votes
Accepted

Enable both brotli and gzip on NGINX?

One of the first articles I found when searching online for nginx and brotli, is this link. It kinda answers your question — I think: Browsers which supports brotli send ‘br’ along with ‘gzip’...
user avatar
  • 5,477
8 votes

What is the best nginx compression gzip level?

If you really can spare CPU resources, you can use 9, but for most sites a value of 2 is enough, since gzip doesn't reduce the file much after level 1. Edit: I looked at Amazon CloudFront and it ...
user avatar
  • 235
8 votes

How to gzip logs created by rotatelogs

You could use rotatelogs option -p to use a program to compress the log after the rotation. (See for reference: https://httpd.apache.org/docs/2.4/programs/rotatelogs.html) -p program If given, ...
user avatar
8 votes

bzip2 too slow. Multiple cores are avaible

Google's brotli is a newer format that has gained some wide support within browsers recently, as it has some impressive compression, some impressive speed, and perhaps the most impressive combination/...
user avatar
  • 337
7 votes

nginx gzip_static: why are the non-compressed files required?

Uncompressed files are not required on Nginx 1.6 with: location ~ \.txt$ { gzip_static on; gunzip on; } Both curl http://localhost/index.txt and curl -H "Accept-Encoding: ...
user avatar
  • 677
7 votes
Accepted

IIS 7.5 inconsistently Gzips files (with PHP & ASP.NET)

I found a likely culprit in the comments of this page; weblog.west-wind.com Essentially, OOTB, IIS will only gzip if the file is requested at least twice in 10 seconds. This is tuned via in web....
user avatar
  • 261
7 votes
Accepted

Faster zlib alternatives

Several years after this question was first asked some faster x86_64 zlib's appeared using the type of optimization BarsMonster suggested: Cloudflare zlib Intel zlib Intel's igzip Google's c-zlib ...
user avatar
  • 1,210
7 votes
Accepted

How to ideally store images on Amazon S3

JPEGs are already inherently a compressed format, so further compression isn't really something that'll give you any significant benefit. You could reencode them with a lower quality setting but that'...
user avatar
  • 32.1k
7 votes

Resume a tar command that was stopped

Is there some parameter that allows my tar file to be resumed after some problem? Nope. What you should do, though, is run your command from within a terminal multiplexer like screen or tmux. That ...
user avatar
  • 108k
7 votes

How to set apache 2.2 gzip compression level?

You can adjust the compression level by simply adding DeflateCompressionLevel. DeflateCompressionLevel 9 This line of code can range from 1 to 9, 9 being the best compression method. By default 9 is ...
user avatar
  • 507
7 votes
Accepted

wget and pipe to tar in the background

You will have to instruct wget to write to STDOUT, you do this by passing -O - to wget, otherwise it will save the file to disk. tar is failing because as executed it's expecting something from STDIN. ...
user avatar
7 votes
Accepted

Does Nginx compress files for every request? (with gzip_on)

Yes Nginx will compress index.html each time it is requested since there is no index.html.gz file. To be honest gzip is not very processor intensive these days and gzipping on the fly (and then ...
user avatar
6 votes

How do I load a sql.gz file to my database? (importing)

On macOS, I used this: zcat < [Database].sql.gz | mysql -u root -p [Database Name in MySQL] Enter your password, and voila!
user avatar
  • 161
6 votes
Accepted

nginx gzip Vary header together with existing Vary headers

RFC2616 states duplicate headers are acceptable and should be treated the same as a single header with a comma-separated list of values: Multiple message-header fields with the same field-name MAY ...
user avatar
  • 4,845
6 votes

gzip not working on nginx for JSON responses

Ah man - I can't believe how dumb this is!!! The data in question was 245 bytes: Content-Length: 245 ...and I had this set: gzip_min_length 256k set. I've changed it to: gzip_min_length 128; and ...
user avatar
5 votes

Why is gzip compression varying in efficiency in IIS?

The compression of static files is handled dynamically (if dynamic compression is enabled) while the file is considered infrequent by IIS. Once the file is considered frequent it will be compressed ...
user avatar
  • 183
5 votes
Accepted

Rolling log file without losing any log event

You'll find it easier to configure logrotate to do the rotation for you. If you create a file called /etc/logrotate.d/php containing something like the following, it'll handle the log rotation ...
user avatar
  • 7,588
5 votes
Accepted

dd command piped over gzip and ssh faster and faster as dd progresses

Your command could be simplified by leaving out the two gzip commands. If compression is useful in your case, it is much simpler to compress data in transit by giving a -C argument to the ssh command, ...
user avatar
  • 29.7k
5 votes

How to zip files with a size limit?

Here's how I did it for a 5GB file (split into 1GB vs 10MB as OP asked)... Example: To split a 5GB file into 1GB files to copy to a FAT32 USB (filename "FIVE_GB_FILE.ISO") Step 1: zip the file (no ...
user avatar
  • 161
5 votes

Enabling Brotli compression on IIS

It appears the Brotli module you referenced requires a paid license, so I haven't tried it, but I encountered a similar issue with my own open source Brotli plugin for IIS. Current browsers advertise ...
user avatar
5 votes
Accepted

How to compress SQL file correctly on CentOS?

The size of your mysqldump file is not going to be the same as your database size as reported by PHPMyAdmin. First, create the dump file with the command: mysqldump -u myuser -p mydb > dumpfile....
user avatar
  • 2,613
4 votes

How do I load a sql.gz file to my database? (importing)

You can use -c, --stdout, --to-stdout option of gunzip command for example: gunzip -c file.sql.gz | mysql -u root -p database
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible