(note, this is not a duplicate of Creating a tar file with checksums included)
I'm familiar with using tar + gzip to create a compressed tar file (
tar cf - files | gzip > something.tar.gz), and gzip does add a master checksum so it will be apparent if the file gets corrupted. This is nearly the behaviour I want.
However, I have a computer with a (really) slow processor, but a fast network card. I've found that if I use tar plus gzip plus socat, that my network transfer is 1/10th the speed (100Mbps) versus leaving the gzip command out of the pipeline (950Mbps).
Some archive utilities, like 7Zip and Zip support an option for zero compression. I don't see that gzip or bzip2 have such an option. But Zip and 7Zip don't support proper streaming like gzip and bzip2 do (I know that 7z can read/write the plaintext from stdio, but it won't write the compressed file to stdout). I must have a proper streaming "compression" program because I'll be using socat to ship the archive to a remote host.
So the question is, is there a way to create a tar archive, while wrapping the output in a archive-like format, not using gzip or bzip2? Or is there some way to tickle gzip or bzip2 into using "no compression"? Or is there a dirt-simple ultra-fast streaming compression utility which might use only RLE encoding?
caveats - it needs to be fully streaming so I can use socat; the solution must be CPU light; solution must use parts available in cygwin and debian repositories