I'm looking for a compression format that supports being tailed. Meaning you dont have to read the entire file to get the last X uncompressed bytes. Is this possible with any of the formats like bzip2, xz, lzma, etc?

I once coded something using gzip that could do this. Basically on a really high level, what it did was cat multiple gzip blocks together, then I had a util that could seek backwards from the end of the file until when the last block started. These files were fully readable by the standard gzip utilities, but I'm hoping theres something a little more standardized available.

The ultimate purpose for this is for log files which I can write out compressed, and then be able to tail them (even when they havent been fully written; i.e. streaming) without having to wait for the whole thing to be read from disk or network.


gzip has an --rsyncable option which does essentially the same. The non-standard part would be the gzip-block-aware "ztail" utility, but it seems like you've dealt with that already.

  • 2
    Of course that option effectively limits you to something like -0.5 compression level, even if you specify -9. – psusi Sep 19 '11 at 13:59
  • where are you getting the version of gzip that supports this? If this was an option at one point, it appears to have been removed. – Patrick Sep 19 '11 at 18:13
  • The Debian-based distros do have it - this is from Ubuntu 10.04: root@backup1:~# gzip -V gzip 1.3.12 root@backup1:~# gzip -h | egrep rsync --rsyncable Make rsync-friendly archive – the-wabbit Sep 20 '11 at 12:51
  • Apparently, some other distro maintainers (e.g. Fedora) seem to have included the patches as well. And there is a patch for an oldish gzip here: samba.org/netfilter/diary/gzip.rsync.patch which might apply to a more recent version with slight modifications, if you really need to self-compile. – the-wabbit Sep 20 '11 at 13:02
  • BTW, discussions suggest that the impact on compression is rather negligible (within 2-3 %), although the mileage for a specific dataset might vary. If you need "tailable" compression with adaptive algorithms like deflate, there is hardly any way around resetting the algorithm every now and then - of course this will induce a compression efficiency hit. – the-wabbit Sep 20 '11 at 13:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.