3

Let's assume I have a several gigabyte tar file, but I also happen to know that the very last file written to the archive is something important that I need. Since tar files are appended sequentially, is there a way I can make tar read into the archive from the end to find this file, instead of starting from the beginning and reading over gigabytes of irrelevant data?

6

No, unfortunately there is not. From Wikipedia

Another weakness of the tar format compared to other archive formats is that there is no centralized location for the information about the contents of the file (a "table of contents" of sorts). So to list the names of the files that are in the archive, one must read through the entire archive and look for places where files start. Also, to extract one small file from the archive, instead of being able to lookup the offset in a table and go directly to that location, like other archive formats, with tar, one has to read through the entire archive, looking for the place where the desired file starts. For large tar archives, this causes a big performance penalty, making tar archives unsuitable for situations that often require random access of individual files.

3
  • I agree with this, except I can't think of a reason why it should be impossible for a program to be able to start from the end of a TAR file to search backwards for those same file markers. – andyortlieb Sep 15 '11 at 17:43
  • I actually tried to do this after I posted my answer. I was not successful, but I only spent about 10 minutes on it. – Louis Marascio Sep 15 '11 at 18:00
  • This answer is not correct because the question is achievable with dd skip. – user1133275 Jun 28 '20 at 16:41
0

We can efficiently seek to last file on archive if tar is created on storage which is seekable, i.e. on hard disks and not on tape. use GNU tar's -n or --seek option. ( see this GNU tar options page ) say for instance, file which is stored last with name last_file.txt , you could use following command

tar -nxvf <your_archive> last_file.txt

Which would simply extract last_file.txt. because tar format contains size of a each file in a header, it is possible to skip entire file efficiently using seek system call, (see tar file format )

To only list all files efficintley in a big archive, use

tar -ntvf <your_archive>
1
  • tar files have a per file header before each file, not a global header so the best tar could do is read every file header seeking over the file data, which is still slow compared to making an index for the tar (which is possible). You can see this with 'xxd', and 'time' on a file more than RAM on a spinning disk. – user1133275 Jun 28 '20 at 16:31
0

Yes; if you know the size of the file you want, you can copy the end of the tar with dd skip. or if you want to read the whole file once for later quick random access you can make an index with:

tar -tRvf "$TAR"

An example script:

#!/bin/bash

#
# tar_extract_via_index.sh
#

TAR="$1"
RE="$2"

if [ ! -f "$TAR" ] ; then
    echo "Not a file $TAR"
    exit 1
fi
if [ "$RE" == "" ] ; then
    echo "Expecting a $RE"
    exit 2
fi
if [ ! -f "$TAR".index ] ; then
    tar -tRvf "$TAR" > "$TAR".index
fi
MATCH="$(grep -P "$RE" "$TAR".index)"
if [ "$(echo "$MATCH" | grep -c .)" != "1" ] ; then
    echo "Multipule matches:"
    echo "$MATCH" | perl -pe 's/^/\t/g' >&2
    exit 3
fi
FILE="$( echo "$MATCH" | perl -pe 's/.* \.\///g;s/.*\///g')"
SKIP="$( echo "$MATCH" | perl -pe 's/:.*//g;s/.* //g')"
COUNT="$(echo "$MATCH" | perl -pe 's/\.\/.*//g;s/.*\/[^ ]+ +//g;s/ .*//g')"
SKIP="$(echo "($SKIP+1)*512" | bc)"
dd if="$TAR" bs=1 status=none skip=$SKIP count=$COUNT of="$FILE"
echo "$FILE"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.