3

I have a folder which i want to create tgz file and compute its sha256:

The folder is converted to tgz using following command

 "tar -c -C #{Shellwords.escape dir} #{Shellwords.escape basename} " \
        "--owner=0 --group=0 --mtime='2000-01-01 00:00:00' | gzip -n > #{Shellwords.escape file}"

Now i ran the above process using 2 separate users and it gave me 2 files: 1 and 2

Both tgz files differ in sizes:

-rw-r--r--@ 1 myuser  \Domain Users  9024 Jul 31 14:28 1.tgz
-rw-r--r--@ 1 myuser  \Domain Users  9037 Jul 31 14:29 2.tgz

If i try to compute diff between files i dont see any diff.Diff was obtained using following command.

diff  <(tar -tvf 1.tgz | sort) <(tar -tvf 2.tgz | sort)

If i compute sha256 using ruby of these 2 files , then it differs.

The question is : Why do i get difference in tgz files when running from diffenrent users.

EDIT: After reading the comments and some googling i found that the order in which the files are added is not fixed everytime.

see this https://reproducible-builds.org/docs/archives/#file-ordering.

I will try this and add details.

  • Have you compared content of this archives? – Alexander Tolkachev Aug 2 '17 at 6:03
  • Just guesses but tar might include some metadata in the archive which would alter the archive's hash without changing the files within it. Or tar might not be deterministic in the way it determines how to compress or how to store the compressed data within the file. – Ginnungagap Aug 2 '17 at 17:00
  • @AlexanderTolkachev :Yes the contents are exactly the same – user93796 Aug 2 '17 at 18:33
  • @Ginnungagap :I am trying to set same metadata by using "--owner=0 --group=0 --mtime='2000-01-01 00:00:00" when tarring. Any way by which i could check the metatadata for tar file ?Where is it stored? Any command? – user93796 Aug 2 '17 at 18:35
  • 1
    The metadata you're trying to set are actually file attributes and do not alter the file contents, the metadata I was speaking of would more likely be stored directly within the file by tar. Also, since AFAIK tar does store file attributes such as umask, dates, owner, group, xattrs, changing any of those between two tars would unavoidably alter the archive. – Ginnungagap Aug 2 '17 at 19:48
2
+50

There are many things that this could be due to.

  • The metadata stored by tar (as well as the metadata stored by gzip, which may include the modification time of the tar archive). I see you're using some GNU tar options that could potentially reset some parts of this metadata, but I'd wager that the options are not exhaustive of all the variable attributes.

  • The order of the files. When you extract the files onto a filesystem, the order hardly matters for most applications (although each directory entry generally comes before or after any other entry within the same dirent on the underlying filesystem). However, the order of the files in a tar archive is not guaranteed.

  • The gzip compression. It's guaranteed by the file format that the compressed files would be decompressed to the originals, however, it is not necessarily guaranteed that their compressed form has to be identical. Moreover, if the input differs in content (even if it stays at the same size), then, likewise, you'll see that the two archives may even be so different as to have different file size.

In summary, if you're trying to determine if the contents of two folders is the same, using .tgz archives is probably not the best way to go about it.

  • Basically, if you're still trying to go with tar, you might want to write your own script that would use a tar library of sorts (e.g., within Perl), and take extra care to only store the contents of the files, in a specific and deterministic order, without any of the attributes; it'll may also be faster to pipe it directly to sha256 instead of compressing it through gzip first. – cnst Aug 9 '17 at 17:51
2

TL;DR: Yes, as you guessed, it is very likely the difference in User IDs that causes the size difference in the resulting files.

Here is a definition, in C structures, of the tar file format:

https://www.gnu.org/software/tar/manual/html_node/Standard.html

You may notice that even in this "definition", there are arguments about the particulars of the tar file headers and metadata information, what exactly is stored and where. But, while there are differing implementations of the tar file format, there is at least agreement that there is indeed metadata information stored about each file or object within the tar file, stored in a dedicated header block before the file content. For your use case, it is relevant that two items that are stored in the tar metadata blocks are user and group file and directory owners.

More details can also be found from the FreeBSD project man page on tar:

https://www.freebsd.org/cgi/man.cgi?query=tar&sektion=5&manpath=FreeBSD+8-current

Tar has a long, winding history, that follows the many turns in the development of serial, non-random-access storage in computing since the 1970s. Backwards-compatibility requirements can cause this sort of thing. :)

ProTip: For comparing directories using hashes, md5deep is your answer. http://md5deep.sourceforge.net/ :)

1

What if you do not sort the tar when running the diff command? It may just have added the files in a diffrent order and gzip then just zips it differently.

  • thats true and it addes the files in different order .plz check my latest edit. – user93796 Aug 9 '17 at 7:37
0

Once method to handle this, would be to add the files in a specific order: (assuming GNU tar and a Bourne-based shell) (This uses find to get the file listing and then sort it in a specific locale)

d="dir1";bn="basename";( cd "$d" && find "$bn" -type f -print0 | \
LC_ALL=C sort -z | \
tar --null -T - --owner=0 --group=0 --mtime='2000-01-01 00:00:00' \
--no-acls --no-xattrs --no-selinux -c | \
gzip -n; ) > out1.tgz

The list of --no--stuff would need to be updated as more features are added to tar and filesystems...

In most cases, if all you are about is the filenames and contents, diff -r might be more suitable... For more than one pair comparing the sha256sum of a sha256sum listing of all the files might be more reliable.

0

I'm not sure tar is the best way to do this. Too many variables under the hood, and not really using it in a way it seems designed to be used. Even more so with the compression.

Depending on your directory structure and time available, this may not be feasible, but have you considered hashing each file, then hashing that list?

One such scheme might be: list all files, sort deterministically, hash each individual file, then hash the output of this hash/filename combo.

This technique ignores all metadata, and deals with purely the content of the file and it's name.

Here is an example command (I'll walk though the individual pieces below)

find -L `pwd` 2> /dev/null | sort | awk '{ print "\""$0"\""}' | xargs md5sum 2> /dev/null > /tmp/out; md5sum /tmp/out | awk '{print $1}'; rm -rf /tmp/out &> /dev/null;
  • find -L \pwd` 2> /dev/null` - Finds a list of all files, ignoring errors
  • sort - Sorts file list by name do avoid the filesystem return order difference issues
  • awk '{ print "\""$0"\""}' - Adds quotes around each line. Not strictly required, but if you have any spaces or special chars in your path, you'll have trouble.
  • xargs md5sum 2> /dev/null > /tmp/out - Actually computes the hash of each line, returns hash to file.
  • md5sum /tmp/out | awk '{print $1}' - Hashes the final list of hashes. The awk is optional, but cleans up the output a bit.
  • rm -rf /tmp/out &> /dev/null - Cleanup temp files

This will result in, as far as I can tell, a "hash" for the directory tree.

According to my testing, it returned the same hash for an untouched directory tree (even after allowing some time to pass), but changed the hash upon changing any of the individual files, or even making a new blank file. The hash then returned back to the original upon deleting the new files and undoing changes.

YMMV.

  • Find command will break on whitespace. find -L . -type f -print0 | sort | xargs -0 md5sum ..... also loses the unnecessary awk and subshell pwd. – quadruplebucky Aug 12 '17 at 19:59
  • @quadruplebucky: find does not break. xargs does break at white by default, but with that (clunky) awk quoting it handles white okay, while it still mishandles quotemarks and backslash. xargs -d'\n' handles everything except newline which is pretty rare in pathnames. Your method would handle everything including newline but only if you make it sort -z. – dave_thompson_085 Aug 13 '17 at 10:07
  • PS: given it's now cheap to make md5 collisions and possible for sha1, if any of your files might ever come from or be influenced by someone else it's best to use sha2, probably sha256 although sha512 or one of the sha512-slashes may be better on 64-bit hardware. – dave_thompson_085 Aug 13 '17 at 10:13
  • a) find does break if pwd returns something with a whitespace. b) yeah, sort's sorta useless with one line, thanks. – quadruplebucky Aug 13 '17 at 19:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.