How to backup/index file metadata

Question

I am advocating to implement a solid backup concept before facing data loss. Having onsite and offsite backups is one pillar of such a concept. However I learned that the metadata is sometimes as valueable as the actual file contents.

E.g. I had a data backup and a recovered btrfs filesystem and needed to decide whether to restore the backup or to keep the recovered data. As I was lacking checksums of the files, I decided to restore from offsite backup.

After the restore I learned, that the file creation date, ownership and permissions are also crucial to keep track of. Luckily, my data is structured and I could set defaults per script.

However to improve my backup plan, I want to store the metadata of my files.

The first action was to sha256sum and stat the data and save it to disk.

How could you index and store the metadata of the files on Linux in an easier and more efficient way?

I've looked into git-annex which looks promising and I've seen people use updatedb to keep track of files (also metdadata?) but maybe someone has better options?

Answer 1

For critical server please use a solution that do a entire machine backup, not just a file copy. That way the metadata, and all related configuration would be saved.

If a virtual machine, some software just plain copy the whole hypervisor data file.

Answer 2

Most backup systems can preserve metadata.

tar archives contain permission and timestamp metadata, but the sha256 checksum you want.

An image of the entire block device contains the original file metadata. However, this possibly has operational challenges. For example, it may be difficult to do a partial restore, and extra steps are required to migrate to a different file system type.

Or take a well documented example of Borg. File metadata preserved includes timestamps and UNIX permissions. A repository can be checked to verify its consistency.

For any of these, you can additionally compute and store your favorite hash.

---

Whatever backup software used, define the recovery point objective then test restores. Restored objects should not only have correct metadata, but contain good data.

Answer 3

You could use a system integrity tool like tripwire, AIDE, etc. to store checksums and metadata for your files. These tools are designed to detect changes to file metadata and contents, so they create an index of this information.