EDIT: clarify context
I have several loosely synchronized filesystems on different machines. (some content is redundant, some is not, things get moved by hand by the users). These are large scientifical datasets (many tens of terabytes); They move across clusters depending on the kind of work we perform on them. They have no standard naming convention (files sometimes get renamed as the various experiments go, or when subsets of files are selected or merged).
I'd like to find a tool that allows me to efficiently find redundancy across remote filesystems, so that we can delete redundant data, and copy non-redundant data when decomissioning storage bricks. (Side note: distributed filesystems like Ceph promise to handle these cases; this will be the future route, but now we have to deal with the existing system as-is)
Since many objects have been moved and renamed by hand, I cannot rely on their file names to compare with diff or rsync. I'd rather use a crypto checksum such as sha256 to identify my data files.
I don't want to checksum the whole dataset every time I run a comparison either. The files, once created, are not likely to change often, so the checksums should be cached.
Is there an existing tool to do this ? Maybe something that stores a checksum in a Posix Extended Attribute (using the timestamp to check the checksum freshness), and a tool that can extract that information to efficiently diff the contents of the filesystems, without caring about the filenames ?