Do any of the commercial IaaS object stores (S3, Azure Blobs etc.) avoid charging multiple times for storing duplicate data (identical files, or parts of files)? For instance, we have a 15 TB dataset of tweets and one of our team wants to make a copy and then make a few modifications to the data. Will we be charged for 30 TB of storage?
Is there a good way to find duplicate chunks on these large object stores, or to compress large datasets in-place? Can we replace duplicate files with some kind of symlinks?