I have a service that generates large data sets where only a few of the files actually change and the rest remain unchanged (but are still in the data folder). We need to keep each version of the data sets, but I'd like to use object versioning to reduce costs for my cloud archival storage. If I upload a new data set with a lot of old files and only a few new ones, will the bucket only generate new versions of the objects which have changed? Will there be the same set of files in the folder, with a few that have multiple versions?
You enable Object Versioning for a bucket. Once enabled, Cloud Storage creates an archived version of an object each time the live version of the object is overwritten or deleted.
While arguably not explicitly saying so, each time the live version of the object is overwritten includes the case when you overwrite an object with exactly the same payload.
To avoid this, you need to upload the objects with a utility that avoids overwriting unchanged files.
gsutil rsync appears to do this, but read the documentation and observe its behavior, because while it can sync with timestamps or checksums, it appears to prefer timestamps, so may not do the right thing if the local files are identical but appear newer.
Every time you upload a file S3 creates a new version. Even if the file has the same content.
To work around that you’ll have to:
- Generate your new dataset locally.
- List all files from the S3 bucket - that will give you Size and ETag which happens to be the file’s md5 sum.
- Compare the local files with the bucket list - if the Size differs or the received ETag differs from calculated md5 sum it’s clearly a different file -> upload it.
Alternatively use tools like
aws s3 sync or
s3cmd sync that do exactly that from the command line or shell scripts.
Hope that helps :)