2

I have a service that generates large data sets where only a few of the files actually change and the rest remain unchanged (but are still in the data folder). We need to keep each version of the data sets, but I'd like to use object versioning to reduce costs for my cloud archival storage. If I upload a new data set with a lot of old files and only a few new ones, will the bucket only generate new versions of the objects which have changed? Will there be the same set of files in the folder, with a few that have multiple versions?

  • 1
    A workaround could be to run a de-duplicating backup system and store the backup on cloud. Restic, for example, though that's beta software. – Tim Mar 19 at 1:43
1

You enable Object Versioning for a bucket. Once enabled, Cloud Storage creates an archived version of an object each time the live version of the object is overwritten or deleted.

https://cloud.google.com/storage/docs/object-versioning

While arguably not explicitly saying so, each time the live version of the object is overwritten includes the case when you overwrite an object with exactly the same payload.

To avoid this, you need to upload the objects with a utility that avoids overwriting unchanged files. gsutil rsync appears to do this, but read the documentation and observe its behavior, because while it can sync with timestamps or checksums, it appears to prefer timestamps, so may not do the right thing if the local files are identical but appear newer.

1

Every time you upload a file S3 creates a new version. Even if the file has the same content.

To work around that you’ll have to:

  1. Generate your new dataset locally.
  2. List all files from the S3 bucket - that will give you Size and ETag which happens to be the file’s md5 sum.
  3. Compare the local files with the bucket list - if the Size differs or the received ETag differs from calculated md5 sum it’s clearly a different file -> upload it.

Alternatively use tools like aws s3 sync or s3cmd sync that do exactly that from the command line or shell scripts.

Hope that helps :)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.