Change Amazon S3 storage class on A LOT of files

Question

We are using Amazon S3 and a great deal of our files have the STANDARD storage class. We want to change this to REDUCED_REDUNDANCY class to save the extra money.

However, is has been hard to find a way to do this that didn't take more than a week to run. Doing it from the AWS Web Console was extremely slow and earlier tests with Bucket Explorer also proved to be really slow.

Maybe there just isn't a fast way of doing it, but earlier results have shown that s3cmd was quite fast. It's just that it doesn't seem to be able to change the storage class (unless you upload/copy a file).

Does anyone know of a fast way of doing it?

Answer 1

The problem you are running into is that the latency of each individual request is high. The solution for this problem on AWS is to parallelize requests to a high degree instead of making requests completely sequentially.

AWS/S3 is set up to handle hundreds and thousands of requests in parallel without any degradation in performance. This will likely require you to do a bit of programming using the API, or you can get some benefit (double, triple, ...) with your existing comfortable UI by making the request several times in different windows for different segments of the S3 objects.