Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We are using Amazon S3 and a great deal of our files have the STANDARD storage class. We want to change this to REDUCED_REDUNDANCY class to save the extra money.

However, is has been hard to find a way to do this that didn't take more than a week to run. Doing it from the AWS Web Console was extremely slow and earlier tests with Bucket Explorer also proved to be really slow.

Maybe there just isn't a fast way of doing it, but earlier results have shown that s3cmd was quite fast. It's just that it doesn't seem to be able to change the storage class (unless you upload/copy a file).

Does anyone know of a fast way of doing it?

share|improve this question
1  
I'd script this using one of the SDKs. It'll likely be significantly faster. –  ceejayoz Sep 26 '11 at 16:33

1 Answer 1

up vote 3 down vote accepted

The problem you are running into is that the latency of each individual request is high. The solution for this problem on AWS is to parallelize requests to a high degree instead of making requests completely sequentially.

AWS/S3 is set up to handle hundreds and thousands of requests in parallel without any degradation in performance. This will likely require you to do a bit of programming using the API, or you can get some benefit (double, triple, ...) with your existing comfortable UI by making the request several times in different windows for different segments of the S3 objects.

share|improve this answer
1  
We did something like this in the end. Created a script that ran through our database and got all the files from that. Then for each of those use s3cmd to change the storage class by doing a cp command with --rr as a parameter. Parallellize with 8 s3cmd's. And in the end (and seems to be the best speedbump) is to run it on an ec2 instance. It will be done in ~4 days. –  freeall Sep 27 '11 at 11:24
    
Fire up 128 s3cmd's and it could get done in 6 hours. Even if you had to run them on 16 instances (unlikely), your total number of instance hours (and therefore cost) would be about the same. –  Eric Hammond Sep 27 '11 at 22:54

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.