Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm trying to speed up a transfer of images from our web server to S3. Some of the images have been uploaded already, but the process was interrupted. We've written new scripts to upload images if they don't already exist in S3, however the if_object_exists() method seems to be quite slow (1-2 seconds). We're thinking of just attempting to upload every image, but make S3 fail if the object exists (hoping this will be faster). Does anyone know if there's a "bucket policy" or some other setting we can use to prevent overwrites?

share|improve this question

closed as off topic by thinice, Scott Pack, mdpc, Chris S Nov 20 '12 at 15:12

Questions on Server Fault are expected to relate to professional server, networking, or related infrastructure administration within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.

You tried using rsync? Sounds like you're re-inventing the wheel. –  Magellan Nov 19 '12 at 19:38
Thanks guys. We've tried most commands, but our problem is that we have to be selecting in 'which' images we upload. Most tools seem to want to upload a full directory, or worse for us (we've got millions of images) generate a list of all files in a directory before filtering which to upload. We've written a custom script that seems to do what we want. Thank you for your help, will close this now. –  Erik Nov 19 '12 at 22:53
add comment