Duplicity not writing to a pre-existing S3 bucket

Question

I'm trying to backup a directory to a pre-existing Amazon S3 bucket using the following command:

duplicity --no-encryption system/ s3+http://MY_BUCKET_NAME/backup

However, I'm getting the following error consistently:

S3CreateError: S3CreateError: 409 Conflict
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>BucketAlreadyOwnedByYou</Code><Message>Your previous request to create the named bucket succeeded and you already own it.</Message><BucketName>vacationlabs</BucketName><RequestId>3C1B8C49469E3374</RequestId><HostId>4dU1TKf3Td6R0yvG9MaLKCYvQfwaCpdM8FUcv53aIOh0LeJ6wtVHHduPSTqjDwt0</HostId></Error>

1. The S3 bucket is empty and does NOT have the backup directory
2. The bucket is in Singapore region

Answer 1

Duplicity also supports Amazon S3 URLs in this format:

s3://host/bucket_name[/prefix]

where host is the S3 endpoint for your region, which you can find in Amazon's list of endpoints.

In your case, the URL you want is:

s3://s3-ap-southeast-1.amazonaws.com/MY_BUCKET_NAME/backup

Note that:

• Duplicity can still only create buckets in the US Standard and EU regions, which means you'll need to manually create the bucket (which you already have).

• For regions other than US Standard, you will want to have Duplicity use Amazon's "new style" subdomain bucket addressing, by adding --s3-use-new-style to your Duplicity command line.

Answer 2

@mgorven is right, and I'm going to try to get this fixed properly. If you'd like to see this fixed, please mark yourself affected and subscribe at https://bugs.launchpad.net/duplicity/+bug/1003159

The s3://<region>/<bucket> syntax doesn't work (any more), you can only do s3+http://<bucket> which uses us-east-1 or eu-west-1 if you specify --s3-european-buckets. You can't get to any other regions.

Answer 3

Based on the duplicity manpage, it looks like it only supports the US and EU S3 regions. I'd guess that it's talking to the US endpoint, detects that the bucket doesn't exist in the US region, and then tries to create it and fails.

Try creating a bucket in the US or EU region and test whether that works (using --s3-european-buckets if appropriate). If that does work and you really want to use the Singapore region, you'll have to hack Duplicity to support it.