0

I am looking at duply and duplicity currently to mirror one bucket to another but it looks like I may be running in to a brick wall?

So far I have been able to get things working backing up local directories to an S3 bucket, that part works great. My real problem is that I am trying to create an S3 bucket backup by backing essentially mirroring two buckets.

When I attempt to run the duply tool I receive the following output:

sudo duply test backup
Start duply v1.5.10, time is 2014-07-01 11:49:48.
Using profile '/home/user/.duply/test'.
Using installed duplicity version 0.6.23, python 2.7.6, gpg 1.4.16 (Home: ~/.gnupg), awk 'GNU Awk 4.0.1', bash '4.3.11(1)-release (i686-pc-linux-gnu)'.
Signing disabled. Not GPG_KEY entries in config.
Test - Encryption with passphrase (OK)
Test - Decryption with passphrase (OK)
Test - Compare (OK)
Cleanup - Delete '/tmp/duply.17999.1404233388_*'(OK)

--- Start running command PRE at 11:49:49.054 ---
Skipping n/a script '/home/jmreicha/.duply/test/pre'.
--- Finished state OK at 11:49:49.077 - Runtime 00:00:00.022 ---

--- Start running command BKP at 11:49:49.098 ---
Command line error: Two URLs specified.  One argument should be a path.
Enter 'duplicity --help' for help screen.
11:49:49.256 Task 'BKP' failed with exit code '2'.
--- Finished state FAILED 'code 2' at 11:49:49.256 - Runtime 00:00:00.157 ---

--- Start running command POST at 11:49:49.285 ---
Skipping n/a script '/home/jmreicha/.duply/test/post'.
--- Finished state OK at 11:49:49.308 - Runtime 00:00:00.022 ---

As you can see duply/duplicity is expecting a local file path as the source. I can produce configs if it is helpful in diagnosing the issue.

Is there any way to get duply/duplicity to run a backup agains one S3 bucket as a source to another S3 target? If not, what tools are capable of syncing two S3 buckets?

Improve this question
4
  • Take a look at s3tools.org/s3cmd.. You can do something like s3cmd --recursive cp s3://mybucket s3://mybucket_backup PS: I haven't tried this myself
    – vijay rajah
    Jul 1, 2014 at 17:27
  • Are you doing this for backup purposes? If so, why not just turn on versioning for this bucket?
    – EEAA
    Jul 1, 2014 at 17:31
  • 1
    @EEAA eh, it's complicated. The thought is to mirror one S3 bucket to another, then have the mirrored bucket be versioned and have it fall off into glacier for long term backups. We basically just want to have a fallback option.
    – jmreicha
    Jul 1, 2014 at 17:34
  • @vijayrajah ok I will take a look at s3cmd. Thanks for the suggestion.
    – jmreicha
    Jul 1, 2014 at 17:35

1 Answer 1

Reset to default
1

If anybody runs across this issue I found a workaround.

First, if you are on Windows and need to copy one bucket to another you can use s3cmd Express. I spoke with one of the developers of s3cmd and he mentioned that there are issues with the Linux/OSX version of s3cmd for huge buckets, so he told me to use Express if at all possible.

Unfortanely I only have a Linux box, so I ended up using aws cli. This is pretty similar to s3cmd as both of them rely on the Python boto library. The command to syncrhonize the bucket is:

aws s3 sync s3://mybucket s3://backup-mybucket

You can append the --dryrun flag to test your command first to make sure it does what you want to.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.