1

I'm trying to configure my Duplicity installation to backup to an Amazon S3 bucket. I keep getting a BackendException error with no further details.

EDIT: I've removed my Duply configuration from this question in an effort to isolate the issue more and switched to a plain duplicity command on a small directory (10 MB, 34 files).

Here's the command I'm trying to run:

duplicity full ./logs "s3://s3-us-east-1.amazonaws.com/bucketname" -v9

And here's the output:

root@ats:/var/ats# duplicity full ./logs "s3://s3-us-east-1.amazonaws.com/bucketname" -v9

Duplicity 0.6 series is being deprecated:
See http://www.nongnu.org/duplicity/

Using archive dir: /root/.cache/duplicity/876c7d0b54276e675d41f6ea6077d52f
Using backup name: 876c7d0b54276e675d41f6ea6077d52f
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.cfbackend Succeeded
Import of duplicity.backends.dpbxbackend Succeeded
Import of duplicity.backends.ftpbackend Succeeded
Import of duplicity.backends.ftpsbackend Succeeded
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.megabackend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.sshbackend Succeeded
Import of duplicity.backends.swiftbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Import of duplicity.backends.~par2wrapperbackend Succeeded
Using temporary directory /tmp/duplicity-sQ3sGs-tempdir
Backend error detail: Traceback (most recent call last):
  File "/usr/local/bin/duplicity", line 1509, in <module>
    with_tempdir(main)
  File "/usr/local/bin/duplicity", line 1503, in with_tempdir
    fn()
  File "/usr/local/bin/duplicity", line 1336, in main
    action = commandline.ProcessCommandLine(sys.argv[1:])
  File "/usr/local/lib/python2.7/dist-packages/duplicity/commandline.py", line 1062, in ProcessCommandLine
    backup, local_pathname = set_backend(args[0], args[1])
  File "/usr/local/lib/python2.7/dist-packages/duplicity/commandline.py", line 955, in set_backend
    globals.backend = backend.get_backend(bend)
  File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py", line 163, in get_backend
    return _backends[pu.scheme](pu)
  File "/usr/local/lib/python2.7/dist-packages/duplicity/backends/_boto_single.py", line 163, in __init__
    self.resetConnection()
  File "/usr/local/lib/python2.7/dist-packages/duplicity/backends/_boto_single.py", line 189, in resetConnection
    raise BackendException(err.message)
BackendException

BackendException:

I've tested Boto (which is what Duplicity uses for S3 connections) with this script:

root@ats:/var/ats# python
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto.s3
>>> conn = boto.s3.connect_to_region("us-east-1")
>>> bucket = conn.get_bucket("bucketname")
>>> for key in bucket.list(): print key, key.storage_class
...
<Key: bucketname,test.txt> STANDARD
>>> exit()

I've made sure that I'm running the latest appropriate versions of duplicity and python, and I've set the AWS_* variables which work fine in Boto.

Am I missing something? What should I do?

8
  • Run duplicity with -v9 for extra verbosity.
    – ceejayoz
    Jul 8 '15 at 20:13
  • @ceejayoz I did "And here's the log I'm getting from Duplicity with the -v9 flag:", unfortunately that is the extra verbosity.
    – A.M.K
    Jul 8 '15 at 20:24
  • My apologies, I missed that in the question. :-(
    – ceejayoz
    Jul 8 '15 at 20:25
  • No problem. Thanks for taking a look, I've been stuck on this one a little while.
    – A.M.K
    Jul 8 '15 at 20:25
  • smells like the "file too big for s3" issue <br> answers.launchpad.net/duplicity/+question/253609 ..ede/duply.net
    – user294647
    Jul 9 '15 at 9:46
1

I don't know why the original command wasn't working, but in the end I figured out that changing the URL format from s3://s3-us-east-1.amazonaws.com/bucketname to s3+http://bucketname fixes the problem.

The reduced working command is now:

duplicity full ./logs "s3+http://bucketname" -v9
1
  • My issue was that I had created a new IAM user but forgot to give them S3 access :P
    – Benno
    Dec 5 '15 at 6:49
1

If you still can't get it to work, try adding: export S3_USE_SIGV4="True" to your duply profile conf file.

This is described in detail here and worked for me.

1
  • I tried that when I was first having problems and it didn't help. Thanks for the suggestion however!
    – A.M.K
    Jul 13 '15 at 22:23
0

Neither the SIGV4 nor the s3+http solutions worked for me.

My solution was to use the region name in the URL, instead of the generic 's3'. In my case, this was eu-west-1 (Ireland).

This looks like:

s3://s3-eu-west-1.amazonaws.com/bucketname

Instead of:

s3://s3.amazonaws.com/bucketname

Even though the second generic URL works in other tools such as s3cmd, it wasn't working with Duplicity / Boto.

You may need to combine this solution with the answer by Joe Hudson if you're using a new region such as Frankfurt or Singapore.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.