Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

It used to be my favorite backup transport agent but now I frequently get this result from s3cmd on the very same Ubuntu server/network:

root@server:/home/backups# s3cmd put bkup.tgz s3://mybucket/
bkup.tgz -> s3://mybucket/bkup.tgz  [1 of 1]
      36864 of 2711541519     0% in    1s    20.95 kB/s  failed
WARNING: Upload failed: /bkup.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.00)
WARNING: Waiting 3 sec...
bkup.tgz -> s3://mybucket/bkup.tgz  [1 of 1]
      36864 of 2711541519     0% in    1s    23.96 kB/s  failed
WARNING: Upload failed: /bkup.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.01)
WARNING: Waiting 6 sec...
bkup.tgz -> s3://mybucket/bkup.tgz  [1 of 1]
      28672 of 2711541519     0% in    1s    18.71 kB/s  failed
WARNING: Upload failed: /bkup.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.05)
WARNING: Waiting 9 sec...
bkup.tgz -> s3://mybucket/bkup.tgz  [1 of 1]
      28672 of 2711541519     0% in    1s    18.86 kB/s  failed
WARNING: Upload failed: /bkup.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.25)
WARNING: Waiting 12 sec...
bkup.tgz -> s3://mybucket/bkup.tgz  [1 of 1]
      28672 of 2711541519     0% in    1s    15.79 kB/s  failed
WARNING: Upload failed: /bkup.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=1.25)
WARNING: Waiting 15 sec...
bkup.tgz -> s3://mybucket/bkup.tgz  [1 of 1]
      12288 of 2711541519     0% in    2s     4.78 kB/s  failed
ERROR: Upload of 'bkup.tgz' failed too many times. Skipping that file.

This happens even for files as small as 100MB, so I suppose it's not a size issue. It also happens when I use put with --acl-private flag (s3cmd version 1.0.1)

I appreciate if you suggest some solution or a lightweight alternative to s3cmd.

share|improve this question
    
Have you tried s3cp? It should do what you want - I believe it also has more verbose error messages. As for the error you are getting, some common causes seem to be a non-existent (e.g. mistyped bucket name), trailing spaces on your authentication values (key/id), or an inaccurate system clock –  cyberx86 Nov 13 '11 at 2:26
    
@cyberx86, I am a bit reluctant to switch to s3cp because I'd rather avoid JVM sitting on myserver just for file copy purpose. As you noticed, the bizarre things is that s3cmd used to work like a charm. –  alfish Nov 13 '11 at 4:23
    
Depending on the complexity of what you want to do, it might be fairly easy to use the Python boto library to upload your files to S3 - you should be able to find an example of it fairly easily. I believe there is even a project to expose some of the boto functions in a command line tool. If you want a Perl script, Tim Kay's aws is quite versatile and easy to use - and should do everything you want. –  cyberx86 Nov 13 '11 at 4:44
    
@cyberx86 the 'aws' not only worked great in putting a large file but also indicated the reason why s3cmd did not work. It gave a warning (sanity-check) that the system clock was about 1700s ahead. When I adjusted the clock, s3cmd started working as before. So I guess the the issue with s3cmd was that when the sending server and S3 host are on the same time zone, they server time should not be ahead. You many put this as an answer and I will accept it. Many thanks –  alfish Nov 13 '11 at 18:31
    
Slightly low brow response here but I had a backup script working (s3cmd) and untouched for over a year... went bung as above and after trying suggestion here discovered permissions had "magically" changed on my S3 bucket... weird. –  user149605 Dec 14 '12 at 6:18

5 Answers 5

up vote 3 down vote accepted

There are a few common problems that result in s3cmd returning the error you mention:

  • A non-existent (e.g. mistyped bucket name; or a bucket that hasn't yet been provisioned)
  • Trailing spaces on your authentication values (key/id)
  • An inaccurate system clock. It is possible to use Wireshark (over an http - not https connection) to see how your system clock lines up with S3's clock - they should match within a few seconds. Consider using NTP to sync your clock if this is an issue.

Alternatives to s3cmd:

  • s3cp - a Java based script that offers good functionality for transferring files to S3, and more verbose error messages than s3cmd
  • aws - a Perl based script, written by Tim Kay, that provides easy access to most AWS (including S3) functions, and is quite popular.

If you wish to write your own script, you can use the Python Boto library which has functions for performing most AWS operations and has many examples available online. There is a project which exposes some of the boto functions on the command line - although, a very small set of functions are currently available.

share|improve this answer
1  
Im my case the problem was due to the inaccurate system clock.Really appreciate your hints. –  alfish Nov 13 '11 at 20:01
    
Thanks. I had same problem. Clock looked right, and I could upload small files, so I don't think it was the other suggestions. However aws worked perfectly. –  Darren Cook Jun 29 '12 at 9:16

This helped in my case:

  1. do s3cmd ls on the bucket
  2. it printed a warning about a redirection
  3. replace the bucket_host in the .s3cfg file with the one from the warning.
  4. repeat s3cmd ls, it should no longer print a warning
  5. reupload file

my .s3cfg now is:

host_bucket = %(bucket)s.s3-external-3.amazonaws.com
share|improve this answer
    
I've replaced the host_bucket to my bucket hostname (like bucketname.s3-sa-east-1.amazonaws.com). –  Rafael Kassner May 14 '13 at 15:47

I had the same problem with the Ubuntu s3cmd command.

Downloading the latest stable version (1.0.1) solved it: http://sourceforge.net/projects/s3tools/files/s3cmd/

share|improve this answer
    
I followed instructions at s3tools.org/repositories#note-deb but it didn't help. (However s3cmd --version tells me 1.0.0, not 1.0.1, so that minor version number might be important; this is on ubuntu 11.04) –  Darren Cook Jun 29 '12 at 9:21

After having tried all the things above, I noticed I'm still having the throttling issue using s3cmd put, but not using s3cmd sync instead. Hope this might be useful to somebody for a quick fix :)

share|improve this answer

I had the same problem and found a solution here in response by samwise.

This problem appeared when I started experiments with IAM. In my case the problem was in ARN. I listed arn:aws:s3:::bucketname instead of arn:aws:s3:::bucketname/*

That's why I had no problems with $ s3cmd ls s://bucketname, but could not upload any file there((

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.