7

It used to be my favorite backup transport agent but now I frequently get this result from s3cmd on the very same Ubuntu server/network:

root@server:/home/backups# s3cmd put bkup.tgz s3://mybucket/
bkup.tgz -> s3://mybucket/bkup.tgz  [1 of 1]
      36864 of 2711541519     0% in    1s    20.95 kB/s  failed
WARNING: Upload failed: /bkup.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.00)
WARNING: Waiting 3 sec...
bkup.tgz -> s3://mybucket/bkup.tgz  [1 of 1]
      36864 of 2711541519     0% in    1s    23.96 kB/s  failed
WARNING: Upload failed: /bkup.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.01)
WARNING: Waiting 6 sec...
bkup.tgz -> s3://mybucket/bkup.tgz  [1 of 1]
      28672 of 2711541519     0% in    1s    18.71 kB/s  failed
WARNING: Upload failed: /bkup.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.05)
WARNING: Waiting 9 sec...
bkup.tgz -> s3://mybucket/bkup.tgz  [1 of 1]
      28672 of 2711541519     0% in    1s    18.86 kB/s  failed
WARNING: Upload failed: /bkup.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.25)
WARNING: Waiting 12 sec...
bkup.tgz -> s3://mybucket/bkup.tgz  [1 of 1]
      28672 of 2711541519     0% in    1s    15.79 kB/s  failed
WARNING: Upload failed: /bkup.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=1.25)
WARNING: Waiting 15 sec...
bkup.tgz -> s3://mybucket/bkup.tgz  [1 of 1]
      12288 of 2711541519     0% in    2s     4.78 kB/s  failed
ERROR: Upload of 'bkup.tgz' failed too many times. Skipping that file.

This happens even for files as small as 100MB, so I suppose it's not a size issue. It also happens when I use put with --acl-private flag (s3cmd version 1.0.1)

I appreciate if you suggest some solution or a lightweight alternative to s3cmd.

Improve this question
5
  • Have you tried s3cp? It should do what you want - I believe it also has more verbose error messages. As for the error you are getting, some common causes seem to be a non-existent (e.g. mistyped bucket name), trailing spaces on your authentication values (key/id), or an inaccurate system clock
    – cyberx86
    Nov 13, 2011 at 2:26
  • @cyberx86, I am a bit reluctant to switch to s3cp because I'd rather avoid JVM sitting on myserver just for file copy purpose. As you noticed, the bizarre things is that s3cmd used to work like a charm.
    – alfish
    Nov 13, 2011 at 4:23
  • Depending on the complexity of what you want to do, it might be fairly easy to use the Python boto library to upload your files to S3 - you should be able to find an example of it fairly easily. I believe there is even a project to expose some of the boto functions in a command line tool. If you want a Perl script, Tim Kay's aws is quite versatile and easy to use - and should do everything you want.
    – cyberx86
    Nov 13, 2011 at 4:44
  • @cyberx86 the 'aws' not only worked great in putting a large file but also indicated the reason why s3cmd did not work. It gave a warning (sanity-check) that the system clock was about 1700s ahead. When I adjusted the clock, s3cmd started working as before. So I guess the the issue with s3cmd was that when the sending server and S3 host are on the same time zone, they server time should not be ahead. You many put this as an answer and I will accept it. Many thanks
    – alfish
    Nov 13, 2011 at 18:31
  • Slightly low brow response here but I had a backup script working (s3cmd) and untouched for over a year... went bung as above and after trying suggestion here discovered permissions had "magically" changed on my S3 bucket... weird.
    – user149605
    Dec 14, 2012 at 6:18

7 Answers 7

Reset to default
7

There are a few common problems that result in s3cmd returning the error you mention:

  • A non-existent (e.g. mistyped bucket name; or a bucket that hasn't yet been provisioned)
  • Trailing spaces on your authentication values (key/id)
  • An inaccurate system clock. It is possible to use Wireshark (over an http - not https connection) to see how your system clock lines up with S3's clock - they should match within a few seconds. Consider using NTP to sync your clock if this is an issue.

Alternatives to s3cmd:

  • s3cp - a Java based script that offers good functionality for transferring files to S3, and more verbose error messages than s3cmd
  • aws - a Perl based script, written by Tim Kay, that provides easy access to most AWS (including S3) functions, and is quite popular.

If you wish to write your own script, you can use the Python Boto library which has functions for performing most AWS operations and has many examples available online. There is a project which exposes some of the boto functions on the command line - although, a very small set of functions are currently available.

Improve this answer
2
  • 1
    Im my case the problem was due to the inaccurate system clock.Really appreciate your hints.
    – alfish
    Nov 13, 2011 at 20:01
  • Thanks. I had same problem. Clock looked right, and I could upload small files, so I don't think it was the other suggestions. However aws worked perfectly.
    – Darren Cook
    Jun 29, 2012 at 9:16
14

This helped in my case:

  1. do s3cmd ls on the bucket
  2. it printed a warning about a redirection
  3. replace the bucket_host in the .s3cfg file with the one from the warning.
  4. repeat s3cmd ls, it should no longer print a warning
  5. reupload file

my .s3cfg now is:

host_bucket = %(bucket)s.s3-external-3.amazonaws.com
Improve this answer
3
  • I've replaced the host_bucket to my bucket hostname (like bucketname.s3-sa-east-1.amazonaws.com).
    – Rafael Kassner
    May 14, 2013 at 15:47
  • We've had the same issue; lots of time trolling for answers; figured this out independently and it seems to work (fingers crossed).
    – user3546411
    Mar 9, 2016 at 2:55
  • We've found the --region arg is necessary in some cases as well. We're now using both the host_bucket config entry and the --region command line arg .. so far this seems to be working.
    – user3546411
    Mar 11, 2016 at 2:28
2

I had the same problem with the Ubuntu s3cmd command.

Downloading the latest stable version (1.0.1) solved it: http://sourceforge.net/projects/s3tools/files/s3cmd/

Improve this answer
1
  • I followed instructions at s3tools.org/repositories#note-deb but it didn't help. (However s3cmd --version tells me 1.0.0, not 1.0.1, so that minor version number might be important; this is on ubuntu 11.04)
    – Darren Cook
    Jun 29, 2012 at 9:21
1

After having tried all the things above, I noticed I'm still having the throttling issue using s3cmd put, but not using s3cmd sync instead. Hope this might be useful to somebody for a quick fix :)

Improve this answer
1

I had the same problem and found a solution here in response by samwise.

This problem appeared when I started experiments with IAM. In my case the problem was in ARN. I listed arn:aws:s3:::bucketname instead of arn:aws:s3:::bucketname/*

That's why I had no problems with $ s3cmd ls s://bucketname, but could not upload any file there((

Improve this answer
1

I had every second upload of multi-part upload with s3cmd sync fail with this error:

WARNING: Upload failed: /large-file.gz?partNumber=13&uploadId=FOOBAR ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.00)
WARNING: Waiting 3 sec...

The next upload would work great, but then one failed again, and so on.

I got it working with --limit-rate= option set to 4m so that uploads are throttled to at most 4MB/s.

So full setting is

s3cmd sync --limit-rate=4m ...
Improve this answer
0

This is also a commonly caused by HTTPS settings of your .s3cfg file.

Try changing the configuration parameter from "use_https = False" to "use_https = True" in the .s3cfg

Remember amazon buckets redirect to Https and hence all the retries. I see this issue quite a bit in the field.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.