This is for questions on Amazon's cloud storage service.

learn more… | top users | synonyms (1)

4
votes
1answer
435 views

Amazon S3 appears to be blocking cloudflare IP addresses. How do I fix it?

I have static content that is being served by Cloudflare. Cloudflare points to Amazon S3 to pick up the static content and serves it via a CNAME (cdn.mydomain.com). The bucket is setup properly and ...
3
votes
1answer
3k views

s3put fails with “Connection reset by peer” for large files

I can upload small files quite fine using the same commands, however whenever I try to upload a 5 GB file to S3 then the s3 script fails: [backup]$ s3put --bucket john-murdoch-bucket --reduced ...
2
votes
1answer
1k views

Amazon Cloudfront with S3 Redirect

I'm trying to do something that should be relatively straight-forward - I want to setup a few domains and subdomains to redirect to the core domain for our site, but I want to put the redirects out on ...
2
votes
1answer
380 views

Postgresql WAL-E - Replication falling behind + can't catch up

Does anyone know how I can have my replications process the WAL segments faster while using WAL-E? https://github.com/wal-e/wal-e I have one master that leverages WAL-E to upload to S3, and I have 6 ...
2
votes
1answer
489 views

How can I whitelist oubound-from-private-subnet traffic to S3 on the NAT instance of an EC2 VPC?

I'm researching privacy-compliant high-security deployment options for AWS VPC. I'm looking for a method of tightening outbound traffic from the private subnets at the NAT instance. For example, how ...
2
votes
1answer
612 views

How backup files with logrotate and Amazon S3?

I would like to create a new standard of two things: how long would be generated and be rotated the logs generated by my applications ? how to transfer the logs to Amazon S3, as a backup server ?. ...
3
votes
0answers
75 views

Set up a different index document per directory with S3 static website hosting

I am using S3 static website hosting, and trying to get the following redirects to work: example.com --> example.com/index.html example.com/foo --> example.com/foo/foo.html example.com/bar ...
3
votes
0answers
28 views

S3 + Cloudflare vs S3 + 2 Web Server + Cloudflate

Currently, I run two websites and each site own it own big server. Currently, all of those two sites has roughly 300GB of images. Its grow rate is 15GB of images per month. I know that eventually, one ...
2
votes
0answers
16 views

AWS: Why is the bucket size on CloudWatch higher than when using S3 tools?

I use the awscli tools to verify the size of my buckets. First the S3 call (all files without versioning, just an ordinary upload): $ aws s3 ls s3://XXX --recursive --human-readable --summarize ...
2
votes
0answers
79 views

Solution for file data backup, versioning and replication

We have a growing set of data files (.wav files, image files etc) which are data, i.e not part of the application code - uploaded and modified by users. The number of files is in the 1000s and the ...
2
votes
0answers
1k views

How can I move s3 objects >5gb from one bucket to another?

Objects larger than 5GB are supported with multi-part uploading. We have 5gb+ files in one bucket already that I'd like to move to another, under the same AWS account. When I issue the command using ...
2
votes
0answers
225 views

Gluster with s3ql mounted brick?

I am trying to test run Gluster with a volume consisting of two local bricks, the first being a plain directory and the second being a s3ql based mount. Having just the plain directory as a unique ...
2
votes
0answers
387 views

Web-based file manager like File Thingie, but for S3?

At work, we use a little PHP script called File Thingie, which is a web-based file manager. Basically, an FTP replacement. It allows users to upload and download files, and most importantly, it has a ...
1
vote
0answers
365 views

s3cmd sync hangs forever. Never finishes

I am trying to backup a directory of around 350GB in size to S3. The command I run is s3cmd --progress -v -H sync /localdir s3://mybucket/ However, the process never moves past INFO: Compiling ...
1
vote
0answers
53 views

svnadmin dump returns error insufficient disk space

I have a SVN repo that is around 35GB after dumping. I know this is huge, but our business requires that all revisions of all files and libraries be tracked. I have Apache Subversion hosted on an ...
1
vote
0answers
202 views

Duplicity -> S3 backup failing with “[Errno 105] No buffer space available”

I have a nightly script on my Ubuntu 10.04 VPS which uses duplicity (0.6.24) to run incremental, encrypted backups to Amazon S3. This script had been working until a month or so ago when it started ...
1
vote
0answers
426 views

Amazon S3: allow user access to subfolder only on any bucket

I'm trying to write a policy that will allow a user to read from any bucket (that part works) but only write to a specific subfolder on any bucket. This is my policy so far (reading works, but it ...
1
vote
0answers
112 views

Handling multiple server blocks in nginx configuration

I am working on setting up a proxy server for my application.The use case is I want a specific set of requests to be redirect to say abc.com and rest of the request to be redirected to xyz.com. Eg:If ...
1
vote
0answers
209 views

Amazon S3 sync theory

I'm setting up a backup of my vital data (database and images, mainly) to S3 from my Ubuntu box. I've set up Amazon S3, and installed S3FS to mount the bucket on my machine. This is working well, but ...
1
vote
0answers
312 views

s3fs-mounted S3 bucket shows bad filesizes

Background: I have an Amazon S3 bucket with data uploaded using the AWS web interface. Now I need to give permissions-based access to paths within my buckets to FTP users. To do this, I'm running ...
0
votes
0answers
10 views

looking for index.php while proxy_pass to s3 bucket

I am configuring my nginx proxy server to fetch data from s3 bucket in case of 502 error. This is my configuration file. error_page 502 @static; location @static{ rewrite ^/a/b /a_b break; ...
0
votes
0answers
7 views

CloudFront + S3 Access Denied Using Signed Cookies

I'm trying to integrate signed cookies for a CloudFront distribution that's fed off of an S3 bucket but getting access denied error messages: <Error> <Code>AccessDenied</Code> ...
0
votes
0answers
27 views

Access denied for the request of data from s3 bucket on high load

I am trying to fetch static files from s3 bucket in case of 502 error. This is how I have configured nginx for requests : error_page 502 = @static; location / { index index.php; try_files $uri ...
0
votes
0answers
9 views

Amazon S3 - Upfront File Organization Architectural Decisions

I'm currently building a web application for a publishing platform, and I am using S3 for content storage. S3 is super simple to use, but I'd like to try to make the best possible file organization ...
0
votes
0answers
24 views

logstash cloudfront codec plugin: Error: Object: #Version: 1.0 is not a legal argument to this wrapper, cause it doesn't respond to “read”

Logstash version 1.5.0.1 I am trying to use the logstash s3 input plugin to download cloudfront logs and the cloudfront codec plugin to filter the stream. I installed the cloudfront codec with ...
0
votes
0answers
11 views

How to use the s3cmd setlifecycle command?

s3cmd has a setlifecycle subcommand that seems like it ought to let me configure a bucket into Amazon Glacier. However I can't find any documentation for it on the web or in the manpage. What's the ...
0
votes
0answers
17 views

aws hosted website not serving web pages to internal network and serving from s3 bucket instead

Sorry about more text to describe..(The issue has been resolved but the causes are un-known) We Have one website hosted via AWS Load Balancer and was working from last year perfectly. Recently an ...
0
votes
0answers
20 views

How to securely allow web apps to read/write to S3? (the new way with IAM)

I've been banging my head against a wall on this for two hours. It's frustrating that Amazon has made this so complicated when it doesn't need to be. Let's say I have a bucket called "test," and a ...
0
votes
0answers
53 views

DNS Based Load balancing on squid - odd issues

I'm attempting to do round-robin DNS based load balancing for two squid boxes running as simple HTTP(S) proxies. For testing I'm using AWS Route 53. To simulate round-robin I created two records ...
0
votes
0answers
30 views

Duplicity for remote backup to AWS S3

I am using Bacula with S3 for backups. I keep bacula volumes for 60 days or so. Using duplicity as I would like to use GPG encryption of the volumes. Is there any way to do an rsync style sync using ...
0
votes
0answers
103 views

AWS security group egress rules for S3

I manually created a new security group using the AWS CLI. I created ingress rules that allow incoming connections only from my company's public IP address using the known ports for SSH (22) and ...
0
votes
0answers
23 views

Duplicity GPG key not available

I am using the following tutorial to get Duplicity to back up my mysql dumps to Amazon S3. http://kappataumu.com/articles/cloud-backups-duplicity-s3.html However I am getting an issue with the ...
0
votes
0answers
28 views

Uploaded 0 object(s) to bucket from Write-S3Object command

When I upload a file with the Write-S3Object command and it seems work properly and in another tool I can see the file but I get the follow message: PS> Write-S3Object -BucketName my-bucket ...
0
votes
0answers
37 views

EC2 websites & RDS databases backup

We have some Amazon EC2 machines on which different websites are working, and their database is being pulled from RDS-MySQL, I know for machine I can have images as backup and for RDP-MySQL it has ...
0
votes
0answers
235 views

Nginx as s3 proxy with private buckets

I'm trying to set up Nginx as a proxy for a static html/js webapp hosted on S3. I read a lots of tutorials and posts and I could make it work if my bucket is set as public. The reason I need Nginx as ...
0
votes
0answers
89 views

duplicity s3 backup from centos

I'm trying to backup server data to S3 but getting no success. Below is the script which I run to start backup on S3. I have create a user using IAM and have granted all permissions to the user. ...
0
votes
0answers
15 views

How can I improve the space usage of duplicate content in S3?

In my project I have duplicated files in various folders in a S3 bucket. Based on the output of s3cmd du, each time I make a copy of some content in S3, the usage increases. I wonder does s3 support ...
0
votes
0answers
15 views

Read data from external bucket on Amazon EMR

I need to read data from a bucket that I don't own, and I get the IAM account key and secrete access key to the bucket. What is the best way I can access the data? How can I add another IAM with ...
0
votes
0answers
84 views

AWS Cloud Front with S3, serving static files- Redirects implemented via java script at browser level

We have our Java application on tomcat server webpages- the application fetch content from multiple services and Database to create page and respond. We are checking for a best method to improve the ...
0
votes
0answers
143 views

Override content type headers when using Apache as a reverse proxy

I'm using Apache 2.2 as a reverse proxy in front of my s3 buckets. S3 serve the hosted files with wrong content-type headers. Is there a way to configure Apache to automatically re write the ...
0
votes
0answers
441 views

IAM user with full s3 access can't create directory

I've a AWS IAM user which has full access to Amazon S3 The policy is the following { "Statement": [ { "Effect": "Allow", "Action": "s3:ListAllMyBuckets", ...
0
votes
0answers
76 views

NGINX: proxy_cache altering headers

I have been having trouble building a proxy between my server and s3 to cache requests. I finally boiled it down to a single point of failure... http{ proxy_cache_path /home/cache levels=1:2 ...
0
votes
0answers
48 views

Benchmark network (internet) connection on server

I have a dedicated ubuntu server I rentet. Now I'm running a lot of ruby background processing on that server, that - among other things - does a lot of get requests to very small images (about 10kb) ...
0
votes
0answers
23 views

Include x-amz-storage-class Header When Calling Amazon S3 from Web.config

Any time my web app calls Amazon S3, it would be nice to conditionally include the following header. I am using a custom domain mapped to an S3 bucket. key: x-amz-storage-class value: ...
0
votes
0answers
159 views

Strict-Transport-Security on CloudFront with S3 origin?

My company's site has a static homepage for speed and cost reasons. We use S3 as the origin for CloudFront. Now, we would like to declare Strict-Transport-Security for the entire domain, but S3 seems ...
0
votes
0answers
56 views

Move query string to path in proxy - Apache

I am setting up Apache to proxy files from Amazon S3. The initial urls have query strings, but query strings can't be passed to S3, so they are stored as paths with question marks in them (which then ...
0
votes
0answers
72 views

Nginx serves response with 200 even after calling ngx.exit()

I am trying to serve static content from S3. I am using nginx and lua script for its configuration. I am new to both nginx and lua. The task that I want to achieve is : Get request URL into nginx. ...
0
votes
0answers
1k views

s3cmd 403 Access Denied error on put but not ls

I can access s3 using s3cmd on an ec2 instance to do s3cmd ls s3://myc bucketname but not using 'put' to add a file. It gives a 403 access denied error. I am using IAM and have group setup to access ...
0
votes
0answers
555 views

s3cmd fails to put files

I'm trying to put mysql dump file on s3 bucket. It gives an error like ; sudo ./backup.sh WARNING: Module python-magic is not available. Guessing MIME types based on file extensions. ...
-1
votes
0answers
4 views

Use s3-parallel-put on Windows

I need to move a large number of files from a Windows EC2 instance to S3. How can I use s3-parallel-put or something similar to do this? I have attempted to use s3-parallel-put already with errors.