This is for questions on Amazon's cloud storage service.

learn more… | top users | synonyms (1)

5
votes
1answer
112 views

Cant connect to S3 with nginx

im trying to serve static and media files from amazon S3 bucket but nginx cant connect to it Here is the error i get <Error> <Code>AccessDenied</Code> <Message> AWS ...
4
votes
1answer
499 views

Amazon S3 appears to be blocking cloudflare IP addresses. How do I fix it?

I have static content that is being served by Cloudflare. Cloudflare points to Amazon S3 to pick up the static content and serves it via a CNAME (cdn.mydomain.com). The bucket is setup properly and ...
3
votes
0answers
117 views

Set up a different index document per directory with S3 static website hosting

I am using S3 static website hosting, and trying to get the following redirects to work: example.com --> example.com/index.html example.com/foo --> example.com/foo/foo.html example.com/bar ...
3
votes
0answers
48 views

S3 + Cloudflare vs S3 + 2 Web Server + Cloudflate

Currently, I run two websites and each site own it own big server. Currently, all of those two sites has roughly 300GB of images. Its grow rate is 15GB of images per month. I know that eventually, one ...
2
votes
0answers
104 views

Solution for file data backup, versioning and replication

We have a growing set of data files (.wav files, image files etc) which are data, i.e not part of the application code - uploaded and modified by users. The number of files is in the 1000s and the ...
2
votes
0answers
309 views

Strict-Transport-Security on CloudFront with S3 origin?

My company's site has a static homepage for speed and cost reasons. We use S3 as the origin for CloudFront. Now, we would like to declare Strict-Transport-Security for the entire domain, but S3 seems ...
2
votes
0answers
2k views

How can I move s3 objects >5gb from one bucket to another?

Objects larger than 5GB are supported with multi-part uploading. We have 5gb+ files in one bucket already that I'd like to move to another, under the same AWS account. When I issue the command using ...
2
votes
0answers
233 views

Gluster with s3ql mounted brick?

I am trying to test run Gluster with a volume consisting of two local bricks, the first being a plain directory and the second being a s3ql based mount. Having just the plain directory as a unique ...
2
votes
0answers
472 views

Web-based file manager like File Thingie, but for S3?

At work, we use a little PHP script called File Thingie, which is a web-based file manager. Basically, an FTP replacement. It allows users to upload and download files, and most importantly, it has a ...
1
vote
0answers
54 views

How to upload a large file using aws commandline when connection may be unreliable?

I've been having trouble uploading a large (800MB) file to s3, using the aws commandline tool. The first attempt completed (after many hours) but was not visible, and I was advised (here) that it has ...
1
vote
0answers
64 views

aws s3 sync: how to exclude proc, sys directories

I can't seem to get the aws sync s3 command to exclude directories. I am backing up a mounted filesystem from OpenVZ which is located at /vz/root/100. My command is: aws s3 sync /vz/root/100 ...
1
vote
0answers
340 views

AWS CLI throws “Unable to locate credentials”, the second time it's run

I'm trying to download some files from S3 to an EC2 instance using the AWS CLI (1.8.7) on startup using a User-Data Script. This instance has a IAM Role & Instance Profile with the appropriate ...
1
vote
0answers
543 views

Nginx as s3 proxy with private buckets

I'm trying to set up Nginx as a proxy for a static html/js webapp hosted on S3. I read a lots of tutorials and posts and I could make it work if my bucket is set as public. The reason I need Nginx as ...
1
vote
0answers
557 views

s3cmd sync hangs forever. Never finishes

I am trying to backup a directory of around 350GB in size to S3. The command I run is s3cmd --progress -v -H sync /localdir s3://mybucket/ However, the process never moves past INFO: Compiling ...
1
vote
0answers
54 views

svnadmin dump returns error insufficient disk space

I have a SVN repo that is around 35GB after dumping. I know this is huge, but our business requires that all revisions of all files and libraries be tracked. I have Apache Subversion hosted on an ...
1
vote
0answers
220 views

Duplicity -> S3 backup failing with “[Errno 105] No buffer space available”

I have a nightly script on my Ubuntu 10.04 VPS which uses duplicity (0.6.24) to run incremental, encrypted backups to Amazon S3. This script had been working until a month or so ago when it started ...
1
vote
0answers
557 views

Amazon S3: allow user access to subfolder only on any bucket

I'm trying to write a policy that will allow a user to read from any bucket (that part works) but only write to a specific subfolder on any bucket. This is my policy so far (reading works, but it ...
1
vote
0answers
132 views

Handling multiple server blocks in nginx configuration

I am working on setting up a proxy server for my application.The use case is I want a specific set of requests to be redirect to say abc.com and rest of the request to be redirected to xyz.com. Eg:If ...
1
vote
0answers
236 views

Amazon S3 sync theory

I'm setting up a backup of my vital data (database and images, mainly) to S3 from my Ubuntu box. I've set up Amazon S3, and installed S3FS to mount the bucket on my machine. This is working well, but ...
1
vote
0answers
330 views

s3fs-mounted S3 bucket shows bad filesizes

Background: I have an Amazon S3 bucket with data uploaded using the AWS web interface. Now I need to give permissions-based access to paths within my buckets to FTP users. To do this, I'm running ...
0
votes
0answers
3 views

Is there any way to restore an entire previous version of a folder in an s3 bucket?

I know folders are not folders within S3, but it would be extremely useful if you could restore a previous version of one when you have versioning enabled. I've been looking at it and I can only see ...
0
votes
0answers
28 views

How to add AWS ELB access logs to logstash with S3 input?

I'm using an ELK stack for server monitoring. My application's access logs which are from AWS ELB are stored in AWS S3. I am trying to add them to logstash with the following input: input { s3 { ...
0
votes
0answers
17 views

Are blocked requests by S3 CORS billable?

I'm getting a large amount of POST requests to my S3 bucket. Ideally I'd like to block them all, so I've been looking into CORS. If a request is being blocked by the CORS policy, do I still pay for ...
0
votes
0answers
19 views

Is there a Policy template standard for secure S3 access?

I've created a policy below which restricts SSL access, and requires SSE. Is there a 'standard' policy template recommended by long time AWS S3 devs for locking down S3 buckets that is commonly ...
0
votes
0answers
14 views

How to make amazon s3 files fast from anywhere in the planet?

I'm working on a project with user created content, users basically upload long mp3 files to the server which can then be played by other users. Recently users not from europe reported slow "seek ...
0
votes
0answers
18 views

Force hadoop to use for S3 connection

I'm trying to upload file to S3 using hadoop: hadoop fs -Dfs.s3a.connection.ssl.enabled=false -Dfs.s3a.proxy.host=127.0.0.1 -Dfs.s3a.proxy.port=8123 -put pig_1421167148680.log ...
0
votes
0answers
21 views

How to get nginx host variable to work with proxy_pass

I am trying to setup an nginx proxy in front of any s3 website that I point at it. In the example below I have my DNS records customsite.com pointing to my proxy. When I don't use any variable for the ...
0
votes
0answers
19 views

Jenkins aws elastic deployment plugin just create a new ec2 on each update

Here is something strange happening when I tries to deploy from Jenkins to Elastic Beanstalk , you can check my previous question for more clarity. ...
0
votes
0answers
35 views

S3FS and Fuse, scale out or scale up

I have an infra where I have multiple machines (eg FTP servers, get/put and Samba/NFS servers) that need to share out the same (HA) storage. So I figured to use s3fs and fuse. I got this all setup and ...
0
votes
0answers
42 views

ERROR — : Response was: 307 — Temporary Redirect when trying to upload to bucket

I get 307 when uploading to a bucket with Amazon's Ruby gem (2.10.2). Another bucket works fine (both were created today). I checked the name and I didn't misspell it. What is causing the error?
0
votes
0answers
46 views

How to use Amazon S3 Ceph Interface on Windows

I have 4 ceph nodes running on virtual machines in our corporate network. I've installed a ceph object gateway on admin node. Can I somehow use it in Amazon S3 style from my windows machine in the ...
0
votes
0answers
73 views

Amazon S3 upload error: An exception occurred while uploading parts to a multipart upload

I am trying to upload a 30 GB file to Amazon S3 using the AWS PHP SDK. require('../vendor/autoload.php'); $client = new Aws\S3\S3Client([ 'version' => 'latest', 'region' => 'us-east-1' ]); ...
0
votes
0answers
47 views

How Do I Mask the CDN Domain with My Own Domain Name?

So i've been trying out cloudfront from AWS. Why is the URL 'example.com' resolving to 'example.com.s3.eu-west-1.amazonaws.com' in the Address Bar? I've tried googling loads of questions with this ...
0
votes
0answers
69 views

looking for index.php while proxy_pass to s3 bucket

I am configuring my nginx proxy server to fetch data from s3 bucket in case of 502 error. This is my configuration file. error_page 502 @static; location @static{ rewrite ^/a/b /a_b break; ...
0
votes
0answers
111 views

CloudFront + S3 Access Denied Using Signed Cookies

I'm trying to integrate signed cookies for a CloudFront distribution that's fed off of an S3 bucket but getting access denied error messages: <Error> <Code>AccessDenied</Code> ...
0
votes
0answers
133 views

Access denied for the request of data from s3 bucket on high load

I am trying to fetch static files from s3 bucket in case of 502 error. This is how I have configured nginx for requests : error_page 502 = @static; location / { index index.php; try_files $uri ...
0
votes
0answers
65 views

Amazon S3 - Upfront File Organization Architectural Decisions

I'm currently building a web application for a publishing platform, and I am using S3 for content storage. S3 is super simple to use, but I'd like to try to make the best possible file organization ...
0
votes
0answers
26 views

How to use the s3cmd setlifecycle command?

s3cmd has a setlifecycle subcommand that seems like it ought to let me configure a bucket into Amazon Glacier. However I can't find any documentation for it on the web or in the manpage. What's the ...
0
votes
0answers
30 views

aws hosted website not serving web pages to internal network and serving from s3 bucket instead

Sorry about more text to describe..(The issue has been resolved but the causes are un-known) We Have one website hosted via AWS Load Balancer and was working from last year perfectly. Recently an ...
0
votes
0answers
34 views

How to securely allow web apps to read/write to S3? (the new way with IAM)

I've been banging my head against a wall on this for two hours. It's frustrating that Amazon has made this so complicated when it doesn't need to be. Let's say I have a bucket called "test," and a ...
0
votes
0answers
99 views

DNS Based Load balancing on squid - odd issues

I'm attempting to do round-robin DNS based load balancing for two squid boxes running as simple HTTP(S) proxies. For testing I'm using AWS Route 53. To simulate round-robin I created two records ...
0
votes
0answers
440 views

AWS security group egress rules for S3

I manually created a new security group using the AWS CLI. I created ingress rules that allow incoming connections only from my company's public IP address using the known ports for SSH (22) and ...
0
votes
0answers
60 views

Duplicity GPG key not available

I am using the following tutorial to get Duplicity to back up my mysql dumps to Amazon S3. http://kappataumu.com/articles/cloud-backups-duplicity-s3.html However I am getting an issue with the ...
0
votes
0answers
46 views

Uploaded 0 object(s) to bucket from Write-S3Object command

When I upload a file with the Write-S3Object command and it seems work properly and in another tool I can see the file but I get the follow message: PS> Write-S3Object -BucketName my-bucket ...
0
votes
0answers
47 views

EC2 websites & RDS databases backup

We have some Amazon EC2 machines on which different websites are working, and their database is being pulled from RDS-MySQL, I know for machine I can have images as backup and for RDP-MySQL it has ...
0
votes
0answers
16 views

How can I improve the space usage of duplicate content in S3?

In my project I have duplicated files in various folders in a S3 bucket. Based on the output of s3cmd du, each time I make a copy of some content in S3, the usage increases. I wonder does s3 support ...
0
votes
0answers
25 views

Read data from external bucket on Amazon EMR

I need to read data from a bucket that I don't own, and I get the IAM account key and secrete access key to the bucket. What is the best way I can access the data? How can I add another IAM with ...
0
votes
0answers
86 views

NGINX: proxy_cache altering headers

I have been having trouble building a proxy between my server and s3 to cache requests. I finally boiled it down to a single point of failure... http{ proxy_cache_path /home/cache levels=1:2 ...
0
votes
0answers
1k views

s3cmd 403 Access Denied error on put but not ls

I can access s3 using s3cmd on an ec2 instance to do s3cmd ls s3://myc bucketname but not using 'put' to add a file. It gives a 403 access denied error. I am using IAM and have group setup to access ...
0
votes
0answers
716 views

s3cmd fails to put files

I'm trying to put mysql dump file on s3 bucket. It gives an error like ; sudo ./backup.sh WARNING: Module python-magic is not available. Guessing MIME types based on file extensions. ...