This is for questions on Amazon's cloud storage service.

learn more… | top users | synonyms (1)

5
votes
1answer
307 views

Cant connect to S3 with nginx

im trying to serve static and media files from amazon S3 bucket but nginx cant connect to it Here is the error i get <Error> <Code>AccessDenied</Code> <Message> AWS ...
5
votes
1answer
555 views

Amazon S3 appears to be blocking cloudflare IP addresses. How do I fix it?

I have static content that is being served by Cloudflare. Cloudflare points to Amazon S3 to pick up the static content and serves it via a CNAME (cdn.mydomain.com). The bucket is setup properly and ...
2
votes
1answer
66 views

What's the fastest way of getting data into an AWS Lambda?

I have something I'll call a "microservice" running on AWS Lambda (using node.js). Basically it serves up condensed summaries drawn from a few hundred megabytes of binary blob. There are a lot of ...
2
votes
1answer
520 views

Postgresql WAL-E - Replication falling behind + can't catch up

Does anyone know how I can have my replications process the WAL segments faster while using WAL-E? https://github.com/wal-e/wal-e I have one master that leverages WAL-E to upload to S3, and I have 6 ...
2
votes
1answer
728 views

How can I whitelist oubound-from-private-subnet traffic to S3 on the NAT instance of an EC2 VPC?

I'm researching privacy-compliant high-security deployment options for AWS VPC. I'm looking for a method of tightening outbound traffic from the private subnets at the NAT instance. For example, how ...
2
votes
1answer
830 views

How backup files with logrotate and Amazon S3?

I would like to create a new standard of two things: how long would be generated and be rotated the logs generated by my applications ? how to transfer the logs to Amazon S3, as a backup server ?. ...
2
votes
1answer
820 views

Amazon S3 Website & MySQL Backup

I have a website (a digital asset management system/gallery - http://www.resourcespace.org) that has a huge amount of images. The total size of the website, including the images is approximately 6gb. ...
2
votes
1answer
602 views

How to access an Eucalyptus Walrus(S3) using Amazon AWS SDK

I have downloaded the Amazon AWS SDK for C#, I have no problem accessing the EC2 part of our private cloud running Eucalyptus, I can list, Images, Instances, Zones ... This is working fine : ...
1
vote
1answer
32 views

s3cmd error on sync: S3 error: 400 (InvalidRequest)

I'm trying to configure s3cmd to backup some directories on my server to my AWS S3 bucket. My problem is that when I do the sync command: s3cmd sync -r -p --no-delete-removed /path/to/dir s3://...
1
vote
1answer
200 views

Nginx reverse proxy causing file downloads to take excessive time to start

Overview My web app allows users to upload files which are stored on s3 via my servers. When a user requests a file my web server retrieves it from s3 then sends it on to the client. I recently ...
1
vote
1answer
66 views

How to parse s3 logs?

I've been trying to parse AWS S3 logs following the documentation, but I've been running into some problems. Specifically, I keep running into new, rare log lines that break my regex. Every time this ...
1
vote
1answer
128 views

ERROR — : Response was: 307 — Temporary Redirect when trying to upload to bucket

I get 307 when uploading to a bucket with Amazon's Ruby gem (2.10.2). Another bucket works fine (both were created today). I checked the name and I didn't misspell it. What is causing the error?
1
vote
1answer
120 views

Route 53 configuration for Cloudfront distribution linked to S3 bucket for naked domain

I am using route 53 for a domain (example.com) linked to my app on Heroku. To handle naked domain redirection, I am using a S3 bucket as explained here: https://aws.amazon.com/blogs/aws/root-domain-...
1
vote
1answer
191 views

Why is AWS S3 Redirect Showing as 302 instead of 301?

I'm implementing a 301 redirect in S3 using this blog post from when the functionality was first announced. It explicitly says the redirect will be a 301, which is what I want. Screenshot of the ...
1
vote
1answer
138 views

AWS S3 bucket permissions

I've created AWS s3 buckets with several thousand objects, initially the access policy was restricted to authenticated users, but later due to some requirement we need to make all objects in this ...
1
vote
1answer
367 views

jboss cluster in ec2 with s3ping

Am trying to get s3ping discovery method working in wildfly 8.0.0. I am using default Amazon Linux AMI and using chef to launch instances in ec2 based on this tutorial http://aws.typepad.com/...
1
vote
1answer
375 views

Where are AMIs stored?

It seems like an very simple question, but yet I could not find the answer. When I make an instance on Amazon AWS and then make some changes and create an AMI where is that AMI stored? S3? Will making ...
1
vote
1answer
152 views

Nginx proxy cache invalidating request to s3

I have set up a proxy from my application server to a private s3 bucket to cache requests. I was having some trouble with it where s3 was rejecting my download requests (403 forbidden) and after some ...
1
vote
1answer
983 views

Cloud-front backed with Nginx (which proxies to S3) randomly missing already cached items?

I wish to serve images from a S3 bucket with Cloudfront as CDN frontend, for that I tried the following: What I wish to acheive (Attempt 2) -- (Misses cloudfront cache randomly) I have the ...
1
vote
1answer
1k views

Move files from ftp server to s3

I would like to set up an ftp server, where users will upload files, and for each file, put it on s3 storage, and delete it from the ftp server. (the server runs on ec2 ubuntu) Here are the stuff I ...
1
vote
1answer
721 views

Debugging logrotate postrotate script

Following is my logrotate conf. /mnt/je/logs/apache/jesites/web/*.log" { missingok rotate 0 size 5M copytruncate notifempty sharedscripts ...
1
vote
1answer
1k views

Using TntDrive to mount Amazon S3 on a Windows Amazon EC2 instance

I currently have 2 Amazon EC2 Windows Server 2008 instances setup with a load balancer. My problem now is that I need both instances to be able to access a single file system source (I'm trying to use ...
1
vote
1answer
910 views

Protected bucket @ S3 - Access Denied

I'm trying to grant RW access to a given bucket to a specific user using the following bucket policy: { "Id": "Policy1322043790167", "Statement": [ { "Sid": "Stmt9999043784080", "...
1
vote
1answer
411 views

Reducing latency for different geographic regions on Amazon Cloud

I have got an application which has three components Application code : Amazon EC2 US-EAST-1 instance Application images, and other static data : Amazon S3 with CloudFront Application Database : ...
0
votes
1answer
11 views

Logstash S3 input plugin re-scanning all bucket objects

I am using the Logstash S3 Input plugin to process S3 access logs. The access logs are all stored in a single bucket, and there are thousands of them. I have set up the plugin to only include S3 ...
0
votes
1answer
28 views

How to use one website with several amazon S3 buckets as a path suffixes?

I would like to host multiple S3 Amazon buckets under one site in next way: Access to bucket1 - http://example.com/bucket1(bucket2...) and so on. What now Amazon allows is access with links like http:/...
0
votes
1answer
21 views

restore backup files in another server using duplicity

I am using duplicity to backup the application files with gpg key to Amazon service S3 server. And then restore the same from S3 to own server. I want to restore the backed up files to another ...
0
votes
1answer
29 views

s3cmd restore recursive not working right

I am running the following command to restore the data from Amazon S3 to the web server. s3cmd restore --recursive s3://my-bucket/Weekly-Sunday/dbexports/ and getting this error: Restore is not ...
0
votes
1answer
63 views

Access s3 bucket from Ec2 service

I'm new to AWS and was looking for a bucket policy to enable my ec2 servers access to the S3 buckets on the same account. I tried using this policy which enables the account: { "Sid": "...
0
votes
1answer
101 views

What is the benefit of VPC Endpoints on AWS?

I learned a long time ago that if you want an EC2 instance to be able to talk to AWS services like SQS, Kinesis and the like, the instance has to either have a public IP address or there needs to be ...
0
votes
1answer
53 views

Cannot copy files from EC2 in VPC to S3 with EC2 role

From an EC2 instance running within a VPC I am in need of accessing an S3 bucket with awscli for copying files from EC2 to S3. I succesfully created an endpoint with an attached custom policy built ...
0
votes
1answer
1k views

Elastic Beanstalk S3 access via .ebextensions

I have a simple file in my .ebextensions folder: 00-myconfig.config Resources: AWSEBAutoScalingGroup: Metadata: AWS::CloudFormation::Authentication: S3Access: ...
0
votes
1answer
279 views

Can I use an IAM role to grant my Heroku app access to my Amazon S3 bucket?

Heroku: Using AWS S3 to Store Static Assets and File Uploads suggests using my AWS security credentials to enable my Heroku app to access my Amazon S3 bucket. However, isn't it better practice (as ...
0
votes
1answer
86 views

Duplicity GPG key not available

I am using the following tutorial to get Duplicity to back up my mysql dumps to Amazon S3. http://kappataumu.com/articles/cloud-backups-duplicity-s3.html However I am getting an issue with the ...
0
votes
1answer
881 views

Setting cache for already uploaded amazon s3 files

In amazon s3 for our site through s3cmd tool we uploaded around 60GB of files including mp3, jpg to a bucket. But now when we checked site slowness files are not caching and further checking we came ...
0
votes
1answer
820 views

AWSCLI s3 sync doesn't create directory

Hi try to sync a local directory to a S3 bucket. The files in the directory get synced perfectly, just the directory isn't created at all. Can you guys give me a hint how to do it? "newdir" contains ...
0
votes
1answer
327 views

Duplicity Backups (via Duply) to S3 - high server load

We have been doing incremental backups using Duply on our main server, to an S3 bucket. However, we've found that there is a high server load during the backups (its an Amazon EC2 server). We're ...
0
votes
1answer
1k views

s3fs “could not determine how to establish security credentials”

I'm trying to use s3fs to mount an S3 bucket on to a standard AWS Amazon Linux AMI (with all the necessary dependencies installed). However when following this tutorial when I run: s3fs mybucketname -...
0
votes
1answer
361 views

rebuild yum index on aws s3

I am trying to rebuild yum repo on aws S3 after adding new packages. Here are few commands I am trying, but it is not helping. [root@chucks ~]$ createrepo --baseurl http://rpmcopy.xxxxx.com.s3-...
0
votes
1answer
105 views

Glacier retrieval notification of s3 objects

I have some s3 buckets which send their objects to glacier after x days of creation. When I go to retrieve those objects on the s3 interface, I need to keep reloading page to see when restoration has ...
0
votes
1answer
269 views

AWS route53 forwarding

Essentially, I need to know how to point one URL/folder to a web server and a different URL/folder to an EC2 app server. I'm trying to setup a SAAS with the following URL configuration. www.mysite....
0
votes
0answers
70 views

Amazon S3 - Upfront File Organization Architectural Decisions

I'm currently building a web application for a publishing platform, and I am using S3 for content storage. S3 is super simple to use, but I'd like to try to make the best possible file organization up-...
0
votes
0answers
42 views

How to use the s3cmd setlifecycle command?

s3cmd has a setlifecycle subcommand that seems like it ought to let me configure a bucket into Amazon Glacier. However I can't find any documentation for it on the web or in the manpage. What's the ...
0
votes
0answers
47 views

aws hosted website not serving web pages to internal network and serving from s3 bucket instead

Sorry about more text to describe..(The issue has been resolved but the causes are un-known) We Have one website hosted via AWS Load Balancer and was working from last year perfectly. Recently an ...
0
votes
0answers
42 views

How to securely allow web apps to read/write to S3? (the new way with IAM)

I've been banging my head against a wall on this for two hours. It's frustrating that Amazon has made this so complicated when it doesn't need to be. Let's say I have a bucket called "test," and a ...
0
votes
0answers
132 views

DNS Based Load balancing on squid - odd issues

I'm attempting to do round-robin DNS based load balancing for two squid boxes running as simple HTTP(S) proxies. For testing I'm using AWS Route 53. To simulate round-robin I created two records ...
0
votes
0answers
800 views

AWS security group egress rules for S3

I manually created a new security group using the AWS CLI. I created ingress rules that allow incoming connections only from my company's public IP address using the known ports for SSH (22) and ...
0
votes
0answers
94 views

NGINX: proxy_cache altering headers

I have been having trouble building a proxy between my server and s3 to cache requests. I finally boiled it down to a single point of failure... http{ proxy_cache_path /home/cache levels=1:2 ...
0
votes
0answers
1k views

s3cmd 403 Access Denied error on put but not ls

I can access s3 using s3cmd on an ec2 instance to do s3cmd ls s3://myc bucketname but not using 'put' to add a file. It gives a 403 access denied error. I am using IAM and have group setup to access ...
-3
votes
0answers
10 views

Will my NAS upload new and changed files or ALL files when doing remote backup

I am trying to do a cost breakdown of cloud storage services. I am wondering, how many PUT requests I am likely to use. I plan to do a daily backup of 620k files (3.5TB) from a QNAP NAS. Will the ...