This is for questions on Amazon's cloud storage service.

learn more… | top users | synonyms (1)

3
votes
2answers
267 views

S3 based file system capable of requesting only part of file

I'm storing large datasets in s3, but on a given computer in my cluster, my program only needs to read a small subset of the data. I first tried s3fs, but it downlooads the entire file first, which ...
1
vote
2answers
138 views

How to list s3fs mounts?

How can I list the s3fs mounts that exist on an ubuntu system? I'd like to know to which bucket each mount is mapped. Specifically, I have a specific mount (e.g. ~/s3/mymount), and would like to know ...
1
vote
2answers
347 views

How to deploy local project to Amazon

I have a small webapp written in Python/Django which works fine on my local machine. I've been tinkering and setting up my server on the free tier of Amazon EC2 by following online tutorials. However, ...
1
vote
2answers
1k views

s3fs Input/output error

I'm trying to set up a backup system with s3fs and the Amazon S3 service. I followed this this guide and this guide. Tailing /var/log/messages I get: Aug 28 13:37:46 server s3fs:###response=403 I ...
0
votes
2answers
149 views

Looking for a high IO s3 “compatible” storage solution

Help me serverfault! I am looking for a high IO (2 million mostly put transactions per day of 7-10k xml chunks) storage solution that is amazon s3 compatible that I can house in my datacenter. I am ...
2
votes
1answer
945 views

Best failover strategy for e-mail servers on AWS to ensure high availability

We have our e-mail server hosted on AWS. Last week Amazon had a failure in their East Coast region which brought down our server along with many others. We now want to implement a failover strategy ...
2
votes
1answer
668 views

Amazon S3 Website & MySQL Backup

I have a website (a digital asset management system/gallery - http://www.resourcespace.org) that has a huge amount of images. The total size of the website, including the images is approximately 6gb. ...
2
votes
1answer
553 views

How to access an Eucalyptus Walrus(S3) using Amazon AWS SDK

I have downloaded the Amazon AWS SDK for C#, I have no problem accessing the EC2 part of our private cloud running Eucalyptus, I can list, Images, Instances, Zones ... This is working fine : ...
1
vote
1answer
28 views

Why is AWS S3 Redirect Showing as 302 instead of 301?

I'm implementing a 301 redirect in S3 using this blog post from when the functionality was first announced. It explicitly says the redirect will be a 301, which is what I want. Screenshot of the ...
1
vote
1answer
82 views

AWS S3 bucket permissions

I've created AWS s3 buckets with several thousand objects, initially the access policy was restricted to authenticated users, but later due to some requirement we need to make all objects in this ...
1
vote
1answer
122 views

jboss cluster in ec2 with s3ping

Am trying to get s3ping discovery method working in wildfly 8.0.0. I am using default Amazon Linux AMI and using chef to launch instances in ec2 based on this tutorial ...
1
vote
1answer
764 views

HTTPS access to S3 bucket through Cloudflare

I'm trying to use Cloudflare to provide https access to an s3 bucket. I've setup the CNAME entry in Cloudfront with pointing to the bucket and I've verified I can access it via http but when I try and ...
1
vote
1answer
126 views

Nginx proxy cache invalidating request to s3

I have set up a proxy from my application server to a private s3 bucket to cache requests. I was having some trouble with it where s3 was rejecting my download requests (403 forbidden) and after some ...
1
vote
1answer
275 views

Backing up onto S3 - do I want to use bucket in a different region?

We are using Amanda to send backups to S3. Each host is acting as its own Amanda "server". All hosts are currently using the same S3-bucket -- each has its own subfolder under it. The dumps are saved ...
1
vote
1answer
742 views

Cloud-front backed with Nginx (which proxies to S3) randomly missing already cached items?

I wish to serve images from a S3 bucket with Cloudfront as CDN frontend, for that I tried the following: What I wish to acheive (Attempt 2) -- (Misses cloudfront cache randomly) I have the ...
1
vote
1answer
642 views

Debugging logrotate postrotate script

Following is my logrotate conf. /mnt/je/logs/apache/jesites/web/*.log" { missingok rotate 0 size 5M copytruncate notifempty sharedscripts ...
1
vote
1answer
348 views

Restore snapshot from other aws account?

I am new in this, so forgive my beginner question. I have an aws account which is currently in us-west-2c. It has a instance running, and also a couple of volumes which i made. I have multiple backups ...
1
vote
1answer
1k views

Using TntDrive to mount Amazon S3 on a Windows Amazon EC2 instance

I currently have 2 Amazon EC2 Windows Server 2008 instances setup with a load balancer. My problem now is that I need both instances to be able to access a single file system source (I'm trying to use ...
1
vote
1answer
755 views

Protected bucket @ S3 - Access Denied

I'm trying to grant RW access to a given bucket to a specific user using the following bucket policy: { "Id": "Policy1322043790167", "Statement": [ { "Sid": "Stmt9999043784080", ...
1
vote
1answer
376 views

Reducing latency for different geographic regions on Amazon Cloud

I have got an application which has three components Application code : Amazon EC2 US-EAST-1 instance Application images, and other static data : Amazon S3 with CloudFront Application Database : ...
0
votes
1answer
28 views

Elastic Beanstalk S3 access via .ebextensions

I have a simple file in my .ebextensions folder: 00-myconfig.config Resources: AWSEBAutoScalingGroup: Metadata: AWS::CloudFormation::Authentication: S3Access: ...
0
votes
1answer
67 views

How best way to rsync from aws s3 to gcloud?

I must execute a rsync ( not only copy or move ) from all my buckets from s3 to google cloud. I need create the same infraestructure of s3 to google. I have like 2Tera in s3, and all days I receive ...
0
votes
1answer
72 views

Can I use an IAM role to grant my Heroku app access to my Amazon S3 bucket?

Heroku: Using AWS S3 to Store Static Assets and File Uploads suggests using my AWS security credentials to enable my Heroku app to access my Amazon S3 bucket. However, isn't it better practice (as ...
0
votes
1answer
53 views

Using Duplicity with AWS S3 and IAM Folder policy

I'm trying to setup backups from multiple hosts to a single bucket. Each host has their own AWS login and credentials. The top most folder (prefix) maps to the AWS user id and I'm trying to limit ...
0
votes
1answer
41 views

AWS S3 Sync options duplicates / weird behaviour

I am trying to sync a folder with 2M+ files to S3, everything went OK but then 40,000 files were not uploaded (server crashed randomly), when I tried to do a sync command again, it started from 0, ...
0
votes
1answer
106 views

Where are AMIs stored?

It seems like an very simple question, but yet I could not find the answer. When I make an instance on Amazon AWS and then make some changes and create an AMI where is that AMI stored? S3? Will making ...
0
votes
1answer
475 views

Setting cache for already uploaded amazon s3 files

In amazon s3 for our site through s3cmd tool we uploaded around 60GB of files including mp3, jpg to a bucket. But now when we checked site slowness files are not caching and further checking we came ...
0
votes
1answer
256 views

AWSCLI s3 sync doesn't create directory

Hi try to sync a local directory to a S3 bucket. The files in the directory get synced perfectly, just the directory isn't created at all. Can you guys give me a hint how to do it? "newdir" contains ...
0
votes
1answer
228 views

Duplicity Backups (via Duply) to S3 - high server load

We have been doing incremental backups using Duply on our main server, to an S3 bucket. However, we've found that there is a high server load during the backups (its an Amazon EC2 server). We're ...
0
votes
1answer
632 views

s3fs “could not determine how to establish security credentials”

I'm trying to use s3fs to mount an S3 bucket on to a standard AWS Amazon Linux AMI (with all the necessary dependencies installed). However when following this tutorial when I run: s3fs mybucketname ...
0
votes
1answer
233 views

rebuild yum index on aws s3

I am trying to rebuild yum repo on aws S3 after adding new packages. Here are few commands I am trying, but it is not helping. [root@chucks ~]$ createrepo --baseurl ...
0
votes
1answer
79 views

Glacier retrieval notification of s3 objects

I have some s3 buckets which send their objects to glacier after x days of creation. When I go to retrieve those objects on the s3 interface, I need to keep reloading page to see when restoration has ...
0
votes
1answer
602 views

Move files from ftp server to s3

I would like to set up an ftp server, where users will upload files, and for each file, put it on s3 storage, and delete it from the ftp server. (the server runs on ec2 ubuntu) Here are the stuff I ...
0
votes
1answer
169 views

AWS route53 forwarding

Essentially, I need to know how to point one URL/folder to a web server and a different URL/folder to an EC2 app server. I'm trying to setup a SAAS with the following URL configuration. ...
0
votes
1answer
188 views

Can AWS S3 files in buckets get deleted by a download timing out?

A few AWS S3 files might have gotten deleted while trying to download everything for a local backup. The download/connection timed-out a couple times. The next day several HTML/JS files are missing ...
0
votes
1answer
252 views

Amazon S3 map url to backend server like Apache's <Location /api> ProxyPass

Is there a way to map a URL to a backend server in S3. I am not looking for redirects, but something on the lines of Apache's Location directive with ProxyPass. ...
0
votes
1answer
686 views

AWS EC2 Mailserver Failover Strategies done right

I'm researching in this topic really hard the last few days and i just want to discuss this with a few specific questions - i did not find any suitable thread here that is covering my needs and ...
0
votes
1answer
676 views

Unable to read S3 file from interactive pig job flow

I'm unable to read a simple test file on S3 from an interactive pig job flow (hadoop, elastic map reduce), and I'm not sure why. I have two S3 buckets. Let's call them unmounted_bucket, and ...
0
votes
1answer
136 views

autoscale static content on ec2 using local cache

Hello Serverfault members. I want to achieve the following setup on EC2: A load balancer starts and stops instances which only serve static content. (a javascript based application) I do not want to ...
0
votes
1answer
427 views

Amazon AWS Store Wordpress uploads on S3

I'm getting ready to move my company's website and order management application to AWS. We use wordpress as the 'face' of our company. Our order management app is for internal use. I'm using Elastic ...
0
votes
1answer
407 views

Upload database backup from mysql to Amazon S3 or Glacier without creating local file

Is there a tool that makes possible to backup a Mysql database to Amazon S3 or Amazon Glacier without having o create a local file with the database contents? Something like that: mysqldump -u ...
0
votes
1answer
200 views

configure s3fs mount for write-only

I am using s3fs to mount a S3 buckets on an Ubunutu server to store my daily backups there. Considering security this would allow a potential hacker to delete all my backups from the mounted bucket. ...
0
votes
1answer
452 views

How to find out what's missing from my iptables, why it's blocking my AWS transfer?

I use ManageWP to backup WordPress on my VPS then upload that to Amazon S3. It's failing with "Error 503 Service Unavailable" at the upload part when I have iptables running. When the iptables ...
0
votes
1answer
329 views

Amazon S3 - can't upload files into the same folder using s3sync.exe

I'm trying to upload a file using S3sync.exe into a certain folder in my S3 bucket. This is the command line : S3Sync.exe -AWSAccessKeyId accKey -AWSSecretAccessKey secKey -RequestEndpoint ...
0
votes
1answer
183 views

s3 multiple write issue

I've built an application which forks process pools to fetch web content. I have 3 ec2 instances mounted onto one s3 bucket via fuse. I'm experiencing intermittent errors writing files. From what I ...
0
votes
1answer
167 views

S3 backup: bucket vs. volume

I've inherited a project that has been pushing encrypted (GPG) DB backups via a backup script to an S3 bucket. That's all been well and good, but now that bootable EBS instances (where the project ...
0
votes
1answer
552 views

S3sync not working

I managed to get s3sync to upload my test folder to Amazon S3 and can see it in the MWS Managment Console. Downloading the data back to a test folder results in the following error message: ...
0
votes
1answer
68 views

Backing Up User Data when data is not in use. Should I be concerned?

This may be a dumb question. I would like to use duplicity to make backups to Amazon S3 of directories, each of which contains a different user's data. Each directory could be written to at any time. ...
0
votes
1answer
438 views

CDN Rerouting on 404 (file not yet in synch with original storage)

Here is the problem. I've setup my app(on EC2) to store uploaded images directly on Amazon S3. I'd like to be able to serve static files(cdn) from my 'home' server so I wrote script that does sync ...
-1
votes
1answer
397 views

AWS s3 - How to serve different file based on url parameter

For legacy reasons I need to serve different files from amazon s3 based on query string parameters. For example myfile.php?type=1 should serve other file than myfile.php?type=2. the files themselves ...