Tagged Questions

This is for questions on Amazon's cloud storage service.

learn more… | top users | synonyms (1)

1
vote
1answer
716 views

Using TntDrive to mount Amazon S3 on a Windows Amazon EC2 instance

I currently have 2 Amazon EC2 Windows Server 2008 instances setup with a load balancer. My problem now is that I need both instances to be able to access a single file system source (I'm trying to use ...
1
vote
2answers
741 views

s3fs changing s3 permissions?

My developer believes that s3fs is changing my bucket's permissions. Is this possible? I want my bucket to be public, but it keeps reverting back to private. Here's my fstab. s3fs#production ...
3
votes
1answer
721 views

Serve up s3 file with nginx

I have some static files sitting in an S3 bucket that I want to serve up from my nginx server. Basically, if a user goes to www.mywebsite.com/this/url I want to serve them up a file from the S3 ...
0
votes
1answer
255 views

Amazon S3 - can't upload files into the same folder using s3sync.exe

I'm trying to upload a file using S3sync.exe into a certain folder in my S3 bucket. This is the command line : S3Sync.exe -AWSAccessKeyId accKey -AWSSecretAccessKey secKey -RequestEndpoint ...
1
vote
2answers
84 views

How to list s3fs mounts?

How can I list the s3fs mounts that exist on an ubuntu system? I'd like to know to which bucket each mount is mapped. Specifically, I have a specific mount (e.g. ~/s3/mymount), and would like to know ...
2
votes
0answers
189 views

Gluster with s3ql mounted brick?

I am trying to test run Gluster with a volume consisting of two local bricks, the first being a plain directory and the second being a s3ql based mount. Having just the plain directory as a unique ...
0
votes
1answer
488 views

Amazon S3 - Automated Backup Using S3Sync, Windows Schedule & BAT file

I am trying to build an automated background backup program to backup my entire website and its heavy photo folders, using Amazon's cloud storage service, S3. I am using S3Sync from S3Tools and have ...
0
votes
4answers
1k views

Easiest to use backup software for live linux server [closed]

I need a backup software that is capable of: doing a full HDD image (from live server, if that's possible at all?)or just copy everything except /dev, /tmp, etc updates the backup incremental just ...
7
votes
2answers
5k views

Which is the fastest way to copy 400G of files from an ec2 elastic block store volume to s3?

I have to copy 400G of files from an elastic block store volume to an s3 bucket... Those are about 300k files of ~1Mb I've tried s3cmd and s3fuse, both of them are really, really slow.. s3cmd ran for ...
-2
votes
1answer
1k views

How do I calculate my bandwidth requirements?

We have a facility that needs to upload approximately 20 3MB photos per minute to Amazon S3. These are coming from 10 different machines on our local network. We need to do this continuously for 16 ...
2
votes
2answers
2k views

How do I create a EC2 S3 backed instance, starting from a Windows VHD?

I have a Windows VHD image created in Hyper-V, and I want it to exist as a S3-backed instance on EC2. I've found several articles (and a tutorial) on how to import my virtual machine to EC2. However, ...
2
votes
1answer
186 views

How can I turn an existing Windows Server 2008 EBS-backed instance into a S3-backed (instance-backed) AMI?

I understand there is no direct conversion to get from EBS to S3. But is there some series of steps that can make this happen, or is rebuilding the image from scratch the best option? Is there a way ...
4
votes
2answers
2k views

Duplicity not writing to a pre-existing S3 bucket

I'm trying to backup a directory to a pre-existing Amazon S3 bucket using the following command: duplicity --no-encryption system/ s3+http://MY_BUCKET_NAME/backup However, I'm getting the following ...
1
vote
2answers
300 views

Configure non-destructive Amazon S3 bucket policy

There's a bucket into which some users may write their data for backup purposes. They use s3cmd to put new files into their bucket. I'd like to enforce a non-destruction policy on these buckets - ...
-4
votes
1answer
131 views

Where to store social network photos and videos? [closed]

I have a social network with 100,000 users. I allow users to upload photos and videos now I feel like it is becoming too much for my dedicated server. Should I get another dedicated server? A content ...
7
votes
3answers
9k views

Why is uploading to S3 so slow?

I am using s3cmd to upload to S3: # s3cmd put 1gb.bin s3://my-bucket/1gb.bin 1gb.bin -> s3://my-bucket/1gb.bin [1 of 1] 366706688 of 1073741824 34% in 371s 963.22 kB/s I am uploading ...
1
vote
1answer
1k views

Deleting multiple objects in a AWS S3 bucket with s3curl.pl?

I have been trying to use the AWS "official" command line tool s3curl.pl to test out the recently announced multi-object delete. Here is what I have done: First, I tested out the s3curl.pl with a ...
0
votes
1answer
394 views

AWS instances: AMI bundles and snapshots and bears, oh my

I've googled a good amount and am still pretty confused and feeling like I'm missing or overlooking something. My goal is to restore a host using a bundle. I have an AMI bundle of a now-nonexistent ...
2
votes
1answer
524 views

Amazon S3 Website & MySQL Backup

I have a website (a digital asset management system/gallery - http://www.resourcespace.org) that has a huge amount of images. The total size of the website, including the images is approximately 6gb. ...
0
votes
1answer
95 views

apache sitemap rewrite to s3

I wanted to submit sitemap to google webmaster, google state only accept sitemap from my site Option 1 - simple rewrite I know this already working: RewriteRule /sitemap.txt http://S3_URL [L] ...
2
votes
2answers
207 views

Amazon CloudFrount on EC2

I know that i can mount s3 buckets with s3fs on EC2 instance. So i have next question can i somehow mount Amazon Cloud Front (my S3 buckets files) in EC2 instance?
4
votes
2answers
2k views

Snapshots == EBS backed instance, persistence, and S3 storage?

I want to clear up some confusion and clarify understanding on EC2 instances being EBS backed. If you start an AMI from a community image, configure your server as you want it configured, then use ...
2
votes
2answers
513 views

CloudFront - Users requests static files non-stop to make your bill high?

I had this weird wondering once. If I am using CloudFront as my CDN, I pay per requests made on my static files. So here comes the wondering. Can one keep requesting the same file in a non-stop ...
2
votes
2answers
3k views

How to calculate Amazon S3 prices

I am trying to estimate how much I need to pay to store images for my customers on Amazon S3. I use their calculator but the total seems to cheap. Feels like I am missing something. Each image is ...
4
votes
2answers
1k views

What design features make Joyent's ZFS and Amazon's EBS (S3) reliable?

I know this isn't exactly an apples to apples comparison, what I'm trying to evaluate is which one is safer - less likely to lose data. Joyent's SmartOS uses ZFS to store data whereas an EC2 machine ...
2
votes
1answer
711 views

Use Microsoft DPM 2010 with Azure or Amazon S3?

Does anyone know of any software that hooks up with Microsoft's DPM 2010 or even the new 2012 to ship data to cloud storage providers like Azure or S3? We utilize D2D backups today but want a more ...
1
vote
2answers
166 views

How to show a “linked from site X” placeholder instead of actual photo?

I’m hosting a bunch of photos on Amazon S3 and also running Amazon CloudFront for them. Also, I’m running a custom domain name. Now, in case someone tries to embed the image to another website, I’d ...
2
votes
1answer
442 views

Using an IP Address to Point to Amazon S3 Storage

Is there a way to get an IP address to refer to Amazon S3 Storage location? The current solution my company is thinking of going with involves setting up a EC2 server with an Elastic IP and using ...
2
votes
1answer
602 views

S3 to cloudfront edge location vs cloudfront to end user transfer ratio

We host videos in amazon S3 and use cloudfront to deliver them. Here is last months numbers: 4,2 TB total traffic from cloudfront to end user. 4 TB of this is in europe, the rest mainly in the US. ...
-1
votes
2answers
176 views

Cross platform, blob storage system

I have a couple of different servers around the house, and a lot of storage on each of these machines. I also have a couple of dedicated servers, and some VMs on these boxes... Most of the boxes and ...
0
votes
1answer
102 views

How do I adjust this python script to backup to Amazon S3

The following script is nearly perfect for my backup needs: http://pastie.org/3368661 However, it only allows you to backup to the server or a remote FTP. I would like to add an option to backup to ...
10
votes
3answers
3k views

Any need to make backup of data on Amazon S3?

I'm hosting 200 GB of product images at S3 (this is my primary file host). Do I need to back that data up somewhere else, or is S3 safe as it is? I have been experimenting with mounting the S3 ...
0
votes
1answer
405 views

A uploaded file is 0 byte on Amazon S3 using django

I've symbolic linked a S3 directory on my EC2 server. But a uploaded file was 0 byte when I uploaded file. It was work well before (I used just web server directory) I don't have any idea dealing ...
3
votes
2answers
646 views

Tracking down “Connection reset” errors in Linux

I'm handling large numbers of concurrent downloads (approx. 500 per server) using Java. All the files are being downloaded from Amazon S3, and the downloading server is an EC2 m1.large instance. ...
7
votes
2answers
8k views

Is mounting S3 buckets directly into EC2 instances safe?

as the title of my question says, I would like to know whether this is a good practice or not, I know I can mount it in an EC2 instance but I was wondering if, as I will be adding more instances to ...
2
votes
3answers
1k views

projects or packages to emulate a tape drive with a Amazon S3 or OpenStack Swift compatible backend?

I've got some data that is being backed up to tape via bacula, and at some point soon I am going to need to migrate to non-tape storage. Bacula supports to-disk archiving, but it occurred to me that ...
1
vote
1answer
666 views

Does anyone have a config to process AWS CloudFront/S3 logs with AWStats (or similar sw)?

I don't want to give an external company like s3stat access to my logs. I know that AWS logs S3 and Cloud Front in a format readable by AWStats. Has anyone used AWSats to analyze them? S3stat used ...
-1
votes
1answer
313 views

Which of snapshots I can safely remove?

If you make periodic snapshots of a volume, the snapshots are incremental so that only the blocks on the device that have changed since your last snapshot are incrementally saved in the new ...
20
votes
4answers
15k views

How to move files between two S3 buckets with minimum cost?

I have millions of files in a Amazon S3 bucket and I'd like to move these files to other buckets and folders with minimum cost or no cost if possible. All buckets are in the same zone. How could I ...
0
votes
1answer
169 views

Is uploading a 500mb file to s3 an expensive operation? Any way to limit cpu usage etc?

On a production server, I'm planning on uploading a single file or a set of files that may be 500 megs in size. I havent' test it just yet, but just want some opinions on whether you think this will ...
5
votes
3answers
4k views

Forcing CloudFront to pass-through the latest HTML file from S3

Background I'm hosting a static site on S3, with CloudFront over the top. The issue I have is with my HTML files. According to CloudFront's FAQ: Amazon CloudFront uses these cache control ...
2
votes
2answers
2k views

Amazon EC2 - What's the point of having Instance storage if it is non-permanent and EBS provides a permanent store?

I'm specifically looking at the Standard Large On-Demand instance which has 850 GB of instance storage. What I really don't understand is why have 850 GB of storage on the instance if that data will ...
-1
votes
3answers
331 views

amazon s3 vs amazon ec2 regarding cost, and which better for image hosting website? [closed]

I want to know which will cost more, s3 or ec2. I need to know by example to estimate my future payments. Also which of them will be better for an image hosting website?
4
votes
3answers
3k views

Uploading 200GB of files to S3

What's the best way to upload 200GB tar.gz files to S3 in Linux? On researching I found that S3 limit on objects has been increased to 5TB and came to know about the multi part upload mechanism to ...
3
votes
1answer
348 views

Using amazon S3 as storage for attachments in a web-mail system

I am looking to use S3 as data storage service for a mail system. The idea is to allow users to upload files that serve as attachments to their emails, which are then available to download from the ...
1
vote
1answer
523 views

Protected bucket @ S3 - Access Denied

I'm trying to grant RW access to a given bucket to a specific user using the following bucket policy: { "Id": "Policy1322043790167", "Statement": [ { "Sid": "Stmt9999043784080", ...
2
votes
0answers
282 views

Web-based file manager like File Thingie, but for S3?

At work, we use a little PHP script called File Thingie, which is a web-based file manager. Basically, an FTP replacement. It allows users to upload and download files, and most importantly, it has a ...
6
votes
5answers
7k views

s3cmd fails too many times

It used to be my favorite backup transport agent but now I frequently get this result from s3cmd on the very same Ubuntu server/network: root@server:/home/backups# s3cmd put bkup.tgz s3://mybucket/ ...
1
vote
1answer
202 views

How to configure S3 or DNS to handle incomplete name (sans www) for web site?

I have a set up a bucket called "www.mydomainname.com" to host my website and I have configured the CNAME such that "www.mydomainname.com" points to the my endopint ...
0
votes
1answer
99 views

Running my own CDN alongside Amazons S3 for nearby visitors

I am running a blog on shared webhosting. The website itself is on the hosting, but my files right now are on Amazons S3 CDN. I have made a bucket lockated in the US, and I have setup a virtual host ...