Questions tagged [amazon-s3]
This is for questions on Amazon's cloud storage service.
1,136
questions
0
votes
1
answer
2k
views
Download files from S3 without listing
I have a s3 bucket with more than a million files and about a thousand files added each day by various applications into various folders.
I would like to download and keep all the files locally as ...
- 421
3
votes
1
answer
2k
views
How do I connect Cloudfront to a private s3 bucket with terraform?
So I'm 90 percent there, but it appears that cloudfront is getting 500 errors from s3? I'm sure I'm just doing something wrong. I've read the various amazon documentation, but to me it seems so vague ...
- 1,566
4
votes
3
answers
10k
views
Problems with Gzip compression on Amazon CloudFront
I had a CloudFront distribution configured with S3 as an origin.
I have enabled "Compress Objects Automatically" as per Amazon guide. Waited for the distribution status to display "online" then ...
- 151
11
votes
2
answers
17k
views
AWS S3 Bucket name already exists
I in S3 Buckets if ill create a new one or Bucket name already exists
but I can I have two visible buckets.
Edit:
In other words, my bucket's been orphaned and I cant see it to delete it and I ...
- 153
-2
votes
1
answer
1k
views
Performant Reverse Proxy to S3
I want to prevent direct interaction with my S3 bucket. While my S3 bucket is properly configured, I want to find a (performant) method to lock down access to a single i.p while still serving files ...
- 157
0
votes
1
answer
1k
views
Can we point godaddy DNS to aws s3 static website?
I have a domain example.com which is registered in godaddy .
I have hosted a static website in s3 bucket .my s3 endpoint is
<bucket-name>.s3-website-<AWS-region>.amazonaws.com .
I want ...
- 121
0
votes
1
answer
754
views
Is AWS CloudFront useful if I need to serve just one region?
We are serving videos from a S3 bucket(in Mumbai) to users in a single region(India). We expect thousands of users to access these files only from India. Is it worth to have a CloudFront distribution? ...
15
votes
4
answers
47k
views
Amazon S3 Console: How to find total number of files with in a folder? [duplicate]
So I can see S3 console and can get into my folder with files listing but I don't find way to find total number of files in it other than going thru pagination which does not work considering ...
- 275
1
vote
1
answer
2k
views
How to copy more than 300GB files one AWS EC2 server to another EC2
I need to copy more than 300 GB files (most them are images) one EC2 instance to another EC2 instance, what will be the best method for doing this?
EBS Snapshot will be not work in my situation ...
- 13
0
votes
1
answer
268
views
IAM user policy to RW to specific folders
I am trying out CloudBerry Drive tool to attach S3 buckets as my network drive. I have a bucket and 2 folders inside that, assuming the bucket name as environment and 2 folders as dev and prod. I have ...
- 764
8
votes
2
answers
3k
views
Proxy for a local mirror of S3 directories
We have an office that has increasing demand for accessing large files from our own Amazon S3 directories. Being able to access them quickly is important for our business, so we believe it is time to ...
- 181
1
vote
1
answer
290
views
Multiple server backups using kubernetes and rsnapshot
I am looking to setup multiple backups, from multiple servers in different geographical locations at different times using something like a docker/rsnapshot and kubernetes (master + a couple nodes at ...
- 11
4
votes
1
answer
6k
views
Rewriting url with results from auth_request
I'm trying to use nginx to proxy requests to files stored on S3, however, I don't want to expose our bucket structure, and I need authorization that the user can access a particular file. I've managed ...
- 41
0
votes
0
answers
119
views
How to make daily database backups readily queryable
We have a production database that generates daily database backups that are uploaded to S3 in form of compressed .sql.bz2 files. We basically have a S3 bucket that has a large number of such backups, ...
- 101
1
vote
3
answers
6k
views
Is possible to reduce AWS data transfer costs using any other AWS resource?
I do have a AWS EC2 instance that contains a very large blog. My data transfer is about 200Gb/month. 80% of it are only media and images.
My data transfer costs are high due the server location in ...
- 168
0
votes
1
answer
425
views
Elastic Beanstalk: IAM policy for deploying a multicontainer Docker environment from a private repository
How do I enable my environment under Elastic Beanstalk to talk to S3 so that it can get the authentication info and then pull my Docker image from a private repository in a multicontainer environment?
...
- 570
2
votes
2
answers
135
views
Backup strategy for user uploaded files
Note: Backing up data stored on Amazon S3 is similar but quite old, and doesn't address common practice.
Our service involves each user upload a couple of files. There are hundreds of thousands of ...
- 23
1
vote
0
answers
107
views
Subdomain pointing to deleted S3 bucket [duplicate]
I am stuck in my present (AP-SOUTH) and deleted (US-EAST) S3 bucket.
I had mistakenly created a bucket in US-EAST (cdn.example.com.s3-website-us-east-1.amazonaws.com) and mapped my subdomain - cdn....
- 11
68
votes
2
answers
20k
views
Why does AWS recommend against public S3 buckets?
"We highly recommend that you never grant any kind of public access to your S3 bucket."
I have set a very granular public policy (s3:GetObject) for one bucket that I use to host a website. Route53 ...
- 791
3
votes
1
answer
4k
views
S3 cross-account bucket permissions
Similar to what is described in this article[0], the company I work for uses a bastion AWS account to store IAM users and other AWS accounts to separate different running environments (prod, dev, etc.)...
- 5,690
1
vote
1
answer
1k
views
Minimum AWS policy actions required for retrieving Glacier files through S3?
I'm trying to create a policy for an IAM account that will allow an employee to have full read permissions for our S3 (lifecycled to Glacier) buckets, with no unnecessary write abilities as to avoid ...
- 123
8
votes
3
answers
17k
views
Streaming PostgreSQL pg_dump to S3
Is it possible or advisable to stream/pipe pg_dump output to S3?
We are dumping large datasets to our instance and the database size is big. So trying to optimize for local disk space (avoid temp ...
- 2,113
7
votes
1
answer
6k
views
Why is only index.html available from an S3 bucket served by CloudFront, and nothing else?
I am trying to expose a static site hosted in S3 though CloudFront.
The S3 bucket (testyop1) is set to host static websites (not in public mode) and its bucket policy is
{
"Version": "2008-10-17"...
- 3,657
1
vote
1
answer
3k
views
Redirect naked domain to www with SSL in cloudfront/route 53
I have setup two buckets on s3 one for my actual angular 2 app: www.example.com and one for redirecting example.com to www.example.com. I have also created two distributions in cloud front and added a ...
- 23
13
votes
1
answer
10k
views
Serving website from Cloudfront and S3 without public bucket
When serving a website via CloudFront from an S3 bucket, I would typically apply the following Bucket Policy to allow access to my files:
{
"Version": "2012-10-17",
"Statement": [
{
...
- 231
5
votes
1
answer
934
views
Syncing / backup from S3 buckets to openstack swift containers
I need to backup/sync few hundred GB of files in S3 buckets to openstack swift containers.
I have thousands of files in several S3 buckets I want to copy/sync over openstack swift containers but I ...
- 51
1
vote
1
answer
2k
views
Redirect naked domain to www in cloudfront/route 53
I have a static website hosted on s3 with cloudfront distribution. I've found tons of answers on redirecting to www with only route 53 in front of s3, but nothing on how to do it with cloudfront.
I'...
- 133
4
votes
2
answers
865
views
Migrate Big Data from AWS S3 to Azure
We have 85TB of data in AWS S3 bucket. What is the best option to move it to Azure?
I see two options right now:
To buy AWS Snowball Edge. Will I be able to import data to Azure from it?
To buy ...
- 141
-1
votes
1
answer
97
views
AWS Lowest latency infrastructure
I'd like to setup the most correct and lowest latency infrastructure for AWS.
I would like to serve static Javascript files over CloudFront, and on top of that - a call to a dynamic server that will ...
1
vote
1
answer
89
views
Error writing from S3 to EBS
I'm trying to copy from an S3 bucket to a mounted EBS volume on a Windows EC2 Instance.
I have the following IAM policies in place:
AmazonEC2FullAccess
AmazonS3FullAccess
...
- 21
3
votes
1
answer
5k
views
Route53 subdomain not resolved with nslookup
My problem
I am trying to build a personal CDN to share static file with my contacts. The design includes an S3 bucket, a CloudFront distribution and a subdomain registered via Route53, all ...
- 13.6k
0
votes
1
answer
537
views
How to identify which account a given S3 bucket lives in?
I come across many S3 buckets... is there a way to tell which AWS account a given S3 bucket was created in? I can only think of iterating through each account I have access to, but I don't have keys ...
- 1,559
1
vote
1
answer
456
views
How can I reduce the number of get requests for s3 bucket? [closed]
I have an apache2 server living on an ec2 instance in AWS as well as an s3 bucket that holds mostly just image files.
What I want to do is implement some sort of caching for a user on my server or ...
- 35
3
votes
1
answer
2k
views
Restricting access to files on Amazon S3 on per-file basis?
I have an interesting requirement: There's various files on the same S3 Bucket. The task is to restrict access to the files on a per-file basis using usernames and passwords.
I tried using an FTP ...
- 151
0
votes
1
answer
1k
views
How to setup DigitalOcean DNS link to AWS CloudFront Distribution
I have a DNS setup with DigitalOcean for the domain
ghazanfarmir.me
What I want to do?
I would like a sub domain i.e. static.ghazanfarmir.me to be linked to the CloudFront Distribution, which then ...
- 103
2
votes
2
answers
8k
views
How to find owner of a aws account with account number
I inherited a couple of AWS environments. I have been recently doing security audits of s3 and found several policies with principals containing aws account numbers I don't know and nobody at my ...
- 63
-1
votes
1
answer
2k
views
Cannot access or delete AWS S3 bucket from console
Learning AWS and mucking about with buckets. I tried to configure a logging bucket for a project, with the following logging.json used for the put-bucket-logging command in the cli:
{
"...
- 149
1
vote
1
answer
738
views
AWS, S3 + VPC Endpoints
Not sure what I'm missing here.
I've got an S3 bucket. I've got an EC2 instance.
I created a VPC endpoint and associated it with ALL the subnets in my VPC, with the default policy:
{
"Statement": ...
- 136
1
vote
0
answers
149
views
cloudwatch log to kinesis and s3
I want to use my cloudwatch logs which are basically website access logs. Need to use these cloudwatch logs for data analytics with kinesis stream since firehose and analytics service is not ...
- 1,577
1
vote
5
answers
3k
views
AWS Windows EC2 instance does not recognize assigned IAM role
Initially I launched a brand-new Windows Server 2016 server EC2. I assigned a S3 full admin IAM role to this instance when launching it. I installed CLI on it. I started a CMD window, and typed in "...
- 568
1
vote
1
answer
2k
views
AWS CLI - S3, why do I need to specify a region?
So my bucket Alpha is in a region called REGION1.
My bucket Beta is in a different region, let's call it REGION2.
I go into my EC2 instance, and use this command:
aws s3 cp s3://Alpha/meh.txt ./meh....
9
votes
2
answers
7k
views
How to backup a s3 bucket to the glacier?
I'm having some difficulty setting up my S3 to make an automatic backup to Glacier. That is, I want a copy of my s3 on the glacier every day. How to do this?
3
votes
1
answer
2k
views
Just created S3 bucket, getting a bunch of access denied logs
I've created an AWS S3 bucket to backup my Postgres DB. It was all going well but then I enabled Logging and started to get a bunch of Access Denied logs.
I thought I had messed something up in my ...
- 131
1
vote
0
answers
235
views
Restore original visitor IP for static site on AWS S3 using Cloudflare
I've a static site on S3, with cloudflare as my DNS provider. The server logs are collected in an S3 bucket. However the visitor IP address in the log files points to cloudflare servers instead of the ...
- 111
1
vote
1
answer
2k
views
What do cloudwatch s3 metrics count as an object?
I was recently using the metrics under the management tab in the aws s3 console and notices that for a few of my buckets the NumberOfObjects (count / day) graph seemed to be way off. I had roughly 50 ...
- 113
0
votes
2
answers
138
views
scaling up when new object is inserted in s3
I want to setup auto scaling in following way.
I am storing website data as zip in aws s3 bucket (versioning not enabled)
when an instance is launched in auto scaling, user data will have shell ...
- 33
3
votes
1
answer
211
views
Amazon storage gateway how do you get the storage usage without using CloudWatch (from the Storage Gateway Volume Page)
I can't find where to get the volume usage (used space) in the Storage Gateway Volume page.
I don't want to use CloudWatch to have the information, there should be a way to add the column "Used space"...
- 31
4
votes
2
answers
9k
views
AWS CodeBuild script fails s3 sync with AccessDenied
I am using CodeBuild with
Image: aws/codebuild/nodejs:7.0.0
service role: code-buid-some-service-role
In my buildspec the following command runs
aws s3 sync ./webroot s3://s3-us-west-2.amazonaws....
- 161
1
vote
1
answer
2k
views
Subdomain from different S3 bucket on same cloudfront distribution
I have example.com served from idxxxx.cloudfront.net, this goes to S3 bucket nr 1. This works well.
I now need subdomain.example.com to also be served from same cloudfront distribution, and point to ...
- 113
2
votes
1
answer
512
views
S3 to Swift middleware proxy
We have an application which is ready for AWS S3 Storage. Recently we had an requirement to write also on a Swift Object Storage. I know that Swift stack has an option to enable S3 compatibility but ...
- 121