Questions tagged [amazon-s3]
This is for questions on Amazon's cloud storage service.
1,062
questions
0
votes
0
answers
31
views
Unable to access few objects in S3
Why some objects are accessible in S3 bucket and some are not. Following are the below errors we faced. Though I checked the access and it seems to be fine. Exception while connecting S3: An error ...
- 1
0
votes
1
answer
20
views
How do I extract a list of all folders of my AWS S3 storage bucket or directories?
Does Amazon provide an easy way extract a list of all folders that have files greater than 500 MB from a s3 bucket? want to limit the scope to the '/files/ftp_upload/' directories also This is so I ...
0
votes
1
answer
32
views
AWS Policy to Read/write RDS
In my scenario , I want a policy that will allow reading and writing of abc-database-backups/rds/postgresql-backup on S3? We'll want the my servers to have that access added.
Is creating a role and ...
- 1
0
votes
1
answer
29
views
How do I open a private static website using boto3?
I have a static website consisting of multiple HTML, CSS, and media files uploaded to a private S3 bucket. I would like to access the website using my web browser.
I'm currently using boto3 to ...
- 101
0
votes
1
answer
47
views
Trying to understand S3 Lifecycle rule
I am working on an e-commerce website.
I store product photos in an s3 bucket. Once the product is deleted, I also delete the photos from s3 bucket.
I have S3 bucket versioning enabled. I am not ...
- 456
1
vote
1
answer
124
views
What is the difference between S3 Multi-Region Access Points and AWS CloudFront?
Can anyone explain the difference between S3 Multi-Region Access Points and AWS CloudFront?
The documentation for a multi-region access point states:
Multi-Region Access Points offer a global S3 ...
- 111
0
votes
1
answer
287
views
How to download millions of s3 files and compress them on the fly?
I have an S3 bucket with millions of files, and I want to download all of them.
Since I don't have enough storage, I would like to download them, compress them on the fly and only then save them. How ...
- 1
1
vote
0
answers
255
views
CloudFront CORS request using signed cookies and withCredentials, not sending back Access-Control-Allow-Credentials unless I include some extra header
I'm having a very strange issue that I can't seem to crack. I configured a private CloudFront distribution to serve content from a private S3 bucket. I am using signed cookies to grant access to the ...
- 61
0
votes
1
answer
53
views
Bandwidth consumption of Amazon S3 bucket mounted with S3fs on a VPS
I have :
VPS server on Vultr with web server and Wordpress installed.
Uploads folder is mounted on Amazon S3 bucket using s3fs.
If a user downloads a 1GB file from my website will the traffic be ...
- 1
1
vote
1
answer
29
views
S3 policy to allow by IP can't be changed
We have an S3 bucket with website content in it that's served to the public through a proxy cache. The access to the bucket is limited to the IPs used by the proxy. This is all working fine.
New IPs ...
- 623
0
votes
1
answer
96
views
S3 Logs event Issue
Is there a way to see what actions the 'g2' IAM user is performing in S3, and which IP(s) they are running from? I have already enabled the logging of S3 actions.
One point I’m still not able to ...
- 61
0
votes
1
answer
19
views
Sending push notification to s3 origin website from lambda
I want to send a websockets notification from lambda function to clients, on a website hosted via s3 origin with cloudfront.
Could someone help me with this, please? Thanks.
- 3
0
votes
1
answer
22
views
aws cloudfront custom ssl pricing for subdomain and www
I have created two s3 origin website linked with cloudfront and used acm certificate.
But I recently found out pricing are different between https://admin.example.com and https://www.example.com.
www....
- 3
0
votes
0
answers
56
views
CloudFront delivery is slow in particular location
In our scenario, I found that our CloudFront delivery particularly in the Japan area is very slow for a few days. which lead our site to become slow in the time of loading the media contents .our s3 ...
- 61
1
vote
0
answers
22
views
Calculation of S3’s byte-hour
S3 uses the concept of Byte-Hour in billing: https://aws.amazon.com/s3/faqs/#Billing
Unfortunately (in typical Amazon AWS style) they left some details unsaid, namely what happens in the case of ...
- 11
0
votes
1
answer
32
views
DNS - GoDaddy for mail and AWS for rest
I have a GoDaddy hosting account but for some of my websites I would like to host them in AWS S3. My question is how to set DNS entries in Route 53 so that I keep web traffic served from S3 but I ...
- 1
1
vote
0
answers
32
views
How do I block TCP ACK causing retransmissions
I am trying to replicate a issue when making requests to S3 that will lead to tcp_retries2 leading to 900 seconds of retransmission attempts before giving up. How would I do this? Is it sufficient to ...
- 11
0
votes
1
answer
14
views
Setting up an architecture for analyses flows of time stream data from multiple sources [closed]
What would be the best practice in terms of AWS for the following:
Many IOT medical devices gather data at around 256kBps
The data is a time series data (a matrix of [Channels X Samples], there can ...
- 1
0
votes
1
answer
150
views
Unable to mount a folder in Ubuntu to as AWS S3 Bucket
I am attempting to mount a folder on a fresh AWS EC2 instance running Ubuntu 18.04, to AWS S3.
I've been following instructions I found at https://cloud.netapp.com/blog/amazon-s3-as-a-file-system and ...
- 101
0
votes
0
answers
138
views
How to Bulk Delete 1.4 Million AWS S3 Access Logs
We hadn't specified a target bucket for our AWS access logs & I believe we set up an infinite loop https://aws.amazon.com/premiumsupport/knowledge-center/s3-server-access-logs-same-bucket leading ...
- 1
1
vote
1
answer
219
views
Will aws s3 be still accessible using pre-signed urls if we create a vpc endpoint gateway?
I have a need to add a vpc endpoint for s3 so i can access it on lambda, but when i try to create the endpoint I get a warning
Warning
When you use an endpoint, the source IP addresses from your ...
- 109
0
votes
1
answer
80
views
S3 life cycle policy - move to SIA only if size bigger than 128KB
Usecase description:
I'm planning to set a life cycle policiy to move objects from frequent access to infrequent access automatically after 30 days. but in the bucket, there are some small files that ...
0
votes
2
answers
291
views
Terraform: "S3:path_to_terraform.tfstate": x509: cannot validate certificate for <IP> because it doesn't contain any IP SANs
I'm using s3 to save my state. since this is a custom S3 (and not AWS) I configured it as shown below (using IP + Port).
when running terraform init I received this error:
Successfully configured the ...
- 103
0
votes
0
answers
19
views
AWS S3 bucket policy does not define Put, yet it is allowed [duplicate]
I have a public static website which images can be uploaded to. For this I enabled static website hosting and set Block public access to OFF. I then added this policy
{
"Version": "...
- 113
0
votes
0
answers
177
views
AWS S3 bucket policy for get and upload
I have two buckets each with their own goals. So far I cannot understand the complicated configuration of bucket settings. It seems there are three things to configure
Block public access
Bucket ...
- 113
0
votes
2
answers
189
views
AWS S3 Static Website vs CloudFront OIA
It seems there are kind of two ways to host a static website on S3/CloudFront.
Create S3 bucket with full public access and Static Website set to Enabled + CloudFront config
Create S3 bucket with no ...
- 113
0
votes
0
answers
38
views
Do I have to set up S3 for Laravel application on Elastic Beanstalk?
I have set up an Elastic Beanstalk environment on top of PHP platform and deployed Laravel code.
I have realized that each time I deploy code to EB environment, the storage becomes empty (naturally ...
- 101
0
votes
1
answer
102
views
API gateway cache every hour
My API is set up as follows:
API Gateway > Lambda function > S3
Every hour the S3 content is updated at the same time. How do I configure API gateway to cache the endpoint's responses but serve ...
- 107
0
votes
1
answer
231
views
how to download archive in aws s3 glacier
first, sorry for my bad english.
i create a vault in glacier service and then i uploaded an archive into vault and in out-put, aws gave back me a archive id. then using the archive-id, i created a job ...
- 69
0
votes
0
answers
48
views
Issue with syncing data from Amazon RDS to Amazon S3
I am trying to sync binlogs in RDS mysql DB with Amazon S3 in real time to query using Amazon Athena. I have tried using debezium as source connector for Apache Kafka connect- kafka stores streaming ...
0
votes
0
answers
37
views
AWS/S3/Cloudfront/GoDadday and subdomains
I am wondering and hoping this is possible.
On AWS I have two s3 buckets to separate a client facing application and an administrative application
mysite-app
mysite-admin
I have a cloudfront ...
- 113
0
votes
1
answer
47
views
Transition of a large bucket to S3 Glacier
I want to move a 72 TB size bucket from S3 Standard to S3 Glacier.
I'm going to create a Lifecycle rule and set "Transition current versions of objects between storage classes" to 1 day.
I ...
- 21
0
votes
2
answers
99
views
Best Practice for AWS IAM access keys for use with AWS SDK
I want to know the best practice used by big company's for programmatic access for multiple AWS services as there are multiple programs needing access to different-2 services so how it is managed? Did ...
- 49
0
votes
0
answers
287
views
400 Bad Request errors (infrequent) on public Amazon S3 assets
We are hosting S3 public assets (images) under a local path using a reverse proxy from NGINX to S3.
We have noticed periodic errors in our logs (400 errors) which are very infrequent, but are causing ...
- 685
0
votes
0
answers
64
views
Deny unencrypted s3 buckets via SCP
Folks just wondering if there’s ability to attach a SCP to OU accounts denying S3 buckets from being created if default encryption is not opted upfront.
From cloud trail it’s apparent that ...
0
votes
0
answers
32
views
AWS S3 static website: unable to access https://mydomain.tld but http://mydomain.tld redirects to https correctly
I've set up a static website on AWS S3, using a custom domain hosted at Namecheap, and Cloudfront serving over https. I am able to access my site at http://mydomain.tld, which redirects (using a DNS ...
- 101
1
vote
0
answers
129
views
S3 through Nginx Proxy doesn't allow more than 10 files to be downloaded simultaneously
I setup a Nginx Proxy in front of my S3 object storage and I'm able to download files without any issues so far.
However, if I try to download the same file from, let's say, 30 different servers, at ...
- 11
0
votes
1
answer
383
views
Why can't my DB back up to my S3 bucket?
I'm trying to test a differential backup and restore on an RDS SQL Server 2019 instance in preparation for migrating some web resources into AWS. Both the EC2 instance (Windows Server 2019) on which I'...
- 143
0
votes
1
answer
368
views
cannot Delete objects on s3 ceph bucket
I have a ceph cluster and running a few s3 bucket on it, 'gitlab-s3-api' user has full permission on everything on ( users=;buckets=;metadata=;usage=;zone=* ) but very oddly it cannot delete any file ...
0
votes
0
answers
29
views
Automate AWS RDS SQL server import from on-premise SQL server
I want to create a full daily backup from an on-premise SQL server, store it on S3, and then restore it onto an existing Amazon RDS SQL Server. Something like this documentation: https://docs.aws....
- 1
0
votes
1
answer
1k
views
Domain to S3 Bucket AccessDenied
Hello i have a problem i recibe AccessDenied in my domain
I upload my site in to a AWS S3 Bucket, i make public, create a static page, but when i access via my domain.com/mypage i get a AccessDenied.
...
- 1
0
votes
1
answer
39
views
Cloudfront distribution with react app and wordpress app hosted on same domain - how to get browsers with react app cached to render wordpress app?
I have a domain that hosts a react website in an s3 bucket via cloudfront. There is also a wordpress site hosted on a subdomain of that domain, and in the cloudfront distribution for the web app, I ...
- 103
0
votes
0
answers
224
views
Mount S3FS mount to EFS additionally and share to other Kubernetes pod
What I need to achieve that is almost done without the EFS/S3FS share:
SFTP pod used by some microservices that process content and deliver back processed content have sftp users that go to tenant ...
4
votes
1
answer
1k
views
AWS: How to redirect many domains to a page on another domain?
My objective
I have a number of domains (e.g. 10 or 20) and I would like to redirect any visitors to anywhere on those pages to one page on another domain (for example my stackoverflow.com profile ...
- 1,110
1
vote
1
answer
652
views
AWS Cloudfront CORS trouble with font files
I have a Javascript application that is delivered to the browser from a CloudFront endpoint. All static files of the application are hosted in an S3 bucket. That involves a whole bunch of javascript ...
- 207
0
votes
0
answers
59
views
AWS S3 links return 403 when application is run using Docker Swarm, but works fine when run with Docker Compose
I have a Django application on EC2 (ubuntu) instance using S3 for static & media files, which is set up currently using Docker Compose. I'm trying to convert it to use Docker Swarm, but in doing ...
2
votes
2
answers
3k
views
How to *actually* exclude a directory in AWS S3 sync?
The aws s3 sync command has an --exclude flag which lets you exclude a folder from the sync. However, even though the files are not uploaded from that directory, the command still looks at and ...
- 121
0
votes
0
answers
145
views
AWS SES Managing and Forwarding Incoming Mail
I'm probably in over my head with this one. Please bare with me while i fumble through my explanation of what it is that i am trying to accomplish here.
We own a domain name. Have access to the back ...
- 1
0
votes
0
answers
169
views
Image resizing and caching in NGINX after authentication and 302 redirect
I'm having issues with resizing and caching images. Here is my non-performant setup without caching:
Client makes request for website.com/static/image.jpeg
NGINX routes to API
API authenticates user, ...
- 101
0
votes
0
answers
24
views
S3 Security Policy for Bucket while allowing signed urls
ugh, bucket policies.
I am trying to write a bucket security policy that would keep an outsider from listing the structure and data in the bucket but at the time my AWS users needs to be able to pull/...
- 109