Questions tagged [amazon-s3]
This is for questions on Amazon's cloud storage service.
1,115
questions
0
votes
0
answers
19
views
Enable MFA S3 delete
I have THALES TOTP hardware tokens (MFA) which I would like to use as and additional protection against the accidental S3 object deletion (https://docs.aws.amazon.com/AmazonS3/latest/userguide/...
- 1
0
votes
2
answers
47
views
ansible - couldn't resolve module/action 'amazon.aws.s3_object' / The module amazon.aws.s3_object was not found in configured module paths
Per requirements, I did installed all the required packages, however amazon.aws.s3_object is one of those things:
couldn't be resolved/found
was not found in configured module paths
step(s), I used ...
- 13k
-1
votes
0
answers
19
views
shell script to copy/backup files from multiple s3s to multiple s3s along with json file for aws access and security key
code for json
{
"bucketname:": {
"accessKeyId:<ACCESS_KEY_ID>,": {}
},
"secretAccessKey:<SECRET_ACCESS_KEY>,": {}
}
need shell script in a server ...
0
votes
0
answers
17
views
AWS AppStream is unable to push session script logs to s3
I have configured session scripts to log stdout and stderr to s3 bucket as described in https://docs.aws.amazon.com/appstream2/latest/developerguide/use-session-scripts.html#enable-S3-bucket-storage-...
- 1
0
votes
1
answer
66
views
Best practice for storing personal information on AWS that we shouldn't be able to see
We store customer information that we shouldn't have access to, and don't want to inconvenience users by making them lose information if they forget their passwords, is there a good way to solve this ...
- 13
0
votes
0
answers
47
views
AWS Lambda's indirect S3 requests
I noticed AWS Lambda causes S3 tier 1 and 2 requests. Do these cost anything after 12 months free tier expires?
Specifically:
Create new AWS Lambda function in AWS console causes 1 tier 1 S3 request
...
- 101
0
votes
1
answer
71
views
Restoring Terraform state from backend
My Terraform Git repository ended-up having some unreconciliable data error and I couldn't push a recent change. I exported my commit to a patch, recloned, applied the patch, and then realized I just ...
- 560
0
votes
0
answers
67
views
botocore.exceptions.ClientError: An error occurred (InvalidToken) when calling the ListObjectsV2 operation
I have an mlflow server that is hosted on aws, that uses s3 in the background for storage. I am trying to run a lambda-function that runs a python script which accesses the mlflow server, loads a ...
- 3
0
votes
2
answers
37
views
How I can Log what and who can access and where my S3 bucket?
Recently I received an email in order to tell me that some IE connections has access to my bucket:
eu-central-1|media.myapp | REST.GET.OBJECT|TLSv1|9|[Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1;...
0
votes
0
answers
35
views
How do I setup a Google Cloud Storage bucket to be accessible via the Simple Storage Service (S3) standard?
I understand from Google documentation that it's possible for google cloud storage to be accessed via the Simple Storage Service, which is just what I need for a program that supports this. The ...
google-cloud-platform
0
votes
1
answer
50
views
Storing 100 million files in the same "directory" under S3-compatible storage?
I have > 100 million image files (book covers) as a flat list of files under a single "directory":
/images/000000093e7d1825b346e9fc01387c7e449e1ed7
/images/...
- 4,447
0
votes
1
answer
52
views
S3 / Cloudfront / Load balancer Architecture explanation needed
My architecture is like so:
I have a static website xyz.com hosted on s3 and served with cloudfront. The website is accessing an api that seats behind a load balancer. I want to make sure that only ...
-2
votes
1
answer
46
views
Does Amazon have a web server named 'AmazonS3'?
Note that I understand what "Amazon S3" is; I'm asking only about the software that handles the HTTP interface.
If I check the HTTP headers for github.com, for example, one of the response ...
- 81
0
votes
0
answers
49
views
How to serve both static and dynamic content via AWS cloudfront?
I have a site hosted on a digital ocean droplet which serves both static and dynamic content. The static content (e.g. /images, /js, /css, /fonts) is served from an s3 bucket with folders /images, /js,...
- 119
0
votes
0
answers
135
views
Whitelist an S3 Bucket in server firewall
I am looking for a way to whitelist an S3 Bucket on my client's server. In this S3 bucket there is a public website, thus, my idea is to find the ip address used for this S3 bucket. However, I have ...
- 3
0
votes
1
answer
241
views
Getting 403 (Forbidden) when try to load an S3 file from AWS CloudFront behaviour
I have created an S3 bucket. All objects are publicly accessible.
I can access the image file via S3 url: https://S3_BUCKET_NAME.s3.eu-west-1.amazonaws.com/xyz.jpg
I can access the image file via ...
- 1
0
votes
1
answer
53
views
Using S3 instead of MinIO for video management
I am relatively new to AWS so I wanted to reach out to the gurus in the community to get some advice on architecture, and I thought I would share my thoughts.
I am building out cloud infrastructure ...
0
votes
1
answer
40
views
Cloud File Storage: Does zipping files reduce data transmission speed / costs?
If I need to store, say, a 5gb file on a cloud file storage location (e.g., aws s3 bucket), will zipping the file reduce the cost / latency when people download it?
Obviously zipping a file reduces ...
- 105
1
vote
1
answer
84
views
AWS IAM: deny users from creating policies on specific resources
I want to be able to give my admin users the permission to create policies in IAM, but I want to make sure that they aren't able to create a policy that affects a specific resource.
To be more ...
- 113
0
votes
0
answers
20
views
Is storing data in S3 cheaper than paying for cross-az data transfer?
Imagine I have two EC2 instances A and B, in the same region/vpc, but in different AZs.
I want to transfer 1TB from A to B.
Now let's imagine two scenarios:
I send the data from A to B directly.
I ...
- 170
1
vote
0
answers
181
views
Amazon S3 life cycle rule for sub folder contents?
With this reference link
My s3 bucket name say testbucket.
If I create prefix for a rule say.
subfolder1/
All the objects in it are applied this rule.
But it is not getting applied for objects inside ...
- 145
1
vote
1
answer
601
views
CloudFront distribution can't access S3 bucket for website static assets
The problem
I'm trying to deploy an S3 bucket hosting my static website asset and a cloudfront distribution to access it, but the distribution still return a bare xml file for an 'access denied' error:...
- 141
0
votes
0
answers
33
views
Mimic user permissions on AWS EC2 instances using IAM roles
I'm setting up an AWS account with several users. Each of these users has policies attached that restrict their access to specific S3 buckets/objects and the EC2 instance types / Autoscaling Groups ...
0
votes
1
answer
578
views
How can I sync two S3-compliant buckets which have different credentials?
I've got a MinIO bucket (which uses the S3 API for access) and I want to copy all its data into an AWS S3 bucket. They both use the same API, but the I can't find a tool that lets me copy directly ...
- 159
0
votes
1
answer
121
views
Seamless switch from NAT GATEWAY to VPC gateway
My team is currently burdened by the NAT Gateway costs and we would like to switch over to VPC Gateway endpoint to reduce the costs associated with all the EC2-S3 communication.
at the same time,
We ...
- 1
0
votes
0
answers
23
views
How to assign an IAM role having the same permission set as the IPs have?
How to update the IAM policy below so that the IAM role, arn:aws:iam::7574333677569:role/dev-abc-webserver, also have permissions?
{
"Version": "2012-10-17",
"Id":...
- 29
0
votes
1
answer
135
views
S3 static website is loading pretty slow on private window mode only
I'm not sure is this is an S3 issue or something with my website but would like to ask the community for some help.
Configuration: I have a static website (about 700KB) hosted on S3. My .css and .js ...
0
votes
0
answers
25
views
AWS S3 > Life Cycle Policy > to delete S3 bucket profiles not utilised over 3 months?
I am looking into possibility to setup a policy to delete the profiles within S3 bucket that have not been used over 3+ months. Is that possible and where would I setup?
Would appreciate your ...
- 1
0
votes
1
answer
201
views
Can't access static site S3, Cloudfront, Route53
I apologize if this needs to be split into two questions, but I'm trying to set up a static site using S3, a CloudFront distribution, and a Route53 domain. If I try to access laszloffy.com, I receive ...
1
vote
0
answers
223
views
AWS S3 Bucket Static Website Index/ Home Access Denied
So i have problem with AWS S3 Bucket Static when i try to access my domain without filename.
http://customdomain/ >> Error access denied
http://customdomain/index.html >> normal
but it's ...
- 23
0
votes
0
answers
29
views
Identify the profiles stored in S3 Bucket?
New to this, and new to AWS in general.
Currently reviewing the profiles in S3 bucket to identify which ones can be deleted.
it is not easy to identify whom it belongs to as its a SHA256 generated ...
- 1
0
votes
1
answer
70
views
Does AWS CLI bring data across local network when doing `aws s3 mv` between two S3 locations?
When moving data between two S3 locations using the AWS CLI, for example:
aws s3 mv s3://bucket/prefix s3://bucket/other-prefix
Is the data downloaded to your local computer and then re-uploaded? Or ...
- 111
1
vote
1
answer
241
views
On AWS Cloudfront, I'd like to route /api/* requests to an external api
I'm fairly new to Cloudfront ( AWS in general ), and looking set up a behaviour such that any requests that is for example
test.com/api/*
goes to an existing api that we also own, except it's on a ...
- 13
0
votes
0
answers
387
views
AWS CLI Unknown options: --include,*.m3u8 For S3
Can the AWS CLI tool be used to only list/retrieve certain file types in S3?
I was trying:
aws s3 ls --summarize --human-readable --recursive s3://cloudfront.abc.com/CDNSource/hls/ --include "*....
- 29
1
vote
0
answers
162
views
making all S3 uploads to digitaloceans spaces default to being public
I'm using DigitalOcean Spaces to upload a bunch of files in different directories. In particular, I'm using WinSCP to do the upload via the S3 protocol.
The problem is that, per WinSCP, all the files ...
- 317
0
votes
0
answers
62
views
Sensitive HTML file served by AWS S3 website hosting
How must I configure an S3 bucket to serve sensitive content via website hosting (where the bucket name is the same as the DNS record)? The website will only be reachable from within intranet.
Since ...
- 11
0
votes
0
answers
88
views
One way s3 sync - Bucket to local only
This is the scenario:
Company A creates a bucket, and periodically uploads images to it, deletes or replaces some, etc
Company B needs to sync with the bucket - they download the images from it and ...
- 123
0
votes
0
answers
1k
views
public s3 bucket file download getting access denied error for files transfered from another account
Hi i have created a bucket in s3 and made public, If i upload any file i am able to download the file with url from public. But when I did a copy of all s3 files from another account (which also had ...
0
votes
1
answer
2k
views
Why does an S3 to S3 copy care about VPCs? Related to error: "VPC endpoints do not support cross-region requests."
Goal: Get files
from Bucket 1 in ca-central-1 in Account A
to Bucket 2 in us-east-1 in Account B
using the AWS CLI from a third machine using an the IAM role with correct S3 read and write ...
- 103
0
votes
0
answers
148
views
AWS S3 IllegalLocationConstraintException Wordpress Updraft Backup
I have successfully connected with S3 and updraft backup but I got a problem.
S3 settings test result: Failure: Region: ap-southeast-3: We
successfully accessed the bucket, but the attempt to create ...
- 157
1
vote
1
answer
246
views
What is the purpose of version dates in AWS policy documents
AWS policies contain a date version. What is its purpose? In this example a bucket policy and a user policy both contain "Version": "2012-10-17" in the JSON. I don't see the ...
0
votes
0
answers
121
views
CloudFront Access Logs Issue
I was tried querying our CloudFront logs using Athena for an error i'm getting but haven't received any results. The query i'm trying using is:
SELECT *
FROM cloudfront_logs
WHERE request_id =
> ...
- 71
1
vote
0
answers
38
views
Reduce Query Syntax for ends_with()
I have the following query:
--query "Contents[?ends_with(Key, 'jpg') || ends_with(Key, 'NEF') || ends_with(Key, 'nef') || ends_with(Key, 'CR2') || ends_with(Key, 'cr2')]"
Is there a way to ...
- 111
0
votes
0
answers
484
views
S3 Fatal error: Failed to connect to proxy URL: "Proxy url"
Good Afternoon,
I have been trying to remove some files from S3, last week I did not have any issues accessing my s3 bucket however now any s3 cmd fails with the error message above, I cant even run ...
- 9
0
votes
0
answers
160
views
How suitable is a S3 backend for rsync operations?
I run a large server providing open source software (https://ftp.halifax.rwth-aachen.de), currently serving more than 30 TByte of data with Multi-Gigabit throughput. Data is synchronized and kept up-...
- 334
1
vote
1
answer
172
views
Since S3 charges by request, couldn't a malicious hacker cause a huge AWS bill just by spamming requests?
What would stop them from doing so, against, say, a static website hosted using S3? Is there a good way to deny some requests such that one avoids getting billed for them?
(Context: I want to host a ...
- 111
1
vote
1
answer
155
views
Why would anyone disable S3 KMS "bucket key" option?
As per the documentation:
Amazon S3 Bucket Keys reduce the cost of Amazon S3 server-side
encryption using AWS Key Management Service (SSE-KMS). This new
bucket-level key for SSE can reduce AWS KMS ...
- 377
1
vote
1
answer
723
views
RDS PostgreSQL Import+Export to/from S3
Is it possible to enable both importing and exporting with an RDS PostgreSQL instance to an S3 bucket? I've been able to use the following pattern to enable one or the other with consistent success:
...
- 394
1
vote
1
answer
1k
views
Google Cloud S3 & rclone: Error 400: Cannot insert legacy ACL for an object when uniform bucket-level access is enabled
I'm trying to configure rclone to access a google cloud bucket (which happens to be managed by terra.bio. My rclone config reads as follows:
[terra]
type = google cloud storage
object_acl = private
...
0
votes
0
answers
50
views
Serve cephFS files over HTTP (or S3?)
I have a CEPH+LXD cluster configuration.
And I would like to serve the files in the container over HTTP (S3,whatever doesn't matter).
Is there a way to do this using the rgw service?
Like can I point ...
- 11