Questions tagged [s3fs]

s3fs is FUSE-based filesystem that stores files in AWS S3.

Filter by
Sorted by
Tagged with
0
votes
0answers
30 views

s3fs enable sftp access only to single user using user/pass getting Permission denied

i try to map s3 to sftp for a single user only, that means only 1 user with the password could sftp to the mounted dir. this is what i configured : created new cool_user group with cool_user user that ...
1
vote
1answer
48 views

S3FS Mount does not seem to work: Am I doing something wrong?

So I've just a brand new VPS which I'll be using for nextcloud. I followed this guide. I did (on a brand new machine): Apt install s3fs Create a /etc/passwd-s3fs file with key:secret key Chmoded it ...
0
votes
1answer
865 views

Mount AWS S3 on boot while behind proxy using s3fs

In order to get a mount to work I have to first set the http_proxy environment variable before I run s3fs. export https_proxy=<proxy address> s3fs -d <bucket> mount-point -o url=https://...
1
vote
2answers
143 views

mysqldump using a lot of space

I have a server and the / partition is 20GB in size. Databases are stored in /mnt/mysql-data partition is 500GB in size. Now here's the problem. Whenever I run mysqldump it fills up / partition to ...
2
votes
0answers
220 views

How does S3FS (or any other S3 FUSE filsystem) compare to AWS FSx for Lustre + S3

I remember trying s3fs a year back, trying to use some S3 bucket as a FUSE filesystem. I remember it being quite laggy, especially when coupled with git operations on it (an oblivious system architect ...
0
votes
0answers
109 views

s3fs cache enabled the size of the cache folder

Ubuntu 18.04 s3fs - V1.85 My /etc/fstab: mybacket /mnt/s3-drive fuse.s3fs _netdev,use_cache=/tmp,del_cache,ensure_diskfree=10000,enable_content_md5,use_sse,passwd_file=/root/.passwd-s3fs,allow_other,...
2
votes
1answer
1k views

Amazon s3 bucket mount point getting unmounted automatically

I use s3fs on aws ubuntu instance to mount s3 bucket. Following is the command I used to mount the bucket. sudo fusermount -u /s3/project1 sudo s3fs -o allow_other project1 /s3/project1 The ...
2
votes
1answer
229 views

Use S3FS Mount As HTTP Server Document Root

My proposed setup: An EC2 server running Apache or nginx. An S3 Bucket in the same region/availability zone mounted with S3FS. The Apache or nginx document root set to the S3FS mount. My question: ...
3
votes
1answer
213 views

How to share a EFS/s3 with code in a auto scaling group?

the main problem is: Infrastructure: Autoscaling (min 1 - max 3) RDS ELB elasticache (redis) elasticsearch I want to share a volume (EFS / S3) with the code of my application; The size of it is ...
0
votes
1answer
400 views

s3fs does not pharse NO_PROXY / no_proxy env variables

I am running s3fs on a node in corporate network which is behind proxy. I am trying to mount S3 bucket on another node next to it. Both nodes are in 10.88.97.65/27 network and should be able to ...
4
votes
2answers
8k views

Can not unmount an S3 directory mounted with s3fs-fuse

I have an s3fs-fuse "device" mounted at /mnt/s3. I want to unmount it so that I can remount it with the mp_umask flag so that chroot will work properly. The command should be: fusermount -u /mnt/s3 ...
3
votes
1answer
2k views

Automatically mounting S3 bucket using s3fs on Amazon CentOS

I have tried all the answers provided in similar questions but none is helpful. I installed S3 Fuse so that I can mount S3 bucket. After the installation, I performed the following the steps: I ...
2
votes
1answer
1k views

How to use SAMBA on an Amazon S3 mounted directory on Ubuntu VM

I had successfully configured a SAMBA share on /home/myusername on an Ubuntu 17.10 VM. I then used s3fs-fuse to mount an S3 bucket on that exact directory, i.e. /home/myusername. It worked too, any ...
1
vote
1answer
237 views

S3 Bucket for hosting media, s3fs mount catch-22

I'm attempting to use an S3 bucket to host media files (images, video and static files) for a small web application and came across a strange catch-22... According to the S3 documentation, to serve a ...
1
vote
0answers
470 views

sharing a S3 storage mount via samba: Permission denied

At work we've recently deployed several S3 storage servers. However, not all of our customers have the ability to work with S3 storage due to old hardware/software. We tried to work around the ...
1
vote
0answers
85 views

Subdomain pointing to deleted S3 bucket [duplicate]

I am stuck in my present (AP-SOUTH) and deleted (US-EAST) S3 bucket. I had mistakenly created a bucket in US-EAST (cdn.example.com.s3-website-us-east-1.amazonaws.com) and mapped my subdomain - cdn....
4
votes
1answer
8k views

Automatically mount bucket with s3fs on boot

I use an Amazon S3 bucket to deliver some of my server's content. I was able to mount it successfully, and grant Apache rights over it, but can't get it mounted properly at reboot. I updated my /etc/...
1
vote
0answers
394 views

rpc.mountd[pid]: /xxx and /yyy have same filehandle for host, using first

I have an S3 bucket mounted using s3fs-fuse at /yyy and a normal drive mounted at /xxx. I am exporting both with NFS however when a client attempts mount, I see the following error: rpc.mountd[pid]: /...
-1
votes
2answers
4k views

AWS S3 no space left on device

I am mounting a S3 bucket on a amazon EC2 instance using S3FS. Things have been working great for now over 18 months. Today I've tried to download a 5GB file using wget directly on the S3 bucket (...
1
vote
1answer
631 views

Best manner to access files in S3 Bucket from EC2 instance

I have tried several packages (goofys, s3fs, fuse) and have not been able to find a solution to my problem. I have an S3 Bucket with several thousand files currently in it and will continue to grow. ...
1
vote
1answer
2k views

Custom file/folder permissions with S3FS?

I have setup an SFTP server using OpenSSH. The home directory for users is /sftp/%user. I'm mounting an S3 bucket at /sftp using S3FS. The problem is that S3FS cascades user permissions down through ...
1
vote
0answers
499 views

Root user permission issues with SFTP and S3FS on Amazon EC2

I have setup an SFTP server using OpenSSH. The home directory for users is /sftp/%user. I'm mounting an S3 bucket at /sftp using S3FS. The problem is that S3FS cascades user permissions down through ...
4
votes
0answers
2k views

Configure SFTP with OpenSSH and an AWS S3 Bucket mounted via S3FS on Amazon EC2

How do I allow multiple SFTP Users with S3FS and OpenSSH? Everything works, except SFTP Users don't have permission to write to their Chrooted Home Directory: remote open("/some_file"): Permission ...
10
votes
2answers
6k views

How to load balance sftp instances on AWS

I like to know is it possible to load balance sftp servers in AWS. I have 2 servers, and each of my servers are using s3fs-fuse to mount the same S3 bucket onto a mount point. Both of my ec2 ...
0
votes
0answers
627 views

S3FS and Fuse, scale out or scale up

I have an infra where I have multiple machines (eg FTP servers, get/put and Samba/NFS servers) that need to share out the same (HA) storage. So I figured to use s3fs and fuse. I got this all setup and ...
0
votes
1answer
163 views

Automate s3fs-fuse install

I'm trying to write script to install s3fs-fuse, but i don't know how to change directory in a bash script. These are the script I want to automate git clone https://github.com/s3fs-fuse/s3fs-fuse....
0
votes
1answer
644 views

Is S3 a proper choice to keep live linux user home directories?

We are building an RDP cluster using Amazon EC2/S3. Users' home directories are on S3 (single bucket) and all EC2 nodes mount the same S3 bucket using S3FS. Each EC2 node uses the appropriate home ...
5
votes
3answers
3k views

Replicated shared filesystem

I'm looking into setting up a shared filesystem/file server on AWS (EC2) infrastructure that offers replication and fairly painless failover. This filesystem would host potentially millions of files ...
3
votes
2answers
5k views

s3fs “could not determine how to establish security credentials”

I'm trying to use s3fs to mount an S3 bucket on to a standard AWS Amazon Linux AMI (with all the necessary dependencies installed). However when following this tutorial when I run: s3fs mybucketname -...
0
votes
1answer
638 views

Does linux count s3fs mounts as local disk usage?

I have an s3 bucket mounted on an EC2 instance whose main disk is an EBS volume. Whenever I run something like du -h /path/to/mount it tells me the size of the folders within those. I'm wondering ...
0
votes
1answer
1k views

AWS: Best solution for syncing files across autoscaled EC2 instances?

I'm working on setting up autoscaling on EC2; Instances will be running a web app with big traffic spikes. The first solution I tried for syncing files across instances (since the web app will write ...
1
vote
0answers
474 views

Amazon S3 sync theory

I'm setting up a backup of my vital data (database and images, mainly) to S3 from my Ubuntu box. I've set up Amazon S3, and installed S3FS to mount the bucket on my machine. This is working well, but ...
5
votes
1answer
2k views

Linux server sync to an Amazon S3 bucket

I am looking for a stable solution to replace a classic server backup to another server using rsync. I have to sync a whole filesystem (more than 1Tb) to Amazon S3. Where am I? Solution 1: I mapped ...
1
vote
1answer
829 views

Fuse module for s3fs

I am trying to install the s3fs software. I however keep coming upon this error: $ sudo /usr/bin/s3fs bucketname /mnt fuse: device not found, try 'modprobe fuse' first $ modprobe fuse FATAL: Module ...
0
votes
1answer
175 views

autoscale static content on ec2 using local cache

Hello Serverfault members. I want to achieve the following setup on EC2: A load balancer starts and stops instances which only serve static content. (a javascript based application) I do not want to ...
1
vote
1answer
365 views

What's the best way to enforce a maximum size on a file directory?

I'd like to create directory with a maximum size of, let's say 500MB. The directory may contain any number and depth of subdirectories. When the maximum size limit of the directory is surpassed, I ...
0
votes
1answer
149 views

Subsequent runs of rsync locally don't reduce data transferred

I have an EC2 instance with data I want to sync to a mounted, but remote, volume, as a backup. rsync seems like the way to go with this, so as a test I took my test file (a Postgres pg_dump file) and ...
0
votes
1answer
480 views

configure s3fs mount for write-only

I am using s3fs to mount a S3 buckets on an Ubunutu server to store my daily backups there. Considering security this would allow a potential hacker to delete all my backups from the mounted bucket. ...
7
votes
4answers
13k views

How to make s3fs work with IAM roles?

We are using IAM roles to maintain the machines. We now planning to mount the s3 to our ec2 instances and do the processing as per our need. We been using s3fs tool for mounting. But that doesn't ...
1
vote
2answers
3k views

Can a S3 mount be used as the document root for Apache?

Has anyone been successful in having their DocumentRoot reside on an S3 mount (using s3fs)? I currently have a mounted bucket at /mnt/s3. I can read and write files to it no problem. In my httpd....
1
vote
4answers
636 views

Shared encrypted filesystem for ec2 instances

Me and my company intend to start using ec2 for distributing some cpu-intensive pieces code, involving a few 10s of gb of data, however I have some concern about the protection of the intellectual ...
1
vote
1answer
959 views

Getting errors running mount on Amazon Linux after installing s3fs (`MOUNT_2.22' not found required by mount)

Followed these instructions to install s3fs on Amazon Linux. It worked fine on one of my systems, however on a different system (following the same instructions) I am getting the following errors ...
2
votes
0answers
471 views

s3fs-mounted S3 bucket shows bad filesizes

Background: I have an Amazon S3 bucket with data uploaded using the AWS web interface. Now I need to give permissions-based access to paths within my buckets to FTP users. To do this, I'm running ...
13
votes
7answers
23k views

S3fs performance improvements or alternative?

I've been trying to use s3fs on an EC2 server but it's really slow. I spent the last 5 hours uploading 100MB of small files. Is there anything I can do to improve the performance or is it something ...
2
votes
2answers
4k views

s3fs changing s3 permissions?

My developer believes that s3fs is changing my bucket's permissions. Is this possible? I want my bucket to be public, but it keeps reverting back to private. Here's my fstab. s3fs#production /mnt/...
2
votes
3answers
3k views

How to list s3fs mounts?

How can I list the s3fs mounts that exist on an ubuntu system? I'd like to know to which bucket each mount is mapped. Specifically, I have a specific mount (e.g. ~/s3/mymount), and would like to know ...
9
votes
2answers
4k views

EC2 - Shared Storage - S3FS or EBS?

Building my web service onto EC2 right now and have a single instance behind a load balancer. I will of course cater for multiple instances. My initial idea was to run all the instances a dumb slaves,...
1
vote
1answer
2k views

Mounting an S3 bucket onto a AWS Ubuntu instance issues

i'm using s3fuse to mount an s3 bucket onto ubuntu, so that i can take advantage of the storage space. here are some environment info: System Info: Linux ip-xxx-xxx-xxx-xxx 2.6.38-8-virtual #42-...
2
votes
1answer
2k views

Writing to an s3 mounted directory

hopefully someone can advise. I've recently mounted an s3 bucket to a directory /s3 on my ubuntu amazon ec2 instance -following: http://code.google.com/p/s3fs/wiki/FuseOverAmazon. This was set up in ...
5
votes
2answers
1k views

S3 based file system capable of requesting only part of file

I'm storing large datasets in s3, but on a given computer in my cluster, my program only needs to read a small subset of the data. I first tried s3fs, but it downlooads the entire file first, which ...