Hot answers tagged

22

My comment could probably use some clarification. I spouted off with the eloquence of an inebriated yak: I have never wanted to set myself on fire so much as I do now. Why? Why would I say such a thing? Mostly because I'm an awful person. However, aside from that, I can explain my outburst by going over the original post piecemeal: I like to know is ...


9

The command that worked was: umount -l /mnt/s3


8

Is S3 a proper choice to keep live linux user home directories? Amazon has announced their EFS service, which is exactly what you need for this. Either use EFS or roll your own NFS server for home directories.


8

This worked for me iam_role=auto Here is my /etc/fstab entry s3fs#my_bucket /s3mount_path fuse _netdev,allow_other,iam_role=auto 0 0 Just make sure you IAM role has proper permissions


6

I just released v0.0.1 of https://github.com/kahing/goofys which was partly motivated by performance problems in s3fs. File creation speedup is 3-6x and time to first byte is 58x. Feedback welcome!


5

There seems to be an iam_role option in the man page: iam_role (default is no role) - set the IAM Role that will supply the credentials from the instance meta-data. A cursory overview of the source suggests that it will get the token and rotate as they expire... also, see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-...


5

I just benchmarked riofs compared to s3fs. My test case was a relatively simple bash script that ran pngquant on each .png it found. On a testbucket with ~70 images of which ~20 where png (spread over a lot of subdirectories, which prob slows things down) the results where: s3fs: 3m54 riofs: 15.9s So for this testcase riofs is ~15x faster! Setup was also ...


5

You want to add _netdev to your fstab: s3fs#my-bucket-name /mnt/s3_bucket fuse _netdev,allow_other,umask=227,uid=33,gid=33,use_cache=/root/cache 0 0


4

It is possible, though not trivial, to setup NFS Clusters in Amazon EC2 using DRBD for synchronous replication and Pacemaker + Corosync for automating failover of the NFS service and exports between nodes (without interrupting client access). If you're planning on replicating synchronously ("real-time"), you'll need both your EC2 instances to be in the ...


4

I'd like to know to which bucket each mount is mapped. # ps -ef | grep s3fs root 9273 1 0 Feb12 ? 00:34:52 s3fs bucket1 /mnt/p1 root 23130 1 0 Feb18 ? 00:53:10 s3fs bucket2 /mnt/p2


4

You can use lsof to figure out which files are still open.


3

https://github.com/kahing/goofys Goofys allows you to mount an S3 bucket as a filey system. It's a Filey System instead of a File System because goofys strives for performance first and POSIX second. Particularly things that are difficult to support on S3 or would translate into more than one round-trip would either fail (random writes) or faked (no per-...


3

Try: s3fs#production /mnt/production fuse use_cache=/tmp,use_rrs=1,allow_other,uid=1000,gid=1000,default_acl=public-read 0 0 I have the same problem on dynamic files (bootstrap), although static files work well once the permissions are set. But the above solutions seem to work for others.


3

Solved the problem. By default s3fs blocks other users from accessing the bucket. Had to make two changes: Uncomment user_allow_other in /etc/fuse.conf While mounting, add -o allow_other in the command s3fs bucketname /local/path -ourl=https://s3.amazonaws.com -o allow_other


2

You can view s3 mounts with the mount command: [user@hostname ~]$ mount | grep ^s3fs s3fs on /mnt/point type fuse.s3fs (rw,relatime,user_id=0,group_id=0) If you don't see it here it is probably not mounted. A related item you will see in the mount output is fusectl: [user@hostname ~]$ mount | grep ^fusectl fusectl on /sys/fs/fuse/connections type ...


2

on ec2, i would sync across instances via pki and rsync, as that would be way faster than s3. another option, if there is a huge amount of data, is to rsync to an ebs mount, and then unmount, detach, re-attach on next instance, mount, rsync, unmount, detach, next instance,.... etc passing an ebs volume around for local rsyncing is pretty easy to script ...


2

This is a bad way to do backups. You should be separating your OS configuration from your valuable data. None of your permissions will be transferred, which in the Linux world are a necessity if you're planning on restoring backups (which you should be - backups without verified restorations are pointless). Firstly, you can synchronise your valuable ...


2

Maybe a little late to the party but since majority of the answers are very old; just wanted to share the command that works right now :- So, to manually mount the s3 bucket using an IAM role you will need to fire the following command :- $ sudo s3fs <bucket-name> /<folder-path>/<folder-name> -o iam_role="<Role-name>" -o url=...


2

I found the best way to resolve this was to use the -o iam_role option to pass the name of the instance role in, plus if you're not in the us-east-1 region, you may also need to specify the url and endpoint options. E.g. s3fs mybucketname -o allow_other myfolder -o iam_role=$iam_role_name -o endpoint=${aws_region} -o url=https://s3-${aws_region} (This is ...


2

That's how you do it -- when the script ends, the current directory reverts back to what it was when the script starts, so perhaps that's why you don't think it's working. Add set -e to the beginning of the script. Otherwise, if any of those commands return an error, the script will just keep on running, moving on to the next command, and that is ...


2

Given the complexity of setting up a replicated NFS server, we're opting to go with S3. The performance of s3fs-fuze was terrible (doing an ls on a directory with over 1,000 files would take close to a minute due to it needing to query metadata for each file, and caching didn't seem to help). However, I then tried out RioFS, which provided me with instant ...


2

EFS performance depends on how much data you have on the volume. The more you store the higher is the performance. That’s probably why with just 1.3GB it’s slow. You can however pay for EFS provisioned IOPS that will increase the performance for an extra cost. Alternatively you can simply store a couple of big files (e.g. 10x 50GB) to increase the volume-...


2

Solved it on my own, the problem was that if you are not using the default server location, try looking at this page: https://wasabi-support.zendesk.com/hc/en-us/articles/360015106031-What-are-the-service-URLs-for-Wasabi-s-different-regions- replace the url with your server's location Fixed my problem!


1

According to this How to upload file directly to Amazon S3 from a remote server, you will have to download the file to local server before uploading it to S3 bucket. Since you have not defined the save path, I afraid you are storing the large file locally at /tmp folder in which you don't have enough space for storing the 5 GB file. You can further refer ...


1

It can easily be done via any of AWS SDK's. You need to do the following: Retreive a list of all objects located in S3 bucket. Filter the list according to your criteria using regular expressions. Dowload files using the filnal list of objects If you are using PHP, the following docs could be useful for you: http://docs.aws.amazon.com/aws-sdk-php/v3/api/...


1

Syntax: s3fs# fuse _netdev,allow_other,passwd_file=/home/ubuntu/.passwd-s3fs,use_cache=/tmp,umask=002,uid=1000 0 0 Example: s3fs#examples /home/ubuntu/s3bucket/examples fuse _netdev,allow_other,passwd_file=/home/ubuntu/.passwd-s3fs,use_cache=/tmp,umask=002,uid=1000 0 0


1

Just some updated information. If you are like me and you have wanted this functionality for a VERY, VERY long time, use Amazon Elastic File System (EFS). It is an NFS mount replicated across multiple availability zones. (Sorry to bump the issue, but the google rank of this answer is high enough that a few people probably are searching for this solution.)


1

Although this question is old, I also had the problem so I figure I would post the solution that worked for me in case another person has the same issue. sudo apt purge s3fs -y sudo apt update -y && sudo apt upgrade -y sudo apt install s3fs -y In the current users home directory, create a txt file with the name .passwd-s3fs with your IAM ...


1

du will query the file system for the size of each file within the specified directory. It will do so recursively through all sub directories. It will tell you the size on disk as reported by the underlying file system. Not every file system driver can report the size on disk, so on certain file system drivers, this is going to be inaccurate. If the ...


1

Yes you can do it, but it's not a simple way. You have to install fuse and s3fs from sources, then configure your S3 bucket and the policies for an IAM user, finaly you have to configure your mount point. Here are some links that may help you: https://www.emfluence.com/blog/amazon-web-services-integrating-s3-and-ec2-few-simple-steps https://github.com/...


Only top voted, non community-wiki answers of a minimum length are eligible