13

I've been trying to use s3fs on an EC2 server but it's really slow. I spent the last 5 hours uploading 100MB of small files.

Is there anything I can do to improve the performance or is it something related to S3fs? If not, what's an alternative I could use?

3
  • define small files, 1K? 500bytes? 1MB? – thinice Jun 6 '12 at 14:21
  • around ~200K... – MB. Jun 6 '12 at 14:22
  • Depending on your storage needs, you may want to just use EBS, unless you need access from other systems simultaneously, in which case setting up a virtual nas with EBS may still be a better option. – Tracker1 Dec 20 '14 at 22:45
10

S3FS may not be the best choice for a large amount of smaller files. The overhead with S3FS is pretty high as well. I'd suggest using something like S3Curl

You can even get parallel transfers going. Just remember it will never be fast like EBS / local storage.

If you need to have it as a 'mountable' storage, the only alternatives to S3FS that I know of is S3Backer or s3ql

4
  • 1
    I'm looking for something that I can mount as a filesystem. – MB. Jun 6 '12 at 14:27
  • Any thoughts on s3ql? – ewwhite Jun 6 '12 at 14:29
  • I'll try S3Backer and S3Sql. Thanks. – MB. Jun 6 '12 at 14:32
  • 5 hours is really slow for ~500, 200KB files even with S3FS. Something else might be factoring in, I'm not aware of any S3FS tunable that would help – thinice Jun 6 '12 at 14:32
6

I just benchmarked riofs compared to s3fs. My test case was a relatively simple bash script that ran pngquant on each .png it found. On a testbucket with ~70 images of which ~20 where png (spread over a lot of subdirectories, which prob slows things down) the results where:

s3fs: 3m54
riofs: 15.9s

So for this testcase riofs is ~15x faster! Setup was also very straightforward, although the docs are somewhat terse.

Regarding the fact that the script still took 15.9s: it's not very efficient, also actually running pngquant on png's is a cpu intensive process.

1
  • Unfortunately riofs fails with AWS message: The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256. – lrkwz Jan 23 '17 at 16:56
4

I just released v0.0.1 of https://github.com/kahing/goofys which was partly motivated by performance problems in s3fs. File creation speedup is 3-6x and time to first byte is 58x. Feedback welcome!

2
  • 3
    As of 2019 - Goofys is the recommended choice. Riotfs hasn't been updated in a while. – timofey.com Sep 30 '19 at 1:38
  • 2021 checking in - Goofys is incredible. According to my droplet graphs I've gone from 20% CPU to 1% CPU, load is now negligible, mem usage is flat at 20% (instead of spiking to 80%), and bandwidth is down from a sustained 1Gbps to under 500 kbps. I don't understand it, but the same workload is running and average operation times dropped from 6.5 seconds to 1.8. And the worst case is 10 vs 110. These are small random reads in very large files. – Aaron McMillin May 12 at 17:39
2

Just adding riofs to the mix. I could not see any benchmark for now but there is an issue/request that anybody can do.

1

I used [1] and it's very good. It's lot of commands and makes accessing aws and s3 a lot easier.

  1. http://timkay.com/aws/
1
  • 1
    Welcome to Server Fault! Whilst this may theoretically answer the question, it would be preferable to include the essential parts of the answer here, and provide the link for reference. – Scott Pack Oct 1 '12 at 2:27
1

I compared s3fs with riofs, and almost switched because of noticeably better performance. Problem is riofs is still not updated to v4 signatures, which means it won't work with Frankfurt or China s3 buckets. According to the owner, an update is on the way. If you don't use Frankfurt or China, riofs is a hands down better option.

1
  • Does not work from eu-central-1 either – lrkwz Jan 23 '17 at 16:57
1

This is kind old but may help who gets here.

I'm using aws cli and its working very good (3 Mb/s)

pip install awscli --upgrade --user
nano ~/.bash_profile
export PATH="~/.local/bin:$PATH"
aws s3 sync /sourcedir/ s3://bucketname/destdir/
1

https://github.com/kahing/goofys

Goofys allows you to mount an S3 bucket as a filey system.

It's a Filey System instead of a File System because goofys strives for performance first and POSIX second. Particularly things that are difficult to support on S3 or would translate into more than one round-trip would either fail (random writes) or faked (no per-file permission). Goofys does not have an on disk data cache (checkout catfs), and consistency model is close-to-open.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.