Sign up ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I've been trying to use s3fs on an EC2 server but it's really slow. I spent the last 5 hours uploading 100MB of small files.

Is there anything I can do to improve the performance or is it something related to S3fs? If not, what's an alternative I could use?

share|improve this question
define small files, 1K? 500bytes? 1MB? – thinice Jun 6 '12 at 14:21
around ~200K... – MB. Jun 6 '12 at 14:22
Depending on your storage needs, you may want to just use EBS, unless you need access from other systems simultaneously, in which case setting up a virtual nas with EBS may still be a better option. – Tracker1 Dec 20 '14 at 22:45

6 Answers 6

up vote 5 down vote accepted

S3FS may not be the best choice for a large amount of smaller files. The overhead with S3FS is pretty high as well. I'd suggest using something like S3Curl

You can even get parallel transfers going. Just remember it will never be fast like EBS / local storage.

If you need to have it as a 'mountable' storage, the only alternatives to S3FS that I know of is S3Backer or s3ql

share|improve this answer
I'm looking for something that I can mount as a filesystem. – MB. Jun 6 '12 at 14:27
Any thoughts on s3ql? – ewwhite Jun 6 '12 at 14:29
I'll try S3Backer and S3Sql. Thanks. – MB. Jun 6 '12 at 14:32
5 hours is really slow for ~500, 200KB files even with S3FS. Something else might be factoring in, I'm not aware of any S3FS tunable that would help – thinice Jun 6 '12 at 14:32

I just benchmarked riofs compared to s3fs. My test case was a relatively simple bash script that ran pngquant on each .png it found. On a testbucket with ~70 images of which ~20 where png (spread over a lot of subdirectories, which prob slows things down) the results where:

s3fs: 3m54
riofs: 15.9s

So for this testcase riofs is ~15x faster! Setup was also very straightforward, although the docs are somewhat terse.

Regarding the fact that the script still took 15.9s: it's not very efficient, also actually running pngquant on png's is a cpu intensive process.

share|improve this answer

I used [1] and it's very good. It's lot of commands and makes accessing aws and s3 a lot easier.

share|improve this answer
Welcome to Server Fault! Whilst this may theoretically answer the question, it would be preferable to include the essential parts of the answer here, and provide the link for reference. – Scott Pack Oct 1 '12 at 2:27

Just adding riofs to the mix. I could not see any benchmark for now but there is an issue/request that anybody can do.

share|improve this answer

I compared s3fs with riofs, and almost switched because of noticeably better performance. Problem is riofs is still not updated to v4 signatures, which means it won't work with Frankfurt or China s3 buckets. According to the owner, an update is on the way. If you don't use Frankfurt or China, riofs is a hands down better option.

share|improve this answer

I just released v0.0.1 of which was partly motivated by performance problems in s3fs. File creation speedup is 3-6x and time to first byte is 58x. Feedback welcome!

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.