Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote 2 down vote favorite
1

I've been trying to use s3fs on an EC2 server but it's really slow. I spent the last 5 hours uploading 100MB of small files.

Is there anything I can do to improve the performance or is it something related to S3fs? If not, what's an alternative I could use?

share|improve this question
define small files, 1K? 500bytes? 1MB? – thinice Jun 6 '12 at 14:21
around ~200K... – MB. Jun 6 '12 at 14:22

2 Answers

up vote 4 down vote accepted

S3FS may not be the best choice for a large amount of smaller files. The overhead with S3FS is pretty high as well. I'd suggest using something like S3Curl

You can even get parallel transfers going. Just remember it will never be fast like EBS / local storage.

If you need to have it as a 'mountable' storage, the only alternatives to S3FS that I know of is S3Backer or s3ql

share|improve this answer
I'm looking for something that I can mount as a filesystem. – MB. Jun 6 '12 at 14:27
Any thoughts on s3ql? – ewwhite Jun 6 '12 at 14:29
I'll try S3Backer and S3Sql. Thanks. – MB. Jun 6 '12 at 14:32
5 hours is really slow for ~500, 200KB files even with S3FS. Something else might be factoring in, I'm not aware of any S3FS tunable that would help – thinice Jun 6 '12 at 14:32
up vote 0 down vote

I used [1] and it's very good. It's lot of commands and makes accessing aws and s3 a lot easier.

  1. http://timkay.com/aws/
share|improve this answer
1  
Welcome to Server Fault! Whilst this may theoretically answer the question, it would be preferable to include the essential parts of the answer here, and provide the link for reference. – Scott Pack Oct 1 '12 at 2:27

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.