Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I need to know what the best (read: highly performant and fault tolerant) filesystem solution based on Amazon's S3 is. I have looked a s3fs, but I notice that it has not been worked on for a while now, and that got me thinking about stability and features.

What other filesystem solutions have you used and what has your experience been with it?

share|improve this question
    

3 Answers 3

up vote 2 down vote accepted

You're not going to get anything that's "highly performant" (at least when compared to modern disk storage systems) when dealing with the latencies inherent with using remote storage. Performance is further hampered by the fact that the target "device" isn't a block device, which necessitates an abstraction layer that even further inhibits performance. S3 is at its best when used for what it was intended for, storing and retrieving individual files using http(s).

Regarding fault tolerance: here's an area where S3 excels. Standard S3 buckets will survive the failure of two of Amazon's facilities, which is quite good for what you're paying. If your needs aren't that stringent, they also recently released the slightly less redundant (but ~30% cheaper) RRS buckets, which are configured to survive the failure of one of Amazon's facilities.

share|improve this answer
    
Erik, thank you for sharing your opinion and thoughts. I should have qualified my "performant" requirement with the caveat that I understand a remote, abstracted "filesystem" will not meet the performance of a local FS. That being said, with the multitude of s3f2 solutions out there, have you used one, and if so, why did you choose that one over another. Bottom line, I will be implementing one, and I would like to know before I do, which one has the best track record. –  QWade Sep 26 '10 at 14:01
    
Understood. Yes, I've used this s3fs version: code.google.com/p/s3fs/wiki/FuseOverAmazon . It worked fairly well. One nice thing is that it supports disk cache, which can help speed up IO a fair amount. –  EEAA Sep 26 '10 at 17:21

Maybe S3QL will be interesting to you. I am not going to praise it here (because I'm the developer), but there is a comparison of S3QL to other S3 file systems at http://code.google.com/p/s3ql/wiki/other_s3_filesystems.

share|improve this answer
    
s3fs's file limit size is currently "only" 64GB, while with S3QL it is S3's own limit (2TB). –  mr-euro May 8 '12 at 21:50

Nasuni is a startup with a relatively low-cost cloud storage appliance. Worth looking into if you need to protect TBs of data on S3 simply.

p.s. I am not affiliated with the company, just a fan of the technology.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.