Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Help me serverfault!

I am looking for a high IO (2 million mostly put transactions per day of 7-10k xml chunks) storage solution that is amazon s3 compatible that I can house in my datacenter.

I am planning to house the data on a ZFS appliance but need something that will be able to handle the high IO needs we have.

I have looked at Eucalyptus (Walrus), but it seems like they are not going to be able to handle our throughput.

Anyone have any other suggestions of things I might be able to look into?

Thanks!

share|improve this question
    
Amazon EBS allows IOPS provisioning , don't know if it's exactly what are you looking for but worth give it a look –  eldblz Aug 26 '13 at 14:36
    
Have you looked at Ceph? –  EEAA Aug 26 '13 at 16:40
    
"I have looked at Eucalyptus (Walrus), but it seems like they are not going to be able to handle our throughput." Based on what? Eucalyptus Walrus should have no trouble with this sort of use case, assuming you provision enough servers. –  ceejayoz Aug 26 '13 at 18:07
add comment

2 Answers

Riak CS is worth looking into. It's S3-compatible, open source so you can try it for free, and you can get commercial support through Basho. I haven't tried it but Riak has been stable in my use case.

share|improve this answer
add comment

I see you've said there is an average of 235 IOPS per file, but I'm unclear from you specifications how large are these xml datasets? You'll want to evaluate thru-put along with IOPS.

What is the time-frame you receive this batch data? You'll have to architect your system to meet your maximum capacity; but if you can throttle your data, you can have a less robust system.

Are you hoping to push to S3 or are you pulling?

What is business cost in dollars for unavailable storage? If its a science experiment, I agree take a look a Ceph, but if you need guaranteed IOPS and thru-put under variable write conditions I'd stay away from all the open-source object file systems unless you have direct experience running one in a such a production environment.

On the surface your need appears to me (assuming this is business, that you have a ow tolerance for downtime and assuming you need to absorb high IOPS, and assuming variable file sizes) that you are going to want enterprise storage.

share|improve this answer
    
Welcome to Server Fault! This is really a comment, not an answer. With a bit more rep, you will be able to post comments. –  slm Sep 22 '13 at 4:19
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.