Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Has anyone come across a near 100% Amazon S3 RESTful API compatible object storage system?

What I am after is a layer that sit on top of any (preferably a POSIX) filesystem that provides Amazon S3 style RESTful API to store (PUT), retrieve (GET), stat (HEAD) and delete (DELETE), with decent authentication.

Commercial projects/ideas are welcome too.

NOTE:

I have so far tried Eucalyptus and Cumulus; of which Eucalyptus seems to be blindly calling itself S3-compatible. The response XML documents are not at all compatible, and is patchy in certain places with no XML documents at all. Cumulus has managed to keep the response documents quite similar, but seems to have forgotten about data integrity!

Let me explain the latter part: both Eucalyptus and Cumulus have no support for integrity verification that Amazon S3 offers. What you can do with S3 is that you can provide a Base64(MD5(FILE)) along with the PUT request, which then gets verified by S3 before they respond with a success. Eucalyptus and Cumulus doesn't support this. With, Eucalyptus we can at least get around this by checking the MD5 given in the response document (not S3-compatible behaviour). In Cumulus, this is not possible as it doesnt respond with anything (like S3). It is made even worse by Cumulus, by not giving out an ETag in the HEAD request.

share|improve this question

closed as off-topic by EEAA, Michael Hampton Jun 1 at 17:25

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions seeking product, service, or learning material recommendations are off-topic because they tend to become obsolete quickly. Instead, describe your situation and the specific problem you're trying to solve." – EEAA, Michael Hampton
If this question can be reworded to fit the rules in the help center, please edit the question.

add comment

11 Answers

Swift is OpenStack's object storage engine, and as of the Bexar release in Feb 2011, it claims an experimental S3-compatible middleware. As OpenStack is getting a lot of attention (Canonical are shifting Ubuntu to it over Eucalyptus this October, for example) it might be worth a look.

share|improve this answer
add comment
up vote 3 down vote accepted

In the end, I had to move forward with Eucalyptus Walrus. Eventhough I ended up finding some fundamental issues with it, they all had possible work-arounds.

Thanks everyone for your valuable input!

share|improve this answer
add comment

[DISCLAIMER: I work for Scality]

Amazon stays very active on their AWS products and makes additions and changes/improvements to their API very frequently. It is then hard to find a near-100% compatible product (especially in commercial products, probably OSS products would be more reactive to any changes). Many object/cloud storage vendors have been toying around with their own protocols for some time (Atmos, DDN even OpenStack's swift..) but they've mostly all abandoned the idea of their own protocol becoming as widely used as S3. They are all (almost?) announcing an S3-compatible interface now or in the months to come. It is after all the industry de facto standard.The only contender left might be CDMI just because of its Standardized nature and the fact it comes from SNIA, a non-profit, non-vendor organization. But it's clearly not being adopted or talked about as much as S3.

There are a few options as far as S3-compatible interface goes, like Mezeo, the ones you've mentioned (Eucalyptus and Cumulus), OpenStack (although it's not out yet i believe). Scality is another one of those.

Scality's S3 implementation (called RS2 - REST Storage Service) is very close to S3 in that our requests/replies are consistent with what you'd get from S3. Some of the functionalities tied to S3's infrastructure are not present for obvious reasons (Region selection for instance). But the necessary commands are there: GET/PUT/DELETE an object, bucket creation and listing ; and react just like S3 would.

I would just add, to answer your second problem, that Scality's product, RING, is a software object storage platform that solves the data integrity issues you're referring to by checksumming and doing background integrity checks of all objects stored on the RING. Storing an object is basically a transactional query, and once an object has been claimed as "successfully stored", it will forever be, with constant integrity checks (on every read for example).

I won't go into details. you can go to our website to understand how we alleviate storage admins from the common data management nightmares and let them sleep at night knowing their data is available. Always. :)

Best, Marc Villemade @mastachand

share|improve this answer
add comment

Newdream's Ceph has a S3-compatible fastcgi gateway which uses the same object storage system as the Ceph Filesystem itself. (http://ceph.newdream.net/wiki/RADOS_Gateway) Still not production ready, but making regular updates.

I found Openstack's object store more reliable in my testing though not S3 compatible with the s3 libraries I used or really with rackspace's own java client for their cloudfiles storage service (which uses their own version of swift). Though the jclouds project which has both s3 and Swift support worked for me.

share|improve this answer
add comment

You could also look at Cloud Foundry vBlob: https://github.com/cloudfoundry/vblob

It's a node.js app (with Ruby wrapper for CF integration) that implements a fair chunk of the S3 protocol on top of any file system that the VM can "see".

share|improve this answer
add comment

SMEStorage provide a compatible S3 API layer on top of any Cloud you choose, even commercial clouds such as Google Drive, DropBox, SkyDrive, or FTP servers even. There is more on their Wiki and on a blog post here. Seems it can be used through their SaaS service even, or can be sited in your DC or premises as a VMWARE appliance.

share|improve this answer
add comment

Old / answered question, but https://github.com/basho/riak_cs was just open sourced at https://github.com/basho/riak_cs : "Riak CS is an object storage system built on top of Riak. It facilitates storing large objects in Riak and presents an S3-compatible interface. It also provides muti-tenancy features such as user accounts, authentication, access control mechanisms, and per account usage reporting."

share|improve this answer
add comment

I am sure you got so many options above still you should check OpenStack Swift, its a OpenSource Object Storage and also supports S3 compatible API. Its used as object storage solution for Rackspace, Hpcloud Korea Telecom and many others.

Documentation http://docs.openstack.org/developer/swift/associated_projects.html Swift3 https://github.com/fujita/swift3 Swift3 Middleware for OpenStack Swift, allowing access to OpenStack swift via the Amazon S3 API.

Hope it helps.

share|improve this answer
add comment

Both s3ql (GNU GPL v3) and s3backer (GNU GPL v2) do what you're looking for.

share|improve this answer
    
Sean; I am not looking for a client library or a client too to interact with storage; more interested in the server itself. –  CodeMedic Nov 25 '11 at 15:10
add comment

[DISCLAIMER: I work for Cloudian]

Cloudian HyperStore object storage software is 100% Amazon S3 API compatible. It implements all the S3 APIs down to the error codes and all the features that Amazon S3 have.

To list some, Cloudian HyperStore supports Multi-part Upload, Object Versioning, S3 compatible ACL, Auto-tiering to S3 and Glacier, S3 Cross-Origin Resource Sharing (CORS), S3 encoding-type, location constraint and many more.

And yes with Cloudian you can provide a Base64(MD5(FILE)) along with the PUT request, which then gets verified before responding with a success.

For more information check this blog post http://www.cloudian.com/blog/?p=64 and Cloudian website at http://www.cloudian.com/products/cloudian-hyperstore.php

share|improve this answer
add comment

Have you checked out s3fs? It doesn't feature the MD5 check that you want ( http://code.google.com/p/s3fs/issues/detail?id=37&q=md5 ) but I suppose it wouldn't be extremely difficult to implement this. Of note is that is is eventually consistent.

share|improve this answer
1  
S3FS is the reverse order to what the poster wants. It makes a filesystem out of an S3 bucket. The OP wants to make a S3 bucket out of a filesystem. –  crb Jul 12 '11 at 12:08
add comment

protected by EEAA Jun 1 at 17:02

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.