14

Has anyone come across a near 100% Amazon S3 RESTful API compatible object storage system?

What I am after is a layer that sit on top of any (preferably a POSIX) filesystem that provides Amazon S3 style RESTful API to store (PUT), retrieve (GET), stat (HEAD) and delete (DELETE), with decent authentication.

Commercial projects/ideas are welcome too.

NOTE:

I have so far tried Eucalyptus and Cumulus; of which Eucalyptus seems to be blindly calling itself S3-compatible. The response XML documents are not at all compatible, and is patchy in certain places with no XML documents at all. Cumulus has managed to keep the response documents quite similar, but seems to have forgotten about data integrity!

Let me explain the latter part: both Eucalyptus and Cumulus have no support for integrity verification that Amazon S3 offers. What you can do with S3 is that you can provide a Base64(MD5(FILE)) along with the PUT request, which then gets verified by S3 before they respond with a success. Eucalyptus and Cumulus doesn't support this. With, Eucalyptus we can at least get around this by checking the MD5 given in the response document (not S3-compatible behaviour). In Cumulus, this is not possible as it doesnt respond with anything (like S3). It is made even worse by Cumulus, by not giving out an ETag in the HEAD request.

10 Answers 10

3

Swift is OpenStack's object storage engine, and as of the Bexar release in Feb 2011, it claims an experimental S3-compatible middleware. As OpenStack is getting a lot of attention (Canonical are shifting Ubuntu to it over Eucalyptus this October, for example) it might be worth a look.

3

In the end, I had to move forward with Eucalyptus Walrus. Eventhough I ended up finding some fundamental issues with it, they all had possible work-arounds.

Thanks everyone for your valuable input!

2

[DISCLAIMER: I work for Scality]

Amazon stays very active on their AWS products and makes additions and changes/improvements to their API very frequently. It is then hard to find a near-100% compatible product (especially in commercial products, probably OSS products would be more reactive to any changes). Many object/cloud storage vendors have been toying around with their own protocols for some time (Atmos, DDN even OpenStack's swift..) but they've mostly all abandoned the idea of their own protocol becoming as widely used as S3. They are all (almost?) announcing an S3-compatible interface now or in the months to come. It is after all the industry de facto standard.The only contender left might be CDMI just because of its Standardized nature and the fact it comes from SNIA, a non-profit, non-vendor organization. But it's clearly not being adopted or talked about as much as S3.

There are a few options as far as S3-compatible interface goes, like Mezeo, the ones you've mentioned (Eucalyptus and Cumulus), OpenStack (although it's not out yet i believe). Scality is another one of those.

Scality's S3 implementation (called RS2 - REST Storage Service) is very close to S3 in that our requests/replies are consistent with what you'd get from S3. Some of the functionalities tied to S3's infrastructure are not present for obvious reasons (Region selection for instance). But the necessary commands are there: GET/PUT/DELETE an object, bucket creation and listing ; and react just like S3 would.

I would just add, to answer your second problem, that Scality's product, RING, is a software object storage platform that solves the data integrity issues you're referring to by checksumming and doing background integrity checks of all objects stored on the RING. Storing an object is basically a transactional query, and once an object has been claimed as "successfully stored", it will forever be, with constant integrity checks (on every read for example).

I won't go into details. you can go to our website to understand how we alleviate storage admins from the common data management nightmares and let them sleep at night knowing their data is available. Always. :)

Best, Marc Villemade @mastachand

2

Newdream's Ceph has a S3-compatible fastcgi gateway which uses the same object storage system as the Ceph Filesystem itself. (http://ceph.newdream.net/wiki/RADOS_Gateway) Still not production ready, but making regular updates.

I found Openstack's object store more reliable in my testing though not S3 compatible with the s3 libraries I used or really with rackspace's own java client for their cloudfiles storage service (which uses their own version of swift). Though the jclouds project which has both s3 and Swift support worked for me.

2

You could also look at Cloud Foundry vBlob: https://github.com/cloudfoundry/vblob

It's a node.js app (with Ruby wrapper for CF integration) that implements a fair chunk of the S3 protocol on top of any file system that the VM can "see".

1

Both s3ql (GNU GPL v3) and s3backer (GNU GPL v2) do what you're looking for.

1
  • 1
    Sean; I am not looking for a client library or a client too to interact with storage; more interested in the server itself.
    – CodeMedic
    Nov 25, 2011 at 15:10
1

Old / answered question, but https://github.com/basho/riak_cs was just open sourced at https://github.com/basho/riak_cs : "Riak CS is an object storage system built on top of Riak. It facilitates storing large objects in Riak and presents an S3-compatible interface. It also provides muti-tenancy features such as user accounts, authentication, access control mechanisms, and per account usage reporting."

1

I am sure you got so many options above still you should check OpenStack Swift, its a OpenSource Object Storage and also supports S3 compatible API. Its used as object storage solution for Rackspace, Hpcloud Korea Telecom and many others.

Documentation http://docs.openstack.org/developer/swift/associated_projects.html Swift3 https://github.com/fujita/swift3 Swift3 Middleware for OpenStack Swift, allowing access to OpenStack swift via the Amazon S3 API.

Hope it helps.

-3

[DISCLAIMER: I work for Cloudian]

Cloudian HyperStore object storage software is 100% Amazon S3 API compatible. It implements all the S3 APIs down to the error codes and all the features that Amazon S3 have.

To list some, Cloudian HyperStore supports Multi-part Upload, Object Versioning, S3 compatible ACL, Auto-tiering to S3 and Glacier, S3 Cross-Origin Resource Sharing (CORS), S3 encoding-type, location constraint and many more.

And yes with Cloudian you can provide a Base64(MD5(FILE)) along with the PUT request, which then gets verified before responding with a success.

For more information check this blog post http://www.cloudian.com/blog/?p=64 and Cloudian website at http://www.cloudian.com/products/cloudian-hyperstore.php

-4

Have you checked out s3fs? It doesn't feature the MD5 check that you want ( http://code.google.com/p/s3fs/issues/detail?id=37&q=md5 ) but I suppose it wouldn't be extremely difficult to implement this. Of note is that is is eventually consistent.

1
  • 2
    S3FS is the reverse order to what the poster wants. It makes a filesystem out of an S3 bucket. The OP wants to make a S3 bucket out of a filesystem.
    – crb
    Jul 12, 2011 at 12:08

Not the answer you're looking for? Browse other questions tagged .