Tuesday, December 17, 2013

Object Storage Gateway now in the 4.0 code base

At Eucalyptus, we are furiously working away on the next major release: Eucalyptus 4.0. One of the important features in 4.0 is an object storage proxy called the Object Storage Gateway. The OSG is the user facing component of Eucalyptus object storage and it implements the S3 API. The OSG is also responsible for user request authentication and Identity and Access Management (IAM) policy enforcement. You may have multiple instances of the OSG in Eucalyptus, one on each host, thus achieving scale out.

On the backend, the OSG can interact with Walrus, the single server legacy implementation, or with other distributed object stores like Riak/RiakCS, Ceph, Swift or with S3 itself. In 4.0, we will be testing with RiakCS, but support for other backends will be tested soon. In addition, we will be adding features to support a hybrid storage solution.

Code has now landed in the "testing" branch. You can get it here: https://github.com/eucalyptus/eucalyptus/tree/testing

Some notes:

  • Users will now need to register and configure at least one Object Storage Gateway (OSG). Instructions can be found here.
  • Users will need to pick the backend or "provider" that they want the OSG to use. This can be "walrus" or "s3".
  • The "Walrus" component is now internal only and used by the OSG for single machine setups.
  • We have tested integration with RiakCS as the backend. You can use your RiakCS installation if you pick "s3" as the backend.
  • Eucalyptus will need to know the access/secret key of the admin user for your RiakCS installation.
  • Eucalyptus does not install or configure Riak/RiakCS. That is left as an exercise to the reader. There are some simple install instructions here. You will need at least 5 nodes for a production setup, but fewer nodes are okay if you are just kicking the tires.
  • Not all S3 API features have been implemented for the OSG, such as logging and versioning. We are bringing these over from the legacy Walrus implementation.
  • This code base currently does not support data migration or conversion. So if you have an existing Walrus installation, your Walrus data will not be migrated, sorry!

Again, configuration instructions are available here if you missed the link above.

Please clone it, install it, test it and let us know what you think!

We are also planning on supporting multipart uploads, bucket lifecycle and bucket policies soon. Stay tuned.

For a list of additional features that are planned for 4.0, please see the Eucalyptus Roadmap.

1 comment: