Monday, January 14, 2013

Eucalyptus brought to you by Eucalyptus

This is an article about using Eucalyptus as a continuous delivery framework for automating software build and testing. At Eucalyptus, we use Eucalyptus itself to build and package Eucalyptus, as well as perform basic automated testing against Eucalyptus.

Eucalyptus as a continuous delivery framework

Eucalyptus is an open architecture and an implementation of Infrastructure-as-a-service (IaaS) cloud computing. It allows flexible, scalable and dynamic provisioning of compute and storage resources programmatically on behalf of users and their applications. The flexible and dynamic nature of the Eucalyptus platform makes it a suitable candidate for a variety of use cases where user applications need to scale as needed, based on demand.

One such use case is continuous delivery of software from a developer's desktop to packaged software that can install on hundreds of servers...for instance, Eucalyptus! With Jenkins, Git, Eucalyptus and some scripting, it is possible to set up a framework to package software from multiple development branches so that an end user is able to deploy it and quickly test features and bug fixes. In addition, a quality engineering organization can leverage Eucalyptus to run automated test suites against software as soon as it is built. 

If quality tests pass, packages can be "promoted" so that the end user has a level of confidence in the software they are installing. Software build, packaging and QA never stops. As opposed to a traditional model where one must wait for all code to be "ready" before packaging and QA work can start, continuous delivery and QA allow us to potentially release software at any point in the development cycle. Software packaging is no longer a bottleneck to software delivery. 

From "git commit" to tested software, automatically, in a matter of minutes.

Eucalyptus packaging development and management

Release Engineering @ Eucalyptus is tasked with creating and maintaining Eucalyptus software packages and repositories. Release engineers design the package installation process and develop packaging scripts that will automatically install the hundred or so pieces of third party software that Eucalyptus depends upon in the correct order. In addition, release engineers maintain the infrastructure necessary for this effort. For this purpose, we use our very own Eucalyptus cloud.

Build and Test Automation

Eucalyptus engineering works on two releases at any given point in time: A maintenance release to fix issues for users who are currently running Eucalyptus and the next development or feature release.

The Eucalyptus release engineering process has continuous delivery as a principle: Automatically build and test software packages whenever a developer commits code to the mainline branch. Software packages are always available in a state that they can be installed. When the decision is made to release the software as an "official" supported released, packages can be promoted to final release status.

Daily commits are available as nightly builds that end users can install to test out bug fixes or new features as they are being developed.

Our Eucalyptus cloud is a single front end installation with 14 nodes.

Eucalyptus User Console connected to Release Engineering cloud

Images registered and compute resources available for package build

We use a EC2 plugin with Jenkins to provision instances in Eucalyptus as worker nodes for Jenkins. The plugin is "tweaked" slightly so that instances are not terminated if they are idle to speed up start up time. This modification can be turned off if the Eucalyptus scheduler is configured to suspend nodes when idle.

Jenkins configured to deploy to Eucalyptus

A top level Jenkins project is configured to monitor commits to mainline Eucalyptus code repositories. This project, in turn, start a number of "downstream" tasks, which include building Eucalyptus from source, generating RPM packages, creating package repositories and triggering quality tests that will use packages from repositories.

Jenkins project set up to monitor the Eucalyptus git repository
Jenkins project set up to build Eucalyptus RPM packages after Eucalyptus binaries are successfully built from source

Jenkins project set up to create package repository, so that programs like "yum" can install Eucalyptus on Linux

RPM package build jobs running on Eucalyptus compute nodes

Eucalyptus open source and plugin packages are made available at a staging area for quality testing. In addition, product documentation builds are also continuously available.

Eucalyptus package repository staging area

A number of past builds are retained, so that packages corresponding to a specific commit hash can be accessed programatically.

Packages corresponding to several git commit hashes are retained and can be retrieved


When package repositories are created, another downstream project triggers QA tests against the software. The quality engineering team at Eucalyptus maintains an in-house QA system that runs a number of "test units" to verify functionality. The QA system has an API that is triggered via Jenkins.

Eucalyptus QA triggered by downstream Jenkins project after packages have been built

The Eucalyptus QA system installs Eucalyptus from package repositories on top of a Linux distribution on bare metal and runs Eucalyptus through its paces.

Eucalyptus QA in action!

The Future: Continuous Deployment & Rolling Upgrades

The Eucalyptus cloud that builds packages, product documentation and QA tests is upgraded whenever the next stable release of Eucalyptus is published. With continuous deployment, we would have the option of automatically upgrading our Eucalyptus installation to the latest tested code base. This requires a high degree of confidence in automated QA, an installation process that is future proof and the ability to upgrade Eucalyptus without service interruptions. Rolling upgrades would enable us to update a Eucalyptus installation without downtime. We are not very far from this goal, but some kinks in the software are yet to be worked out.


  1. Nice blog here man. Keep them coming.

    Just curious - do you envision Walrus being integrated? For example, using it to host the past builds ( using something like this plugin - - to access the yum packages)?


    Harold Spencer, Jr.

  2. Hi Harold,

    Thanks for the comment and the feedback!

    Yes, perhaps it makes sense to integrate Walrus as a backing store for yum repos.

    Currently, we are using a NFS share that is accessible to package builder instances. Using IAM and Walrus will avoid the need for maintaining a NFS share.

    I haven't tried that plugin. I'll give it a try. We will also need a way for instances to securely upload to Walrus once packages are built. NFS was just a lot more convenient.

    Dog fooding is a side effect of what we are trying to achieve. The primary use case was having dynamic compute available, without having to reserve a bunch of physical hosts a priori. Packaging tends to be a bursty workload. The same hosts are also used for building docs, running static analysis jobs against the code, etc.


  3. Great post, Neil. Thanks for giving us a peek at the process and tools used to deliver Euca.

    You mentioned that the Jenkins EC2 plugin required a "tweak". Could you elaborate on what the tweak entailed?

    - Colby

    1. Hi Colby,

      The tweak involved just getting rid of the logic that terminates instances that have been running for over 30 min. It matters when you are using EC2 because you get billed for it.

      We like to keep the instances around because we have capacity. As I mentioned, if you want to do power aware scheduling, it is probably a good idea to make this timeout configurable and not disable it, so that a NC with no running instances can be suspended.

      Hope that helps