This is an article about using Eucalyptus as a continuous delivery framework for automating software build and testing. At Eucalyptus, we use Eucalyptus itself to build and package Eucalyptus, as well as perform basic automated testing against Eucalyptus.
Eucalyptus as a continuous delivery frameworkEucalyptus is an open architecture and an implementation of Infrastructure-as-a-service (IaaS) cloud computing. It allows flexible, scalable and dynamic provisioning of compute and storage resources programmatically on behalf of users and their applications. The flexible and dynamic nature of the Eucalyptus platform makes it a suitable candidate for a variety of use cases where user applications need to scale as needed, based on demand.
One such use case is continuous delivery of software from a developer's desktop to packaged software that can install on hundreds of servers...for instance, Eucalyptus! With Jenkins, Git, Eucalyptus and some scripting, it is possible to set up a framework to package software from multiple development branches so that an end user is able to deploy it and quickly test features and bug fixes. In addition, a quality engineering organization can leverage Eucalyptus to run automated test suites against software as soon as it is built.
If quality tests pass, packages can be "promoted" so that the end user has a level of confidence in the software they are installing. Software build, packaging and QA never stops. As opposed to a traditional model where one must wait for all code to be "ready" before packaging and QA work can start, continuous delivery and QA allow us to potentially release software at any point in the development cycle. Software packaging is no longer a bottleneck to software delivery.
From "git commit" to tested software, automatically, in a matter of minutes.
Eucalyptus packaging development and management
Release Engineering @ Eucalyptus is tasked with creating and maintaining Eucalyptus software packages and repositories. Release engineers design the package installation process and develop packaging scripts that will automatically install the hundred or so pieces of third party software that Eucalyptus depends upon in the correct order. In addition, release engineers maintain the infrastructure necessary for this effort. For this purpose, we use our very own Eucalyptus cloud.
Build and Test AutomationEucalyptus engineering works on two releases at any given point in time: A maintenance release to fix issues for users who are currently running Eucalyptus and the next development or feature release.
The Eucalyptus release engineering process has continuous delivery as a principle: Automatically build and test software packages whenever a developer commits code to the mainline branch. Software packages are always available in a state that they can be installed. When the decision is made to release the software as an "official" supported released, packages can be promoted to final release status.
Daily commits are available as nightly builds that end users can install to test out bug fixes or new features as they are being developed.
Our Eucalyptus cloud is a single front end installation with 14 nodes.
|Eucalyptus User Console connected to Release Engineering cloud|
|Images registered and compute resources available for package build|
We use a EC2 plugin with Jenkins to provision instances in Eucalyptus as worker nodes for Jenkins. The plugin is "tweaked" slightly so that instances are not terminated if they are idle to speed up start up time. This modification can be turned off if the Eucalyptus scheduler is configured to suspend nodes when idle.
|Jenkins configured to deploy to Eucalyptus|
A top level Jenkins project is configured to monitor commits to mainline Eucalyptus code repositories. This project, in turn, start a number of "downstream" tasks, which include building Eucalyptus from source, generating RPM packages, creating package repositories and triggering quality tests that will use packages from repositories.
|Jenkins project set up to monitor the Eucalyptus git repository|
|Jenkins project set up to build Eucalyptus RPM packages after Eucalyptus binaries are successfully built from source|
|Jenkins project set up to create package repository, so that programs like "yum" can install Eucalyptus on Linux|
|RPM package build jobs running on Eucalyptus compute nodes|
Eucalyptus open source and plugin packages are made available at a staging area for quality testing. In addition, product documentation builds are also continuously available.
|Eucalyptus package repository staging area|
|Packages corresponding to several git commit hashes are retained and can be retrieved|
When package repositories are created, another downstream project triggers QA tests against the software. The quality engineering team at Eucalyptus maintains an in-house QA system that runs a number of "test units" to verify functionality. The QA system has an API that is triggered via Jenkins.
|Eucalyptus QA triggered by downstream Jenkins project after packages have been built|
The Eucalyptus QA system installs Eucalyptus from package repositories on top of a Linux distribution on bare metal and runs Eucalyptus through its paces.
|Eucalyptus QA in action!|
The Future: Continuous Deployment & Rolling Upgrades
The Eucalyptus cloud that builds packages, product documentation and QA tests is upgraded whenever the next stable release of Eucalyptus is published. With continuous deployment, we would have the option of automatically upgrading our Eucalyptus installation to the latest tested code base. This requires a high degree of confidence in automated QA, an installation process that is future proof and the ability to upgrade Eucalyptus without service interruptions. Rolling upgrades would enable us to update a Eucalyptus installation without downtime. We are not very far from this goal, but some kinks in the software are yet to be worked out.