OpenStack

Is open source software for building private and public clouds.

• http://openstack.org

Projects

Contributors

• Individual Contributors
• Commercial Entities
• Number, quality, and area of contributions can change daily

Gated Trunk

• Ensures Code Quality
• Protects developers
  • Devs always start from working code
• Protects tree
  • Bad code doesn't land
• Egalitarian
  • Process is the same for everyone
  • Process is transparent
  • Process is automated

Everything Is Automated

Gerrit

• Developed by Google for Android
• Stand-alone patch review system
• Integration points: hooks, JSON queries, event-stream
• Extensible review categories, default: Verified, Code-Review

Process Flow

• Code is written and locally tested in a virtualenv
• Code is submitted for code review to gerrit
• Code is run through patch-uploaded automated checks
• Code is peer-reviewed
• Code is accepted or rejected by core team
• Code is run through pre-merge automated checks
• Code is merged or rejected
• Code is run through post-merge analysis

Jenkins

Types of jobs:

• Check / Gate
  • unit tests
  • integraton tests
  • code style
• Post
  • docs
  • tarballs
  • pypi

Types of Tests

• Unit tests
• Integration tests
  • May be able to run on virtual servers, should run on real servers
  • Difficult or impossible for a developer to run

Specific Challenges/Solutions

• Testing effect of merging change
• Using cloud builders
• Large numbers of similar projects
• Disparate hardware configurations

Gerrit Git Prep

• Test the result of the change, not the change
• https://github.com/openstack/openstack-ci-puppet/blob/master/modules/jenkins_slave/files/slave_scripts/gerrit-git-prep.sh
• For each gerrit change:
  • grabs target branch
  • cleans tree
  • merges change to be tested

Interrelated Integration Testing

• Devstack-gate
  • Spin up fresh node
  • Run devstack
  • Run integration tests
• Gate proposed change against current state of other projects

Devstack-Gate Problems

• Tests are slow
• Clouds are unreliable
• Github is unreliable
• Pypi is unreliable
• Distro mirrors are unreliable
• ALL NETWORK ACCESS IS FAILURE

Devstack-Gate Solutions

• Create a new node
• Pre-fetch all needed packages, repos
• Snapshot to cloud image
• Maintain a pool of cloud nodes
• Slave can only be used for one test run
• Set of python and shell scripts triggered by jenkins

Alternate Caching Idea

• Create a volume
• Pre-fetch all needed packages, repos
• Attatch to test node
• Detach when complete
• Re-use on next test node

Providers

• OpenStack tests currently run on 2 zones from 2 providers each
• Maintain up to date images for each base OS
• Set min/max targets for number of servers from each zone

Launching Nodes

• Periodically check if new nodes need to be launched
• Launch them, wait for them to come on-line
• Timeout waiting after 10 minutes
• Test that the node really works (ssh connectivity)

Node Pooling

• Add to Jenkins as a slave (avoid exhaustion errors)
• Give the node a label (e.g., devstack-precise)
• Keep a database of which nodes in which state
  • New
  • Ready
  • Used
  • Error
  • Delete
• TODO: periodically test that new nodes are still viable

Devstack Nodes

Use

• Mark node as used in the database (inprogress job)
• Change the label to prevent re-use by Jenkins (e.g., devstack-used)
• Run the tests
• Test timeout
• Mark node for deletion in database (complete job)

Deletion

• Identify all nodes that are in deleted or error state
• If node exists: delete it
• If node no longer exists: remove from database

Devstack Jobs

Jclouds-Plugin

• EC2 plugin useless to us (sorry)
• Adrian Cole and Cloudsoft wrote jclouds-plugin
• Provisions on any provider via jclouds
• Image pre-creation and pooling features
• Single use slaves
• Slave provisioning as build step

Jclouds Configuration

Jclouds Configuration

Thanks!

These slides available at: https://github.com/openstack-ci/publications