Scaling OpenStack Development with Git, Gerrit, and Jenkins

Monty Taylor <mordred@inaugust.com>
James E. Blair <corvus@inaugust.com>

Projects

Contributors

Individual Contributors
Commercial Entities
Number, quality, and area of contributions can change daily

Release Management

Time Based Releases
Six Month Cadence
- Tied to Ubuntu Releases
Design summits each cycle
Continuously Open Trunk
- Develop directly on master
One Month Milestone Releases
Post-release Stable Branches

Vision

Consistent Tooling
Consistent Process
Consistent Product
Multiplier Effect

Consistent Tooling

Minimize meta-development
Process divergence == wasted developer time
Lowers onboarding time
Consolidate tool development
Minimize project-specific weird build crud

Development Infrastructure Systems

Gerrit (code review, git) http://review.openstack.org
Jenkins (CI, automation) http://jenkins.openstack.org
- pre-merge testing
- post-merge testing
- post-merge artifact management
Orchestra (bare metal deployment)
Launchpad (bugs, blueprints, releases, translations, SSO) http://launchpad.net/openstack
Documentation servers
IRC bots
planet.openstack.org
Package repositories
Etherpad

Environment

Ubuntu
Python
- pep8 standards
- openstack.common
virtualenv/pip
IRC (#openstack-dev, #openstack-meeting)
devstack
gated trunk based on master

Gated Trunk

Ensures Code Quality
Protects developers
- Devs always start from working code
Protects tree
- Bad code doesn't land
Egalitarian
- Process is the same for everyone
- Process is transparent
- Process is automated

Everything Is Automated

Process Flow

Code is written and locally tested in a virtualenv
Code is submitted for code review to gerrit
Code is peer-reviewed
Code is accepted or rejected by core team
Code is run through pre-merge automated checks
Code is merged or rejected
Code is run through post-merge automated checks

Gerrit

Developed by Google for Android
Stand-alone patch review system
Integration points: hooks, JSON queries, event-stream
Extensible review categories, default: Verified, Code-Review

Pre-merge Check

Approved Reviews

Types of Jenkins Gerrit Triggers

Patchset uploaded
Change merged
Comment added (review state)

OpenID SSO Integration

Bug Integration - Gerrit

Bug Integration - Launchpad

Blueprints - Gerrit

Blueprints - Launchpad

Blueprints - Gerrit Topics

Git Review

External Git subcommand
Developers can easily incorporate code review into git workflow
Zero-configuration
Can be used for any project, being adopted by other projects

corvus@shiprock:~/rs/github/quantum$ git commit -a
[new-versionpy ddf1dce] Base version.py on glance.
 3 files changed, 28 insertions(+), 107 deletions(-)
 delete mode 100644 version.py

corvus@shiprock:~/rs/github/quantum$ git review
remote: Resolving deltas:   0% (0/3)
remote: 
remote: New Changes:
remote:   https://review.openstack.org/3072
remote: 
To ssh://corvus@review.openstack.org:29418/openstack/quantum.git
 * [new branch]      HEAD -> refs/for/master/bug/916018

Types of Tests

Unit tests
Functional tests
- Can be run on real or virtual servers
- Easy for developers to run
Integration tests
- May be able to run on virtual servers, should run on real servers
- Difficult or impossible for a developer to run

Integration Testing

Install and test code
Targets
- Single cloud server
- Multiple cloud servers
- Bare metal machines
  - HP
  - Rackspace
  - Cisco
  - Citrix
Tests
- tempest
- bindings unittests (jclouds, libcloud, fog)

Bare-metal Test process

One time (Orchestra driven):

PXE boot
Install Ubuntu
Install OpenStack de
Snapshot LVM volume

Each test:

Kexec boot into LVM snapshot
Install OpenStack
Run test suite

Thanks!

These slides available at: https://github.com/openstack-ci/publications