DevOps Guide¶

Introduction¶

Fuel-Devops is a sublayer between application and target environment (currently only supported under libvirt).

This application is used for testing purposes like grouping virtual machines to environments, booting KVM VMs locally from the ISO image and over the network via PXE, creating, snapshotting and resuming back the whole environment in single action, create virtual machines with multiple NICs, multiple hard drives and many other customizations with a few lines of code in system tests.

After 6.0 release, fuel-devops was divided into 2.5.x and 2.9.x versions. Two separate versions of fuel-devops provide backward compatibility for system tests which have been refactored since the last major release. Look here how to migrate from older devops.

For sources please refer to fuel-devops repository on github.

Installation¶

The installation procedure can be implemented via PyPI in Python virtual environment (suppose you are using Ubuntu 12.04 or Ubuntu 14.04):

Before using it, please install the following required dependencies:

sudo apt-get install --yes \
git \
libyaml-dev \
libffi-dev \
python-dev \
python-pip \
qemu \
libvirt-bin \
libvirt-dev \
vlan \
bridge-utils \
genisoimage

sudo apt-get update && sudo apt-get upgrade -y

Configuration¶

Basically devops requires that the following system-wide settings are configured:

• Default libvirt storage pool is active (called ‘default’)
• Current user has a permission to run KVM VMs with libvirt
• PostgreSQL server running with appropriate grants and schema for devops
• (Optional) Nested Paging is enabled

sudo virsh pool-define-as --type=dir --name=default --target=/var/lib/libvirt/images
sudo virsh pool-autostart default
sudo virsh pool-start default

sudo usermod $(whoami) -a -G libvirtd,sudo

sudo apt-get install --yes postgresql

pg_version=$(dpkg-query --show --showformat='${version;3}' postgresql)
pg_createcluster $pg_version main --start
sudo sed -ir 's/peer/trust/' /etc/postgresql/9.*/main/pg_hba.conf
sudo service postgresql restart

sudo apt-get install --yes libsqlite3-0

export DEVOPS_DB_NAME=$WORKING_DIR/fuel-devops
export DEVOPS_DB_ENGINE="django.db.backends.sqlite3

django-admin.py syncdb --settings=devops.settings
django-admin.py migrate devops --settings=devops.settings

./bin/django-admin syncdb --settings=devops.settings
./bin/django-admin migrate devops --settings=devops.settings

cat /etc/modprobe.d/qemu-system-x86.conf
options kvm_intel nested=1

sudo apt-get install --yes cpu-checker
sudo modprobe kvm_intel
sudo kvm-ok && cat /sys/module/kvm_intel/parameters/nested

INFO: /dev/kvm exists
KVM acceleration can be used
Y

Environment creation via Devops + Fuel_QA or Fuel_main¶

Depending on the Fuel release, you may need a different repository.

1. Clone GIT repository

For 6.1 and later, the fuel-qa is required:

git clone https://github.com/openstack/fuel-qa
cd fuel-qa/

In case of 6.0 or earlier, please use fuel-main repository:

git clone https://github.com/openstack/fuel-main -b stable/6.0
cd fuel-main/

2. Install requirements (follow Devops installation in virtualenv section for the WORKING_DIR variable)

. $WORKING_DIR/fuel-devops-venv/bin/activate
pip install -r ./fuelweb_test/requirements.txt --upgrade

3. Check Configuration section
4. Prepare environment

Download Fuel ISO from Nightly builds or build it yourself (please, refer to Building the Fuel ISO)

Next, you need to define several variables for the future environment:

• the path where is located your iso (e.g. $WORKING_DIR/fuel-community-7.0.iso)
• the number of nodes instantiated for the environment (e.g. 5)

export ISO_PATH=$WORKING_DIR/fuel-community-7.0.iso
export NODES_COUNT=5

Optionally you can specify the name of your test environment (it will be used as a prefix for the domains and networks names created by libvirt, defaults is fuel_system_test).

export ENV_NAME=fuel_system_test
export VENV_PATH=$WORKING_DIR/fuel-devops-venv

If you want to use separated files for snapshots you need to set env variable and use the following required versions:

• fuel-devops >= 2.9.17
• libvirtd >= 1.2.12

This change will switch snapshots created by libvirt from internal to external mode.

export SNAPSHOTS_EXTERNAL=true

Alternatively, you can edit this file to set them as a default values

fuelweb_test/settings.py

Start tests by running this command

./utils/jenkins/system_tests.sh -t test -w $(pwd) -j fuelweb_test -i $ISO_PATH -o --group=setup

For more information about how tests work, read the usage information

./utils/jenkins/system_tests.sh -h

Important notes for Sahara tests¶

• It is not recommended to start tests without KVM.
• For the best performance Put Sahara image savanna-0.3-vanilla-1.2.1-ubuntu-13.04.qcow2 (md5: 9ab37ec9a13bb005639331c4275a308d) in /tmp/ before start, otherwise (If Internet access is available) the image will download automatically.

Important notes for Murano tests¶

• Murano is deprecated in Fuel 9.0.
• Put Murano image ubuntu-murano-agent.qcow2 (md5: b0a0fdc0b4a8833f79701eb25e6807a3) in /tmp before start.
• Running Murano tests on instances without an Internet connection will fail.
• For Murano tests execute ‘export SLAVE_NODE_MEMORY=5120’ before starting.
• If you need an image For Heat autoscale tests check prebuilt-jeos-images.

Run single OSTF tests several times¶

• Export environment variable OSTF_TEST_NAME. Example: export OSTF_TEST_NAME=’Request list of networks’
• Export environment variable OSTF_TEST_RETRIES_COUNT. Example: export OSTF_TEST_RETRIES_COUNT=120
• Execute test_ostf_repetable_tests from tests_strength package

Run tests

sh "utils/jenkins/system_tests.sh" -t test \
     -w $(pwd) \
     -j "fuelweb_test" \
     -i "$ISO_PATH" \
     -V $(pwd)/venv/fuelweb_test \
     -o \
     --group=create_delete_ip_n_times_nova_flat

Upgrade from system-wide devops to devops in Python virtual environment¶

To migrate from older devops, follow these steps:

1. Remove system-wide fuel-devops (e.g. python-devops)

You must remove system-wide fuel-devops and switch to separate venvs with different versions of fuel-devops, for Fuel 6.0.x (and older) and 6.1 release.

Repositories ‘fuel-main’ and ‘fuel-qa’, that contain system tests, must use different Python virtual environments, for example:

• ~/venv-nailgun-tests - used for 6.0.x and older releases. Contains version 2.5.x of fuel-devops
• ~/venv-nailgun-tests-2.9 - used for 6.1 and above. Contains version 2.9.x of fuel-devops

If you have scripts which use system fuel-devops, fix them, and activate Python venv before you start working in your devops environment.

By default, the network pool is configured as follows:

• 10.108.0.0/16 for devops 2.5.x
• 10.109.0.0/16 for 2.9.x

Please check other settings in devops.settings, especially the connection settings to the database.

Before using devops in Python venv, you need to install system dependencies

2. Update fuel-devops and Python venv on CI servers

To update fuel-devops, you can use the following examples:

# DevOps 2.5.x for system tests from 'fuel-main' repository
if [ -f ~/venv-nailgun-tests/bin/activate ]; then
  echo "Python virtual env exist"
else
  rm -rf ~/venv-nailgun-tests
  virtualenv --no-site-packages ~/venv-nailgun-tests
fi
source ~/venv-nailgun-tests/bin/activate
pip install -r https://raw.githubusercontent.com/openstack/fuel-main/master/fuelweb_test/requirements.txt --upgrade
django-admin.py syncdb --settings=devops.settings --noinput
django-admin.py migrate devops --settings=devops.settings --noinput
deactivate

# DevOps 2.9.x for system tests from 'fuel-qa' repository
if [ -f ~/venv-nailgun-tests-2.9/bin/activate ]; then
  echo "Python virtual env exist"
else
  rm -rf ~/venv-nailgun-tests-2.9
  virtualenv --no-site-packages ~/venv-nailgun-tests-2.9
fi
source ~/venv-nailgun-tests-2.9/bin/activate
pip install -r https://raw.githubusercontent.com/openstack/fuel-qa/master/fuelweb_test/requirements.txt --upgrade
django-admin.py syncdb --settings=devops.settings --noinput
django-admin.py migrate devops --settings=devops.settings --noinput
deactivate

3. Setup new repository of system tests for 6.1 release

All system tests for 6.1 and higher were moved to fuel-qa repo.

To upgrade 6.1 jobs, follow these steps:

• make a separate Python venv, for example in ~/venv-nailgun-tests-2.9

• install requirements of system tests

• if you are using system tests on CI, please configure your CI to use new Python venv, or export path to the new Python venv in the variable VENV_PATH (follow Devops installation in virtualenv section for the WORKING_DIR variable):

  export VENV_PATH=$WORKING_DIR/fuel-devops-venv-2.9
  

Known issues¶

• Some versions of libvirt contain a bug that breaks QEMU virtual machine XML. You can see this when tests crush with a libvirt: QEMU Driver error: unsupported configuration: host doesn’t support invariant TSC. See: Bug 1133155.

  Workaround: upgrade libvirt to the latest version.

• If the same version of fuel-devops is used with several different databases (for example, with multiple sqlite3 databases, or with a separated database for each devops in different python virtual environments), there will be a collision between Libvirt bridge names and interfaces.

  Workaround: use the same database for the same version of the fuel-devops.

  • for 2.9.x, export the following env variables:

    export DEVOPS_DB_NAME=fuel_devops
    export DEVOPS_DB_USER=fuel_devops
    export DEVOPS_DB_PASSWORD=fuel_devops
    

  • for 2.5.x, edit the dict for variable DATABASES:

    vim $WORKING_DIR/fuel-devops-venv/lib/python2.7/site-packages/devops/settings.py