5.26. OpenStack tenant networking performance

status

ready

version

1.0

Abstract

This document describes test plan for measuring performance of tenant networking of the OpenStack cloud.

Conventions
  • Topology is how instances are plugged into tenant network

  • L2 topology is tenant network topology when instances located in the same L2 domain

  • L3 east-west topology is tenant network topology when instances located in different L2 domains connected to the same Neutron router

  • L3 north-south topology is tenant network topology when instances located in different L2 domains connected to different Neutron routers, thus the traffic goes outside of the cloud and back

5.26.1. Test Plan

This test plan covers base topologies (L2, L3 east-west and L3 north-south) and collection of common network parameters like bandwidth and latency.

5.26.1.1. Test Environment

5.26.1.1.1. Preparation

This test plan is performed against existing OpenStack cloud.

5.26.1.1.2. Environment description

The environment description includes hardware specification of servers, network parameters, operation system and OpenStack deployment characteristics.

5.26.1.1.2.1. Hardware

This section contains list of all types of hardware nodes.

Parameter

Value

Comments

model

e.g. Supermicro X9SRD-F

CPU

e.g. 6 x Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz

role

e.g. compute or network

5.26.1.1.2.2. Network

This section contains list of interfaces and network parameters. For complicated cases this section may include topology diagram and switch parameters.

Parameter

Value

Comments

network role

e.g. provider or public

card model

e.g. Intel

driver

e.g. ixgbe

speed

e.g. 10G or 1G

MTU

e.g. 9000

offloading modes

e.g. default

5.26.1.1.2.3. Software

This section describes installed software.

Parameter

Value

Comments

OS

e.g. Ubuntu 14.04.3

OpenStack

e.g. Liberty

Hypervisor

e.g. KVM

Neutron plugin

e.g. ML2 + OVS

L2 segmentation

e.g. VLAN or VxLAN or GRE

virtual routers

e.g. legacy or HA or DVR

5.26.1.2. Test Case 1: single node L2 instance-to-instance performance

5.26.1.2.1. Description

This test case is executed on a single pair of instances launched on one compute node and plugged into the same tenant network. The measurement is done during 60 seconds, average values are calculated.

5.26.1.2.2. List of performance metrics

Priority

Value

Measurement Units

Description

1

Latency

ms

The network latency

1

TCP bandwidth

Mbits/s

TCP network bandwidth

1

UDP bandwidth

packets per sec

Number of UDP packets with 32 bytes payload

2

UDP delay jitter

ms

Packet delay variation

2

UDP packet loss

%

Percentage of lost UDP packets

2

TCP retransmits

packets per sec

Number of retransmitted TCP packets

5.26.1.3. Test Case 2: L2 instance-to-instance performance

5.26.1.3.1. Description

This test case is executed on a single pair of instances launched on different compute nodes and plugged into the same tenant network. The measurement is done during 60 seconds, average values are calculated.

5.26.1.3.2. List of performance metrics

Priority

Value

Measurement Units

Description

1

Latency

ms

The network latency

1

TCP bandwidth

Mbits/s

TCP network bandwidth

1

UDP bandwidth

packets per sec

Number of UDP packets with 32 bytes payload

2

UDP delay jitter

ms

Packet delay variation

2

UDP packet loss

%

Percentage of lost UDP packets

2

TCP retransmits

packets per sec

Number of retransmitted TCP packets

5.26.1.4. Test Case 3: L2 concurrent performance

5.26.1.4.1. Description

This test case is executed on pairs of instances. Every instance is deployed on a different compute nodes (one instance per node). All instances are plugged into the same tenant network. The measurement is run simultaneously on different number of instances, starting with one pair and increasing in geometric progression until all pairs are involved. On every iteration the measurement is run during 60 seconds, then average numbers are stored.

5.26.1.4.2. List of performance metrics

Test case result is series of following measurements done at different levels of concurrency. The output may be shown in table form and/or as chart showing dependency of parameters from concurrency.

Priority

Value

Measurement Units

Description

1

Latency

ms

The network latency

1

TCP bandwidth

Mbits/s

TCP network bandwidth

1

UDP bandwidth

packets per sec

Number of UDP packets with 32 bytes payload

2

UDP delay jitter

ms

Packet delay variation

2

UDP packet loss

%

Percentage of lost UDP packets

2

TCP retransmits

packets per sec

Number of retransmitted TCP packets

5.26.1.5. Test Case 4: single node L3 east-west instance-to-instance performance

5.26.1.5.1. Description

This test case is executed on a single pair of instances launched on one compute node and plugged into different tenant networks. Networks are connected by a single router. The measurement is done during 60 seconds, average values are calculated.

5.26.1.5.2. List of performance metrics

Priority

Value

Measurement Units

Description

1

Latency

ms

The network latency

1

TCP bandwidth

Mbits/s

TCP network bandwidth

1

UDP bandwidth

packets per sec

Number of UDP packets with 32 bytes payload

2

UDP delay jitter

ms

Packet delay variation

2

UDP packet loss

%

Percentage of lost UDP packets

2

TCP retransmits

packets per sec

Number of retransmitted TCP packets

5.26.1.6. Test Case 5: L3 east-west instance-to-instance performance

5.26.1.6.1. Description

This test case is executed on a single pair of instances launched on different compute nodes and plugged into different tenant networks. Networks are connected by a single router. The measurement is done during 60 seconds, average values are calculated.

5.26.1.6.2. List of performance metrics

Priority

Value

Measurement Units

Description

1

Latency

ms

The network latency

1

TCP bandwidth

Mbits/s

TCP network bandwidth

1

UDP bandwidth

packets per sec

Number of UDP packets with 32 bytes payload

2

UDP delay jitter

ms

Packet delay variation

2

UDP packet loss

%

Percentage of lost UDP packets

2

TCP retransmits

packets per sec

Number of retransmitted TCP packets

5.26.1.7. Test Case 6: L3 east-west concurrent performance

5.26.1.7.1. Description

This test case is executed on pairs of instances. Every instance is deployed on a different compute nodes (one instance per node). All instances are grouped by pairs, one member is plugged in tenant network A, the other into tenant network B. Networks A and B are plugged into a single router. The measurement is run simultaneously on different number of instances, starting with one pair and increasing in geometric progression until all pairs are involved. On every iteration the measurement is run during 60 seconds, then average numbers are stored.

5.26.1.7.2. List of performance metrics

Test case result is series of following measurements done at different levels of concurrency. The output may be shown in table form and/or as chart showing dependency of parameters from concurrency.

Priority

Value

Measurement Units

Description

1

Latency

ms

The network latency

1

TCP bandwidth

Mbits/s

TCP network bandwidth

1

UDP bandwidth

packets per sec

Number of UDP packets with 32 bytes payload

2

UDP delay jitter

ms

Packet delay variation

2

UDP packet loss

%

Percentage of lost UDP packets

2

TCP retransmits

packets per sec

Number of retransmitted TCP packets

5.26.1.8. Test Case 7: single node L3 north-south instance-to-instance performance

5.26.1.8.1. Description

This test case is executed on a single pair of instances launched on one compute node and plugged into different tenant networks. Each networks is connected to own router, thus traffic goes via external network. The destination instance is reached by floating IP. The measurement is done during 60 seconds, average values are calculated.

5.26.1.8.2. List of performance metrics

Priority

Value

Measurement Units

Description

1

Latency

ms

The network latency

1

TCP bandwidth

Mbits/s

TCP network bandwidth

1

UDP bandwidth

packets per sec

Number of UDP packets with 32 bytes payload

2

UDP delay jitter

ms

Packet delay variation

2

UDP packet loss

%

Percentage of lost UDP packets

2

TCP retransmits

packets per sec

Number of retransmitted TCP packets

5.26.1.9. Test Case 8: L3 north-south instance-to-instance performance

5.26.1.9.1. Description

This test case is executed on a single pair of instances launched on different compute nodes and plugged into different tenant networks. Each networks is connected to own router, thus traffic goes via external network. The destination instance is reached by floating IP. The measurement is done during 60 seconds, average values are calculated.

5.26.1.9.2. List of performance metrics

Priority

Value

Measurement Units

Description

1

Latency

ms

The network latency

1

TCP bandwidth

Mbits/s

TCP network bandwidth

1

UDP bandwidth

packets per sec

Number of UDP packets with 32 bytes payload

2

UDP delay jitter

ms

Packet delay variation

2

UDP packet loss

%

Percentage of lost UDP packets

2

TCP retransmits

packets per sec

Number of retransmitted TCP packets

5.26.1.10. Test Case 9: L3 north-south concurrent performance

5.26.1.10.1. Description

This test case is executed on pairs of instances. Every instance is deployed on a different compute nodes (one instance per node). All instances are grouped by pairs, one member is plugged into tenant network A, the other into tenant network B. Networks A and B are plugged into different routers. Instances from network B are reached via floating IP. The measurement is run simultaneously on different number of instances, starting with one pair and increasing in geometric progression until all pairs are involved. On every iteration the measurement is run during 60 seconds, then average numbers are stored.

5.26.1.10.2. List of performance metrics

Test case result is series of following measurements done at different levels of concurrency. The output may be shown in table form and/or as chart showing dependency of parameters from concurrency.

Priority

Value

Measurement Units

Description

1

Latency

ms

The network latency

1

TCP bandwidth

Mbits/s

TCP network bandwidth

1

UDP bandwidth

packets per sec

Number of UDP packets with 32 bytes payload

2

UDP delay jitter

ms

Packet delay variation

2

UDP packet loss

%

Percentage of lost UDP packets

2

TCP retransmits

packets per sec

Number of retransmitted TCP packets

5.26.1.11. Test Case 10: Neutron QoS testing

5.26.1.11.1. Description

This test case is used to verify Neutron QoS feature as scale. The feature allows to limit the traffic bandwidth in a particular network. To simplify testing instances are deployed in the same network (L2 domain). The target bandwidth is specified in kBits/s.

Note

In order to work Neutron QoS extension must be installed and properly configured.

5.26.1.11.2. List of performance metrics

Priority

Value

Measurement Units

Description

1

Latency

ms

The network latency

1

TCP bandwidth

Mbits/s

TCP network bandwidth

1

UDP bandwidth

packets per sec

Number of UDP packets with 32 bytes payload

2

UDP delay jitter

ms

Packet delay variation

2

UDP packet loss

%

Percentage of lost UDP packets

2

TCP retransmits

packets per sec

Number of retransmitted TCP packets

It’s expected that achieved TCP bandwidth is not higher that the one set in Neutron QoS policy.

5.26.2. Tools

This section contains tools that can be used to perform the test plan.

5.26.2.1. Shaker

This section describes how to perform OpenStack tenant networking performance with Shaker tool.

5.26.2.1.1. Test environment preparation

To run tests you will need a machine located outside of the cloud. This machine must be routable from OpenStack instances and need to have one open port to accept connections from Shaker agents. See more details in Shaker deployment guide.

Shaker is distributed as Python package and available through PyPi (https://pypi.org/project/pyshaker/). It is recommended to be installed inside virtualenv.

$ virtualenv venv
$ . venv/bin/activate
$ pip install pyshaker

The connection to OpenStack can be configured using standard openrc file. (Refer to http://docs.openstack.org/cli-reference/content/cli_openrc.html on how to retrieve it). Alternatively the configuration can be passed to Shaker via set of CLI parameters --os-tenant-name, --os-username, --os-password, --os-auth-url and --os-region-name. Connection to SSL endpoints is configured by parameters --os-cacert and --os-insecure. Note that it is highly recommended to run Shaker with admin user. Use of non-admin user is also possible though, see Running Shaker by non-admin user.

Before starting tests a master image must be built. The process downloads Ubuntu cloud image, installs all necessary packages and stores snapshot into Glance. This snapshot is used by shaker to boot instances.

$ shaker-image-builder

Running shaker tool starts a server that accepts connections from agents located on instances. In order to work user needs to specify host address and port number. The result may be stored in raw format (--json), in ReST (--book) or as interactive HTML report (--report).

5.26.2.1.2. Test Case 1: L2 instance-to-instance performance

Test case specification: Test Case 2: L2 instance-to-instance performance

Execution:

shaker --server-endpoint <host:port> --scenario openstack/perf_l2 --book <report folder>

5.26.2.1.3. Test Case 2: L2 concurrent performance

Test case specification: Test Case 3: L2 concurrent performance

Execution:

shaker --server-endpoint <host:port> --scenario openstack/full_l2 --book <report folder>

5.26.2.1.4. Test Case 3: L3 east-west instance-to-instance performance

Test case specification: Test Case 5: L3 east-west instance-to-instance performance

Execution:

shaker --server-endpoint <host:port> --scenario openstack/perf_l3_east_west --book <report folder>

5.26.2.1.5. Test Case 4: L3 east-west concurrent performance

Test case specification: Test Case 6: L3 east-west concurrent performance

Execution:

shaker --server-endpoint <host:port> --scenario openstack/full_l3_east_west --book <report folder>

5.26.2.1.6. Test Case 5: L3 north-south instance-to-instance performance

Test case specification: Test Case 8: L3 north-south instance-to-instance performance

Execution:

shaker --server-endpoint <host:port> --scenario openstack/perf_l3_north_south --book <report folder>

5.26.2.1.7. Test Case 6: L3 north_south concurrent performance

Test case specification: Test Case 9: L3 north-south concurrent performance

Execution:

shaker --server-endpoint <host:port> --scenario openstack/full_l3_north_south --book <report folder>

5.26.2.1.8. Test Case 7: Neutron QoS testing

Test case specification: Test Case 10: Neutron QoS testing

Execution:

shaker --server-endpoint <host:port> --scenario openstack/qos/perf_l2 --book <report folder>

5.26.3. Reports

Test plan execution reports: