6.4.1. OpenStack control plane performance report for 1000 nodes¶

Abstract: This document includes OpenStack control plane performance test results for 1000 compute nodes cluster. All tests have been performed regarding OpenStack control plane performance test plan

6.4.1.1. Environment description¶

Environment contains 4 types of servers:

rally node
controller node
osd node
hypervisor
compute node

Amount of servers each role¶
Role	Servers count	Server Type
rally	1	bare-metal
controller	3	bare-metal
osd	20	bare-metal
compute	1000	virtual

6.4.1.1.1. Hardware configuration of each server¶

All bare-metal servers have the same configuration describing in table below

Description of servers hardware¶
server	vendor,model	HP,DL380 Gen9
CPU	vendor,model	Intel,E5-2680 v3
	processor_count	2
	core_count	12
	frequency_MHz	2500
RAM	vendor,model	HP,752369-081
RAM	amount_MB	262144
NETWORK	interface_name	p1p1
	vendor,model	Intel,X710 Dual Port
	bandwidth	10G
STORAGE	dev_name	/dev/sda
	vendor,model	raid10 - HP P840 12 disks EH0600JEDHE
	SSD/HDD	HDD
	size	3,6TB

All virtual servers has the same configuration. QEMU-KVM used as hypervisor. For datailed info about VM configuration look into sample domain.xml.

6.4.1.1.2. Network configuration of each server¶

Lab network configuration you can find in scheme below:

Here is the part of switch configuration for each switch port which connected to p1p1 interface of a bare-metal server:

switchport mode trunk
switchport trunk native vlan 600
switchport trunk allowed vlan 600-602,630-649
spanning-tree port type edge trunk
spanning-tree bpduguard enable
no snmp trap link-status

6.4.1.1.3. Software configuration on servers with controller, osd and compute roles¶

Services on servers by role¶
Role	Service name
controller	horizon keystone nova-api nava-scheduler nova-cert nova-conductor nova-consoleauth nova-consoleproxy cinder-api cinder-backup cinder-scheduler cinder-volume glance-api glance-glare glance-registry neutron-dhcp-agent neutron-l3-agent neutron-metadata-agent neutron-openvswitch-agent neutron-server heat-api heat-api-cfn heat-api-cloudwatch ceph-mon rados-gw memcached rabbitmq_server mysqld galera corosync pacemaker haproxy
osd	ceph-osd
compute	nova-compute neutron-l3-agent neutron-metadata-agent neutron-openvswitch-agent

Software version on servers with controller, compute and compute-osd roles¶
Software	Version
OpenStack	Mitaka
Ceph	Hammer
Ubuntu	Ubuntu 14.04.3 LTS

You can find outputs of some commands and /etc folder in the following archives:

controller-1.tar.gz
controller-2.tar.gz
controller-3.tar.gz
compute-1.tar.gz
osd-1.tar.gz

6.4.1.1.4. Software configuration on servers with rally role¶

On this server should be installed Rally. How to do it you can find in Rally installation documentation

Software version on server with rally role¶
Software	Version
Rally	0.4.0
Ubuntu	Ubuntu 14.04.3 LTS

6.4.1.2. Testing process¶

Some test parameters¶
Name	Value
Volume size to create in Cinder	1GB
Flavor to create VM from	m1.tiny
Image name to create VM from	cirros

Create work directory on server with Rally role. In future we will call that directory as WORK_DIR
Create directory “plugins” in WORK_DIR and copy to that directory nova_performance.py plugin.
Create directory “scenarios” in WORK_DIR and copy to that directory boot_attach_live_migrate_and_delete_server_with_secgroups.json, create-and-delete-image.json and keystone.json scenarios.

Create deployment.json file in WORK_DIR and fill it with OpenStack environment info. It should looks like this:

{
  "admin": {
    "password": "password",
    "tenant_name": "tenant",
    "username": "user"
  },
  "auth_url": "http://1.2.3.4:5000/v2.0",
  "region_name": "RegionOne",
  "type": "ExistingCloud",
  "endpoint_type": "internal",
  "admin_port": 35357,
  "https_insecure": true
}

Create job-params.yaml file in WORK_DIR and fill it with scenarios info. It should looks like this:

---
  concurrency: 5
  compute: 1000
  gre_enabled: false
  http_server_with_glance_images: "1.2.3.4"

Perform tests:

rally  --plugin-paths /home/rally/plugins task start \
keystone.json \
--task-args '{"gre_enabled": true, "compute": 375, "concurrency": 5}' \
--out=keystone.html

rally  --plugin-paths /home/rally/plugins task start \
boot_attach_live_migrate_and_delete_server_with_secgroups.json \
--task-args '{"gre_enabled": true, "compute": 375, "concurrency": 5}' \
--out=boot_attach_live_migrate_and_delete_server_with_secgroups.html

rally  --plugin-paths /home/rally/plugins task start \
create-and-delete-image.json \
--task-args '{"gre_enabled": true, "compute": 375, "concurrency": 5, "http_server_with_glance_images": "1.2.3.4"}' \
--out=create-and-delete-image.html

As a result of this part we got the following HTML file:

rally_report.html

6.4.1.3. Test results¶

All values in tables below are in seconds.

6.4.1.3.1. Cinder¶

Operation	Mean	90%ile	50%ile	Max	Min
create_volume	2.59589	2.72586	2.57214	3.05174	2.40753
delete_volume	2.31282	2.42514	2.30285	2.77269	0.4474

6.4.1.3.2. Glance¶

Operation	Mean	90%ile	50%ile	Max	Min
create_image	44.797	59.5237	43.129	73.7454	26.3647
delete_image	1.51749	1.98148	1.51694	2.93414	0.649655

6.4.1.3.3. Keystone¶

Operation	Mean	90%ile	50%ile	Max	Min
keystone	0.0772378	0.10375	0.071027	0.624643	0.0493538

6.4.1.3.4. Neutron¶

Operation	Mean	90%ile	50%ile	Max	Min
create_20_rules	4.45449	4.75529	4.4336	5.72318	3.71847
create_2_security_groups	0.358809	0.405863	0.350541	1.10707	0.261818
delete_2_security_groups	0.314609	0.397726	0.308533	0.602876	0.191209

6.4.1.3.5. Nova¶

Operation	Mean	90%ile	50%ile	Max	Min
attach_volume	2.77096	2.93019	2.753	3.5367	2.50419
boot_server	13.1012	14.1159	12.9169	17.6851	11.2657
delete_server	2.4879	2.62365	2.42707	4.98843	2.32484
detach_volume	2.64998	2.80518	2.62848	3.30333	2.39798
find_host_to_migrate	1.19893	1.36617	1.21302	1.77083	0.896176
live_migrate	14.6121	15.3808	14.9722	26.7783	12.3919

6.4.1.4. Issues which have been found during the tests¶

Issues which have been found during the tests¶
Issue description	Root cause, Link	Link to bug	Is fixed
Live migration failure. Port range intersection.	Qemu and nova use the same port range.	1627476	Yes

Test result from run where bug mentioned above was not fixed: rally_report.html

6.4.1. OpenStack control plane performance report for 1000 nodes