Submitting a software bug¶
Please read this entire page before submitting an OpenStack Charms bug. Applying the guidelines given here will considerably reduce the time needed to triage your bug. Reach out to the team (see Contact us) if you have any questions.
Preliminaries¶
First consult these resources:
Release notes - the observed issue may be known to the dev team, and a workaround may be available
Existing bugs - someone else may have already encountered the issue - avoid creating duplicate bugs
The observed issue may already be fixed. Ideally, upgrade to the most recent revision available and confirm whether the problem persists. See the Charms upgrade documentation.
Consider testing for a regression before submitting a bug as it is an extremely valuable data point. The main variables are Ubuntu series, OpenStack release, and charm revision/channel.
Cloud stack¶
Understand the cloud stack to ensure you are filing your bug in the right place.
Charmed OpenStack is the result of a multi-layered stack, and a problem’s root cause may reside in any one of its layers:
charm payload - OpenStack service (e.g. Nova Compute)
charms - OpenStack charm (e.g. nova-compute)
Juju - modelling software
MAAS - Juju-managed cloud
hardware and network topology
The bug you are submitting is for an individual OpenStack charm (the charms layer).
When filing the bug, select the charm (from the ‘Project’ dropdown menu) that you think is most closely related to your problem. For example, if the problem seems related to a malfunctioning hypervisor, select OpenStack Nova Compute Charm (i.e. the nova-compute charm).
Essential information¶
This section covers the minimal amount of information that every bug should have.
Please provide:
an overview of your environment
a detailed description of the problem and how it differs from your expected outcome
specific software versions in use
Ubuntu
OpenStack
Juju (see Juju versions)
MAAS
if available, concise samples of identified command output errors or log messages directly related to the problem
if known, specific steps that will allow someone else to reproduce the problem
See section Supporting data for details on providing:
the output to commands juju status --relations and juju export-bundle
the logs of any troubled application unit (unit agent logs)
If any of the questions below are answered in the affirmative, provide relevant details:
Did the appearance of the problem coincide with a change made to the cloud (configuration or topological)?
Has AppArmor configuration been modified on the cloud nodes?
Are Policy overrides in use?
Based on all the above, use a descriptive yet succinct bug title.
Juju versions¶
It is possible for the client, the controller model, and the workload model to each have a different version.
Assuming that the currently active controller is that of the OpenStack cloud, and the workload model is called ‘openstack’, you can determine each version as follows:
The client:
juju version
The controller model:
juju show-model controller | grep agent-version
The workload model:
juju show-model openstack | grep agent-version
Supporting data¶
When providing supporting data, use a separate file for each different type of data and attach them to the bug. Include a summary of what each file contains (in the bug description) if you think it will facilitate bug triage.
Avoid using a third-party service (e.g. pastebin or imagebin) as data hosted in this way is not considered permanent.
The below sections cover the most common types of supporting data.
CLI commands¶
status¶
The juju status command is a staple when communicating the state of a model. Here we also include the relations:
juju status --relations > juju-status_relations.txt
export-bundle¶
The juju export-bundle command inspects a model and generates a bundle file from it. This will give a good understanding as to how the cloud was deployed.
juju export-bundle --filename juju-export-bundle.txt
config¶
The juju config command retrieves a charm’s configuration options and their corresponding current values. These options alter how the charm and its payload behave together. Not only does this information help in understanding the environment but it will also reveal an incorrectly set option.
To retrieve the configuration for a charm (ceph-osd here):
juju config ceph-osd > juju-config_ceph-osd.txt
crashdump¶
The juju crashdump command generates a comprehensive, yet unsanitised, report on an entire Juju model. It is available via a Juju plugin. Install it alongside the Juju client:
sudo snap install juju-crashdump --classic
For example, to analyse the currently active model and tag the report with a unique string (assuming the issue involves the ovn-central charm):
juju crashdump --small --as-root -o ~/tmp -u ovn-central
This will produce the file ~/tmp/juju-crashdump-ovn-central.tar.xz.
Omitting the --small option will lead to the inclusion of a massive amount
of Juju debug information (see Dealing with large file attachments). To get
more command help: juju crashdump --help.
Omitting the --as-root option will prevent certain logs (and effectively
more sensitive information) from being collected.
Note
To avoid copying the file across networks in order to attach it to the bug
(the file is probably not immediately available to your browser), the
command’s -b option can be used to send it directly to an existing bug.
Logs¶
Logs are often an essential type of supporting data. With Charmed OpenStack there are two main categories: Juju agent logs and OpenStack service logs.
Agent logs¶
Juju agents record events from the perspective of Juju. Unit agents are generally more useful in the present context as they interface with the payload (OpenStack) whereas machine agents are concerned with the provisioning of the Juju machine.
To retrieve unit agent logs (nova-compute/0 here):
juju debug-log --replay --no-tail --include nova-compute/0 | tee bug_1234567_nova-compute_0.log
To retrieve machine agent logs (machine-8 here):
juju debug-log --replay --no-tail --include machine-8 | tee bug_1234567_machine_8.log
Create an archive (e.g. with the tar command) of the desired logs and attach it to the bug.
You can set the logging verbosity (of the currently active model) for both
types of agents. Here we set the level of both the machine agent (<root>)
and the unit agent (unit) to ‘DEBUG’:
juju model-config logging-config="<root>=DEBUG;unit=DEBUG"
The How to manage logs page in the Juju documentation has more details.
Service logs¶
Service logs are the native logs of the OpenStack service in question. They are
found in their standard locations under /var/log on each individual
machine. Create an archive (e.g. with the tar command) of the
desired logs and attach it to the bug.
To increase the verbosity of these logs for an application (nova-compute here):
juju config nova-compute debug=true
CLI session¶
A CLI session is a series of terminal-based commands and their respective outputs. This is very useful in conveying an exact chronology of what was done/attempted and what the results were.
Screenshots¶
Screenshots are typically used when the subject is graphical in nature such as the web UIs available with MAAS, OpenStack Horizon, and Ceph Dashboard.
Dealing with large file attachments¶
Attaching an oversized file to the bug can be problematic (Launchpad may time out). In such cases, the common split utility can be of use. Consider the below juju-crashdump report:
-rw-rw-r-- 1 ubuntu ubuntu 167M Feb 7 22:06 juju-crashdump-7c9c30a8-686c-4d28-8765-b31c1791ca85.tar.xz
To break it into 64MiB chunks (and add some prefix and suffix information to the resulting files):
split -b 64M --numeric-suffixes=1 --additional-suffix=-juju-crashdump \
juju-crashdump-7c9c30a8-686c-4d28-8765-b31c1791ca85.tar.xz split-
This yields three manageable files:
-rw-rw-r-- 1 ubuntu ubuntu 64M Feb 8 16:32 split-01-juju-crashdump
-rw-rw-r-- 1 ubuntu ubuntu 64M Feb 8 16:32 split-02-juju-crashdump
-rw-rw-r-- 1 ubuntu ubuntu 39M Feb 8 16:32 split-03-juju-crashdump
Please include an explanatory bug comment:
I have split a juju-crashdump file into three and attached them. To
reconstruct:
$ cat split-0?-juju-crashdump > juju-crashdump.tar.xz
