Having good tests in place is absolutely critical for ensuring a stable, maintainable codebase. Hopefully that doesn’t need any more explanation.
However, what defines a “good” test is not always obvious, and there are a lot of common pitfalls that can easily shoot your test suite in the foot.
If you already know everything about testing but are fed up with trying to debug why a specific test failed, you can skip the intro and jump straight to Debugging Unit Tests.
There are three main types of tests, each with their associated pros and cons:
These are isolated, stand-alone tests with no external dependencies. They are written from the perspective of “knowing the code”, and test the assumptions of the codebase and the developer.
Pros:
Cons:
These are generally also isolated tests, though sometimes they may interact with other services running locally. The key difference between functional tests and unit tests, however, is that functional tests are written from the perspective of the user (who knows nothing about the code) and only knows what they put in and what they get back. Essentially this is a higher-level testing of “does the result match the spec?”.
Pros:
Cons:
This layer of testing involves testing all of the components that your codebase interacts with or relies on in conjunction. This is equivalent to “live” testing, but in a repeatable manner.
Pros:
Cons:
A few simple guidelines:
Limiting our focus just to unit tests, there are a number of things you can do to make your unit tests as useful, maintainable, and unburdensome as possible.
Use a single, consistent set of test data. Grow it over time, but do everything you can not to fragment it. It quickly becomes unmaintainable and perniciously out-of-sync with reality.
Make your test data as accurate to reality as possible. Supply all the attributes of an object, provide objects in all the various states you may want to test.
If you do the first suggestion above first it makes the second one far less painful. Write once, use everywhere.
To make your life even easier, if your codebase doesn’t have a built-in ORM-like function to manage your test data you can consider building (or borrowing) one yourself. Being able to do simple retrieval queries on your test data is incredibly valuable.
Mocking is the practice of providing stand-ins for objects or pieces of code you don’t need to test. While convenient, they should be used with extreme caution.
Why? Because overuse of mocks can rapidly land you in a situation where you’re not testing any real code. All you’ve done is verified that your mocking framework returns what you tell it to. This problem can be very tricky to recognize, since you may be mocking things in setUp methods, other modules, etc.
A good rule of thumb is to mock as close to the source as possible. If you have a function call that calls an external API in a view , mock out the external API, not the whole function. If you mock the whole function you’ve suddenly lost test coverage for an entire chunk of code inside your codebase. Cut the ties cleanly right where your system ends and the external world begins.
Similarly, don’t mock return values when you could construct a real return value of the correct type with the correct attributes. You’re just adding another point of potential failure by exercising your mocking framework instead of real code. Following the suggestions for testing above will make this a lot less burdensome.
Think long and hard about what you really want to verify in your unit test. In particular, think about what custom logic your code executes.
A common pitfall is to take a known test object, pass it through your code, and then verify the properties of that object on the output. This is all well and good, except if you’re verifying properties that were untouched by your code. What you want to check are the pieces that were changed, added, or removed. Don’t check the object’s id attribute unless you have reason to suspect it’s not the object you started with. But if you added a new attribute to it, be damn sure you verify that came out right.
It’s also very common to avoid testing things you really care about because it’s more difficult. Verifying that the proper messages were displayed to the user after an action, testing for form errors, making sure exception handling is tested... these types of things aren’t always easy, but they’re extremely necessary.
To that end, Horizon includes several custom assertions to make these tasks easier. assertNoFormErrors(), assertMessageCount(), and assertNoMessages() all exist for exactly these purposes. Moreover, they provide useful output when things go wrong so you’re not left scratching your head wondering why your view test didn’t redirect as expected when you posted a form.
Use assertNoFormErrors() immediately after your client.post call for tests that handle form views. This will immediately fail if your form POST failed due to a validation error and tell you what the error was.
Use assertMessageCount() and assertNoMessages() when a piece of code is failing inexplicably. Since the core error handlers attach user-facing error messages (and since the core logging is silenced during test runs) these methods give you the dual benefit of verifying the output you expect while clearly showing you the problematic error message if they fail.
Use Python’s pdb module liberally. Many people don’t realize it works just as well in a test case as it does in a live view. Simply inserting import pdb; pdb.set_trace() anywhere in your codebase will drop the interpreter into an interactive shell so you can explore your test environment and see which of your assumptions about the code isn’t, in fact, flawlessly correct.
If the error is in the Selenium test suite, you’re likely getting very little information about the error. To increase the information provided to you, edit horizon/test/settings.py to set DEBUG = True and set the logging level to ‘DEBUG’ for the default ‘test’ logger. Also, add a logger config for Django:
},
'loggers': {
+ 'django': {
+ 'handlers': ['test'],
+ 'propagate': False,
+ },
'django.db.backends': {
There are a number of typical (and non-obvious) ways to break the unit tests. Some common things to look for:
Horizon uses mox as its mocking framework of choice, and while it offers many nice features, its output when a test fails can be quite mysterious.
This occurs when you stubbed out a piece of code, and it was subsequently called in a way that you didn’t specify it would be. There are two reasons this tends to come up:
This one is the opposite of the unexpected method call. This one means you told mox to expect a call and it didn’t happen. This is almost always the result of an error in the conditions of the test. Using the assertNoFormErrors() and assertMessageCount() will make it readily apparent what the problem is in the majority of cases. If not, then use pdb and start interrupting the code flow to see where things are getting off track.
The integration tests currently live in the Horizon repository, see here, which also contains instructions on how to run the tests. To make integration tests more understandable and maintainable, the Page Object pattern is used throughout them.
Warning
To enable integration tests support before running them, please copy openstack_dashboard/local/local_settings.d/_20_integration_tests_scaffolds.py.example to openstack_dashboard/local/local_settings.d/_20_integration_tests_scaffolds.py and then run ./manage.py collectstatic –clear && ./manage.py compress.
Horizon repository also provides two shell scripts, which are executed in pre_test_hook and post_test_hook respectively. Pre hook is generally used for modifying test environment, while post hook is used for running actual integration tests with tox and collecting test artifacts. Thanks to the incorporating all modifications to tests into Horizon repository, one can alter both tests and test environment and see the immediate results in Jenkins job output.
Within any web application’s user interface (UI) there are areas that the tests interact with. A Page Object simply models these as objects within the test code. This reduces the amount of duplicated code; if the UI changes, the fix needs only be applied in one place.
Page Objects can be thought of as facing in two directions simultaneously. Facing towards the developer of a test, they represent the services offered by a particular page. Facing away from the developer, they should be the only thing that has a deep knowledge of the structure of the HTML of a page (or part of a page). It is simplest to think of the methods on a Page Object as offering the “services” that a page offers rather than exposing the details and mechanics of the page. As an example, think of the inbox of any web-based email system. Amongst the services that it offers are typically the ability to compose a new email, to choose to read a single email, and to list the subject lines of the emails in the inbox. How these are implemented should not matter to the test.
Because the main idea is to encourage the developer of a test to try and think about the services that they are interacting with rather than the implementation, Page Objects should seldom expose the underlying WebDriver instance. To facilitate this, methods on the Page Object should return other Page Objects. This means that we can effectively model the user’s journey through the application.
Another important thing to mention is that a Page Object need not represent an entire page. It may represent a section that appears many times within a site or page, such as site navigation. The essential principle is that there is only one place in your test suite with knowledge of the structure of the HTML of a particular (part of a) page. With this in mind, a test developer builds up regions that become reusable components (example of a base form). These properties can then be redefined or overridden (e.g. selectors) in the actual pages (subclasses) (example of a tabbed form).
The page objects are read-only and define the read-only and clickable elements of a page, which work to shield the tests. For instance, from the test perspective, if “Logout” used to be a link but suddenly becomes an option in a drop-down menu, there are no changes (in the test itself) because it still simply calls the “click_on_logout” action method.
This approach has two main aspects:
There is little that is Selenium-specific in the Pages, except for the properties. There is little coupling between the tests and the pages. Writing the tests becomes like writing out a list of steps (by using the previously mentioned action methods). One of the key points, particularly important for this kind of UI driven testing is to isolate the tests from what is behind them.
Even perfectly designed Page Objects are not a guarantee that your integration test will not ever fail. This can happen due to different causes:
The first and most anticipated kind of failure is the inability to perform a testing scenario by a living person simply because some OpenStack service or Horizon itself prevents them from doing so. This is exactly the kind that integration tests are designed to catch. Let us call them “good” failures.
All other kinds of failures are unwanted and could be roughly split into the two following categories:
An inconvenient thing about reading test results in the console.html file attached to every gate-horizon-dsvm-integration finished job is that the test failure may appear either as failure (assertion failed), or as error (expected element didn’t show up). In both cases an inquirer should suspect a legitimate failure first (i.e., treat errors as failures). Unfortunately, no clear method exists for the separation of “good” from from “bad” failures. Each case is unique and full of mysteries.
The Horizon testing mechanism tries to alleviate this ambiguity by providing several facilities to aid in failure investigation:
The best way to solve the cause of test failure is running and debugging the troublesome test locally. You could use pdb or Python IDE of your choice to stop test execution in arbitrary points and examining various Page Objects attributes to understand what they missed. Looking at the real page structure in browser developer tools also could explain why the test fails. Sometimes it may be worth to place breakpoints in JavaScript code (provided that static is served uncompressed) to examine the objects of interest. If it takes long, you may also want to increase the webdriver’s timeout so it will not close browser windows forcefully. Finally, sometimes it may make sense to examine the contents of logs directory, especially apache logs - but that is mostly the case for the “good” failures.
So, you are going to write your first integration test and looking for some guidelines on how to do it. The first and the most comprehensive source of knowledge is the existing codebase of integration tests. Look how other tests are written, which Page Objects they use and learn by copying. Accurate imitation will eventually lead to a solid understanding. Yet there are few things that may save you some time when you know them in advance.
Below is the filesystem structure that test helpers rely on.:
horizon/
└─ openstack_dashboard/
└─ test/
└─ integration_tests/
├─ pages/
│ ├─ admin/
│ │ ├─ __init__.py
│ │ └─ system/
│ │ ├─ __init__.py
│ │ └─ flavorspage.py
│ ├─ project/
│ │ └─ compute/
│ │ ├─ __init__.py
│ │ ├─ access_and_security/
│ │ │ ├─ __init__.py
│ │ │ └─ keypairspage.py
│ │ └─ imagespage.py
│ └─ navigation.py
├─ regions/
├─ tests/
├─ config.py
└─ horizon.conf
New tests are put into integration_tests/tests, where they are grouped by the kind of entities being tested (test_instances.py, test_networks.py, etc). All Page Objects to be used by tests are inside pages/directory, the nested directory structure you see within it obeys the value of Navigation.CORE_PAGE_STRUCTURE you can find at pages/navigation.py module. The contents of the CORE_PAGE_STRUCTURE variable should in turn mirror the structure of standard dashboard sidebar menu. If this condition is not met, the go_to_<pagename>page() methods which are generated automatically at runtime will have problems matching the real sidebar items. How are these go_to_*page() methods are generated? From the sidebar’s point of view, dashboard content could be at most four levels deep: Dashboard, Panel Group, Panel and Tab. Given the mixture of these entities in existing dashboards, it was decided that:
As you might have noticed, method name components are chosen from normalized items of the CORE_PAGE_STRUCTURE dictionary, where normalization means replacing spaces with _ symbol and & symbol with and, then downcasing all symbols.
Once the go_to_*page() method’s name is parsed and the proper menu item is matched in a dashboard, it should return the proper Page Object. For that to happen a properly named class should reside in a properly named module located in the right place of the filesystem. More specifically and top down:
TableRegion binds to the HTML Horizon table using the TableRegion‘s name attribute. To bind to the proper table this attribute has to be the same as the name attribute of a Meta subclass of a corresponding tables.DataTable descendant in the Python code. TableRegion provides all the needed facilities for solving the following table-related tasks.
when interacting with modal and non-modal forms three flavors of form wrappers can be used.
BaseFormRegion is used for simplest forms which are usually ‘Submit’ / ‘Cancel’ dialogs with no fields to be filled.
FormRegion is the most used wrapper which provides interaction with the fields within that form. Every field is backed by its own wrapper class, while the FormRegion acts as a container which initializes all the field wrappers in its __init__() method. Field mappings passed to __init__() could be
- either a tuple of string labels, in that case the same label is used for referencing the field in test code and for binding to the HTML input (should be the same as name attribute of that widget, could be seen in Django code defining that form in Horizon)
- or a dictionary, where the key will be used for referencing the test field and the value will be used for binding to the HTML input. Also it is feasible to provide values other than strings in that dictionary - in this case they are meant to be a Python class. This Python class will be initialized as any BaseRegion is usually initialized and then the value’s key will be used for referencing this object. This is useful when dealing with non-standard widgets in forms (like Membership widget in Create/Edit Project form or Networks widget in Launch Instance form).
TabbedFormRegion is a slight variation of FormRegion, it has several tabs and thus can accept a tuple of tuples / dictionaries of field mappings, where every tuple corresponds to a tab of a real form, binding order is that first tuple binds to leftmost tab, which has index 0. Passing default_tab other than 0 to TabbedFormRegion.__init__ we can make the test form to be created with the tab other than the leftmost being shown immediately. Finally the method switch_to allows us to switch to any existing form’s tab.
MessageRegion is a small region, but is very important for asserting that everything goes well in Horizon under test. Technically, the find_message_and_dismiss method belongs to BasePage class, but whenever it is called, regions.messages module is imported as well to pass a messages.SUCCESS / messages.ERROR argument into. The method returns True / False depending on if the specified message was found and dismissed (which could be then asserted for).
First, for more details on writing a Horizon plugin please refer to Horizon Plugin. Second, there are 2 possible setups when running integration tests for Horizon plugins.
The first setup, which is suggested to be used in gate of *-dashboard plugins is to get horizon as a dependency of a plugin and then run integration tests using horizon.conf config file inside the plugin repo. This way the plugin augments the location of Horizon built-in Page Objects with the location of its own Page Objects, contained within the plugin_page_path option and the Horizon built-in nav structure with its own nav structure contained within plugin_page_structure. Then the plugin integration tests are run against core Horizon augmented with just this particular plugin content.
The second setup may be used when it is needed to run integration tests for Horizon + several plugins. In other words, content from several plugins is merged into core Horizon content, then the combined integration tests from core Horizon and all the involved plugins are run against the resulting dashboards. To make this possible both options plugin_page_path and plugin_page_structure have MultiStrOpt type. This means that they may be defined several times and all the specified values will be gathered in a list, which is iterated over when running integration tests. In this setup it’s easier to run the tests from Horizon repo, using the horizon.conf file within it.
Also keep in mind that plugin_page_structure needs to be a strict JSON string, w/o trailing commas etc.