Settings¶
Hypothesis tries to have good defaults for its behaviour, but sometimes that’s not enough and you need to tweak it.
The mechanism for doing this is the settings
object.
You can set up a @given based test to use this using a settings decorator:
@given
invocation as follows:
from hypothesis import given, settings
@given(integers())
@settings(max_examples=500)
def test_this_thoroughly(x):
pass
This uses a settings
object which causes the test to receive a much larger
set of examples than normal.
This may be applied either before or after the given and the results are the same. The following is exactly equivalent:
from hypothesis import given, settings
@settings(max_examples=500)
@given(integers())
def test_this_thoroughly(x):
pass
Available settings¶
-
class
hypothesis.
settings
(parent=None, **kwargs)[source]¶ A settings object controls a variety of parameters that are used in falsification. These may control both the falsification strategy and the details of the data that is generated.
Default values are picked up from the settings.default object and changes made there will be picked up in newly created settings.
-
database_file
¶ database: An instance of hypothesis.database.ExampleDatabase that will be used to save examples to and load previous examples from. May be None in which case no storage will be used. default value: ‘/build/python-hypothesis-bAc2Rw/python-hypothesis-3.6.0/docs/.hypothesis/examples’
-
database
¶ An ExampleDatabase instance to use for storage of examples. May be None.
If this was explicitly set at settings instantiation then that value will be used (even if it was None). If not and the database_file setting is not None this will be lazily loaded as an SQLite backed ExampleDatabase using that file the first time this property is accessed on a particular thread.
-
max_examples
¶ Once this many satisfying examples have been considered without finding any counter-example, falsification will terminate. default value: 200
-
max_iterations
¶ Once this many iterations of the example loop have run, including ones which failed to satisfy assumptions and ones which produced duplicates, falsification will terminate. default value: 1000
-
max_shrinks
¶ Once this many successful shrinks have been performed, Hypothesis will assume something has gone a bit wrong and give up rather than continuing to try to shrink the example. default value: 500
-
min_satisfying_examples
¶ Raise Unsatisfiable for any tests which do not produce at least this many values that pass all assume() calls and which have not exhaustively covered the search space. default value: 5
-
perform_health_check
¶ If set to True, Hypothesis will run a preliminary health check before attempting to actually execute your test. default value: True
-
stateful_step_count
¶ Number of steps to run a stateful program for before giving up on it breaking. default value: 50
-
strict
¶ If set to True, anything that would cause Hypothesis to issue a warning will instead raise an error. Note that new warnings may be added at any time, so running with strict set to True means that new Hypothesis releases may validly break your code.
You can enable this setting temporarily by setting the HYPOTHESIS_STRICT_MODE environment variable to the string ‘true’. default value: False
-
suppress_health_check
¶ A list of health checks to disable default value: []
-
timeout
¶ Once this many seconds have passed, falsify will terminate even if it has not found many examples. This is a soft rather than a hard limit - Hypothesis won’t e.g. interrupt execution of the called function to stop it. If this value is <= 0 then no timeout will be applied. default value: 60
-
Seeing intermediate result¶
To see what’s going on while Hypothesis runs your tests, you can turn
up the verbosity setting. This works with both find()
and @given
.
(The following examples are somewhat manually truncated because the results of verbose output are, well, verbose, but they should convey the idea).
>>> from hypothesis import find, settings, Verbosity
>>> from hypothesis.strategies import lists, booleans
>>> find(lists(booleans()), any, settings=settings(verbosity=Verbosity.verbose))
Found satisfying example [True, True, ...
Shrunk example to [False, False, False, True, ...
Shrunk example to [False, False, True, False, False, ...
Shrunk example to [False, True, False, True, True, ...
Shrunk example to [True, True, True]
Shrunk example to [True, True]
Shrunk example to [True]
[True]
>>> from hypothesis import given
>>> from hypothesis.strategies import integers
>>> @given(integers())
... @settings(verbosity=Verbosity.verbose)
... def test_foo(x):
... assert x > 0
...
>>> test_foo()
Trying example: test_foo(x=-565872324465712963891750807252490657219)
Traceback (most recent call last):
...
File "<stdin>", line 3, in test_foo
AssertionError
Trying example: test_foo(x=565872324465712963891750807252490657219)
Trying example: test_foo(x=0)
Traceback (most recent call last):
...
File "<stdin>", line 3, in test_foo
AssertionError
Falsifying example: test_foo(x=0)
Traceback (most recent call last):
...
AssertionError
The four levels are quiet, normal, verbose and debug. normal is the default, while in quiet Hypothesis will not print anything out, even the final falsifying example. debug is basically verbose but a bit more so. You probably don’t want it.
You can also override the default by setting the environment variable
HYPOTHESIS_VERBOSITY_LEVEL
to the name of the level you want. So e.g.
setting HYPOTHESIS_VERBOSITY_LEVEL=verbose
will run all your tests printing
intermediate results and errors.
Building settings objects¶
settings can be created by calling settings with any of the available settings values. Any absent ones will be set to defaults:
>>> from hypothesis import settings
>>> settings()
settings(average_list_length=25.0, database_file='/home/david/projects/hypothesis/.hypothesis/examples.db', derandomize=False, max_examples=200, max_iterations=1000, max_shrinks=500, min_satisfying_examples=5, stateful_step_count=50, strict=False, timeout=60, verbosity=Verbosity.normal)
>>> settings().max_examples
200
>>> settings(max_examples=10).max_examples
10
You can also copy settings off other settings:
>>> s = settings(max_examples=10)
>>> t = settings(s, max_iterations=20)
>>> s.max_examples
10
>>> t.max_iterations
20
>>> s.max_iterations
1000
>>> s.max_shrinks
500
>>> t.max_shrinks
500
Default settings¶
At any given point in your program there is a current default settings, available as settings.default. As well as being a settings object in its own right, all newly created settings objects which are not explicitly based off another settings are based off the default, so will inherit any values that are not explicitly set from it.
You can change the defaults by using profiles (see next section), but you can also override them locally by using a settings object as a context manager
>>> with settings(max_examples=150):
... print(settings.default.max_examples)
... print(settings().max_examples)
...
150
150
>>> settings().max_examples
200
Note that after the block exits the default is returned to normal.
You can use this by nesting test definitions inside the context:
from hypothesis import given, settings
with settings(max_examples=500):
@given(integers())
def test_this_thoroughly(x):
pass
All settings objects created or tests defined inside the block will inherit their defaults from the settings object used as the context. You can still override them with custom defined settings of course.
Warning: If you use define test functions which don’t use @given inside a context block, these will not use the enclosing settings. This is because the context manager only affects the definition, not the execution of the function.
settings Profiles¶
Depending on your environment you may want different default settings. For example: during development you may want to lower the number of examples to speed up the tests. However, in a CI environment you may want more examples so you are more likely to find bugs.
Hypothesis allows you to define different settings profiles. These profiles can be loaded at any time.
Loading a profile changes the default settings but will not change the behavior of tests that explicitly change the settings.
>>> from hypothesis import settings
>>> settings.register_profile("ci", settings(max_examples=1000))
>>> settings().max_examples
200
>>> settings.load_profile("ci")
>>> settings().max_examples
1000
Instead of loading the profile and overriding the defaults you can retrieve profiles for specific tests.
>>> with settings.get_profile("ci"):
... print(settings().max_examples)
...
1000
Optionally, you may define the environment variable to load a profile for you. This is the suggested pattern for running your tests on CI. The code below should run in a conftest.py or any setup/initialization section of your test suite. If this variable is not defined the Hypothesis defined defaults will be loaded.
>>> from hypothesis import settings
>>> settings.register_profile("ci", settings(max_examples=1000))
>>> settings.register_profile("dev", settings(max_examples=10))
>>> settings.register_profile("debug", settings(max_examples=10, verbosity=Verbosity.verbose))
>>> settings.load_profile(os.getenv(u'HYPOTHESIS_PROFILE', 'default'))
If you are using the hypothesis pytest plugin and your profiles are registered
by your conftest you can load one with the command line option --hypothesis-profile
.
$ py.test tests --hypothesis-profile <profile-name>