Watcher Decision Engine has an external scoring engine plugin interface which gives anyone the ability to integrate an external scoring engine in order to make use of it in a strategy.
This section gives some guidelines on how to implement and integrate custom scoring engines with Watcher. If you wish to create a third-party package for your plugin, you can refer to our documentation for third-party package creation.
Because scoring engines execute a purely mathematical tasks, they typically do not have any additional dependencies. Additional requirements might be defined by specific scoring engine implementations. For example, some scoring engines might require to prepare learning data, which has to be loaded during the scoring engine startup. Some other might require some external services to be available (e.g. if the scoring infrastructure is running in the cloud).
In order to create a new scoring engine you have to:
Here is an example showing how you can write a plugin called NewScorer:
# filepath: thirdparty/new.py
# import path: thirdparty.new
from watcher.decision_engine.scoring import base
class NewScorer(base.ScoringEngine):
def get_name(self):
return 'new_scorer'
def get_description(self):
return ''
def get_metainfo(self):
return """{
"feature_columns": [
"column1",
"column2",
"column3"],
"result_columns": [
"value",
"probability"]
}"""
def calculate_score(self, features):
return '[12, 0.83]'
As you can see in the above example, the calculate_score() method returns a string. Both this class and the client (caller) should perform all the necessary serialization or deserialization.
Optionally, it’s possible to implement a container plugin, which can return a list of scoring engines. This list can be re-evaluated multiple times during the lifecycle of Watcher Decision Engine and synchronized with Watcher Database using the watcher-sync command line tool.
Below is an example of a container using some scoring engine implementation that is simply made of a client responsible for communicating with a real scoring engine deployed as a web service on external servers:
class NewScoringContainer(base.ScoringEngineContainer):
@classmethod
def get_scoring_engine_list(self):
return [
RemoteScoringEngine(
name='scoring_engine1',
description='Some remote Scoring Engine 1',
remote_url='http://engine1.example.com/score'),
RemoteScoringEngine(
name='scoring_engine2',
description='Some remote Scoring Engine 2',
remote_url='http://engine2.example.com/score'),
]
Here below is the abstract ScoringEngine class:
A base class for all the Scoring Engines.
A Scoring Engine is an instance of a data model, to which the learning data was applied.
Please note that this class contains non-static and non-class methods by design, so that it’s easy to create multiple Scoring Engine instances using a single class (possibly configured differently).
Calculates a score value based on arguments passed.
Scoring Engines might be very different to each other. They might solve different problems or use different algorithms or frameworks internally. To enable this kind of flexibility, the method takes only one argument (string) and produces the results in the same format (string). The consumer of the Scoring Engine is ultimately responsible for providing the right arguments and parsing the result.
| Parameters: | features (str) – Input data for Scoring Engine |
|---|---|
| Returns: | A score result |
| Return type: | str |
Defines the configuration options to be associated to this loadable
| Returns: | A list of configuration options relative to this Loadable |
|---|---|
| Return type: | list of oslo_config.cfg.Opt instances |
Returns the description of the Scoring Engine.
The description might contain any human readable information, which might be useful for Strategy developers planning to use this Scoring Engine. It will be also visible in the Watcher API and CLI.
| Returns: | A Scoring Engine description |
|---|---|
| Return type: | str |
Returns the metadata information about Scoring Engine.
The metadata might contain a machine-friendly (e.g. in JSON format) information needed to use this Scoring Engine. For example, some Scoring Engines require to pass the array of features in particular order to be able to calculate the score value. This order can be defined in metadata and used in Strategy.
| Returns: | A Scoring Engine metadata |
|---|---|
| Return type: | str |
Returns the name of the Scoring Engine.
The name should be unique across all Scoring Engines.
| Returns: | A Scoring Engine name |
|---|---|
| Return type: | str |
Here below is the abstract ScoringContainer class:
A base class for all the Scoring Engines Containers.
A Scoring Engine Container is an abstraction which allows to plugin multiple Scoring Engines as a single Stevedore plugin. This enables some more advanced scenarios like dynamic reloading of Scoring Engine implementations without having to restart any Watcher services.
Defines the configuration options to be associated to this loadable
| Returns: | A list of configuration options relative to this Loadable |
|---|---|
| Return type: | list of oslo_config.cfg.Opt instances |
Returns a list of Scoring Engine instances.
| Returns: | A list of Scoring Engine instances | ||
|---|---|---|---|
| Return type: |
|
In order for the Watcher Decision Engine to load your new scoring engine, it must be registered as a named entry point under the watcher_scoring_engines entry point of your setup.py file. If you are using pbr, this entry point should be placed in your setup.cfg file.
The name you give to your entry point has to be unique and should be the same as the value returned by the get_name() method of your strategy.
Here below is how you would proceed to register NewScorer using pbr:
[entry_points]
watcher_scoring_engines =
new_scorer = thirdparty.new:NewScorer
To get a better understanding on how to implement a more advanced scoring engine, have a look at the DummyScorer class. This implementation is not really using machine learning, but other than that it contains all the pieces which the “real” implementation would have.
In addition, for some use cases there is a need to register a list (possibly dynamic, depending on the implementation and configuration) of scoring engines in a single plugin, so there is no need to restart Watcher Decision Engine every time such list changes. For these cases, an additional watcher_scoring_engine_containers entry point can be used.
For the example how to use scoring engine containers, please have a look at the DummyScoringContainer and the way it is configured in setup.cfg. For new containers it could be done like this:
[entry_points]
watcher_scoring_engine_containers =
new_scoring_container = thirdparty.new:NewContainer
The Watcher Decision Engine service will automatically discover any installed plugins when it is restarted. If a Python package containing a custom plugin is installed within the same environment as Watcher, Watcher will automatically make that plugin available for use.
At this point, Watcher will scan and register inside the Watcher Database all the scoring engines you implemented upon restarting the Watcher Decision Engine.
In addition, watcher-sync tool can be used to trigger Watcher Database synchronization. This might be used for “dynamic” scoring containers, which can return different scoring engines based on some external configuration (if they support that).