Mitaka Series Release Notes

0.2.1

Prelude

The initial mitaka release (0.2) did not support ElasticSearch 2.x. ElasticSearch made several changes to internal index data mappings that were not backwards compatible with ElasticSearch 1.x. This release has all the changes required for Searchlight to support both ElasticSearch 1.x and 2.x.

Searchlight now supports oslo pools instead of requiring a different topic for each listener. This simplifies deployment configuration. Previously, services would have to be specifically configured to send notifications to the Searchlight topic. With this change, Searchlight can share topics with other services such as Ceilometer.

Upgrade Notes

  • To support ElasticSearch 1.x and 2.x, several internal data mappings have been updated. If you previously deployed Searchlight, you will need to reindex your resource data using searchlight-manage index sync. You may re-index everything or limit it to OS::Nova::Server, OS::Glance::Image, OS::Glance::Metadef, OS::Cinder:Volume, and OS::Cinder::Snapshot.

  • To start using a pool with a shared topic instead of separate topics:

    • Update the respective service configuration files for searchlight enabled plugins to only publish to a single topic (e.g. set notification_topics = notifications). If you have Ceilometer enabled, this must be the same topic which Ceilometer uses. You also typically must restart each service for the change to take effect. For example, in nova.conf, glance-api.conf, cinder.conf, neutron.conf, and designate.conf:

    notification_topics = notifications

    • Update searchlight.conf. In the [resource_plugin] section, set notifications_topic to match the shared topic that you set in the service configurations. In the [listener] sections, set notifications_pool to your desired pool name. The default name starting in Newton is searchlight. The notifications_pool does not have to match anything from other services configuration files. For example:

    [resource_plugin]
notifications_topic = notifications

[listener]
notifications_pool = searchlight

Bug Fixes

  • Bug 1570213 Apply query to highlight query

  • Bug 1532010 Ensure consistency in mapping field types

  • Bug 1570674 Fix unicode error when booting instance from volume

  • Bug 1570199 Fix inconsistent mapping in image plugin

  • Bug 1568709 Remove port.create.end handler from nova

  • Bug 1565015 Add volume.retype event

  • Bug 1583215 Correct Cinder exchange value

  • Bug 1583215 Enable notification messaging pools

0.2.0

Prelude

Adds cinder plugins for volumes and snapshots.

Added separation of administrator-only data at an index level, which allows full faceting and searching of sensitive data by those with access to it.

In order to allow policy files to more be useful in Searchlight, we now support limited control over individual resource types via oslo policy control.

Remove the ‘index’ API. It is currently a passthrough to elasticsearch, allowing direct index access and bypassing any document transforms. As such it is dangerous, and only marginally useful. Administrators wishing to do this should do so directly in Elasticsearch.

Swift plugin for searchlight

Ability added to reindex data while allowing indexing of notifications, and therefore returning accurate search results during the indexing operation.

A fix for a couple of parent-child plugin relationship issues related to initial indexing. Child plugins can no longer be indexed separately from their parents.

Config options for searchlight-api and searchlight-listener are grouped under config groups [api] and [listener].

The default config file for searchlight was searchlight-api.conf despite the fact that this config file was used by both the API and the listener processes. Therefore, we decided to rename the default config file to searchlight.conf. This change also prompted us to to follow the convention used by many other projects and simplify searchlight-api-paste.ini to api-paste.ini.

New Features

  • Cinder plugins for volumes and snapshots.

  • Plugins with sensitive, administrator-only data are indexed twice; once with that data and once without. Searches are directed to the appropriate version depending on the context of the user.

  • Adds neutron plugins for networks, subnets, ports and routers.

  • Policy rules can now be specified per resource type and per operation per resource type.

  • Removes the /index api and related tests. If an admin needs to directly modify data in Elasticsearch they should do so in Elasticsearch.

  • Three resource types are introduced for swift plugin. OS::Swift:Account ->OS::Swift::Container –>OS::Swift::Object

  • Reindexing of data now takes place in a separate Elasticsearch index. During indexing, incoming events are processed into both the old and new indices. Once indexing is complete, an Elasticsearch alias is used to switch searches seamlessly onto the new index, at which time the old one is removed.

  • Plugin configuration refers to resource_group_name rather than index_name to represent groups of resources separate from the logical Elasticsearch index. This setting is used to determine index and alias names.

Known Issues

  • Backups cannot currently be implemented because the cinder API does not provide enough information for RBAC. They will be added when that is resolved.

  • Neutron resources do not provide dates (created_at or updated_at). created_at is left empty; updated_at is taken from the server (and will be adjusted to use the notification timestamp at a later time)

  • RBAC is still separate from policy restriction; this is targeted for the Newton release.

  • The Swift service currently doesn’t send notifications. Follow the swift plugin documentation for current solutions.

  • Reindexing must now take place across all resource types that share the index (it is not possible to re-index a specific resource type). This is a limitation that Elasticsearch 2.x would likely have forced upon us anyway. A subsequent patch will allow more efficient reindexing for associated resource types.

Upgrade Notes

  • Options ‘bind_host’, ‘bind_port’, ‘backlog’, ‘tcp_keepidle’, ‘ca_file’, ‘cert_file’, ‘key_file’, ‘workers’, ‘max_header_line’, ‘public_endpoint’, and ‘http_keepalive’ are moved to [api] group. Options ‘listener_workers’ is renamed to ‘workers’ and moved to [listener] group. Options ‘host’ and ‘http_timeout’ are removed since they are not currently used anywhere.

  • During an upgrade to Mitaka, operators must rename the previous configured searchlight-api.conf to searchlight.conf and searchlight-api-paste.ini to api-paste.ini.

Security Issues

  • Removes the admin-only ‘/index’ api, which allowed an unnecessary amount of access to Elasticsearch.

Bug Fixes

  • Child plugins cannot be indexed on their own, only as a result of indexing their parent. This fixes an issue whereby indexing only a parent plugin could leave orphaned children (or vice versa).

  • Applies mappings for parent/child plugins simultaneously. Under elasticsearch 2.x, child mappings must be created at the same time or before parents, not the other way round.

  • The workers for listener was always 1, so config listener_workers was not work as expected. Now operators can set ‘workers’ under group [listener] to running multiple listener processes.