Elasticsearch Publishing

The Web UI Search Screen offers search functionality that requires STEP data to be published to an Elasticsearch cluster that is accessible from the STEP application servers. For the end user, the Search Screen provides search results that can be refined and modified on-the-fly.

Important: Before starting the configuration outlined in this topic, contact your Stibo Systems account manager or partner manager for assistance. Activation and configuration for the Search screen, Elasticsearch, and corresponding components / functionality should not be done without the assistance of Stibo Systems.

Important: To ensure current STEP data is displayed on the Web UI Search Screen, verify that the event processor indicated by the active Elasticsearch configuration is enabled, is set to Read Events, and is running on a reasonable schedule.

Data displayed in workbench is fully aligned with data available from the Search Screen only when all events are processed.

The amount of data being published directly impacts the amount of time required for the events to be processed. When publishing large amounts of data, it is recommended to invoke the event process during user down time, such as overnight or on the weekend.

Publish to the Elasticsearch Database

Follow the steps below to publish data from STEP to the Elasticsearch database:

  1. Generate events for the STEP data required in the Search Screen using the republish action as defined in the Event-Based OIEP Forward, Rewind, Purge, and Republish topic of the Data Exchange documentation here. The republish background process generates events for the configured products and/or classifications.
  2. Publish STEP data to the Elasticsearch database by invoking the event processor running the Elasticsearch Configuration as defined in the Running an Event Processor topic of the System Setup / Super User documentation here. The event processor background process creates indexes and publishes products and/or classifications to the Elasticsearch database.

Reindex the Elasticsearch Database

For an existing Elasticsearch database, use one of the following set of steps to reindex Elasticsearch when required. For example, when changes are made by Stibo Systems to the Elasticsearch schema. Both methods remove all data (products and classifications) from the Elasticsearch database and then publishes the current STEP data to Elasticsearch based on the active configuration.

Under rare circumstances, a system can be safely reverted to the previous configuration and index. For example, when a schema change involves functionality not implemented by the system. In such a scenario, the 'Create New Indexes and Republish' method would be used.

Important: When you need to republish a large number of objects, or all objects, first review the number of unread events on the OIEP looking specifically for delete events (see the Event-Based OIEP Queued Events topic in the Data Exchange documentation here).

When there are no unread delete events or old indexes have been deleted, consider skipping the current unread events (via the Forward option) to reduce the time it takes for the processor to finish and avoid processing multiple events for each object (see the Event-Based OIEP Forward, Rewind, Purge, and Republish topic in the Data Exchange documentation here). Then republish all objects to generate one event for each object.

Be aware that if there are unread delete events and you keep old indexes, the Elasticsearch index will contain items that should have been deleted but were not because the delete events were skipped with the Forward action.

For example, publishing to an Elasticsearch database typically involves all products. Before republishing to Elasticsearch, first process delete events, Forward the remaining unread events, and then Republish all by selecting the top-level product node.

Delete Indexes and Republish

This method removes all existing indexes and reuses the active Index Prefix. None of the selected object types and data specifications are changed, but additional configuration may be required.

  1. In Kibana, use the 'Manage Indices' option to delete all Elasticsearch indexes with the Index Prefix of your active Elasticsearch Configuration as defined in the Initial Configuration step of the wizard. For more information, see the Creating an Elasticsearch Configuration topic here.
  2. Generate events for the STEP data required in the Search Screen using the republish action as defined in the Event-Based OIEP Forward, Rewind, Purge, and Republish topic of the Data Exchange documentation here.
  3. Publish STEP data to the Elasticsearch database by invoking the event processor running the Elasticsearch Configuration as defined in the Running an Event Processor topic of the System Setup / Super User documentation here. New indexes are created using the Index Prefix in the configuration.

Create New Indexes and Republish

This method allows you to keep the current index although it will no longer be active. Additional indexes will be created with a new Index Prefix. None of the selected object types and data specifications are changed, but additional configuration may be required.

  1. In STEP, use the Duplicate option on the Maintain menu to duplicate your active Elasticsearch Configuration. Alternatively, export the configuration using STEPXML, modify the ID of the configuration, and reimport the STEPXML.
  2. On the new configuration, change the Index Prefix of your active Elasticsearch Configuration as defined in the Initial Configuration step of the wizard. For more information, see the Creating an Elasticsearch Configuration topic here.
  3. Generate events for the STEP data required in the Search Screen using the republish action as defined in the Event-Based OIEP Forward, Rewind, Purge, and Republish topic of the Data Exchange documentation here.
  4. Publish STEP data to the Elasticsearch database by invoking the event processor running the Elasticsearch Configuration as defined in the Running an Event Processor topic of the System Setup / Super User documentation here. New indexes are created using the Index Prefix in the configuration.