Data stream visualization with dynamic mappings

OpenSearch Dashboards
5

The images lead me to think of 2 possibilities:

  1. The index pattern you have is logs-seat-map-serice.raw-dev which is a fixed pattern.

    • This index pattern only matches ONE single index or alias named logs-seat-map-serice.raw-dev.
    • To have an index pattern match the logs-seat-map-serice.raw-dev-00001 index and similarly named index, the pattern should be named logs-seat-map-serice.raw-dev-*.

  2. If you are using ISM and in fact logs-seat-map-serice.raw-dev is an alias that covers logs-seat-map-serice.raw-dev-00001 (which is most likely your case):

    • The index pattern appears to have been created when not much data was ingested (or any at all)

Irrespective of which one applies to you, when an index pattern is created, it takes note of the fields that it can find and records that. After some data is ingested, if any new fields are introduced, the index pattern won’t be aware of them because it doesn’t proactively scan your data - that would be very expensive.

To instruct the index pattern to revisit the data and update its internal records, you can use the tiny refresh button on the top right:

refresh-fields
refresh-fields1229×667 25.1 KB

You would know that you need to refresh the index pattern if you see any unknown fields in Discover:

unknown-fields
unknown-fields650×786 29.1 KB

Beside all that, many of the aggregations of OpenSearch might not like text fields and as a result, they might not appear as fields when building visualizations. I would encourage you to add mapping to specific fields that you care for. For example ActionName sounds like a field that should be mapped as a keyword.

Since it is not practical to add mapping to each and every index, you could create a template or index template that specifically matches your indices and add the field mapping to it. I would recommend using the same pattern as your index pattern in your template.

If you choose to make templates or index templates, keep in mind that they wouldn’t impact existing indices and are only applied to indices created after the creating / updating the template. For existing indices, you can either delete them and reingest, or copy them to a temporary name (reindex), clean up old stuff, and reindex back.

1 Like