Data stream visualization with dynamic mappings

OpenSearch Dashboards
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

OpenSearch 1.3

Describe the issue:

We’re not able to reference the data stream in visualizations.
I can work around this by simply creating an index pattern that matches the data stream name.
This displays all the dynamic fields just fine in Discovery.
However when we try to use the fields in actual components it refuses to see any fields that aren’t explicitly mapped on the index pattern.

Configuration:

{
  "persistent" : {
    "action.auto_create_index" : "true",
    "aes.jetty.admission_control.global_cpu_usage.window_duration" : "300s",
    "cluster.max_shards_per_node" : "1000",
    "cluster.metadata.perf_analyzer.state" : "3",
    "cluster.routing.allocation.awareness.force.zone.values" : "xx-xxxxx-xx",
    "cluster.routing.allocation.cluster_concurrent_rebalance" : "2",
    "cluster.routing.allocation.disk.watermark.flood_stage" : "1.7138442993164062gb",
    "cluster.routing.allocation.disk.watermark.high" : "3.4276885986328125gb",
    "cluster.routing.allocation.disk.watermark.low" : "5.141532897949219gb",
    "cluster.routing.allocation.load_awareness.provisioned_capacity" : "2",
    "cluster.routing.allocation.load_awareness.skew_factor" : "50.0",
    "cluster.routing.allocation.node_concurrent_recoveries" : "2",
    "cluster.routing.allocation.node_initial_primaries_recoveries" : "4",
    "indices.recovery.max_bytes_per_sec" : "60mb",
    "plugins.index_state_management.template_migration.control" : "-1",
    "search_backpressure.node_duress.heap_threshold" : "0.85"
  },
  "transient" : {
    "aes.jetty.admission_control.global_cpu_usage.window_duration" : "300s",
    "cluster.routing.allocation.awareness.force.zone.values" : "xx-xxxxx-xx",
    "cluster.routing.allocation.cluster_concurrent_rebalance" : "2",
    "cluster.routing.allocation.disk.watermark.flood_stage" : "1.7138442993164062gb",
    "cluster.routing.allocation.disk.watermark.high" : "3.4276885986328125gb",
    "cluster.routing.allocation.disk.watermark.low" : "5.141532897949219gb",
    "cluster.routing.allocation.exclude.di_number" : "",
    "cluster.routing.allocation.load_awareness.provisioned_capacity" : "2",
    "cluster.routing.allocation.load_awareness.skew_factor" : "50.0",
    "cluster.routing.allocation.node_concurrent_recoveries" : "2",
    "cluster.routing.allocation.node_initial_primaries_recoveries" : "4",
    "indices.recovery.max_bytes_per_sec" : "60mb",
    "search_backpressure.node_duress.heap_threshold" : "0.85"
  }
}

Relevant Logs or Screenshots:

Sorry, new user restrictions meant I had to spread this over multiple posts

Discovery, using workaround index pattern with dynamic fields

image
image1705×786 156 KB

2

API showing data stream has fields discovered

image
image1721×597 93.9 KB

3

Visualization refusing to see dynamic fields

image
image514×779 28.5 KB

4

Workaround index pattern

image
image1229×667 42.3 KB

5

The images lead me to think of 2 possibilities:

  1. The index pattern you have is logs-seat-map-serice.raw-dev which is a fixed pattern.

    • This index pattern only matches ONE single index or alias named logs-seat-map-serice.raw-dev.
    • To have an index pattern match the logs-seat-map-serice.raw-dev-00001 index and similarly named index, the pattern should be named logs-seat-map-serice.raw-dev-*.

  2. If you are using ISM and in fact logs-seat-map-serice.raw-dev is an alias that covers logs-seat-map-serice.raw-dev-00001 (which is most likely your case):

    • The index pattern appears to have been created when not much data was ingested (or any at all)

Irrespective of which one applies to you, when an index pattern is created, it takes note of the fields that it can find and records that. After some data is ingested, if any new fields are introduced, the index pattern won’t be aware of them because it doesn’t proactively scan your data - that would be very expensive.

To instruct the index pattern to revisit the data and update its internal records, you can use the tiny refresh button on the top right:

refresh-fields
refresh-fields1229×667 25.1 KB

You would know that you need to refresh the index pattern if you see any unknown fields in Discover:

unknown-fields
unknown-fields650×786 29.1 KB

Beside all that, many of the aggregations of OpenSearch might not like text fields and as a result, they might not appear as fields when building visualizations. I would encourage you to add mapping to specific fields that you care for. For example ActionName sounds like a field that should be mapped as a keyword.

Since it is not practical to add mapping to each and every index, you could create a template or index template that specifically matches your indices and add the field mapping to it. I would recommend using the same pattern as your index pattern in your template.

If you choose to make templates or index templates, keep in mind that they wouldn’t impact existing indices and are only applied to indices created after the creating / updating the template. For existing indices, you can either delete them and reingest, or copy them to a temporary name (reindex), clean up old stuff, and reindex back.