Nested objects are not displayed correctly in Discover dashboard

OpenSearch Dashboards Reporting Plugin
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): OpenSearch v1.3.6, OpenSearch Dashboard v1.3.6, MacOS, Chrome

Describe the issue:

  1. Go to Dev Tools. Create an index testindex2 with mappings
PUT testindex2
{
  "mappings": {
    "properties": {
      "patients": {
        "properties": {
          "name": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "age": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "smoker": {
            "type": "boolean"
          }
        }
      }
    }
  }
}
  1. Insert a document.
PUT testindex2/_doc/100
{ 
  "patients": [ 
    {"name" : "John Doe", "age" : "56", "smoker" : true},
    {"name" : "Mary Major", "age" : "85", "smoker" : false}
  ] 
}
  1. Click on ‘Stack Management’ and create an index pattern for testindex2
  2. Go to the ‘Discover’ page
  3. Click on ‘Filter by Type’ and disable ‘Hide missing fields’
  4. Add patients.age and patients.name from the ‘Available fields’

Referred to the documentation on nested objects.

Expected behavior
There should be 1 hit with 2 columns - patients.name has ["John Doe", "Mary Major"] value, and patients.age has [56, 85] value. However, they are all empty.

Configuration:

Relevant Logs or Screenshots:

image
image2543×1099 176 KB

2

@chelsey, the Discover page is not capable of showing nested objects in the tabular format. The documentation you linked to which mentions “When these objects are stored, they are flattened” is talking about the inner workings of OpenSearch and not what you get from OpenSearch when you query the data. The Discover page simply queries OpenSearch to get the objects that match your query and the JSON output is exactly what it gets back from OpenSearch.

3

Hi @AMoo-Miki , thanks for your response.

Yes, the JSON output is correct for patients. But I would expect that the nested field patients.age returns an aggregated value which is an array [56, 85], not an empty value.

When I search the index, using GET testindex2/_search with the field patients.age from the Dev console, it returns the correct array. The same result should be expected in the Discover page.

This was able to work in Kibana. Is it a missing feature in OpenSearch Dashboard?

4

Yes, the JSON output is correct for patients.

Great so we have the correct data recorded

But I would expect that the nested field patients.age returns an aggregated value which is an array [56, 85], not an empty value.

Since Discover is unable to handle nested fields, it doesn’t see those fields and shows them as empty. The effort for revamping Discover is tracked under “Data Explorer - Revamping Discover to be able to be used for all of OpenSearch Dashboards data exploration use cases · Issue #3056 · opensearch-project/OpenSearch-Dashboards · GitHub”.

When I search the index, using GET testindex2/_search with the field patients.age from the Dev console, it returns the correct array. The same result should be expected in the Discover page.

When I run the query against OpenSearch with

GET testindex2/_search
{
  "_source": "patients.age"
}

… I get an array of objects and not an array of numbers:

...
        "_source" : {
          "patients" : [
            {
              "age" : "56"
            },
            {
              "age" : "85"
            }
          ]
        }
...

This nested object structure contributes to Discover’s inability to render them in the table.

This was able to work in Kibana. Is it a missing feature in OpenSearch Dashboard?

The Kibana versions that I have used couldn’t handle nested objects in the tabular format either; maybe the newer versions do.