Are rollups supported in Dashboards(Kibana)?

Plugins Index Management
1

Hi. I’ve been messing with rollup jobs, but I don’t seem to be able to use the resulting rollup- index as part of an index pattern in Dashboards for use with a visualization. According to documentation, when creating a new index pattern there is supposed to be a dropdown on the “Create index pattern” button to select “Rollup index pattern”. However, that dropdown does not appear for me. Is this something that has to be explicitly enabled in Dashboards/Kibana? Or is it just not yet supported?

My data index, and its corresponding rollup index, are both in a remotely connected cross-search cluster.
The rollup index pattern I would make in Dashboards would be like: cluster1:rollup-app1-logs,cluster1:app1-logs
Is it relevant that a remotely connected cluster is involved? Both clusters are the last available version of OpenDistro, 1.13.2

2

[I’m moving this to the index management category - you’re more likely to get a good answer in that category]

1 Like
3

Maybe answering my own question, I think searching rollups is not (fully?) supported in OpenDistro 1.13.2

curl --no-progress-meter --insecure -u barry:redacted -X POST "https://localhost:9200/rollup-my-index-family,my-index-family-*/_search?pretty" -H 'Content-Type: application/json' -d @aggs_query7.json
{
  "took" : 254546,
  "timed_out" : false,
  "_shards" : {
    "total" : 210,
    "successful" : 180,
    "skipped" : 0,
    "failed" : 30,
    "failures" : [
      {
        "shard" : 0,
        "index" : "rollup-my-index-family",
        "node" : "grifaXJ7TNKEZVd2r20_oA",
        "reason" : {
          "type" : "illegal_argument_exception",
          "reason" : "Searching rollup index with other indices is not supported currently"
        }
      }
    ]
  },........

Is there an equivalent of _rollup_search in OpenDistro/OpenSearch? I can’t seem to find it. This documentation seems to indicate _search should do the trick: Index Rollups - Open Distro Documentation

I’m starting to guess that the rollup job stuff is enabled/supported, but actually searching the resulting rollup index is not (fully?) supported in the last final OpenDistro release (1.13.2).

4

Hi @mhoydis,

Yes, we do support rollups in dashboards.

Just for some clarification, you mentioned there being a “Rollup index pattern” selection in dropdown for index patterns. That is for Elastic’s Rollup feature.

For our implementation you should be able to select the index as you normally would, but we do currently have a limitation which is you cannot search across rollup and non-rollup indices yet in the same queries. There is some extra logic for merging time intervals which we haven’t tackled yet which is why it’s not supported and you see that message “Searching rollup index with other indices is not supported currently”.

5

Hi, @dbbaughe. Thanks for that insight. I’ll try to work around that limitation in the short term.

I tried to create a new Index Pattern in Kibana for just my rollup index: rollup-my-index-family
Kibana successfully created the Index Pattern, but when I go to use it in Discover, I get this error message:

: Rollup search must have size explicitly set to 0, but found 500

Error: Bad Request
    at Fetch._callee3$ (https://kibana7.redacted.net/36136/bundles/core/core.entry.js:6:59535)
    at tryCatch (https://kibana7.redacted.net/36136/bundles/plugin/opendistroQueryWorkbenchKibana/opendistroQueryWorkbenchKibana.plugin.js:1:32004)
    at Generator.invoke [as _invoke] (https://kibana7.redacted.net/36136/bundles/plugin/opendistroQueryWorkbenchKibana/opendistroQueryWorkbenchKibana.plugin.js:1:35968)
    at Generator.forEach.prototype.<computed> [as next] (https://kibana7.redacted.net/36136/bundles/plugin/opendistroQueryWorkbenchKibana/opendistroQueryWorkbenchKibana.plugin.js:1:33129)
    at fetch_asyncGeneratorStep (https://kibana7.redacted.net/36136/bundles/core/core.entry.js:6:52652)
    at _next (https://kibana7.redacted.net/36136/bundles/core/core.entry.js:6:52968)

This is Kibana (opendistroforelasticsearch-kibana 1.13.2) , the last available release of OpenDistro.
Is this behavior different in OpenSearch RC1?

6

Hey @mhoydis,

So when you rollup data into the rollup index, we have to change the structure of the data to roll it up. e.g. you might choose sum, max, min on one field, but we have to store that as 3 fields or you might choose avg on a field and we have to store the sum and the value count of that field to compute the avg for when you try to do average of averages.

Because of this as you can imagine if you tried to execute FooQuery against your original indices, it would not work as is against the new rollup index because all the data has been restructured. To make this work we actually transform the query during the request to match the new structure internally. That comes at a cost though of not being able to easily explore the data in discovery currently. We require the size to be set to 0, i.e. we just support queries on it. So you can use it for visualizations/dashboards. If you absolutely must be able to query the rollup index you can set the rollup search enabled cluster setting (opendistro.rollup.search.enabled) to false and we won’t intercept the request and transform it anymore. Then you should be able to look around on discover, but just know the limitations from above.

8

OK, that makes sense that the rollup index would only be applicable to the Visualization feature. Thanks.