How to add layers to a map in OpenSearch Dashboard 1.2.0

Thank you for the help Lionel. I am having trouble with the DSL query. If I want to return back the actual documents, not an aggregation, I was unable to find a way to do this with the DSL query. I tried google to no avail.

In your example (which I am very grateful for you providing), I see:

{
  "sample": {
    "index": "opensearch_dashboards_sample_data_ecommerce",
    "query": {
      "bool": {
        "must": [
          "_DASHBOARD_CONTEXT_",
          "_TIME_RANGE_[order_date]"
        ]
      }
    },
    "aggs": {
      "customer": {
        "terms": {
          "field": "customer_full_name.keyword",
            "size": 5
        }
      }
    }
  }
}

If I wanted to get back all the documents so I could put them on a map, what is the key, it isn’t “aggs” is it? Sorry for all the questions.

OK, lets say I want to use the transform vis to show a list of tracks. So I have a query like this:

{
“sample”: {
“index”: “tracks”,
“query”: {
“bool”: {
“must”: [
]
}
},

“fields”: [
“trackId”,
“correlFactor”
],
“_source”: false
}
}

And a JS like this:
({
ecommerce: function() {

return this.response.sample.hits.hits[0].fields.trackId; <-- What do I put in here to return???

}
}
);

And my template is:

List of best buyers from function

{{#meta.ecommerce}} {{/meta.ecommerce}}
Full name Number or orders
{{key}} {{doc_count}}

I guess what I am struggling with here is how to access the various track data that is being returned?

Hello John,

Question 1: If I wanted to get back all the documents so I could put them on a map, what is the key, it isn’t “aggs” is it?
You do not not need to have an aggs in your query but you will be limited by the number of docs a query can return (10000). That’s the reason why it is preferable to use aggs compared to simple query. This is what most of the vizs in Opensearch are doing (aggs).

By default in Opensearch, if you do not specify the size to be returned the value is 10.
If you want to get more, use size:

{
  "sample": {
    "index": "opensearch_dashboards_sample_data_ecommerce",
    "size": 250,
    "query": {
      "bool": {
        "must": [
          "_DASHBOARD_CONTEXT_",
          "_TIME_RANGE_[order_date]"
        ]
      }
    },
    "aggs": {
      "customer": {
        "terms": {
          "field": "customer_full_name.keyword",
            "size": 5
        }
      }
    }
  }
}

By default you do not get the total number neither. In order to get the total number in the result, use track_total_hits:

{
  "sample": {
    "index": "opensearch_dashboards_sample_data_ecommerce",
    "size": 250,
    "track_total_hits" : true,
    "query": {
      "bool": {
        "must": [
          "_DASHBOARD_CONTEXT_",
          "_TIME_RANGE_[order_date]"
        ]
      }
    },
    "aggs": {
      "customer": {
        "terms": {
          "field": "customer_full_name.keyword",
            "size": 5
        }
      }
    }
  }
}

The result are then in the response field (this.response), you will get an object named sample with:

  • an aggregations object called customer with an array containing aggs results: this.response.sample.aggregations.customer.buckets
  • a hits object with the docs also within an array : this.response.sample.aggregations.hits.hits

Question 2: I guess what I am struggling with here is how to access the various track data that is being returned?

If you do not have more than 10000 docs in your index you can set the size in the query (if you have more this is still achievable but more complex and you will need to use Opensearch scroll searches and XHR request in Javascript).

{
“sample”: {
“index”: “tracks”,
“query”: {
"size": 10000,
“bool”: {
“must”: [
]
}
},

“fields”: [
“trackId”,
“correlFactor”
],
“_source”: false
}
}

Then, in the JS part, you need to return the data from the response object. You will need to format the data depending on how you want them to be displayed. E.g., if you need to display the list of trackIds and trackFactors.

({
  track: function() {
     let data = this.response.sample.hits.hits; // Get the docs array into data var
     let to_be_displayed = []; // Initialize the array used to display the data
    data.forEach(item => to_be_displayed.push({ "myTrackId": item._source.trackId, "myCorrelFactor": item._source.correlFactor }) // Loop over all the data to build the array
    return to_be_displayed; // This is returning the array of docs from the hit Object (hits is an array)
  }
});

Finally in the template part:

    {{#meta.track}} 
        {{myTrackId}} : {{myCorrelFactor}}
    {{/meta.track}} 

Hope it helps.

PS: Use the developer tools in your browser in order to see the structure of the objects you are manipulating (console tab by using console.log).

Lionel

Wow Lionel. Thanks for all this help. I got everything to work as needed. This plugin really is very nice.

The plugin is working well. It should become an integrated part of OpenSearch Dashboards.

One more question. We have a visualization where the some of the search parameters are passed in via the URL. We append the parameters to the url, like this:

  • server:port/app/vizualize/etc/etc/?id=14&time=123454445

Is there a way to access this in the plugin itself?

Thanks.

Hello,
This is not part of the plugin but pure javascript:

const url = window.location.href;
const queryString=url.split('?')[1]
const urlParams = new URLSearchParams(queryString);
const time = urlParams.get('time')
console.log(time);

Rgds.
Lionel

1 Like

This is really great @lguillaud, thanks for this!

I’ve checked out the plugin and it’s really awesome! I think @jrcartmell mentioned to get this into https://github.com/opensearch-project and into the main distribution but I unfortunately think that it will be caught in our security scans so it might be preferable to keep out as a community project. And I say this not to imply this plugin is bad or unsafe to use but due to the nature of the plugin, these security scans might cause a lot overhead and could force this plugin to become incredibly restrictive which would defeat its purpose. That said if we want to attempt to incorporate it, we can talk about it more.

If we prefer not to incorporate it, perhaps we can escalate or if someone has the bandwidth to take on this issue: https://github.com/opensearch-project/OpenSearch-Dashboards/issues/1148, where it’s not specific to Wazuh plugins but it can accept any plugin and run through a command to install these custom plugins. This could be a happy medium to make the process of installing this plugin a little bit easier (or on par with using an executable). I also believe there is a vision for this repo: https://github.com/opensearch-project/opensearch-build to enable a mechanism that allows people to select the plugins they want to install feed into the pipeline and get a distribution that they desire. But that could not be the case anymore or way down the line.

Let me know if any questions about my above remarks.

Thanks!

Hello @kavilla,

First of all, just to remind I am not the creator of the plugin but I have just ported the plugin from Kibana to OSD by changing a few filenames (as mentioned here in the credits section https://github.com/lguillaud/osd_transform_vis#readme).

Regarding the fact to get this plugin into https://github.com/opensearch-project, I think it could be a good idea but I understand it would have to be reviewed (code review and cleaning, JS improvements, security, …). Being incorporated would allow the plugin to be updated and released at the same time as new OSD versions (which was sometimes really a pain for the the developers to update with the breaking changes in Kibana plugins management). You have also pointed out the plugin might be caught in your security scans and indeed I guess it will but again the purpose of this plugin is to let some doors open to allow flexibility in coding visualisations.

As you said, a trade-off could be to be to ease the plugin installation which could also be a good solution.

As far as I am concerned, I would be more confident if this plugin could be a real part of OSD as it could be “industrialized”. I know more about using this plugin than the real plumbing behind it.

Rgds.

Lionel

I too agree with @kavilla that this plugin might not be the best fit for the main distribution or opensearch-project. Having used this plugin quite a bit, it is honestly quite useful and powerful, but at the same time poses a security nightmare since you can import any arbitrary script and run it on any visualization.

Hello @ashwinpc,

I understand well this plugin can raise many security concerns but you can still play with the Kibana configuration (csp rules) in order to restrict the librairies to import (this is not ideal but useful as a start).

Rgds.
Lionel

True. While security risks are one of the main issues including the plugin directly in the github org, there are a few other concerns I had.

  1. We already have a way of building custom visualizations using vega.
  2. Taking community built plugins and incorporating them in the main github org might not be the right thing to do. It suggests that only plugins in the main github org can be trusted. I have a feeling that this will severely limit community plugin adoption.
  3. Folding into the org might suggest that this is a recommended way to create visualizations and i dont think it is. If there are gaps in the current visualizations, I’d rather we fill those gaps, rather than lean on another library which is another bundle the dashboard has to load for each visualization

Dont get me wrong. This is a beautiful plugin that definitely has its place. Ideally I’d love to see discovering, installing and maintaining plugins to become more accessible. This lets the community grow with their own plugins that can be just as reliable as those maintained by opensearch-project. This plugin is a great example of one such plugin and I’d rather put the effort to help maintaining this as a community plugin than folding it into the org and letting it get locked down due to the org’s security policies.

Improving the plugin ecosystem seems to be in the works as outlined in OpenSearch Dashboards 2022 Initiatives · Issue #1405 · opensearch-project/OpenSearch-Dashboards · GitHub point 2.6.

I agree with all your arguments even if I must admit vega is not my cup of tea.
No problem to keep the plugin as it is (community plugin).

Thanks @lguillaud for this plugin. It saved the day as I needed to reproduce a map visualization from Kibana that had layers and I could not do it with vega. Now, I am not a skilled user, so perhaps there is a way to do this with vega or something else in OS Dashboards. But I think offering a solution without the ability to throw layers on top of a map is quite a deficiency.

As to the security arguments those do make sense. However, I remember someone in IT I worked with many years ago. He suggested locking down a network for security purposes. The result of his solution was that no one could use the network. Perfect security. So why did we have the network in the first place? If people don’t use OS/OS Dashboards because of the inability to add layers to maps, then the security issue is solved.

I know, I know, there is this wonderful plugin from @lguillaud. And as good as it is, it does require skill to add the plugin into OS Dashboards.

I also understand that there would be lots of work to integrate this plugin, I just thought I’d offer my support for the plugin as it is very neat. Thanks again @lguillaud.

@lguillaud - One followup question on the plugin. When I enable the “Refresh Every” on the time drop-down, every time the transform updates it reverts back to the original zoom level and center point. Is there some way to prevent this from happening? I am sure there is but I was not smart enough to figure it out.

Hello @jrcartmell,

In the after_render function where you display the map, you will have to check in the first place if the map exist in the DOM and if so, save the zoom level and center point location (check if Leaflet API can do that).
Then, at the time you ask the map to be displayed, apply the saved settings if any otherwise display with default settings.
Easy to say what to do but a bit more tricky to implement :wink:.

Lionel