@tvc_apisani The link to Elasticsearch exporter is in GO and it’s available as docker.
Point 2) do you think that will be easy to migrate from ELK+XPack to Opendistro in the future?
Elastic already prevent migrate Kibana to OD for 7.10.2 due to flattened mapping usage.
In the near future, it’s easy to implement a “change” in index mapping and prevent the shift from Elastic Elasticsearch and OpenDistro.
The link to Elasticsearch exporter is in GO and it’s available as docker
I know. That’s the reason I was argumenting it could be a bit complicated for some users to setup such configuration: they would need ODFE + docker exporter + prometheus (standalone or via docker) + grafana (standalone or via docker) : not exactly a 1-2-3 setup compared to xpath (i repeat: for some users). As i said above: it is mainly a matter of which kind of users ODFE would primarely target: the average user of the tech-savv user (or, to simply put it: users with small clusters and relative experience or IT staff of large deployments).
Elastic already prevent migrate Kibana to OD for 7.10.2 due to flattened mapping usage.
Really ? Didn’t known about this one. Interesting.
Going through the thread I have a question regarding Prometheus Exporters:
If, for example in this proposal, I would like to add a new stats endpoint for a new OpenSearch metric, this would require a new exporter for OpenSearch. Does anyone see value in forking the exporter in favor of the OpenSearch project for additional reasons? If so - where would you suggest the exporter be maintained?
From my perspective any software can export prometheus metrics to be scraped as needed. Jaeger does this for example and it allows for the easy collection of data about the operation of the system. This allows for agentless collection of the metrics in a standard format: Monitoring Jaeger — Jaeger documentation (jaegertracing.io) There are also standard ways to do this with mixins which is another way to create a nice monitoring system for OpenSearch : 2018-02 Prometheus Monitoring Mixins Design Document - Google Docs
we’re currently using ES + elasticsearch-prometheus-exporter.
there’s a ticket open with the latter to support OpenSearch, but so far this hasn’t been done (and it has even been suggested that it might be easier to just create a fork of the plugin since it seems that you can’t have the same code base for ES & OpenSearch plugins due to the imports, etc.?):
https://github.com/vvanholl/elasticsearch-prometheus-exporter/issues/308
i’m wondering what the right way forward is here as OpenSearch is moving towards the 1.0.0 release:
i presume we’re not the only one who want to have some observability of our OpenSearch cluster in prometheus and this is a blocker for the move to OpenSearch.
Hi, I have deployed a fork to use with opensearch here GitHub - aparo/opensearch-prometheus-exporter: OpenSearch Prometheus Exporter (fork of https://github.com/vvanholl/elasticsearch-prometheus-exporter) with a release for RC1 https://github.com/aparo/opensearch-prometheus-exporter/releases
I will be easy installable as plugin.
I also have a pipeline to build and release new versions based on OpenSearch releases.
Best regards,
Albero
I was going to suggest that we do this work for logz.io but I’m glad to see you have done so. I would suggest we move this into the main repo! Nice job @aparo !
← You get a trophy!
Can we put this on the community projects page (to be released today-ish)?
thanks a lot, @aparo!
are there any plans to make this an “official” OpenSearch plugin (i.e. host it under https://github.com/opensearch-project and have it maintained by the OpenSearch maintainers)? i’m not even sure if there’s already a process for plugins to become part of the official OpenSearch project?
@ralph going from being a project that “works with” to a project that is “part of” is an interesting question. I think there will be some more news on that general area soon-ish.
One thing I would separate out being part of the github org vs maintainer-ship, those are two distinct issues. I think it’s possible that a project will become part of the opensearch org, but not be maintained by anyone who is currently a OpenSearch maintainer. So, net-new maintainers for these type of projects seems more likely.
Hi Kyle - I’m curious where these discussions are happening if you’re expecting news on this topic soon-ish? I haven’t seen the discussions here on the forums, but maybe I missed something?
Well, there are some discussions in these two issues:
https://github.com/opensearch-project/opensearch-plugins/issues/20
https://github.com/opensearch-project/opensearch-plugin-template-java/issues/4
The forthcoming work/news that I was referring to is about making this a less adhoc process. That’s mostly stuff the Amazon team needs to iron out.
while there’s now a plugin for OpenSearch to export prometheus metrics (GitHub - aparo/opensearch-prometheus-exporter: OpenSearch Prometheus Exporter (fork of https://github.com/vvanholl/elasticsearch-prometheus-exporter)) i didn’t find one for OpenSearch Dashboards. i did find GitHub - pjhampton/kibana-prometheus-exporter: Prometheus metrics for Kibana for Kibana but didn’t find an OpenSearch Dashboards fork of it yet. are there any plans to create also an OpenSearch Dashboards fork of this?
i still think that it’d be good to move the now-released fork from @aparo to the opensearch project so that it’s clear that this is the one-and-only prometheus exporter for opensearch and it’s easy for everyone to find.
talking about easy-to-find: i just noticed that prometheus has a list of known exporters and the new exporter isn’t listed there yet: Exporters and integrations | Prometheus
it might make sense to get it added to that list (if the plugin gets moved to opensearch i’d suggest to wait for that so that the link doesn’t have to be updated again; also, if a fork of the kibana exporter is created then both could be added at the same time).
Huh. The Prometheus Exporter by @aparo slipped by me.
I’m not sure that every OpenSearch related project should go under opensearch-project github repo. That’s a lot of centralization and implies some degree of Amazon involvement, which doesn’t match my gut check. A project like OpenSearch should have a ton of external tools that compete with each other - if it’s housed in the opensearch-projects github repo, it’s like crowning a winner.
However, the right place (from my PoV) for this seems like the community projects page:
i believe that all enterprise-grade installations will need monitoring and prometheus seems to be the de-facto standard for that. seeing that there was already only one single implementation of that for elasticsearch i doubt that we’ll suddenly get multiple for opensearch. and i don’t see the value in it either, unless there’d be something fundamentally broken with the current implementation (in which case we should IMHO fix it there, because having something fundamentally broken is obviously bad!).
so here i do see the rationale for adding it centrally. it’s the same logic as for the other plugins which are under the organisation.
otherwise you’d also have to move out your security plugin since there are / will be alternative security plugins available 
and i believe you said yourself (in some other thread a while ago) that just because a repo is under the opensearch project doesn’t mean that it must be run by AWS (and it shouldn’t since opensearch isn’t a pure AWS project!) and that there should be a path for projects to be moved there if they fit in.
Check MK support elasticsearch well. I used it for monitoring cluster ES
The version on screen is ODFE 7.8
Elasticsearch is way differently organized than OpenSearch. I’d love to see multiple different implementations (I get the feeling that wouldn’t be the case over in ES, but maybe I’m wrong).
As it stands right now, at least some AWS folks do have admin rights on anything in the OpenSearch github org (which seems reasonable for reasons described later) . That’s the extent of the involvement in things like the client libs so far - the team cleaned them up so they were not full of sticky trademark issues, etc. And now the community is taking them over as maintainers as there are people committed to working on them (of course AWS folks will collaborate to some extent, but the real stars are the community maintainers).
Client libs are absolutely core to the use of OpenSearch, so even if those folks stepped away, someone at AWS would step up. So there is a commitment from AWS. If the org would contain the Prometheus Exporter, I think a similar set of considerations would need to be weighed.
I’m not ruling out bringing in the Prometheus Exporter (that’s not my role) into the org, but I think it would be a bigger decision, based on the precedence setup by the clients. Personally, I’d like to see OpenSearch be less opinionated but I’m not immovable in that opinion either!
