Hi,
i need help on this configuration. My scope is to monitor OpenSearch and OpensSearch Dashboard with Metricbeat, in the same way it’s possibile to do with ELK stack.
I use this docker-compose.yml
version: '3'
services:
opensearch:
container_name: opensearch
image: opensearchproject/opensearch:latest
environment:
- discovery.type=single-node
- bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
hard: 65536
ports:
- 9200:9200
- 9600:9600 # required for Performance Analyzer
networks:
- opensearch-net
dashboards:
image: opensearchproject/opensearch-dashboards:latest
container_name: opensearch-dashboards
ports:
- 5601:5601
expose:
- "5601"
environment:
OPENSEARCH_HOSTS: '["https://opensearch:9200"]'
depends_on:
- opensearch
networks:
- opensearch-net
metricbeat:
build: .
image: docker.elastic.co/beats/metricbeat-oss:8.3.1
user: root
environment:
- ELASTICSEARCH_HOSTS=opensearch:9200
- KIBANA_HOST=dashboards:5601
- ELASTICSEARCH_USERNAME=admin
- ELASTICSEARCH_PASSWORD=admin
volumes:
- metricbeat-data01:/usr/share/metricbeat/data
- /var/run/docker.sock:/var/run/docker.sock
networks:
- opensearch-net
depends_on:
- opensearch
volumes:
opensearch-data1:
opensearch-data2:
metricbeat-data01:
driver: local
networks:
opensearch-net:
I have this error on OS: Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:
version: '3'
services:
opensearch-node1:
image: opensearchproject/opensearch:2.0.1
container_name: opensearch-node1
environment:
- cluster.name=opensearch-cluster
- node.name=opensearch-node1
- bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
- "DISABLE_INSTALL_DEMO_CONFIG=true" # disables execution of install_demo_configuration.sh bundled with security plugin, which installs demo certificates and security configurations to OpenSearch
- "DISABLE_SECURITY_PLUGIN=true" # disables security plugin entirely in OpenSearch by setting plugins.security.disabled: true in opensearch.yml
- "discovery.type=single-node" # disables bootstrap checks that are enabled when network.host is set to a non-loopback address
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
hard: 65536
volumes:
- opensearch-data1:/usr/share/opensearch/data
ports:
- 9200:9200
- 9600:9600 # required for Performance Analyzer
networks:
- opensearch-net
opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:2.0.1
container_name: opensearch-dashboards
ports:
- 5601:5601
expose:
- "5601"
environment:
- 'OPENSEARCH_HOSTS=["http://opensearch-node1:9200"]'
- "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true" # disables security dashboards plugin in OpenSearch Dashboards
networks:
- opensearch-net
metricbeat:
build: .
image: docker.elastic.co/beats/metricbeat-oss:8.3.1
user: root
environment:
- ELASTICSEARCH_HOSTS=opensearch-node1:9200
- KIBANA_HOST=dashboards:5601
- ELASTICSEARCH_USERNAME=admin
- ELASTICSEARCH_PASSWORD=admin
volumes:
- metricbeat-data01:/usr/share/metricbeat/data
- /var/run/docker.sock:/var/run/docker.sock
networks:
- opensearch-net
depends_on:
- opensearch-node1
volumes:
opensearch-data1:
metricbeat-data01:
driver: local
networks:
opensearch-net:
i have this error:
ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(http://opensearch-node1:9200)): Connection marked as failed because the onConnect callback failed: Metricbeat requires the default distribution of Elasticsearch. Please update to the default distribution of Elasticsearch for full access to all free features, or switch to the OSS distribution of Metricbeat.
2022-07-05T10:46:36.956Z INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(elasticsearch(http://opensearch-node1:9200)) with 23 reconnect attempt(s)
I use the OSS version, so i don’t understand the error.
Are you trying to send Metricbeat data directly into OpenSearch? Beats does a version check to make sure that it is connecting to Elasticsearch. If you want to send traffic from Beats over to OpenSearch, you’ll want to do that via Logstash using the OpenSearch Logstash Output plugin. You can find more information on the agents and ingestion tools documentation page.
Another option would be to use another open source agent like Fluent Bit and send that into OpenSearch. Anurag from Calytpia wrote a blog post on doing just that.
I would like to monitor with metricbeat Opensearch, OS Dashboard and also Logstash that in this case are all docker in different machine. I have read in this article Can’t connect Metricbeat to opensearch - OpenDistro - OpenSearch that it is possible but I am not managing to obtain the same result, also I do not understand why it gives me a version error considering that I am using the OSS version of Metricbeat.
Thank you
@yago82 did you manage to make lastest metricbeat work with opensearch? we also having same issues. @jdbright when it is done through logstash how index template managed? normally metricbeat creaates a index template when directly connected to elasticsearch