How connect Metricbeat to Opensearch for monitoring OS and OS Dashboard?

Hi,
i need help on this configuration. My scope is to monitor OpenSearch and OpensSearch Dashboard with Metricbeat, in the same way it’s possibile to do with ELK stack.

I use this docker-compose.yml

version: '3'
services:
  opensearch:
    container_name: opensearch
    image: opensearchproject/opensearch:latest
    environment:
      - discovery.type=single-node
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net
  dashboards:
    image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch:9200"]'
    depends_on:
      - opensearch
    networks:
      - opensearch-net
  metricbeat:
    build: .
    image: docker.elastic.co/beats/metricbeat-oss:8.3.1
    user: root
    environment:
      - ELASTICSEARCH_HOSTS=opensearch:9200
      - KIBANA_HOST=dashboards:5601
      - ELASTICSEARCH_USERNAME=admin
      - ELASTICSEARCH_PASSWORD=admin
    volumes:
      - metricbeat-data01:/usr/share/metricbeat/data
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - opensearch-net
    depends_on:
      - opensearch

volumes:
  opensearch-data1:
  opensearch-data2:
  metricbeat-data01:
    driver: local

networks:
  opensearch-net:

I have this error on OS: Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:

I have this error on Metricbeat: Failed to connect to backoff(elasticsearch(http://opensearch:9200)): Get “http://opensearch:9200”: EOF

Thank you.

UPDATE:

now with this new docker-compose:

version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:2.0.1
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      - "DISABLE_INSTALL_DEMO_CONFIG=true" # disables execution of install_demo_configuration.sh bundled with security plugin, which installs demo certificates and security configurations to OpenSearch
      - "DISABLE_SECURITY_PLUGIN=true" # disables security plugin entirely in OpenSearch by setting plugins.security.disabled: true in opensearch.yml
      - "discovery.type=single-node" # disables bootstrap checks that are enabled when network.host is set to a non-loopback address
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net

  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:2.0.1
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      - 'OPENSEARCH_HOSTS=["http://opensearch-node1:9200"]'
      - "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true" # disables security dashboards plugin in OpenSearch Dashboards
    networks:
      - opensearch-net
  metricbeat:
    build: .
    image: docker.elastic.co/beats/metricbeat-oss:8.3.1
    user: root
    environment:
      - ELASTICSEARCH_HOSTS=opensearch-node1:9200
      - KIBANA_HOST=dashboards:5601
      - ELASTICSEARCH_USERNAME=admin
      - ELASTICSEARCH_PASSWORD=admin
    volumes:
      - metricbeat-data01:/usr/share/metricbeat/data
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - opensearch-net
    depends_on:
      - opensearch-node1

volumes:
  opensearch-data1:
  metricbeat-data01:
    driver: local

networks:
  opensearch-net:

i have this error:

ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(http://opensearch-node1:9200)): Connection marked as failed because the onConnect callback failed: Metricbeat requires the default distribution of Elasticsearch. Please update to the default distribution of Elasticsearch for full access to all free features, or switch to the OSS distribution of Metricbeat.

2022-07-05T10:46:36.956Z INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(elasticsearch(http://opensearch-node1:9200)) with 23 reconnect attempt(s)

I use the OSS version, so i don’t understand the error.

Thank you

Hi @yago82,

Are you trying to send Metricbeat data directly into OpenSearch? Beats does a version check to make sure that it is connecting to Elasticsearch. If you want to send traffic from Beats over to OpenSearch, you’ll want to do that via Logstash using the OpenSearch Logstash Output plugin. You can find more information on the agents and ingestion tools documentation page.

Another option would be to use another open source agent like Fluent Bit and send that into OpenSearch. Anurag from Calytpia wrote a blog post on doing just that.

Hope that helps.

HI @jdbright thank you for reply.

I would like to monitor with metricbeat Opensearch, OS Dashboard and also Logstash that in this case are all docker in different machine. I have read in this article Can’t connect Metricbeat to opensearch - OpenDistro - OpenSearch that it is possible but I am not managing to obtain the same result, also I do not understand why it gives me a version error considering that I am using the OSS version of Metricbeat.
Thank you

Thanks @yago82. Have you set ECS to disabled in your Logstash config? Are you using the Logstash Output plugin for OpenSearch?