Can't connect Metricbeat to opensearch

OpenDistro
1

Hi,

I have a docker compose file where I start the whole stack including Metricbeat and Filebeat.
I have the same configuration on both beat containers but Metricbeat says ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(http://opensearch:9200)): 401 Unauthorized: Unauthorized.

Filebeat on the other hand is able to connect to opensearch.
What am I missing?

metricbeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

logging.level: debug
setup.ilm.enabled: false

setup.kibana:
    host: "dashboards:5601"
    username: admin
    password: admin
    headers:
      securitytenant: global

processors:
  - add_cloud_metadata: ~
  - add_docker_metadata: ~

# enabled modules for monitoring (e.g. elasticsearch-xpack)
metricbeat.modules:
- module: elasticsearch
  xpack.enabled: true
  period: 10s
  hosts: ["http://opensearch:9200"]

output.elasticsearch:
  hosts: ["opensearch:9200"]
  username: 'admin'
  password: 'admin'

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

logging.level: info

setup.ilm.enabled: false

setup.kibana:
    host: "dashboards:5601"
    username: admin
    password: admin
    headers:
      securitytenant: global

filebeat.inputs:
- type: udp
  enabled: true
  host: ":55014"    
    
processors:
  - dissect:
      tokenizer: "%{apptimestamp} %{host.machine} %{process.name}[%{process.pid|long}]: [%{log.level}] [%{log.logger}] %{log.message}"
      field: "message"
      target_prefix: ""
      trim_chars: "\n"
      overwrite_keys: true
  - timestamp:
      field: apptimestamp
      layouts:
        - '2006-01-02T15:04:05.999-07:00'
      test:
        - '2021-07-14T14:03:36.185188+02:00'
  
output.elasticsearch:
  hosts: ["opensearch:9200"]
  username: "admin"
  password: "admin"

docker compose:

  filebeat:
    restart: "always"
    depends_on: 
      - opensearch
    image: docker.elastic.co/beats/filebeat-oss:7.12.1
    command: filebeat -e -strict.perms=false
    container_name: filebeat
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
    networks: 
      - opensearch-net
    ports:
      - "6513:6513/tcp"  
      - "55014:55014/udp" 

  metricbeat:
    restart: "always"
    image: docker.elastic.co/beats/metricbeat-oss:7.12.1
    command: metricbeat -e -strict.perms=false
    environment:
      ELASTICSEARCH_HOSTS: http://opensearch:9200
    volumes:
      - ./metricbeat.yml:/usr/share/metricbeat/data/metricbeat.yml
    networks:
      - opensearch-net
    depends_on:
      - opensearch
2

I think this OpenSearch is expecting an SSL (https) connection and that’s why you’re getting a 401

3

Isn‘t it strange that filebeat works and metricbeat not?
Because I deactivated SSL on opensearch.
Or is Metricbeat talking somehow differently to opensearch than filebeat does?

4

@smudi No - they should be the same. I’m 99% sure something is awry in the configuration. Are they identical?

5

Long time no see …
But to clear this up:
I had to configure metricbeat and filebeat differently.

In metricbeat I had to configure environment inside the compose file:

    environment:
      - ELASTICSEARCH_HOSTS=opensearch:9200
      - KIBANA_HOST=dashboards:5601
      - ELASTICSEARCH_USERNAME=admin
      - ELASTICSEARCH_PASSWORD=admin

And in filebeat I can write in config file:

output.elasticsearch:
  hosts: ["opensearch:9200"]
  username: admin
  password: admin
1 Like
6

Awesome. That seems straight forward!

7

HI @smudi how do you deactived SSL? It’s possibile to have the complete configuration? I’m trying a similar configuration but I have some problems between metricbeat and OpenSearch.

version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:2.0.1
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      - "DISABLE_INSTALL_DEMO_CONFIG=true" # disables execution of install_demo_configuration.sh bundled with security plugin, which installs demo certificates and security configurations to OpenSearch
      - "DISABLE_SECURITY_PLUGIN=true" # disables security plugin entirely in OpenSearch by setting plugins.security.disabled: true in opensearch.yml
      - "discovery.type=single-node" # disables bootstrap checks that are enabled when network.host is set to a non-loopback address
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net

  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:2.0.1
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      - 'OPENSEARCH_HOSTS=["http://opensearch-node1:9200"]'
      - "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true" # disables security dashboards plugin in OpenSearch Dashboards
    networks:
      - opensearch-net
  metricbeat:
    build: .
    image: docker.elastic.co/beats/metricbeat-oss:8.3.1
    user: root
    environment:
      - ELASTICSEARCH_HOSTS=opensearch-node1:9200
      - KIBANA_HOST=dashboards:5601
      - ELASTICSEARCH_USERNAME=admin
      - ELASTICSEARCH_PASSWORD=admin
    volumes:
      - metricbeat-data01:/usr/share/metricbeat/data
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - opensearch-net
    depends_on:
      - opensearch-node1

volumes:
  opensearch-data1:
  metricbeat-data01:
    driver: local

networks:
  opensearch-net:

I have this error:

ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(http://opensearch-node1:9200)): Connection marked as failed because the onConnect callback failed: Metricbeat requires the default distribution of Elasticsearch. Please update to the default distribution of Elasticsearch for full access to all free features, or switch to the OSS distribution of Metricbeat.

2022-07-05T09:49:34.189Z INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(elasticsearch(http://opensearch-node1:9200)) with 4 reconnect attempt(s)

Thank you.