Auditbeat + AWS Opensearch connection timeout

Open Source Elasticsearch and Kibana
1

I am trying to connect my auditbeat agents with Opensearch on AWS. I have setup a user with the required permissions within the OS security plugin in the front end. I can connect with curl and via a browser to the endpoint from the hosts but sudo auditbeat setup -e just fails due to timeout.

I am using the OSS version of auditbeat 7.12.1. I have tried two different hosts.

curl -XGET https://endpoint--url--aws-region.es.amazonaws.com/ -u 'usernam:Password' --insecure
{
  "name" : "9ec417c96db259a33fghiy6b1db2dgh6680",
  "cluster_name" : "87655688886:clustername3",
  "cluster_uuid" : "s-R7WFTFQ2ulebAbFVSzmQ",
  "version" : {
    "number" : "7.10.2",
    "build_type" : "tar",
    "build_hash" : "unknown",
    "build_date" : "2021-11-11T13:03:46.725100Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

The errors in auditbeat, it just times out, no actual reason given:

ERROR	[esclientleg]	eslegclient/connection.go:261	error connecting to Elasticsearch at https://aws-endpoint-url.es.amazonaws.com:9200: Get "https://aws-endpoint-url.es.amazonaws.com:9200": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

ERROR	instance/beat.go:971	Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at https://aws-endpoint-url.es.amazonaws.com:9200: Get "https://aws-endpoint-url.amazonaws.com:9200": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)]
Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at https://aws-endpoint-url.amazonaws.com:9200: Get "https://aws-endpoint-url.es.amazonaws.com:9200": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)]
2

Two things:

  1. Double confirm you’re using the OSS variant (it’s a tad hard to find) - I know you say you’re using it but a lot of folks accidentally download the wrong version.
  2. Follow the instructions in the docs to put OpenSearch into compatibility mode.
3

Thanks for the reply :grinning:

When I configured the domain I enabled compatibility mode in AWS. It is reporting version 7.10.2.

The auditbeat version I am using is definitely the OSS version 7.12.1, I made the error of using the newer version at the beginning.

I may have to move away from AWS managed and just install on EC2 for now, but that defeats what I was trying to do by handing over the admin of the servers to AWS.

4

For the pruposes of troubleshooting I have tried creating my own Opeasearch cluster on an EC2 using docker. The error from Auditbeat now is different so this is progress. I now get the X509 certificate error. I tried disabling SSL verification but it still does not work.

ssl.verification_mode: none

Auditbeat OSS version 7.12.1
Docker Opensearch 1.2.0

sudo auditbeat setup -e
........
2021-12-22	ERROR	[esclientleg]	eslegclient/connection.go:261	error connecting to Elasticsearch at https://ip.address:9200: Get "https://ip.address:9200": x509: certificate signed by unknown authority
2021-12-22	ERROR	instance/beat.go:971	Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at https://ip.address:9200: Get "https://ip.address:9200": x509: certificate signed by unknown authority]
Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at https://ip.address:9200: Get "https://ip.address:9200": x509: certificate signed by unknown authority]
5

Hey all - have been working on this and consulting with support.

  • It DOES work using the source/tar version of Auditbeat 7.12.1. Still working on the RPM version.
  • The dashboards cannot be automatically created using “auditbeat setup -e”. But it will create the indexes and you have to create your own dashboards.
  • you need to add some custom configs.

Custom configs below to create the indexes and disable ILM:

setup.ilm:
  enabled: false

# ======================= Elasticsearch template setting =======================
setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false
  name: "auditbeat"
  pattern: "auditbeat-*"
2 Likes
6

Glad to see you’re making progress!