Unable to ship logs from Filebeat OSS to AWS managed elasticsearch

Hi Team,

I have my managed ES running and I login using username and password followed by MFA authentication through DUO app.

I have installed filebeat OSS 7.10.2 in couple of linux servers and I’m trying to ship the logs from csv files to AWS managed Elasticsearch 7.10.2 however im getting 401 authentication error.

Is this due to MFA set up. Can we by pass this ? pls help

Moving this to the security category, but could you also provide a little more details - i.e. the filebeat error message?

Hello @shaimoh

Could you share your config.yml and filebeat.yml files?

Hi @pablo

This is a AWS managed ES so I dont have the config.yml. below is the filebeat.yml


# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log
    - /opt/t24user/Temenos/T24/bnk/UD/JT.*.CSV
    logtype: "runlog"
    device: "UAT-"   
  enabled: true
  reload.enabled: true
  reload.period: 10s
  setup.ilm.enabled: false
  setup.ilm.check_exists: false

 # ============================== Filebeat modules ==============================

  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

 # ======================= Elasticsearch template setting =======================

  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false

# ================================== Outputs ===================================
# ---------------------------- Elasticsearch Output ----------------------------
  # Array of hosts to connect to.
  hosts: ["vpc-nnelasticsearch-5a327eo3upk4iujb5gx3c6si3q.eu-west-1.es.amazonaws.com:443"]

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "636853_cu"
  password: "*******"
#----------------------------- Index ---------------------------------------------------

    - index: "uat-batch-runlog.55-%{+YYYY.MM}"
        fields.device: "UAT-"

This is the filebeat error message

q2021-08-20T12:10:51.440+0100 ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://vpc-nnelasticsearch-5a327eo3upk4iujb5gx3c6si3q.eu-west-1.es.amazonaws.com:443)): 401 Unauthorized:

@shaimoh as this is an issue with managed service I think you would need to raise a support ticket.

However just looking at your config, have you tried supplying certificate, similar to this ticket

Hi @shaimoh

Have you seen this article?

It’s filebeat - logstash - AWS scenario, however HTTP 401 is mentioned there.

1 Like

Hi @shaimoh

Did you get a solution for the 401 unauthorised error?

I am getting the similar error message from Filebeat OSS while sending the logs to AWS managed OpenSearch cluster.

I appreciate your response on this.