Unable to ship logs from Filebeat OSS to AWS managed elasticsearch

Security
1

Hi Team,

I have my managed ES running and I login using username and password followed by MFA authentication through DUO app.

I have installed filebeat OSS 7.10.2 in couple of linux servers and I’m trying to ship the logs from csv files to AWS managed Elasticsearch 7.10.2 however im getting 401 authentication error.

Is this due to MFA set up. Can we by pass this ? pls help

2

Moving this to the security category, but could you also provide a little more details - i.e. the filebeat error message?

3

Hello @shaimoh

Could you share your config.yml and filebeat.yml files?

4

Hi @pablo

This is a AWS managed ES so I dont have the config.yml. below is the filebeat.yml

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log
  paths:
    - /opt/t24user/Temenos/T24/bnk/UD/JT.*.CSV
  fields:
    logtype: "runlog"
    device: "UAT-10.106.121.55"   
  enabled: true
  reload.enabled: true
  reload.period: 10s
  setup.ilm.enabled: false
  setup.ilm.check_exists: false

 # ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

 # ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false

# ================================== Outputs ===================================
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["vpc-nnelasticsearch-5a327eo3upk4iujb5gx3c6si3q.eu-west-1.es.amazonaws.com:443"]

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "636853_cu"
  password: "*******"
  
#----------------------------- Index ---------------------------------------------------

  indices:
    - index: "uat-batch-runlog.55-%{+YYYY.MM}"
      when.contains:
        fields.device: "UAT-10.106.121.55"
5

This is the filebeat error message

q2021-08-20T12:10:51.440+0100 ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://vpc-nnelasticsearch-5a327eo3upk4iujb5gx3c6si3q.eu-west-1.es.amazonaws.com:443)): 401 Unauthorized:

6

@shaimoh as this is an issue with managed service I think you would need to raise a support ticket.

However just looking at your config, have you tried supplying certificate, similar to this ticket

7

Hi @shaimoh

Have you seen this article?

It’s filebeat - logstash - AWS scenario, however HTTP 401 is mentioned there.

1 Like
8

Hi @shaimoh

Did you get a solution for the 401 unauthorised error?

I am getting the similar error message from Filebeat OSS while sending the logs to AWS managed OpenSearch cluster.

I appreciate your response on this.