Filebeat 7.12.0 log parsing issue to opensearch 3.0.0

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
i am using opensearch 3.0.0 and filebeat 7.12.0

Describe the issue:
While send logs to opensearch using filebeat getting below error in filebeat status

ERROR        [elasticsearch]        elasticsearch/client.go:224        failed to perform any bulk index operations: 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}

Configuration:

filebeat configuration

# ============================== Filebeat inputs ===============================
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/*.log

# ============================== Output to OpenSearch ===========================
output.elasticsearch:
  # OpenSearch URL and port
  hosts: ["https://127.0.0.1:9200"]

  protocol: "https"
  # Optional authentication (username and password) if needed
  username: "user"
  password: "passwd"

  # TLS settings
  ssl.enabled: true
  ssl.verification_mode: full  # Optional, set to 'none' if not verifying certificates
  ssl.certificate_authorities: ["/etc/opensearch/ca.crt"]  # Specify your CA certificate if necessary
  ssl.certificate: "/etc/opensearch/node.crt"        # Optional if client certificates are needed
  ssl.key: "/etc/opensearch/node.key"                 # Optional if client certificates are needed

  # Set the index pattern for OpenSearch (or Elasticsearch)
  index: "filebeat-%{+yyyy.MM.dd}"

# ============================== Filebeat paths ================================
path.data: /var/lib/filebeat         # Specify a directory for Filebeat's internal data

path.logs: /var/log/filebeat         # Logs directory for Filebeat itself

setup.template.enabled: false
  #setup.template.type: "doc"
output.elasticsearch.document_type: "doc"

Relevant Logs or Screenshots:

image
image1315×276 22.3 KB

2

Hey @kirankalelkar ,

Elastic has introduced a version check in Filebeat 7.13. So it won’t allow you to connect with OpenSearch 3.X. Last compatible version was filebeat 7.12.1 which worked for Opensearch 2.X.

As a result compatibility has been removed in 3.X.

You can try using Logstash with the OpenSearch output plugin and newer Filebeat as a workaround.

3

i tried using filebeat 7.12.1 and opensearch 2.19.3 but getting same issue

image
image1315×184 15.2 KB

4

Hi @kirankalelkar ,

I tested Filebeat and it works with 7.12.0 and 7.12.1 to Opensearch 2.19.2 and 2.19.3.

I am also doing this using Filebeats-oss. So please test with Filebeat-oss and let me know if you’re still having issues.

Supported versions can be found here - Tools - OpenSearch Documentation .

Also you will see in the doc, but please also ensure you have “override_main_response_version”: true set in your opensearch.yml or set it using the following -

PUT _cluster/settings
{
  "persistent": {
    "compatibility": {
      "override_main_response_version": true
    }
  }
}

Leeroy.

5 
Thank You. i am able to receive the logs.
1 Like