ILM is not supported by the Elasticsearch version in use: Elasticsearch 2.15.0 does not support ILM

Security
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
AWS Managerd Service 2.15
Filebeat version. 8.15.3

Infra:
Opensearch - On AWS Managed Service
EKS Cluster
Filebeat Helm chart: helm-charts/filebeat at main · elastic/helm-charts · GitHub

Describe the issue:
AWS managed Opensearch version 2.15. I have deployed Filebeat using helm chart. Filebeat is not able to connect to Opensearch.
Below logs are from the filebeat pods:

filebeat test output
elasticsearch: my-opensearch -endpoint:443…
parse url… OK
connection…
parse host… OK
dns lookup… OK
addresses: 104.26.10.172, 17.26.10.172, 172.0.8.171
dial up… OK
TLS…
security: server’s certificate chain verification is enabled
handshake… OK
TLS version: TLSv1.3
dial up… OK
talk to server… ERROR Connection marked as failed because the onConnect callback failed: could not connect to a compatible version of Elasticsearch: unauthorized access, could not connect to the xpack endpoint, verify your credentials

Configuration:
filebeat values.yaml file:

----
filebeat:
  image:
    repository: docker.elastic.co/beats/filebeat
    tag: "8.15.3"
    pullPolicy: IfNotPresent

filebeatConfig:
  filebeat.yml: |
    filebeat.autodiscover:
     providers:
       - type: kubernetes
         node: ${NODE_NAME}
         hints.enabled: true
         hints.default_config:
           type: container
           id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
           paths:
           - /var/log/containers/*-${data.kubernetes.container.id}.log

   processors:
      - add_cloud_metadata: {}
      - add_host_metadata: {}

   output.elasticsearch:
      host: '${NODE_NAME}'
      hosts: ["https://my- opensearch- endpoint:443"]
      protocol: "https"
      username: "${ELASTICSEARCH_USERNAME}"
      password: "${ELASTICSEARCH_PASSWORD}"
      setup.ilm.overwrite: true
      setup.ilm.enabled: false
extraEnvs:
  - name: "ELASTICSEARCH_USERNAME"
    valueFrom:
      secretKeyRef:
        name: elasticsearch-master-credentials
        key: username
  - name: "ELASTICSEARCH_PASSWORD"
    valueFrom:
      secretKeyRef:
        name: elasticsearch-master-credentials
        key: password
 #Rediness probe:
readinessProbe:
  exec:
    command:
      - sh
      - -c
      - |
        #!/usr/bin/env bash -e
        curl --fail 127.0.0.1:5066
  serviceAccount:
    create: false
    name: filebeatsa

Relevant Logs or Screenshots:

talk to server… ERROR Connection marked as failed because the onConnect callback failed: could not connect to a compatible version of Elasticsearch: unauthorized access, could not connect to the xpack endpoint, verify your credentials

2

@singh.gursharan34 You need to use the supported version of the Filebeat software. Filebeat 7.14+ has version check function when connecting to Elasticsearch and it will fail to work with OpenSearch.