Filebeat not pussing every logs to elasticsearch there is discrepancy in the logs

[Nandeeshb]

Hi Team,

Really need a help on this , as Im not able to figure out whats the exact issue.

Filebeat version: 8.3.3
Elastic search version installed on vms: 8.12.0

We are using filebeat to push the logs to elastic search based on the tags with namespace in kubernetes.

Everytime we are missing the logs , only few logs are pushing and not all the logs.

What might be the issue? Please help me on this…

below is the config file which we are using.

filebeat.yml: |-
filebeat.inputs:
- type: container
paths:
- /var/log/containers/uat
tags: [“uat”]
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: “/var/log/containers/”
- type: container
paths:
- /var/log/containers/qa
tags: [“qa”]
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: “/var/log/containers/”
- type: container
paths:
- /var/log/containers/dev
tags: [“dev”]
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: “/var/log/containers/”

processors:
  - add_cloud_metadata:
  - add_host_metadata:

cloud.id: ${ELASTIC_CLOUD_ID}
cloud.auth: ${ELASTIC_CLOUD_AUTH}

output.elasticsearch:
  hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:443}']
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}
  protocol: "https"
  indices:
  - index: "uat-logs-%{+yyyy.MM.dd}"
    when.contains:
      tags: "uat"
  - index: "qa-logs-%{+yyyy.MM.dd}"
    when.contains:
      tags: "qa"
  - index: "dev-logs-%{+yyyy.MM.dd}"
    when.contains:
      tags: "dev"