Buffer does not have enough capacity left for the number of records error

Data Prepper
,
1

Versions :
I am using opensearchproject/data-prepper:2 docker image for deploying data prepper in k8s cluster pod.

Describe the issue:
Using a fluentbit → data-prepper → AWS OpenSearch stack. This used to work fine until recently I have started noticing timed out waiting for slots errors. See error logs below.

Configuration:
Using following simple config.
The expressions in ${...} will be replaced with actual values during deployment.

  pipeline.yaml: |
    log-pipeline:
      source:
        http:
          ssl: false
      processor:
        - grok:
            match:
              log: [ '${GROK_EXPRESSION_KEY_VALUE}' ]
        - parse_json:
            source: context
        - date:
            from_time_received: true
            destination: "@timestamp"
      sink:
        - opensearch: 
            hosts: [ "${OPENSEARCH_ENDPOINT}" ]
            insecure: false
            username: "${ES_USERNAME}"
            password: "${ES_PASSWORD}"
            index: "${OPENSEARCH_INDEX_NAME}"
        - stdout:
  config.yaml: |
    ssl: false

Relevant Logs or Screenshots:
Errors:

2024-11-25T11:54:55,748 [pool-7-thread-174] ERROR org.opensearch.dataprepper.plugins.source.loghttp.LogHTTPService - Failed to write the request of size 798112 due to: Pipeline [log-pipeline] - Buffer does not have enough capacity left for the number of records: 3039, timed out waiting for slots.
2024-11-25T11:54:55,821 [pool-7-thread-183] ERROR org.opensearch.dataprepper.plugins.source.loghttp.LogHTTPService - Failed to write the request of size 2487931 due to: Pipeline [log-pipeline] - Buffer does not have enough capacity left for the number of records: 9475, timed out waiting for slots.
...
...
...