Versions :
I am using opensearchproject/data-prepper:2
docker image for deploying data prepper in k8s cluster pod.
Describe the issue:
Using a fluentbit → data-prepper → AWS OpenSearch stack. This used to work fine until recently I have started noticing timed out waiting for slots
errors. See error logs below.
Configuration:
Using following simple config.
The expressions in ${...}
will be replaced with actual values during deployment.
pipeline.yaml: |
log-pipeline:
source:
http:
ssl: false
processor:
- grok:
match:
log: [ '${GROK_EXPRESSION_KEY_VALUE}' ]
- parse_json:
source: context
- date:
from_time_received: true
destination: "@timestamp"
sink:
- opensearch:
hosts: [ "${OPENSEARCH_ENDPOINT}" ]
insecure: false
username: "${ES_USERNAME}"
password: "${ES_PASSWORD}"
index: "${OPENSEARCH_INDEX_NAME}"
- stdout:
config.yaml: |
ssl: false
Relevant Logs or Screenshots:
Errors:
2024-11-25T11:54:55,748 [pool-7-thread-174] ERROR org.opensearch.dataprepper.plugins.source.loghttp.LogHTTPService - Failed to write the request of size 798112 due to: Pipeline [log-pipeline] - Buffer does not have enough capacity left for the number of records: 3039, timed out waiting for slots.
2024-11-25T11:54:55,821 [pool-7-thread-183] ERROR org.opensearch.dataprepper.plugins.source.loghttp.LogHTTPService - Failed to write the request of size 2487931 due to: Pipeline [log-pipeline] - Buffer does not have enough capacity left for the number of records: 9475, timed out waiting for slots.
...
...
...