Fluentbit rejected_execution_exception error

OpenSearch
1

Hello, when sending logs via fluentbit to opensearch I’m getting a lot of these messages:

Apr 20 09:36:55 fluentbit-static02 td-agent-bit[4487]: [2023/04/20 09:36:55] [error] [output:opensearch:opensearch.2] HTTP status=429 URI=/_bulk, response:
Apr 20 09:36:55 fluentbit-static02 td-agent-bit[4487]: {"error":{"root_cause":[{"type":"rejected_execution_exception","reason":"rejected execution of coordinating operation [coordinating_and_primary_bytes=1717946997, replica_bytes=0, all_bytes=1717946997, coordinating_operation_bytes=2764011, max_coordinating_and_primary_bytes=1717986918]"}],"type":"rejected_execution_exception","reason":"rejected execution of coordinating operation [coordinating_and_primary_bytes=1717946997, replica_bytes=0, all_bytes=1717946997, coordinating_operation_bytes=2764011, max_coordinating_and_primary_bytes=1717986918]"},"status":429}
Apr 20 09:36:55 fluentbit-static02 td-agent-bit[4487]: [2023/04/20 09:36:55] [ warn] [engine] chunk '4487-1681983401.821589894.flb' cannot be retried: task_id=141, input=tail.5 > output=opensearch.2

I dont really understand the root cause and have no idea of the potential fix.
also my opensearch-dashboard is getting slow, I assume because opensearch has problem to ingest data.
but previously the same server running elasticsearch + filebeat had no problem handling the exact same logs data

2

So a quick look and this is what I am seeing. 429 is a too many requests status. That with the rejected_execution_exception makes me think that you are sending too much data too fast to the _bulk endpoint and its exhausting your threadpool.

What does your cluster look like currently? Are you running a single/multiple nodes?

3

Seems that the indexing pressure limit is reached, when the inflight indexing requests consume too much memory, OpenSearch will reject new indexing requests, the limit defaults to 10% of JVM heap, maybe you can increase the memory of JVM heap in your cluster, or reducing the batch size when bulking in the client-side, i.e. fluent-bit. OpenSearch does not document this feature, take a look at this:Indexing pressure | Elasticsearch Guide [7.10] | Elastic.

1 Like
4

You can also refer to OpenSearch blog on indexing back pressure here

1 Like
5

oh perfect thats what i was trying too! thank you I will have a look!