Max Coordinating and Primary Bytes

OpenSearch
, ,
1

Hello,

I am currently conducting performance tests on an OpenSearch cluster running in a Kubernetes environment. The cluster is configured with three indexing nodes, and logs are collected via Fluent Bit. The purpose of these tests is to observe the system’s data processing capacity and its limits.

During the tests, the following error messages were observed in the Fluent Bit logs:

May 25 10:27:00 fluent-bit[414235]: {"error":{"root_cause":[{"type":"rejected_execution_exception","reason":"rejected execution of coordinating operation [coordinating_and_primary_bytes=536769568, replica_bytes=0, all_bytes=536769568, coordinating_operation_bytes=2656432, max_coordinating_and_primary_bytes=536870912]"}],"type":"rejected_execution_exception","reason":"rejected execution of coordinating operation [coordinating_and_primary_bytes=536769568, replica_bytes=0, all_bytes=536769568, coordinating_operation_bytes=2656432, max_coordinating_and_primary_bytes=536870912]"},"status":429}

However, the output of the _cluster/stats API indicates that the system has a higher resource usage capacity than this (512MB) limit:

"jvm": {
  "max_uptime": "10.1d",
  "max_uptime_in_millis": 877134375,
  "versions": [
    {
      "version": "21.0.6",
      "vm_name": "OpenJDK 64-Bit Server VM",
      "vm_version": "21.0.6+7-LTS",
      "vm_vendor": "Eclipse Adoptium",
      "bundled_jdk": true,
      "using_bundled_jdk": true,
      "count": 3
    }
  ],
  "mem": {
    "heap_used": "8.8gb",
    "heap_used_in_bytes": 9554800536,
    "heap_max": "15gb",
    "heap_max_in_bytes": 16106127360
  },
  "threads": 282
}

I have no idea how to increase this limit. Could you please help me?

Also my fluent-bit.conf file is like below

[SERVICE]
    Flush         1
    Log_Level     info
    Parsers_File  parsers.conf

[INPUT]
    Name           dummy
    Rate           2500
    Copies         10
    Dummy          {"message":"25k EPS test log"}

[OUTPUT]
    Name           opensearch
    Match          *
    Host           fqdn
    Port           443
    Index          test-25k
    Type           _doc
    HTTP_User      admin
    HTTP_Passwd    pass
    tls            On
    Logstash_Format Off
    Suppress_Type_Name On