Getting 429 Too many requests while searching data from index

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): 1.3

Describe the issue:
We are getting 429 Too many Requests error. is it temporary error, but not sure why we are getting it? is there any API limit in Opensearch?

System.Exception: Invalid OpenSearch.Client response built from a unsuccessful (429) low level call on POST: /my_index/_search?typed_keys=true
# Audit trail of this API call:
 - [1] BadResponse: Node: https://xyz.us-west-2.es.amazonaws.com/ Took: 00:00:00.0211567
# OriginalException: OpenSearch.Net.OpenSearchClientException: Request failed to execute. Call: Status code 429 from: POST /my_index/_search?typed_keys=true

Configuration:

{
    "my_index": {
        "settings": {
            "index": {
                "number_of_shards": "1",
                "blocks": {
                    "read_only_allow_delete": "false",
                    "write": "false"
                },
                "provided_name": "my_index",
                "creation_date": "1689943982592",
                "number_of_replicas": "1",
                "uuid": "bcdd",
                "version": {
                    "created": "135248027"
                }
            }
        }
    }
}

Relevant Logs or Screenshots:

2

This error can be triggered by high CPU usage or high memory usage. For example, there’re too many search requests queued in the search thread_pool, or the memory circuit breaker is triggered because of the high JVM memory usage. You can check the complete error message or call the following API to diagnose:

GET _cat/thread_pool
GET _nodes/stats/breaker
3

What further steps we can take by analyzing stats?

4

If the error is triggered by high CPU usage, then hot nodes or hot shards exist in your cluster, you can check the shard distribution to make sure that the shards are balanced between nodes.

If the error is triggered by high memory usage, then you can check which sub circuit breaker was triggered and why: Circuit breaker settings - OpenSearch Documentation, you may increase the circuit breaker limit temporarily or consider increasing the memory size.

5

The 429 error indicates hitting OpenSearch rate limits, often due to high request volume. This is typically temporary. To resolve it, monitor and reduce your API request frequency, implement retry logic, or increase the rate limit settings if possible. Check OpenSearch documentation for specific rate limit configurations.