Search getting 429 Too Many Requests

[CarolynR]

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch 2.11.1
Custom Dashboards
OS redhat 9.3, but running inside a docker container

Describe the issue:
Initially when my dashboard is displayed, all the visualizations are fine. If I click on the ‘Refresh’ button after only a few seconds, some of the visualizations will show an error of 429 ‘Too Many Requests’.
If I wait another minute and click on Refresh, all the visualizations will once again be OK.

It appears that Opensearch isn’t waiting for all the threads to complete.

Configuration:
These same dashboards worked fine under Opensearch 2.8
With upgrading to Opensearch 2.11.1, we have implemented security that uses roles and tenants.

Relevant Logs or Screenshots:

When all the visualizations are working correctly, I see numbers in active and queue.

When I start seeing the 429, I do not see numbers in active and queue.

[Gsmitt]

Hey @CarolynR

@CarolynR

error of 429 ‘Too Many Requests’

This might be from memory usage to exceed the maximum size of the Java heap.
Have you tried increasing you heap?

[CarolynR]

Yes, I increased the java heap to 4x its original size and the behavior didn’t change. I am wondering what kind of clean up it does such that after a few minutes I can successfully refresh the dashboard.

Is there any other information I can provide that will help someone diagnose this problem?

Does the application of roles and tenants use many more resources?

[CarolynR]

Just case someone else runs into something like this, it is caused by rate limiting in the application we use for additional security. We needed to increase the default values.