Search getting 429 Too Many Requests

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch 2.11.1
Custom Dashboards
OS redhat 9.3, but running inside a docker container

Describe the issue:
Initially when my dashboard is displayed, all the visualizations are fine. If I click on the ‘Refresh’ button after only a few seconds, some of the visualizations will show an error of 429 ‘Too Many Requests’.
If I wait another minute and click on Refresh, all the visualizations will once again be OK.

It appears that Opensearch isn’t waiting for all the threads to complete.

These same dashboards worked fine under Opensearch 2.8
With upgrading to Opensearch 2.11.1, we have implemented security that uses roles and tenants.

Relevant Logs or Screenshots:


When all the visualizations are working correctly, I see numbers in active and queue.

When I start seeing the 429, I do not see numbers in active and queue.

Hey @CarolynR

This might be from memory usage to exceed the maximum size of the Java heap.
Have you tried increasing you heap?

Yes, I increased the java heap to 4x its original size and the behavior didn’t change. I am wondering what kind of clean up it does such that after a few minutes I can successfully refresh the dashboard.

Is there any other information I can provide that will help someone diagnose this problem?

Does the application of roles and tenants use many more resources?

Just case someone else runs into something like this, it is caused by rate limiting in the application we use for additional security. We needed to increase the default values.