Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
“opensearch-project/opensearch-php”: “^2.2”
Describe the issue:
Attempting to index 800,000 records, but the process is failing with exception 429 Too Many Requests. I’ve tried chunking the requests, adding sleeps, and even chunking into separate service calls, but they all fail at around 185-190,000 request with the 429. It’s making me wonder if there’s some throttling going on.
Configuration:
Relevant Logs or Screenshots:
Could you show the complete error message? Does the error message show that the queue of the write thread_pool is full?
Hi @bagsmode,
How are you indexing?
I do not believe there are any built-in throttling mechanisms (you could build that logic in your code/script).
Have you considered using _bulk API Bulk - OpenSearch Documentation ?
Is scaling out by adding more nodes to your cluster an option in your current infrastructure?
best,
mj
I was trying to reproduce the error for the detail, but I had also reached out to our Cloud staff. They found we were only using a T3 Small, and have since upgraded to a Large. This seems to have cleaned up the issues.
I’ll take a look at the _bulk as well.
Appreciate your help.