Agentic search runs so slow

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): opensearch 3.3.0

Describe the issue:

I’m trying out this new agentic search thing with the GPT-5 model through the OpenAI API. The issue is it’s very slow – usually takes at least 25 seconds for a simple search on one text field (not even multiple ones). And it often hits the 30-second timeout. For comparison, my own RESTful API handles the same search (no LLM involved) in just over 100ms. I get that hitting the LLM, getting the DSL response, and running it on opensearch cluster all adds up, but is my 20+ seconds wait time typical? If so, I guess it is literally “experimental”.

2

@summerist.l I’ve tried that with AWS Bedrock us.anthropic.claude-sonnet-4-20250514-v1:0 and openai.gpt-oss-120b-1:0, and they were running fine. (below 10 sec)

I was testing the Query Planning Tool.

Could you share your agent configuration?

3

Hi @pablo, thanks for the response. For model creating, I used the demo code here:

Please check the json of the agent below (the 2 model_ids are the same):

{
“name”: “gpt-5-agent-test1”,
“type”: “conversational”,
“description”: “Use this for Agentic Search”,
“tools”: [
{
“description”: “”,
“include_output_in_agent_response”: false,
“type”: “QueryPlanningTool”,
“parameters”: {
“search_templates”: “”,
“model_id”: “xxxxxxxxxxxxx”,
“generation_type”: “llmGenerated”
}
},
{
“description”: “”,
“include_output_in_agent_response”: false,
“type”: “SearchIndexTool”
},
{
“description”: “”,
“include_output_in_agent_response”: false,
“type”: “ListIndexTool”
},
{
“description”: “”,
“include_output_in_agent_response”: false,
“type”: “IndexMappingTool”
}
],
“llm”: {
“model_id”: “xxxxxxxxxxxxxxxxxx”,
“parameters”: {
“max_iteration”: “15”
}
},
“memory”: {
“type”: “conversation_index”
},
“parameters”: {
“_llm_interface”: “openai/v1/chat/completions”
}
}

4

Thanks for looking into agentic search @summerist.l .We are aware of the latency, this is expected, as it comes from the multiple LLM calls made by the conversation agent.

If your primary goal is intelligent retrieval with contextual understanding, we recommend using the conversation agent.

However, if lower latency is more important for your use case, you can use the flow agent with the query planning tool:
https://docs.opensearch.org/latest/vector-search/ai-search/agentic-search/flow-agent/

Keep in mind that the flow agent is optimized for performance and won’t maintain conversational context like the conversation agent does.

Also, now from 3.3 you just need one model id for the agent and Query Planner Tool would use the same Using conversational agents - OpenSearch Documentation

5

That is a perfect

1 Like
6

@owaiskazi19 Thank you so much for the clarification!

1 Like
7

@summerist.l @pablo @jakabasej5 we are looking for gathering use cases for agentic search. If you can let us know your use cases for using the same would be really helpful. Thank you.