Very slow first few search queries

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch 2.12.0

Describe the issue:
Hi all, we recently completed the installation of OpenSearch V2.12.0 into our environment, and here’s the complete data flow:

Kafka Topic → Logstash → OpenSearch

We have done some testing in Development environment and the search query was really fast, even with hundreds of GB of data, the search time usually took only less than 10ms.

However, when we did the same search query in this new environment, supposedly with much more powerful hardware, the search response time is not looking good. This is what we observed:

  1. The first few queries usually take about 4-7 seconds.
  2. It gets better over some time until it got to about 20ms.
  3. After a minute or so of inactivity, the response time again got back to 4-7 seconds.

This looks like it has something to do with caching, but the question is, how can we ensure consistent search response time? Or this is expected behavior?

The search query:

{
    "query": {
        "bool": {
            "filter": [
                {
                    "term": {
                        "id": {
                            "value": "sample-id"
                        }
                    }
                }
            ]
        }
    }
}

We also checked the CPU utilization of all OpenSearch nodes and they are consistently low: not more than 5% every time we checked.

Please advise if there is anything that we are missing?

Configuration:
3 nodes 128 cores CPU x 500GB memory running in HOT state
3 nodes 96 cores CPU x 500GB memory running in WARM state
Number of shards: 3
Number of replicas: 1
Index is set to rollover daily / whenever primary shard is more than 10GB.
Data ingestion rate is about 250GB per day.

Relevant Logs or Screenshots: