Slow fetching performance

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

OpenSearch 2.17 and OpenSearch 2.19

Describe the issue:

Slow fetching performance

Configuration:

AWS OpenSearch Service
3 Dedicated Master nodes: r7g.4xlarge
39 Data nodes: i4g.4xlarge
1116 primary shards
2 replicas
Most shards are between 10-50 GB (some are smaller)
~1 billion documents

Relevant Logs or Screenshots:

Query performance is good when size=0 but when size=1 and _source=false the query time increases significantly. Only the meta fields are being returned, no _source or stored fields.

Query:

{
  "query": {
    "query_string": {
      "query": "the OR a OR is OR in OR at OR on"
    }
  }
}

_count for the above query is 1,057,557,417

Performance with size = 0 (_search?_source=false&request_cache=false&size=0)

Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
search   /_search - size=0                                                                 51     0(0.00%) |    187     158     440    180 |    0.86        0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------

Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
search   /_search - size=0                                                                     180    190    190    190    200    210    260    440    440    440    440     51
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------

Performance with size = 1 (_search?_source=false&request_cache=false&size=1)

Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
search   /_search - size=1                                                                 11     0(0.00%) |  14508   11888   17015  15000 |    0.06        0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------

Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
search   /_search - size=1                                                                   15000  15000  16000  16000  16000  17000  17000  17000  17000  17000  17000     11
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------

Any help would be appreciated. Let me know if any additional information is needed.

Thanks