Issue Running opensearch-benchmark http_logs on with opensearch container

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Is there a standard setup for running opensearch-benchmark http_logs against running opensearch docker containers?

Describe the issue:

Benchmark hangs and according to the log coordinating node is getting a python3.9 timeout error.

Configuration:

Using a client/server topology on the same network - verified that both port and ip addr is reachable. This configuration works running NYC-Taxis.

docker pull opensearchproject/opensearch:latest

docker run -d -p 9200:9200 -p 9600:9600 -e “discovery.type=single-node” -e “OPENSEARCH_INITIAL_ADMIN_PASSWORD=” opensearchproject/opensearch:latest

opensearch-benchmark execute-test --pipeline=benchmark-only --workload=http_logs --target-host=https://<ip_address>:9200 --client-options=basic_auth_user:admin,basic_auth_password:<auth_passwd>,verify_certs:false --kill-running-processes

Relevant Logs or Screenshots:

Hangs on “Running check-cluster-health [ 0% done]”

From the Log:

2025-01-08 18:12:57,596 ActorAddr-(T|:37549)/PID:7258 osbenchmark.worker_coordinator.worker_coordinator INFO iteration-count-based schedule will determine when the schedule for [check-cluster-health] terminates.
2025-01-08 18:13:08,476 -not-actor-/PID:7258 opensearch WARNING GET https://10.16.105.153:9200/_cluster/health?wait_for_status=green&wait_for_no_relocating_shards=true [status:N/A request:10.801s]
Traceback (most recent call last):
File “/home/user/open/lib64/python3.9/site-packages/aiohttp/client_reqrep.py”, line 1059, in start
message, payload = await protocol.read() # type: ignore[union-attr]
File “/home/user/open/lib64/python3.9/site-packages/aiohttp/streams.py”, line 671, in read
await self._waiter
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/home/user/open/lib64/python3.9/site-packages/opensearchpy/_async/http_aiohttp.py”, line 294, in perform_request
async with self.session.request(
File “/home/user/open/lib64/python3.9/site-packages/aiohttp/client.py”, line 1425, in aenter
self._resp: _RetType = await self._coro
File “/home/user/open/lib64/python3.9/site-packages/aiohttp/client.py”, line 730, in _request
await resp.start(conn)
File “/home/user/open/lib64/python3.9/site-packages/aiohttp/client_reqrep.py”, line 1074, in start
self._continue = None
File “/home/user/open/lib64/python3.9/site-packages/aiohttp/helpers.py”, line 671, in exit
raise asyncio.TimeoutError from exc_val
asyncio.exceptions.TimeoutError

Localhost cluster health check:
Issuing a curl on localhost stdout:

{
“name” : “2d6f3ab15f2d”,
“cluster_name” : “docker-cluster”,
“cluster_uuid” : “AOLKcancRJiWWO0xZuxIOQ”,
“version” : {
“distribution” : “opensearch”,
“number” : “2.18.0”,
“build_type” : “tar”,
“build_hash” : “99a9a81da366173b0c2b963b26ea92e15ef34547”,
“build_date” : “2024-10-31T19:08:04.231254959Z”,
“build_snapshot” : false,
“lucene_version” : “9.12.0”,
“minimum_wire_compatibility_version” : “7.10.0”,
“minimum_index_compatibility_version” : “7.0.0”
},

Benchmark additionally runs fine when workload=nyc_taxis. Any help or guidance is greatly appreciated.

We are seeing similar phenomenon with nyc_taxis on amazonlinux2023 and ubuntu20.04.6.
This only started last night, before that we were runnin with no issues.

Configuration

sudo docker run -dq --name opensearch -p 9200:9200 -p 9600:9600 -v /mnt/opensearch/data:/usr/share/opensearch/data -e “discovery.type=single-node” -e “OPENSEARCH_INITIAL_ADMIN_PASSWORD=******” opensearchproject/opensearch:2.14.0

sudo docker run -dq --name opensearch-benchmark --network=host -v /mnt/benchmark/benchmark:/opensearch-benchmark/.benchmark opensearchproject/
opensearch-benchmark:1.6.0 execute-test --target-hosts https://localhost:9200 --pipeline benchmark-only --workload nyc_taxis --client-options ‘basic_auth_user:admin,basic_auth_password:*******,verify_certs:false,timeout:30’ --distri
bution-version 2.14.0;

Logs
2025-01-09 22:46:48,905 -not-actor-/PID:63 opensearch WARNING GET localhost:9200/_cluster/health?wait_for_status=green&wait_for_no_relocating_shards=true [status:408 request:30.012s]
2025-01-09 22:46:48,905 ActorAddr-(T|:41643)/PID:63 osbenchmark.worker_coordinator.runner INFO [cluster-health] has timed out. Retrying in [0.50] seconds.
2025-01-09 22:47:19,409 -not-actor-/PID:63 opensearch WARNING GET localhost:9200/_cluster/health?wait_for_status=green&wait_for_no_relocating_shards=true [status:408 request:30.002s]
2025-01-09 22:47:19,409 ActorAddr-(T|:41643)/PID:63 osbenchmark.worker_coordinator.runner INFO [cluster-health] has timed out. Retrying in [0.50] seconds.
2025-01-09 22:47:49,913 -not-actor-/PID:63 opensearch WARNING GET https:localhost:9200/_cluster/health?wait_for_status=green&wait_for_no_relocating_shards=true [status:408 request:30.002s]
2025-01-09 22:47:49,913 ActorAddr-(T|:41643)/PID:63 osbenchmark.worker_coordinator.runner INFO [cluster-health] has timed out. Retrying in [0.50] seconds.

2025-01-09 22:51:23,439 -not-actor-/PID:63 opensearch WARNING GET localhost:9200/_cluster/health?wait_for_status=green&wait_for_no_relocating_shards=true [status:N/A request:30.002s]
Traceback (most recent call last):
File “/usr/local/lib/python3.11/site-packages/opensearchpy/_async/http_aiohttp.py”, line 296, in perform_request
async with self.session.request(
File “/usr/local/lib/python3.11/site-packages/aiohttp/client.py”, line 1197, in aenter
self._resp = await self._coro
^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/site-packages/aiohttp/client.py”, line 608, in _request
await resp.start(conn)
File “/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py”, line 971, in start
with self._timer:
File “/usr/local/lib/python3.11/site-packages/aiohttp/helpers.py”, line 735, in exit
raise asyncio.TimeoutError from None
TimeoutError

(had to edit a little bit the log because as a new user I can only put two links…)