Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Is there a standard setup for running opensearch-benchmark http_logs against running opensearch docker containers?
Describe the issue:
Benchmark hangs and according to the log coordinating node is getting a python3.9 timeout error.
Configuration:
Using a client/server topology on the same network - verified that both port and ip addr is reachable. This configuration works running NYC-Taxis.
docker pull opensearchproject/opensearch:latest
docker run -d -p 9200:9200 -p 9600:9600 -e “discovery.type=single-node” -e “OPENSEARCH_INITIAL_ADMIN_PASSWORD=” opensearchproject/opensearch:latest
opensearch-benchmark execute-test --pipeline=benchmark-only --workload=http_logs --target-host=https://<ip_address>:9200 --client-options=basic_auth_user:admin,basic_auth_password:<auth_passwd>,verify_certs:false --kill-running-processes
Relevant Logs or Screenshots:
Hangs on “Running check-cluster-health [ 0% done]”
From the Log:
2025-01-08 18:12:57,596 ActorAddr-(T|:37549)/PID:7258 osbenchmark.worker_coordinator.worker_coordinator INFO iteration-count-based schedule will determine when the schedule for [check-cluster-health] terminates.
2025-01-08 18:13:08,476 -not-actor-/PID:7258 opensearch WARNING GET https://10.16.105.153:9200/_cluster/health?wait_for_status=green&wait_for_no_relocating_shards=true [status:N/A request:10.801s]
Traceback (most recent call last):
File “/home/user/open/lib64/python3.9/site-packages/aiohttp/client_reqrep.py”, line 1059, in start
message, payload = await protocol.read() # type: ignore[union-attr]
File “/home/user/open/lib64/python3.9/site-packages/aiohttp/streams.py”, line 671, in read
await self._waiter
asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File “/home/user/open/lib64/python3.9/site-packages/opensearchpy/_async/http_aiohttp.py”, line 294, in perform_request
async with self.session.request(
File “/home/user/open/lib64/python3.9/site-packages/aiohttp/client.py”, line 1425, in aenter
self._resp: _RetType = await self._coro
File “/home/user/open/lib64/python3.9/site-packages/aiohttp/client.py”, line 730, in _request
await resp.start(conn)
File “/home/user/open/lib64/python3.9/site-packages/aiohttp/client_reqrep.py”, line 1074, in start
self._continue = None
File “/home/user/open/lib64/python3.9/site-packages/aiohttp/helpers.py”, line 671, in exit
raise asyncio.TimeoutError from exc_val
asyncio.exceptions.TimeoutError
Localhost cluster health check:
Issuing a curl on localhost stdout:
{
“name” : “2d6f3ab15f2d”,
“cluster_name” : “docker-cluster”,
“cluster_uuid” : “AOLKcancRJiWWO0xZuxIOQ”,
“version” : {
“distribution” : “opensearch”,
“number” : “2.18.0”,
“build_type” : “tar”,
“build_hash” : “99a9a81da366173b0c2b963b26ea92e15ef34547”,
“build_date” : “2024-10-31T19:08:04.231254959Z”,
“build_snapshot” : false,
“lucene_version” : “9.12.0”,
“minimum_wire_compatibility_version” : “7.10.0”,
“minimum_index_compatibility_version” : “7.0.0”
},
Benchmark additionally runs fine when workload=nyc_taxis. Any help or guidance is greatly appreciated.