# Issue Running opensearch-benchmark http\_logs on with opensearch container

**URL:** <https://forum.opensearch.org/t/issue-running-opensearch-benchmark-http-logs-on-with-opensearch-container/23047>\
**Category:** OpenSearch\
**Tags:** troubleshoot\
**Created:** [January 8, 2025, 6:18pm UTC](https://forum.opensearch.org/t/issue-running-opensearch-benchmark-http-logs-on-with-opensearch-container/23047 "2025-01-08T18:18:34Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![titoreinhart](https://avatars.discourse-cdn.com/v4/letter/t/258eb7/32.png) [@titoreinhart](https://forum.opensearch.org/u/titoreinhart)\
**Post date:** [January 8, 2025, 6:18pm UTC](https://forum.opensearch.org/t/issue-running-opensearch-benchmark-http-logs-on-with-opensearch-container/23047/1 "2025-01-08T18:18:34Z")

</div>

**Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser):

Is there a standard setup for running opensearch-benchmark http\_logs against running opensearch docker containers?

**Describe the issue** :

Benchmark hangs and according to the log coordinating node is getting a python3.9 timeout error.

**Configuration** :

Using a client/server topology on the same network - verified that both port and ip addr is reachable. This configuration works running NYC-Taxis.

docker pull opensearchproject/opensearch:latest

docker run -d -p 9200:9200 -p 9600:9600 -e “discovery.type=single-node” -e “OPENSEARCH\_INITIAL\_ADMIN\_PASSWORD=” opensearchproject/opensearch:latest

opensearch-benchmark execute-test --pipeline=benchmark-only --workload=http\_logs --target-host=https://\<ip\_address\>:9200 --client-options=basic\_auth\_user:admin,basic\_auth\_password:\<auth\_passwd\>,verify\_certs:false --kill-running-processes

**Relevant Logs or Screenshots** :

Hangs on “Running check-cluster-health [0% done]”

From the Log:

2025-01-08 18:12:57,596 ActorAddr-(T|:37549)/PID:7258 osbenchmark.worker\_coordinator.worker\_coordinator INFO iteration-count-based schedule will determine when the schedule for [check-cluster-health] terminates.  
2025-01-08 18:13:08,476 -not-actor-/PID:7258 opensearch WARNING GET [https://10.16.105.153:9200/\_cluster/health?wait\_for\_status=green&wait\_for\_no\_relocating\_shards=true](https://10.16.105.153:9200/_cluster/health?wait_for_status=green&wait_for_no_relocating_shards=true) [status:N/A request:10.801s]  
Traceback (most recent call last):  
File “/home/user/open/lib64/python3.9/site-packages/aiohttp/client\_reqrep.py”, line 1059, in start  
message, payload = await protocol.read() # type: ignore[union-attr]  
File “/home/user/open/lib64/python3.9/site-packages/aiohttp/streams.py”, line 671, in read  
await self.\_waiter  
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):  
File “/home/user/open/lib64/python3.9/site-packages/opensearchpy/\_async/http\_aiohttp.py”, line 294, in perform\_request  
async with self.session.request(  
File “/home/user/open/lib64/python3.9/site-packages/aiohttp/client.py”, line 1425, in **aenter**  
self.\_resp: \_RetType = await self.\_coro  
File “/home/user/open/lib64/python3.9/site-packages/aiohttp/client.py”, line 730, in \_request  
await resp.start(conn)  
File “/home/user/open/lib64/python3.9/site-packages/aiohttp/client\_reqrep.py”, line 1074, in start  
self.\_continue = None  
File “/home/user/open/lib64/python3.9/site-packages/aiohttp/helpers.py”, line 671, in **exit**  
raise asyncio.TimeoutError from exc\_val  
asyncio.exceptions.TimeoutError

Localhost cluster health check:  
Issuing a curl on localhost stdout:

{  
“name” : “2d6f3ab15f2d”,  
“cluster\_name” : “docker-cluster”,  
“cluster\_uuid” : “AOLKcancRJiWWO0xZuxIOQ”,  
“version” : {  
“distribution” : “opensearch”,  
“number” : “2.18.0”,  
“build\_type” : “tar”,  
“build\_hash” : “99a9a81da366173b0c2b963b26ea92e15ef34547”,  
“build\_date” : “2024-10-31T19:08:04.231254959Z”,  
“build\_snapshot” : false,  
“lucene\_version” : “9.12.0”,  
“minimum\_wire\_compatibility\_version” : “7.10.0”,  
“minimum\_index\_compatibility\_version” : “7.0.0”  
},

Benchmark additionally runs fine when workload=nyc\_taxis. Any help or guidance is greatly appreciated.

---

<div class="post-metadata">

**Author:** ![dancoh](https://avatars.discourse-cdn.com/v4/letter/d/ed655f/32.png) [@dancoh](https://forum.opensearch.org/u/dancoh)\
**Post date:** [January 9, 2025, 11:13pm UTC](https://forum.opensearch.org/t/issue-running-opensearch-benchmark-http-logs-on-with-opensearch-container/23047/2 "2025-01-09T23:13:33Z")

</div>

We are seeing similar phenomenon with nyc\_taxis on amazonlinux2023 and ubuntu20.04.6.  
This only started last night, before that we were runnin with no issues.

**Configuration**

sudo docker run -dq --name opensearch -p 9200:9200 -p 9600:9600 -v /mnt/opensearch/data:/usr/share/opensearch/data -e “discovery.type=single-node” -e “OPENSEARCH\_INITIAL\_ADMIN\_PASSWORD=\*\*\*\*\*\*” opensearchproject/opensearch:2.14.0

sudo docker run -dq --name opensearch-benchmark --network=host -v /mnt/benchmark/benchmark:/opensearch-benchmark/.benchmark opensearchproject/  
opensearch-benchmark:1.6.0 execute-test --target-hosts [https://localhost:9200](https://localhost:9200) --pipeline benchmark-only --workload nyc\_taxis --client-options ‘basic\_auth\_user:admin,basic\_auth\_password:\*\*\*\*\*\*\*,verify\_certs:false,timeout:30’ --distri  
bution-version 2.14.0;

**Logs**  
2025-01-09 22:46:48,905 -not-actor-/PID:63 opensearch WARNING GET localhost:9200/\_cluster/health?wait\_for\_status=green&wait\_for\_no\_relocating\_shards=true [status:408 request:30.012s]  
2025-01-09 22:46:48,905 ActorAddr-(T|:41643)/PID:63 osbenchmark.worker\_coordinator.runner INFO [cluster-health] has timed out. Retrying in [0.50] seconds.  
2025-01-09 22:47:19,409 -not-actor-/PID:63 opensearch WARNING GET localhost:9200/\_cluster/health?wait\_for\_status=green&wait\_for\_no\_relocating\_shards=true [status:408 request:30.002s]  
2025-01-09 22:47:19,409 ActorAddr-(T|:41643)/PID:63 osbenchmark.worker\_coordinator.runner INFO [cluster-health] has timed out. Retrying in [0.50] seconds.  
2025-01-09 22:47:49,913 -not-actor-/PID:63 opensearch WARNING GET https:localhost:9200/\_cluster/health?wait\_for\_status=green&wait\_for\_no\_relocating\_shards=true [status:408 request:30.002s]  
2025-01-09 22:47:49,913 ActorAddr-(T|:41643)/PID:63 osbenchmark.worker\_coordinator.runner INFO [cluster-health] has timed out. Retrying in [0.50] seconds.  
…  
2025-01-09 22:51:23,439 -not-actor-/PID:63 opensearch WARNING GET localhost:9200/\_cluster/health?wait\_for\_status=green&wait\_for\_no\_relocating\_shards=true [status:N/A request:30.002s]  
Traceback (most recent call last):  
File “/usr/local/lib/python3.11/site-packages/opensearchpy/\_async/http\_aiohttp.py”, line 296, in perform\_request  
async with self.session.request(  
File “/usr/local/lib/python3.11/site-packages/aiohttp/client.py”, line 1197, in **aenter**  
self.\_resp = await self.\_coro  
^^^^^^^^^^^^^^^^  
File “/usr/local/lib/python3.11/site-packages/aiohttp/client.py”, line 608, in \_request  
await resp.start(conn)  
File “/usr/local/lib/python3.11/site-packages/aiohttp/client\_reqrep.py”, line 971, in start  
with self.\_timer:  
File “/usr/local/lib/python3.11/site-packages/aiohttp/helpers.py”, line 735, in **exit**  
raise asyncio.TimeoutError from None  
TimeoutError

(had to edit a little bit the log because as a new user I can only put two links…)

---

<div class="post-metadata">

**Author:** ![titoreinhart](https://avatars.discourse-cdn.com/v4/letter/t/258eb7/32.png) [@titoreinhart](https://forum.opensearch.org/u/titoreinhart)\
**Post date:** [January 22, 2025, 5:29pm UTC](https://forum.opensearch.org/t/issue-running-opensearch-benchmark-http-logs-on-with-opensearch-container/23047/3 "2025-01-22T17:29:11Z")

</div>

@dancoh did you find a resolution?

---

<div class="post-metadata">

**Author:** ![dancoh](https://avatars.discourse-cdn.com/v4/letter/d/ed655f/32.png) [@dancoh](https://forum.opensearch.org/u/dancoh)\
**Post date:** [January 23, 2025, 7:20am UTC](https://forum.opensearch.org/t/issue-running-opensearch-benchmark-http-logs-on-with-opensearch-container/23047/4 "2025-01-23T07:20:19Z")

</div>

Hi @titoreinhart, kind of - it is a workaround - we are running the cluster with “plugins.security.disabled=true” flag and the benchmark without the SSL (https) connectivity: “–target-hosts [http://localhost:9200](http://localhost:9200)”

---

<div class="post-metadata">

**Author:** ![titoreinhart](https://avatars.discourse-cdn.com/v4/letter/t/258eb7/32.png) [@titoreinhart](https://forum.opensearch.org/u/titoreinhart)\
**Post date:** [January 24, 2025, 7:32pm UTC](https://forum.opensearch.org/t/issue-running-opensearch-benchmark-http-logs-on-with-opensearch-container/23047/5 "2025-01-24T19:32:57Z")

</div>

ok same - thanks for the confirmation 🙂

---

<div class="post-metadata">

**Author:** ![KateWinslet](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/katewinslet/32/6694_2.png) [@KateWinslet](https://forum.opensearch.org/u/KateWinslet)\
**Post date:** [January 29, 2025, 7:03pm UTC](https://forum.opensearch.org/t/issue-running-opensearch-benchmark-http-logs-on-with-opensearch-container/23047/6 "2025-01-29T19:03:52Z")

</div>

> [@titoreinhart](#):
>
> **Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser):
> 
> Is there a standard setup for running opensearch-benchmark http\_logs against running opensearch docker containers?
> 
> **Describe the issue** :
> 
> Benchmark hangs and according to the log coordinating node is getting a python3.9 timeout error.
> 
> **Configuration** :
> 
> Using a client/server topology on the same network - verified that both port and ip addr is reachable. This configuration works running NYC-Taxis.
> 
> docker pull opensearchproject/opensearch:latest
> 
> docker run -d -p 9200:9200 -p 9600:9600 -e “discovery.type=single-node” -e “OPENSEARCH\_INITIAL\_ADMIN\_PASSWORD=” opensearchproject/opensearch:latest
> 
> opensearch-benchmark execute-test --pipeline=benchmark-only --workload=http\_logs --target-host=https://\<ip\_address\>:9200 --client-options=basic\_auth\_user:admin,basic\_auth\_password:\<auth\_passwd\>,verify\_certs:false --kill-running-processes
> 
> **Relevant Logs or Screenshots** :
> 
> Hangs on “Running check-cluster-health [0% done]”
> 
> From the Log:
> 
> 2025-01-08 18:12:57,596 ActorAddr-(T|:37549)/PID:7258 osbenchmark.worker\_coordinator.worker\_coordinator INFO iteration-count-based schedule will determine when the schedule for [check-cluster-health] terminates.  
> 2025-01-08 18:13:08,476 -not-actor-/PID:7258 opensearch WARNING GET [https://10.16.105.153:9200/\_cluster/health?wait\_for\_status=green&wait\_for\_no\_relocating\_shards=true](https://10.16.105.153:9200/_cluster/health?wait_for_status=green&wait_for_no_relocating_shards=true) [status:N/A request:10.801s] [ISO 45001 Lead Auditor Training](https://www.kelmacgroup.com/training/ISO-45001-Lead-Auditor-Training-Course)  
> Traceback (most recent call last):  
> File “/home/user/open/lib64/python3.9/site-packages/aiohttp/client\_reqrep.py”, line 1059, in start  
> message, payload = await protocol.read() # type: ignore[union-attr]  
> File “/home/user/open/lib64/python3.9/site-packages/aiohttp/streams.py”, line 671, in read  
> await self.\_waiter  
> asyncio.exceptions.CancelledError
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):  
> File “/home/user/open/lib64/python3.9/site-packages/opensearchpy/\_async/http\_aiohttp.py”, line 294, in perform\_request  
> async with self.session.request(  
> File “/home/user/open/lib64/python3.9/site-packages/aiohttp/client.py”, line 1425, in **aenter**  
> self.\_resp: \_RetType = await self.\_coro  
> File “/home/user/open/lib64/python3.9/site-packages/aiohttp/client.py”, line 730, in \_request  
> await resp.start(conn)  
> File “/home/user/open/lib64/python3.9/site-packages/aiohttp/client\_reqrep.py”, line 1074, in start  
> self.\_continue = None  
> File “/home/user/open/lib64/python3.9/site-packages/aiohttp/helpers.py”, line 671, in **exit**  
> raise asyncio.TimeoutError from exc\_val  
> asyncio.exceptions.TimeoutError
> 
> Localhost cluster health check:  
> Issuing a curl on localhost stdout:
> 
> {  
> “name” : “2d6f3ab15f2d”,  
> “cluster\_name” : “docker-cluster”,  
> “cluster\_uuid” : “AOLKcancRJiWWO0xZuxIOQ”,  
> “version” : {  
> “distribution” : “opensearch”,  
> “number” : “2.18.0”,  
> “build\_type” : “tar”,  
> “build\_hash” : “99a9a81da366173b0c2b963b26ea92e15ef34547”,  
> “build\_date” : “2024-10-31T19:08:04.231254959Z”,  
> “build\_snapshot” : false,  
> “lucene\_version” : “9.12.0”,  
> “minimum\_wire\_compatibility\_version” : “7.10.0”,  
> “minimum\_index\_compatibility\_version” : “7.0.0”  
> },
> 
> Benchmark additionally runs fine when workload=nyc\_taxis. Any help or guidance is greatly appreciated.

Increase the timeout setting in the benchmark or verify if the cluster health is stable to resolve the network-related issue causing the timeout error.

---

<div class="post-metadata">

**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\
**Post date:** [January 30, 2025, 10:32am UTC](https://forum.opensearch.org/t/issue-running-opensearch-benchmark-http-logs-on-with-opensearch-container/23047/7 "2025-01-30T10:32:31Z")

</div>

@dancoh @titoreinhart I’ve got the same when running a single node cluster. However, it worked when I had 2 node cluster.  
I think benchmark script is waiting for Green health check. In the single node you get Yellow as not all indices are in Green state.

Single node:

```auto
curl --insecure -u admin:Eliatra123 https://localhost:9200/_cat/indices
green open logs-221998 N8WwN_HlT2mihrW414BggQ 5 0 0 0 1kb 1kb
green open .opensearch-observability 9gBnn-PdRkOmw6h3RTMJKg 1 0 0 0 208b 208b
green open .plugins-ml-config 6V1gv32cTWuUD9VodO0SqQ 1 0 1 0 4kb 4kb
green open logs-211998 wi-GdCF6SIW4RmZnPpsZIw 5 0 0 0 1kb 1kb
green open reindexed-logs 5uxflNR3Sv69FStX9U_t9Q 5 0 0 0 1kb 1kb
green open logs-231998 21trI9JRQL2ONPVqEEgJuw 5 0 0 0 1kb 1kb
yellow open security-auditlog-2025.01.30 46VjxnjWTaKh15e2gU-7iA 1 1 16 0 115.2kb 115.2kb
green open logs-241998 OF_Td9r-Sj612BZBB8HVMQ 5 0 0 0 1kb 1kb
green open logs-181998 mZERbzmBR8ecvQdZLrevLA 5 0 0 0 1kb 1kb
green open logs-201998 tv311_bmSiSwT2-mDYBT1w 5 0 0 0 1kb 1kb
green open .opendistro_security bL4HF9TpQP6GZa19Lb2CLg 1 0 10 0 82.4kb 82.4kb
green open logs-191998 xe6l9TJnRTSzTSV2DWjwQQ 5 0 0 0 1kb 1kb

curl --insecure -u admin:Eliatra123 https://localhost:9200/_cluster/health?pretty
{
  "cluster_name" : "docker-cluster",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 45,
  "active_shards" : 45,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 1,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 97.82608695652173
}

```

Two nodes cluster:

```auto
curl --insecure -u admin:Eliatra123 https://docker5.pablo.local:9200/_cat/indices
green open logs-221998 6jYYo1CQT_W_SvspIuuoug 5 0 0 0 1kb 1kb
green open reindexed-logs uxL_DD5YS4CWWxgEe6rNpw 5 0 0 0 1kb 1kb
green open security-auditlog-2025.01.30 QRyE9uSVSeupTNJH_4VFAQ 1 1 16 0 186.6kb 86kb
green open logs-241998 XBAqTLUcR0ye8BctH4iLsQ 5 0 0 0 1kb 1kb
green open logs-181998 Q7duo8W8RgqBLG_OX9GkCQ 5 0 2666530 0 218.2mb 218.2mb
green open logs-201998 G5tI7JLlQjKTyDgu3UDvGg 5 0 5240248 0 388.3mb 388.3mb
green open .opendistro_security 4e5w9l9xQBqsxl6LXhGglQ 1 1 10 0 118.8kb 59.4kb
green open .kibana_1 akvKKk-TTS2NR8iPNTJa0A 1 1 0 0 416b 208b
green open .plugins-ml-config KwxJbco0ScKEIbqoMeNn3w 1 1 1 0 8.1kb 4kb
green open .opensearch-observability CS3rl5VxSBOAFt9dsUq8Hw 1 1 0 0 416b 208b
green open logs-211998 VyYJBoBUS4S4Nptt6JQFfA 5 0 0 0 1kb 1kb
green open logs-231998 7WMjc6XiRUuzETZKqjdPww 5 0 0 0 1kb 1kb
green open logs-191998 9CpQbp1aQIOr2F3iYeQLAw 5 0 9554199 0 700.4mb 700.4mb

curl --insecure -u admin:Eliatra123 https://docker5.pablo.local:9200/_cluster/health?pretty
{
  "cluster_name" : "opensearch-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 46,
  "active_shards" : 52,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

```

When you look at the shards in single node cluster, you’ll notice that security plugin audit logs are set with 1 primary and 1 replica shard. Replica is in Unassigned state.  
Disabling security plugin resolved your issue as security plugin audit index wasn’t created.

I would avoid disabling security plugin as results may vary.

```auto
curl --insecure -u admin:Eliatra123 https://localhost:9200/_cat/shards
logs-221998 0 p STARTED 0 208b 172.17.0.2 c3d46fadcddb
logs-221998 1 p STARTED 0 208b 172.17.0.2 c3d46fadcddb
logs-221998 2 p STARTED 0 208b 172.17.0.2 c3d46fadcddb
logs-221998 3 p STARTED 0 208b 172.17.0.2 c3d46fadcddb
logs-221998 4 p STARTED 0 208b 172.17.0.2 c3d46fadcddb
reindexed-logs 0 p STARTED 0 208b 172.17.0.2 c3d46fadcddb
reindexed-logs 1 p STARTED 0 208b 172.17.0.2 c3d46fadcddb
reindexed-logs 2 p STARTED 0 208b 172.17.0.2 c3d46fadcddb
reindexed-logs 3 p STARTED 0 208b 172.17.0.2 c3d46fadcddb
reindexed-logs 4 p STARTED 0 208b 172.17.0.2 c3d46fadcddb
security-auditlog-2025.01.30 0 p STARTED 16 115.2kb 172.17.0.2 c3d46fadcddb
security-auditlog-2025.01.30 0 r UNASSIGNED

```

Try using docker-compose example from OpenSearch documentation.

> **[Docker](https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/#sample-docker-composeyml)**
>
> Docker

---

<div class="post-metadata">

**Author:** ![dancoh](https://avatars.discourse-cdn.com/v4/letter/d/ed655f/32.png) [@dancoh](https://forum.opensearch.org/u/dancoh)\
**Post date:** [February 2, 2025, 4:41pm UTC](https://forum.opensearch.org/t/issue-running-opensearch-benchmark-http-logs-on-with-opensearch-container/23047/8 "2025-02-02T16:41:06Z")

</div>

Thanks @pablo , what you say makes sense on hand, but on the other hand my setup worked fine for more than a year with a single-node-cluster and then suddenly stopped working. Hence, I assume that the issue is some expired certificate somewhere, and not the missing node.

---

<div class="post-metadata">

**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\
**Post date:** [February 5, 2025, 8:00am UTC](https://forum.opensearch.org/t/issue-running-opensearch-benchmark-http-logs-on-with-opensearch-container/23047/9 "2025-02-05T08:00:12Z")

</div>

@dancoh If your script stops at the same point as @titoreinhart then I don’t think it is related to your certificates.

> [@titoreinhart](#):
>
> Hangs on “Running check-cluster-health [0% done]”

Before the benchmark script runs the cluster health check, it deletes and creates indices on the target cluster.

```auto
[WARNING] flush_total_time is 33 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
Running delete-index [100% done]
Running create-index [100% done]
Running check-cluster-health [0% done]

```

```auto
green open logs-181998 4qdKaidqQfiV8KslZj0ehg 5 0 0 0 1kb 1kb
green open logs-191998 Whcdog-JQkmlyhKyFAxFUw 5 0 0 0 1kb 1kb
green open logs-201998 p_6RC1NsRVSUx6fPimdm_w 5 0 0 0 1kb 1kb
green open logs-211998 VF18n5VRQg27lYqjKjwdmg 5 0 0 0 1kb 1kb
green open logs-221998 iuqNFWUuQ0mTUHkrjwOJSw 5 0 0 0 1kb 1kb
green open logs-231998 ondT7otcTEahVc_MgmPstg 5 0 0 0 1kb 1kb
green open logs-241998 8lzhCIIbSYScymPhPEL0PQ 5 0 0 0 1kb 1kb

```

The expired certificate shouldn’t allow the benchmark script to run these actions before a cluster health check.

What is your cluster status? If it’s yellow, you can always change the number of replica shards to a number that will fit in your cluster.