# AWS OpenSearch service lost one node during indexation of knn vectors

**URL:** <https://forum.opensearch.org/t/aws-opensearch-service-lost-one-node-during-indexation-of-knn-vectors/16165>\
**Category:** OpenSearch\
**Created:** [October 5, 2023, 4:42pm UTC](https://forum.opensearch.org/t/aws-opensearch-service-lost-one-node-during-indexation-of-knn-vectors/16165 "2023-10-05T16:42:20Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Garance](https://avatars.discourse-cdn.com/v4/letter/g/c2a13f/32.png) [@Garance](https://forum.opensearch.org/u/Garance)\
**Post date:** [October 5, 2023, 4:42pm UTC](https://forum.opensearch.org/t/aws-opensearch-service-lost-one-node-during-indexation-of-knn-vectors/16165/1 "2023-10-05T16:42:20Z")

</div>

**Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser):  
OpenSearch 1.3 on AWS

**Describe the issue** :  
I have a OpenSearch Service instance on AWS with 2 nodes, during each indexation with knn vectors, one node was lost near the end of indexation. Thanks to the automatic remediation of red clusters, the lost node was restored 30 minutes later.

The vector index contains some text fields and a knn field, with more than 2 millions documents, the total size is 11gb in index.  
I have a Python3 program to get data from an index with scroll, and run vector indexation with \_bulk

How to avoid the problem of lost node?

**Configuration** :  
OpenSearch Service instance: type c6g.xlarge.search on AWS, 2 nodes with 4 vCPU, 8 Gb RAM, 200 Gb storage, 6000 IOPS, 256 Mo/s of each node

Schema:

```auto
{
  "settings": {
    "index": {
      "number_of_shards": 2,
      "number_of_replicas": 0,
      "knn": true,
      "knn.algo_param.ef_search": 512
    }
  },
  "mappings": {
    "dynamic": "false",
    "properties": {
      "title": {
        "type": "text"
      },
      "text": {
        "type": "text"
      }
      "vector": {
        "type": "knn_vector",
        "dimension": 256,
        "method": {
          "name": "hnsw",
          "space_type": "l2",
          "engine": "nmslib",
          "parameters": {
            "ef_construction": 2000,
            "m": 16
          }
        }
      }
    }
  }
}

```

**Relevant Logs or Screenshots** :  
IndexingRate

 ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/7/7b05427c19a324379fddcc79a07a3daf8fdf5805.png)

---

<div class="post-metadata">

**Author:** ![radu.gheorghe](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/radu.gheorghe/32/3516_2.png) [@radu.gheorghe](https://forum.opensearch.org/u/radu.gheorghe)\
**Post date:** [October 10, 2023, 9:52am UTC](https://forum.opensearch.org/t/aws-opensearch-service-lost-one-node-during-indexation-of-knn-vectors/16165/2 "2023-10-10T09:52:40Z")

</div>

Why did the node go down? Do you see anything in the logs?

---

<div class="post-metadata">

**Author:** ![Garance](https://avatars.discourse-cdn.com/v4/letter/g/c2a13f/32.png) [@Garance](https://forum.opensearch.org/u/Garance)\
**Post date:** [October 10, 2023, 12:07pm UTC](https://forum.opensearch.org/t/aws-opensearch-service-lost-one-node-during-indexation-of-knn-vectors/16165/3 "2023-10-10T12:07:46Z")

</div>

I didn’t know the reason. Here are some logs in CloudWatch:

[2023-10-02T21:19:33,271][WARN][o.o.c.NodeConnectionsService] [19a43ee7408bca1891925f5ed1897d14] failed to connect to {e7736a956ab97de531be4c788aa799ed}{XLPkuGNkSPulv8HsUpJChw}{0DFrYfW3Q8C4mAlZ\_gwv5Q}{ **IP** }{ **IP** }{dimr}{dp\_version=20210501, distributed\_snapshot\_deletion\_enabled=false, cold\_enabled=false, adv\_sec\_enabled=false, **AMAZON\_INTERNAL** , cross\_cluster\_transport\_address= **IP** , awareness\_features\_enabled=true, global\_cpu\_usage\_ac\_supported=true, shard\_indexing\_pressure\_enabled=true, **AMAZON\_INTERNAL** , search\_backpressure\_feature\_present=true} (tried [25] times)  
ConnectTransportException[[xxx][**IP**] handshake failed. unexpected remote node {xxx}{xxx}{xxx}{ **IP** }{ **IP** }{dimr}{dp\_version=20210501, distributed\_snapshot\_deletion\_enabled=false, cold\_enabled=false, adv\_sec\_enabled=false, **AMAZON\_INTERNAL** , cross\_cluster\_transport\_address= **IP** , awareness\_features\_enabled=true, global\_cpu\_usage\_ac\_supported=true, shard\_indexing\_pressure\_enabled=true, **AMAZON\_INTERNAL** , search\_backpressure\_feature\_present=true}]

[2023-10-02T21:20:17,508][WARN][o.o.i.c.IndicesClusterStateService] [xxx] [my\_knn\_index][0] marking and sending shard failed due to [shard failure, reason [merge failed]]  
org.apache.lucene.index.MergePolicy$MergeException: java.lang.RuntimeException: java.lang.RuntimeException: [KNN] Adding footer to serialized graph failed: org.apache.lucene.index.MergePolicy$MergeAbortedException: Merge aborted.  
…

During the indexation, CPUUtilization was 60%, JVMMemoryPressure was between 80% and 100%, FreeStorageSpace was 220 Gib.

---

<div class="post-metadata">

**Author:** ![vamshin](https://avatars.discourse-cdn.com/v4/letter/v/c5a1d2/32.png) [@vamshin](https://forum.opensearch.org/u/vamshin)\
**Post date:** [October 11, 2023, 5:38am UTC](https://forum.opensearch.org/t/aws-opensearch-service-lost-one-node-during-indexation-of-knn-vectors/16165/4 "2023-10-11T05:38:52Z")

</div>

@Garance Is it possible to repro in latest versions(\>=2.5 version)? We can certainly take look but want to make sure if this is something we fixed in 2.x.

---

<div class="post-metadata">

**Author:** ![Garance](https://avatars.discourse-cdn.com/v4/letter/g/c2a13f/32.png) [@Garance](https://forum.opensearch.org/u/Garance)\
**Post date:** [October 11, 2023, 12:10pm UTC](https://forum.opensearch.org/t/aws-opensearch-service-lost-one-node-during-indexation-of-knn-vectors/16165/5 "2023-10-11T12:10:29Z")

</div>

@vamshin Thank you for the response, I will test the latest version.