# ReIndexing index data from ElasticSearch 7.14.0 to OpenSearch 2.14.0 using ReIndex API

**URL:** <https://forum.opensearch.org/t/reindexing-index-data-from-elasticsearch-7-14-0-to-opensearch-2-14-0-using-reindex-api/20407>\
**Category:** Open Source Elasticsearch and Kibana\
**Tags:** migration\
**Created:** [July 20, 2024, 7:32am UTC](https://forum.opensearch.org/t/reindexing-index-data-from-elasticsearch-7-14-0-to-opensearch-2-14-0-using-reindex-api/20407 "2024-07-20T07:32:49Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![nadiw98](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/nadiw98/32/7802_2.png) [@nadiw98](https://forum.opensearch.org/u/nadiw98)\
**Post date:** [July 20, 2024, 7:32am UTC](https://forum.opensearch.org/t/reindexing-index-data-from-elasticsearch-7-14-0-to-opensearch-2-14-0-using-reindex-api/20407/1 "2024-07-20T07:32:49Z")

</div>

I’m trying to reindex data from ES to OS through reindex API. When indexing data to the ES, previously we had set custom routing shards allocations for the documents. When reindexing the data to OS, I want to keep the same routing shards allocations for the destination index. How can I achieve this through reindex API?

---

<div class="post-metadata">

**Author:** ![gaobinlong](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/gaobinlong/32/5242_2.png) [@gaobinlong](https://forum.opensearch.org/u/gaobinlong)\
**Post date:** [July 22, 2024, 2:17am UTC](https://forum.opensearch.org/t/reindexing-index-data-from-elasticsearch-7-14-0-to-opensearch-2-14-0-using-reindex-api/20407/2 "2024-07-22T02:17:00Z")

</div>

Reindex API keeps the routing value by default, routing in the source documents will be fetched and set in the bulk requests when writing to the destination index.

---

<div class="post-metadata">

**Author:** ![nadiw98](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/nadiw98/32/7802_2.png) [@nadiw98](https://forum.opensearch.org/u/nadiw98)\
**Post date:** [July 22, 2024, 7:19am UTC](https://forum.opensearch.org/t/reindexing-index-data-from-elasticsearch-7-14-0-to-opensearch-2-14-0-using-reindex-api/20407/4 "2024-07-22T07:19:25Z")

</div>

However, the issue is, that documents keep getting deleted from the destination index while reindexing. op\_type is also set to ‘index’ in reindex request

---

<div class="post-metadata">

**Author:** ![gaobinlong](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/gaobinlong/32/5242_2.png) [@gaobinlong](https://forum.opensearch.org/u/gaobinlong)\
**Post date:** [July 22, 2024, 7:33am UTC](https://forum.opensearch.org/t/reindexing-index-data-from-elasticsearch-7-14-0-to-opensearch-2-14-0-using-reindex-api/20407/5 "2024-07-22T07:33:53Z")

</div>

How did you find the documents keep getting deleted? Could you show the request parameters when calling reindex API?

---

<div class="post-metadata">

**Author:** ![nadiw98](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/nadiw98/32/7802_2.png) [@nadiw98](https://forum.opensearch.org/u/nadiw98)\
**Post date:** [July 22, 2024, 7:37am UTC](https://forum.opensearch.org/t/reindexing-index-data-from-elasticsearch-7-14-0-to-opensearch-2-14-0-using-reindex-api/20407/6 "2024-07-22T07:37:56Z")

</div>

While on reindexing I’m looking at the indices status through _GET \_cat/indices?v_.  
The request parameters are,

POST \_reindex  
{  
“source”: {  
“remote”: {  
“host”: “\<source\_ip\>:9200”  
},  
“index”: “\<index\_name\>”  
},  
“dest”: {  
“index”: “\<index\_name\>”,  
“op\_type”: “index”  
}  
}

---

<div class="post-metadata">

**Author:** ![gaobinlong](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/gaobinlong/32/5242_2.png) [@gaobinlong](https://forum.opensearch.org/u/gaobinlong)\
**Post date:** [July 22, 2024, 8:38am UTC](https://forum.opensearch.org/t/reindexing-index-data-from-elasticsearch-7-14-0-to-opensearch-2-14-0-using-reindex-api/20407/7 "2024-07-22T08:38:15Z")

</div>

I think the reason why you see there’re deleted documents in the target index is that you called reindex API multiple times so some document IDs are duplicated when you call reindex API again, but it doesn’t matter, if the document with same ID exists in the target index, then the document will be updated, if not the document will be created. For update operation, actually it’s transformed to an delete operation and index operation internally, so you can see there’re many `deleted` documents in the target index.

---

<div class="post-metadata">

**Author:** ![nadiw98](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/nadiw98/32/7802_2.png) [@nadiw98](https://forum.opensearch.org/u/nadiw98)\
**Post date:** [July 22, 2024, 9:46am UTC](https://forum.opensearch.org/t/reindexing-index-data-from-elasticsearch-7-14-0-to-opensearch-2-14-0-using-reindex-api/20407/8 "2024-07-22T09:46:09Z")

</div>

Thank you for your reply. But the thing is due to this deleted doc count getting increased, the storage is getting increased significantly.  
Here are stats,  
In Source Index - 397222 docs in 176.5mb  
In Destination - 397222 docs with 1191666 deleted docs, all together in 1.3gb

Since the reindex API is called once, are you suggesting that the API itself do bulk indexing?

---

<div class="post-metadata">

**Author:** ![gaobinlong](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/gaobinlong/32/5242_2.png) [@gaobinlong](https://forum.opensearch.org/u/gaobinlong)\
**Post date:** [July 23, 2024, 2:38am UTC](https://forum.opensearch.org/t/reindexing-index-data-from-elasticsearch-7-14-0-to-opensearch-2-14-0-using-reindex-api/20407/9 "2024-07-23T02:38:19Z")

</div>

1191666= 397222\*3, so I think the reindex API was called 4 times, if you were using the Dev tools in OpenSearch-Dashboards, please add a parameter `wait_for_completion=false` when calling the reindex API, because the default behavior of that API is to wait until the reindex process completes, but if takes more than 30 seconds, OpenSearch-Dashboards will retry so the API is called again.

```auto
POST _reindex?wait_for_completion=false

```

, when `wait_for_completion` is false, the reindex API will return a task ID, you can check the progress by calling the tasks API:

```auto
GET _tasks/{taskId}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/33547ea01a5b12dcca2958411d3edd97ae2ea8c1.png) [@system](https://forum.opensearch.org/u/system)\
**Post date:** [September 21, 2024, 2:38am UTC](https://forum.opensearch.org/t/reindexing-index-data-from-elasticsearch-7-14-0-to-opensearch-2-14-0-using-reindex-api/20407/10 "2024-09-21T02:38:29Z")

</div>

Closed on September 21, 2024, 2:38am UTC

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.
