I have an Elasticsearch cluster with 3 nodes, each running version 7.16.1, and Kibana version 7.4.2 on each node. Our company has decided to migrate to OpenSearch. After researching the migration process, I found that Elasticsearch 7.10.2 can be migrated to OpenSearch 1.x, and thereafter, it can be upgraded to OpenSearch 2.x.
I found three methods:
Snapshot: However, snapshots only support up to Elasticsearch 7.12.
Reindexing from the Elasticsearch cluster to the OpenSearch cluster, but this takes a lot of time since I have almost 1TB of data.
Using Logstash.
I am wondering which method is better for migrating from Elasticsearch 7.16.1 to OpenSearch without data loss. Actually, I’m not sure if the above methods will work effectively.
I would use Logstash, because 1TB isn’t all that much, if both clusters are in the same datacenter or close enough.
This will allow you to experiment with new OpenSearch features (e.g. maybe you’ll want to use match_only_text if you’re searching logs) before you make the migration. And you do that migration in one go (ES to latest OpenSearch, rather than in-place upgrades twice, which as you said isn’t really OK since you’re on 7.16>7.10).
Hi, thank you very much for your response.
Both clusters are located on the same NAS server. I believe that reindexing via Logstash should be quick. We’re not searching logs. We have an online shop, and our programmers use an API to retrieve some JSON files and compare them, as far as I understand. So, the plan would be as follows: I should create a brand new OpenSearch cluster, install Logstash with OpenSearch plugins, and transfer the data from Elasticsearch through Logstash to the OpenSearch cluster, right?
Alright
I need to discuss with our programmers whether they really want to migrate to OpenSearch because some of them do not agree. :))
If we decide to definitely migrate to OpenSearch, I will test it and share the results here.
First of all, thank you very much for your solution. I migrated from an ES-Cluster 7.16 through Logstash 7.16 to OS-Cluster 2.12. All the indexes, along with their templates, were migrated, and this method proved to be really fast. The only drawback was that when creating a pipeline and starting Logstash, it sends all the documents to OpenSearch but doesn’t stop even after all documents have been migrated. It keeps sending documents repeatedly. However, I was able to solve this problem with a script. The process stopped when the number of documents in OS matched exactly with the number in ES.
