Data Type Mismatch During Migration from Elasticsearch to OpenSearch

OpenSearch
1

:red_question_mark: Problem Summary:

I am migrating data from Elasticsearch (v7.17.9) to OpenSearch (v2.x). The migration is mostly successful; however, not all data types are correctly preserved in OpenSearch. This results in mapping conflicts and incomplete field representations, especially for fields with dynamic mapping in Elasticsearch.

:package: Environment:

  • Source: Elasticsearch v7.17.9
  • Target: OpenSearch v2.x
  • Migration method: Python script using elasticsearch and opensearch-py clients with scroll and bulk indexing
  • Index involved: nscdata

:test_tube: Observed Issues:

  1. Some fields are coming in as text in OpenSearch even though they were keyword or integer in Elasticsearch.
  2. Nested fields or arrays are flattened or interpreted incorrectly.
  3. Mappings are not preserved despite trying to copy settings and mappings using:

python

CopyEdit

es.indices.get_mapping(index='nscdata')
es.indices.get_settings(index='nscdata')

:gear: Steps Taken:

  • Extracted source mappings using GET nscdata/_mapping
  • Created new index in OpenSearch using these mappings
  • Used scroll API to extract documents and bulk indexed them into OpenSearch
  • Verified data with GET nscdata/_mapping on both ends

:clipboard: Sample Mapping Difference:

Elasticsearch (source):

json

CopyEdit

"device_id": {
  "type": "keyword"
}

OpenSearch (target):

json

CopyEdit

"device_id": {
  "type": "text"
}

:red_exclamation_mark: Question:

  • How can I fully preserve the original Elasticsearch mappings, especially field types, when migrating data to OpenSearch?
  • Is there a recommended tool or plugin for migration that maintains full mapping integrity?
  • Is this a known issue when using Python scripts and opensearch-py for migration?
2

Do you still have field-type issues after creating indices with specific mappings?

Are you aware of the Migration Assistant tool?