Index Template and Disable Dynamic mapping

OpenSearch
1

Version 2.11.0

I’m trying to set an index_template on our OpenSearch cluster with a set of defined fields that I want indexed but I want any fields that don’t fall into those defined to still be stored in the document, just not indexed.

Here is a sample of the index template that I put in

PUT /_index_template/standardtemplate
{
   "index_patterns":[
      "mypattern*"
   ],
   "priority": 100,
   "template": {
   "settings":{
      "index":{
         "mapping":{
            "total_fields":{
               "limit":"2000"
            }
         }
      }
   },
   "mappings":{
      "dynamic": false,
      "properties":{
           ....
      }
}
}

It was my understanding that setting the “mappings.dynamic” property to false would achieve this but I’ve noticed that it’s not working. Any suggestions?

Thank you!

2

@couten Could you share an example of the desired output index?

When the “mappings.dynamic” is set to false, all undefined fields will be stored only in the _source.
This is my example.

PUT /_index_template/standardtemplate
{
   "index_patterns":[
      "mypattern*"
   ],
   "priority": 100,
   "template": {
     "settings":{
        "index":{
           "mapping":{
              "total_fields":{
                 "limit":"2000"
              }
           }
        }
     },
     "mappings":{
        "dynamic": false,
        "properties": {
          "first": {
            "type": "text",
            "fields": {
              "keyword":{
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
     }
  }
}

POST mypattern1/_doc
{
  "first": "not_first",
  "second": "second"
}

GET mypattern1

{
  "mypattern1": {
    "aliases": {},
    "mappings": {
      "dynamic": "false",
      "properties": {
        "first": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    },
    "settings": {
      "index": {
        "replication": {
          "type": "DOCUMENT"
        },
        "mapping": {
          "total_fields": {
            "limit": "2000"
          }
        },
        "number_of_shards": "1",
        "provided_name": "mypattern1",
        "creation_date": "1708972480456",
        "number_of_replicas": "1",
        "uuid": "Ij961oBrTfie0EkVXg_r4Q",
        "version": {
          "created": "136327827"
        }
      }
    }
  }
}

GET mypattern1/_search

{
  "took": 5,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "mypattern1",
        "_id": "tYay5o0BbsIFAQjgmD07",
        "_score": 1,
        "_source": {
          "first": "not_first",
          "second": "second"
        }
      }
    ]
  }
}
3

@pablo :
I copy/pasted your example (index_template creation + sending that mypattern1/_doc) into a AWS 2.9 Opensearch cluster. I then did:

GET /mypattern1/_mapping

which gives:

{
  "mypattern1": {
    "mappings": {
      "properties": {
        "first": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "second": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "source_host": {
          "type": "ip"
        }
      }
    }
  }
}

=> this did create a “second” field, which should not have happened.

Notes:

  1. I tried replicating on my local with docker images … and it’s working fine, whatever the version !
    Full example for version 2.13:

export PORT=9202
docker run -it -p ${PORT}:9200 -p 9602:9600 -e "discovery.type=single-node" -e OPENSEARCH_INITIAL_ADMIN_PASSWORD='6rrc?244aZRSDFEm8oX=tzNmh1?%' --name opensearch-node2.13 -d opensearchproject/opensearch:2.13.0

curl -X PUT -H 'content-type: application/json; charset=UTF-8' -ku admin:'6rrc?244aZRSDFEm8oX=tzNmh1?%' "https://localhost:${PORT}/_index_template/standardtemplate" --data '{
   "index_patterns":[
      "mypattern*"
   ],
   "priority": 100,
   "template": {
     "settings":{
        "index":{
           "mapping":{
              "total_fields":{
                 "limit":"2000"
              }
           }
        }
     },
     "mappings":{
        "dynamic": false,
        "properties": {
          "first": {
            "type": "text",
            "fields": {
              "keyword":{
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
     }
  }
}'

curl -X POST -H 'content-type: application/json; charset=UTF-8' -ku admin:'6rrc?244aZRSDFEm8oX=tzNmh1?%' "https://localhost:${PORT}/mypattern1/_doc" --data '{
   "first": "not_first",
   "second": "second"
}'

curl -X GET -H 'content-type: application/json; charset=UTF-8' -ku admin:'6rrc?244aZRSDFEm8oX=tzNmh1?%' "https://localhost:${PORT}/mypattern1/_mapping" | jq

This is weird .
2) I remember (?) setting this up for AWS opensearch 2.3 and it was working fine. We now upgraded to 2.9, and it stopped working …

4

@Jerome I don’t have access to managed AWS OpenSearch. Maybe there is a general setting that overrides index dynamic mapping. I would suggest contacting AWS support.