Truncate documents when a document exceeds index.mapping.depth.limit

OpenSearch
, ,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Amazon Managed Opensearch domain, OpenSearch 2.11 (latest)
Service Software Version OpenSearch_2_11_R20231113-P2

Describe the issue:
I’m looking for documentation on the mapping template settings such as index.mapping.total_fields.limit and index.mapping.depth.limit
Searching for mapping in the documentation doesn’t yield any useful results documenting these configuration options and how they work.
Ideally I’d love to find something similar to what is here:
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-settings-limit.html#:~:text=The%20maximum%20depth%20for%20a,Default%20is%2020%20.

Ultimately, my goal is to resolve this issue I’m getting when some logs I’m ingesting cause the following:
“The depth of the field has exceeded the allowed limit of [10]. This limit can be set by changing the [index.mapping.depth.limit] index level setting.”

I’m wondering if there may be a way to still ingest the document but truncate it in some way such that fields beyond the depth limit are not mapped/ignored.

Configuration:

"settings": {
        "index.mapping.total_fields.limit": 6000,
        "index.mapping.depth.limit": 10,
		"number_of_shards": 1,
		"number_of_replicas": 1
    }
2

There’s no setting can control what happens when the depth of a document exceeds the limit, maybe we can try to use the ingest pipeline to remove the subfields which trigger the limitation, for example, define a script processor, check the depth of the incoming document, remove the subfields if the depth is greater than the limit, this method is not friendly because it would reduce the indexing performance and is a little bit complex. Maybe add a new setting something like index.mapping.depth.ignore_fields_beyond_limit to control the behavior is better, would you mind creating a feature request in the github repo: Issues · opensearch-project/OpenSearch · GitHub?

1 Like
3

Thanks @gaobinlong, I’ve raised the feature request in the project here: [Feature Request] Add an index setting to ignore field mapping for documents with fields beyond the index.mapping.depth.limit · Issue #13089 · opensearch-project/OpenSearch · GitHub