# Keyword field unable to be used for aggregation

**URL:** <https://forum.opensearch.org/t/keyword-field-unable-to-be-used-for-aggregation/14700>\
**Category:** OpenSearch\
**Tags:** troubleshoot\
**Created:** [June 15, 2023, 5:58am UTC](https://forum.opensearch.org/t/keyword-field-unable-to-be-used-for-aggregation/14700 "2023-06-15T05:58:19Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Overhear1431](https://avatars.discourse-cdn.com/v4/letter/o/ce7236/32.png) [@Overhear1431](https://forum.opensearch.org/u/Overhear1431)\
**Post date:** [June 15, 2023, 5:58am UTC](https://forum.opensearch.org/t/keyword-field-unable-to-be-used-for-aggregation/14700/1 "2023-06-15T05:58:19Z")

</div>

**Versions** :  
v 1.0.0  
Amazon OpenSearch Service  
Managed Cluster

**Describe the issue** :  
I currently have a search that fires every five minutes via an AWS lambda that makes a request to the \_search endpoint for my index. This search has been working successfully for four months, but suddenly today it has started giving me the following error:

`Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [webaclId] in order to load field data by uninverting the inverted index. Note that this can use significant memory.`

This error only happens when I run the search via my lambda. When I run the same search using the dev tools in the dashboard it runs just fine.

As best as I can tell, I’ve configured webaclId as a keyword field, not a text field, as per my mapping shared below, so I can’t figure out the issue. Has anyone else experienced this or found a solution?

The search is as follows:

```auto
{
  "query": {
    "constant_score": {
      "filter": {
        "bool": {
          "must": [
            {
              "wildcard": {
                "rateBasedRuleList.limitKey": "*"
              }
            },
            {
              "match_phrase": {
                "nonTerminatingMatchingRules.action": "COUNT"
              }
            },
            {
              "match_phrase": {
                "terminatingRuleId": "Default_Action"
              }
            },
            {
              "range": {
                "timestamp": {
                  "gte": "now-1h"
                }
              }
            }
          ]
        }
      }
    }
  },
  "aggs": {
    "webaclId_count": {
      "terms": {
        "field": "webaclId"
      },
      "aggs": {
        "single_webaclId_count": {
          "value_count": {
            "field": "webaclId"
          }
        },
        "limit": {
          "bucket_selector": {
            "buckets_path": {
              "count": "single_webaclId_count"
            },
            "script": "params.count > 100"
          }
        }
      }
    }
  }
}

```

The relevant section of the mapping template is as follows:

```auto
{
  "myindex" : {
    "mappings" : {
      "properties" : {
        "nonTerminatingMatchingRules" : {
          "properties" : {
            "action" : {
              "type" : "keyword"
            },
            "ruleId" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "ruleMatchDetails" : {
              "properties" : {
                "conditionType" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "location" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "matchedData" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "sensitivityLevel" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            }
          }
        },
        "rateBasedRuleList" : {
          "properties" : {
            "limitKey" : {
              "type" : "keyword"
            },
            "limitValue" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "maxRateAllowed" : {
              "type" : "long"
            },
            "rateBasedRuleId" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "rateBasedRuleName" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "terminatingRuleId" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "timestamp" : {
          "type" : "date"
        },
        "webaclId" : {
          "type" : "keyword"
        }
      }
    }
  }
}

```

**Full error** :

```auto
{
    "error": {
        "root_cause": [
            {
                "type": "illegal_argument_exception",
                "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [webaclId] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
            }
        ],
        "type": "search_phase_execution_exception",
        "reason": "",
        "phase": "fetch",
        "grouped": true,
        "failed_shards": [
            {
                "shard": 0,
                "index": "redacted",
                "node": "redacted",
                "reason": {
                    "type": "illegal_argument_exception",
                    "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [webaclId] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
                }
            }
        ],
        "caused_by": {
            "type": "null_pointer_exception",
            "reason": null
        }
    },
    "status": 400
}

```

---

<div class="post-metadata">

**Author:** ![gaobinlong](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/gaobinlong/32/5242_2.png) [@gaobinlong](https://forum.opensearch.org/u/gaobinlong)\
**Post date:** [June 15, 2023, 7:47am UTC](https://forum.opensearch.org/t/keyword-field-unable-to-be-used-for-aggregation/14700/2 "2023-06-15T07:47:44Z")

</div>

Maybe not all of the indices you search against have the same mapping, you can check whether the field `webaclId ` in all of the indices are `keyword` type or not.

---

<div class="post-metadata">

**Author:** ![Overhear1431](https://avatars.discourse-cdn.com/v4/letter/o/ce7236/32.png) [@Overhear1431](https://forum.opensearch.org/u/Overhear1431)\
**Post date:** [June 16, 2023, 8:51am UTC](https://forum.opensearch.org/t/keyword-field-unable-to-be-used-for-aggregation/14700/3 "2023-06-16T08:51:18Z")

</div>

Thank you for the good suggestion.

I went to have a look at the index pattern and found this. Based on “Searchable” and “Aggregatable” being green, I assume this is correctly a keyword across the whole index pattern that I am searching.

 ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/6/6336bb425b74bdc2824b818d533463affacda193.png)

If it were different in some of the indexes I was searching I imagine I would see a conflict for this field.

---

<div class="post-metadata">

**Author:** ![Overhear1431](https://avatars.discourse-cdn.com/v4/letter/o/ce7236/32.png) [@Overhear1431](https://forum.opensearch.org/u/Overhear1431)\
**Post date:** [June 16, 2023, 8:52am UTC](https://forum.opensearch.org/t/keyword-field-unable-to-be-used-for-aggregation/14700/4 "2023-06-16T08:52:02Z")

</div>

Further screenshot of the settings when I go to edit the field

 ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/0/09fbca712a3e2472fbf22c3e27959ba97f0485da.png)

---

<div class="post-metadata">

**Author:** ![radu.gheorghe](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/radu.gheorghe/32/3516_2.png) [@radu.gheorghe](https://forum.opensearch.org/u/radu.gheorghe)\
**Post date:** [June 17, 2023, 5:03pm UTC](https://forum.opensearch.org/t/keyword-field-unable-to-be-used-for-aggregation/14700/5 "2023-06-17T17:03:53Z")

</div>

You can see the mapping in Dev Tools as well, something like:

```auto
GET /index-pattern*/_mapping

```

You should see a response with all the indices matching your pattern. Search for your field name there and see which one is a `text` field, because my assumption is that your lambda function hits one of these fields.

If there’s nothing in there, I’d also look for the same field name in other indices (e.g. by getting all the mappings via `GET /_mapping`

---

<div class="post-metadata">

**Author:** ![Overhear1431](https://avatars.discourse-cdn.com/v4/letter/o/ce7236/32.png) [@Overhear1431](https://forum.opensearch.org/u/Overhear1431)\
**Post date:** [June 19, 2023, 4:57am UTC](https://forum.opensearch.org/t/keyword-field-unable-to-be-used-for-aggregation/14700/6 "2023-06-19T04:57:00Z")

</div>

> You can see the mapping in Dev Tools as well, something like:  
> GET /index-pattern\*/\_mapping  
> You should see a response with all the indices matching your pattern. Search for your field name there and see which one is a `text` field, because my assumption is that your lambda function hits one of these fields.

Using this method I found some instances of the following on some really old indexes:

```auto
"webaclId" : {
  "type" : "text",
  "fields" : {
    "keyword" : {
      "type" : "keyword",
      "ignore_above" : 256
    }
  }
}

```

I assume the solution then would be to reindex these older indexes to match the mapping below that I’m expecting/need for my query?

```auto
"webaclId" : {
  "type" : "keyword"
}

```

---

<div class="post-metadata">

**Author:** ![radu.gheorghe](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/radu.gheorghe/32/3516_2.png) [@radu.gheorghe](https://forum.opensearch.org/u/radu.gheorghe)\
**Post date:** [June 29, 2023, 8:55am UTC](https://forum.opensearch.org/t/keyword-field-unable-to-be-used-for-aggregation/14700/7 "2023-06-29T08:55:59Z")

</div>

That’s correct (sorry for the late reply).

---

<div class="post-metadata">

**Author:** ![Overhear1431](https://avatars.discourse-cdn.com/v4/letter/o/ce7236/32.png) [@Overhear1431](https://forum.opensearch.org/u/Overhear1431)\
**Post date:** [October 10, 2023, 7:51am UTC](https://forum.opensearch.org/t/keyword-field-unable-to-be-used-for-aggregation/14700/8 "2023-10-10T07:51:52Z")

</div>

Is it possible to perform a bulk reindex/mapping update for a number of old indexes?

---

<div class="post-metadata">

**Author:** ![radu.gheorghe](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/radu.gheorghe/32/3516_2.png) [@radu.gheorghe](https://forum.opensearch.org/u/radu.gheorghe)\
**Post date:** [October 10, 2023, 8:12am UTC](https://forum.opensearch.org/t/keyword-field-unable-to-be-used-for-aggregation/14700/9 "2023-10-10T08:12:26Z")

</div>

Yes, I think you can hit an index pattern as a source of your reindex API call, and then the destination can be an alias. If an alias rotates automatically (because it’s managed by ISM), then you should be good.
