Set max_analyzer_offset or disable highlighting

OpenSearch Dashboards
1

I’m on OpenSearch AWS hosted version 2.15.

I have some documents with large text fields. Sometimes when I query through Dashboards, it fails with an error like “The length of [log] field of [0] doc of […] index has exceeded [1000000] - maximum allowed to be analyzed for highlighting. This maximum can be set by changing the [index.highlight.max_analyzed_offset] index level setting. For large texts, indexing with offsets or term vectors is recommended!”

I don’t really need highlighting; I’d rather disable it entirely instead of getting an error and no results.

I found this in the docs that sounds like what I want: Highlight query matches - OpenSearch Documentation

The maximum number of characters that will be analyzed for a highlight request is defined by index.highlight.max_analyzed_offset. When this limit is reached, an error is returned. Set the max_analyzer_offset to a lower value than index.highlight.max_analyzed_offset to avoid the error.

And this PR Adds a new parameter, max_analyzer_offset, for the highlighter by hauck-jvsh · Pull Request #3893 · opensearch-project/OpenSearch · GitHub from 2022 seems like it is implemented:

Adds a new parameter, max_analyzer_offset, for the highlighter
This new parameter can be used to avoid aborting when highlighting large fields, or just to speedup the highlight doing it only on the beginning of the field.

But where do I set it? I have tried

  • AWS console: doesn’t exist under cluster settings
  • AWS CLI: aws opensearch update-domain-config with --advanced-options '{"index.highlight.max_analyzer_offset": "1000"}' returns Unrecognized advanced option
  • DevTools: logged in as opensearch (superuser) said 401 unauthorized
PUT _cluster/settings
{
  "persistent": {
    "search.highlight.max_analyzed_offset": 1000
  }
}

I tried at the index level:

PUT /index-name-001033/_settings
{
  "index": {
    "highlight.max_analyzed_offset": 1000
  }
}

but I still saw the error when trying to query from Dashboards, even when using only a filter (no full text terms).

How can I disable highlighting entirely and/or set max_analyzer_offset to a small value so that my Dashboards queries won’t crash?

2

I wasn’t able to determine much for certain, but it looks like there’s some ambiguity surrounding whether it’s set_max_analyzer_offset or set_max_analyzed_offset - it looks like you’re setting analyzer - try again with analyzed to see if it gets you anywhere?

A shot in the dark for ya. Hope it helps. :slight_smile: