[Q] Filter too complex / Shards failed

General Feedback
1

I’m new to this forum (& Open Search in general), so please go easy on me :slight_smile:

[Problem]
I’m trying to sort for some IPs but the Query DSL filter i normally use doesn’t work with this quantity of IPs (ca. 16000)

{
  "query": {
    "terms": {
      "ip.keyword": [
          "1.1.1.1",
          "1.1.1.2"
          etc...
      ]
    }
  }
}

[Question]
Are there any workarounds to this problem?
maybe a way to divide the filter into smaller bits?

[Error messages]

  • too_complex_to_determinize_exception at shard 0
  • Type: too_complex_to_determinize_exception
  • Reason: Determinizing automaton with 212586 states and 212585 transitions would require more than 10000 effort.
  • Shards Failed: 1
  • Shards Skipped: 14
  • Shards Total: 15

any help will be appreciated, thanks :slight_smile:

2

Hey @Sebastian

Using this documentation here
Example:

GET winlogbeat-2023.07.12/_search
{
    "query": {
      "term": {
        "host.ip": "192.168.1.100,192.168.1.200"
     }
    }
  }
3

I’m confused, I don’t think the terms query uses automatons, this is used more for wildcard, fuzzy or regex queries. For terms there’s a index.max_terms_count that defaults to 65K which should be enough for what you need.

Are you sure that the error message that you’re describing (can you paste it here?) corresponds to this query and not a different one?

4

Hi Radu, here is an image of the “show details” window

Screenshot 2023-07-17 at 15.43.09
Screenshot 2023-07-17 at 15.43.091498×1348 152 KB

5

does this help?
Thanks.

6

Can you somehow get the JSON request and response as well? I assume those two would be in the two other tabs?