I have more than 10K values for this document in my address_index.
Note: Query works If I use another document ID where I have less than 1024 values.
Configuration:
Relevant Logs or Screenshots:
"reason": {
"type": "query_shard_exception",
"reason": "failed to create query: maxClauseCount is set to 1024",
"index": "address_index",
"index_uuid": "4oi75d-kk45pgpEPvA",
"caused_by": {
"type": "too_many_clauses",
"reason": "maxClauseCount is set to 1024"
}
}
Can you run _analyze API if it’s possible to modify indices.query.bool.max_clause_count from the default(1024) to 2048 or above?
You must be careful that this setting will have an impact on memory usage in clusters so it’s just for checking how many clauses are created by the query you ran. After you test, now’s the time to determine the adequate number for max_clause_count.
As you know, OpenSearch’s _search queries are always rewritten to Lucene’s BooleanQuery internally.
I am migrating from elasticSearch 7.10 and this same query works there without changing default value of indices.query.bool.max_clause_count. I am really not sure to modify max_clause_count because my document could have any number of values. I am trying to understand why am I getting this error with openSearch?
Posting it in more readable view. Wondering why TermsQuery wasn’t rewritten to bitset. It’s worth an issue for sure.
"type": "too_many_clauses",
"reason": "too_many_clauses: maxClauseCount is set to 1024",
"stack_trace": "NotSerializableExceptionWrapper[too_many_clauses: maxClauseCount is set to 1024]
at org.apache.lucene.search.BooleanQuery$Builder.add(BooleanQuery.java:115)
at org.apache.lucene.search.BooleanQuery$Builder.add(BooleanQuery.java:129)
at org.opensearch.index.mapper.MappedFieldType.termsQuery(MappedFieldType.java:237)
at org.opensearch.index.mapper.IpFieldMapper$IpFieldType.termsQuery(IpFieldMapper.java:257)
at org.opensearch.index.query.TermsQueryBuilder.doToQuery(TermsQueryBuilder.java:476)
at org.opensearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:117)
at org.opensearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:346)
at org.opensearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:328)
at org.opensearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:117)
at org.opensearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:346)
at org.opensearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:326)
at org.opensearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:117)
at org.opensearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:346)
at org.opensearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:329)
at org.opensearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:117)
at org.opensearch.index.query.QueryShardContext.lambda$toQuery$3(QueryShardContext.java:467)
at org.opensearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:479)
at org.opensearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:466)
at org.opensearch.search.SearchService.parseSource(SearchService.java:1238)
at org.opensearch.search.SearchService.createContext(SearchService.java:985)
at org.opensearch.search.SearchService.executeQueryPhase(SearchService.java:593)
at org.opensearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:566)
at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74)
at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89)
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78)
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59)
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908)
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.lang.Thread.run(Thread.java:833)
"
}```
here IpFieldMapper it falls back to MappedFieldType which is prone to maxClauses error. Presumably if you avoid slashes in those IPs it should work.
UPD Do you have an idea how many concrete IPs and mask/s you have in those lists?
What can work is if you have a separate fields for concrete IPs and mask/s - in this case concrete IPs isn’t limited by max_caluses, but mask/s still encounter this limitation. Then you can bool/should two term lookups. It’s not ideal workaround but this may work.