I have more than 10K values for this document in my address_index.
Note: Query works If I use another document ID where I have less than 1024 values.
Configuration:
Relevant Logs or Screenshots:
"reason": {
"type": "query_shard_exception",
"reason": "failed to create query: maxClauseCount is set to 1024",
"index": "address_index",
"index_uuid": "4oi75d-kk45pgpEPvA",
"caused_by": {
"type": "too_many_clauses",
"reason": "maxClauseCount is set to 1024"
}
}
Can you run _analyze API if itâs possible to modify indices.query.bool.max_clause_count from the default(1024) to 2048 or above?
You must be careful that this setting will have an impact on memory usage in clusters so itâs just for checking how many clauses are created by the query you ran. After you test, nowâs the time to determine the adequate number for max_clause_count.
As you know, OpenSearchâs _search queries are always rewritten to Luceneâs BooleanQuery internally.
I am migrating from elasticSearch 7.10 and this same query works there without changing default value of indices.query.bool.max_clause_count. I am really not sure to modify max_clause_count because my document could have any number of values. I am trying to understand why am I getting this error with openSearch?
Posting it in more readable view. Wondering why TermsQuery wasnât rewritten to bitset. Itâs worth an issue for sure.
"type": "too_many_clauses",
"reason": "too_many_clauses: maxClauseCount is set to 1024",
"stack_trace": "NotSerializableExceptionWrapper[too_many_clauses: maxClauseCount is set to 1024]
at org.apache.lucene.search.BooleanQuery$Builder.add(BooleanQuery.java:115)
at org.apache.lucene.search.BooleanQuery$Builder.add(BooleanQuery.java:129)
at org.opensearch.index.mapper.MappedFieldType.termsQuery(MappedFieldType.java:237)
at org.opensearch.index.mapper.IpFieldMapper$IpFieldType.termsQuery(IpFieldMapper.java:257)
at org.opensearch.index.query.TermsQueryBuilder.doToQuery(TermsQueryBuilder.java:476)
at org.opensearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:117)
at org.opensearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:346)
at org.opensearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:328)
at org.opensearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:117)
at org.opensearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:346)
at org.opensearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:326)
at org.opensearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:117)
at org.opensearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:346)
at org.opensearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:329)
at org.opensearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:117)
at org.opensearch.index.query.QueryShardContext.lambda$toQuery$3(QueryShardContext.java:467)
at org.opensearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:479)
at org.opensearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:466)
at org.opensearch.search.SearchService.parseSource(SearchService.java:1238)
at org.opensearch.search.SearchService.createContext(SearchService.java:985)
at org.opensearch.search.SearchService.executeQueryPhase(SearchService.java:593)
at org.opensearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:566)
at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74)
at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89)
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78)
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59)
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908)
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.lang.Thread.run(Thread.java:833)
"
}```
here IpFieldMapper it falls back to MappedFieldType which is prone to maxClauses error. Presumably if you avoid slashes in those IPs it should work.
UPD Do you have an idea how many concrete IPs and mask/s you have in those lists?
What can work is if you have a separate fields for concrete IPs and mask/s - in this case concrete IPs isnât limited by max_caluses, but mask/s still encounter this limitation. Then you can bool/should two term lookups. Itâs not ideal workaround but this may work.