Node crashes with StackOverflowError

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch 1.2.4

Describe the issue:
Node crashes with StackOverflowError. This error seems to be happening when parsing a search query.

Configuration:
Default

Relevant Logs or Screenshots:

2023-11-30T16:31:40,668 worker][T#1] [W] ope.htt.AbstractHttpServerTransport - [UID=] - caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.200.11.101:50140, remoteAddress=/10.100.4.101:13072}
java.lang.Exception: java.lang.StackOverflowError
	at org.opensearch.http.netty4.Netty4HttpRequestHandler.exceptionCaught(Netty4HttpRequestHandler.java:69) [transport-netty4-client-1.2.4.jar:1.2.4]
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:381) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:71) [transport-netty4-client-1.2.4.jar:1.2.4]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.72.Final.jar:4.1.72.Final]
	at java.lang.Thread.run(Thread.java:887) [?:?]
Caused by: java.lang.StackOverflowError
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:831) ~[jackson-core-2.12.5.jar:2.12.5]
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:762) ~[jackson-core-2.12.5.jar:2.12.5]
	at org.opensearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:64) ~[opensearch-x-content-1.2.4.jar:1.2.4]
	at org.opensearch.common.xcontent.AbstractObjectParser.parseArray(AbstractObjectParser.java:439) ~[opensearch-x-content-1.2.4.jar:1.2.4]
	at org.opensearch.common.xcontent.AbstractObjectParser.lambda$declareObjectArrayOrNull$16(AbstractObjectParser.java:326) ~[opensearch-x-content-1.2.4.jar:1.2.4]
	at org.opensearch.common.xcontent.ObjectParser.lambda$declareField$9(ObjectParser.java:414) ~[opensearch-x-content-1.2.4.jar:1.2.4]
	at org.opensearch.common.xcontent.ObjectParser.parseValue(ObjectParser.java:582) ~[opensearch-x-content-1.2.4.jar:1.2.4]
	at org.opensearch.common.xcontent.ObjectParser.parseArray(ObjectParser.java:576) ~[opensearch-x-content-1.2.4.jar:1.2.4]
	at org.opensearch.common.xcontent.ObjectParser.parseSub(ObjectParser.java:611) ~[opensearch-x-content-1.2.4.jar:1.2.4]
	at org.opensearch.common.xcontent.ObjectParser.parse(ObjectParser.java:352) ~[opensearch-x-content-1.2.4.jar:1.2.4]
	at org.opensearch.common.xcontent.ObjectParser.parse(ObjectParser.java:284) ~[opensearch-x-content-1.2.4.jar:1.2.4]
	at org.opensearch.index.query.BoolQueryBuilder.fromXContent(BoolQueryBuilder.java:320) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.search.SearchModule.lambda$registerQuery$17(SearchModule.java:1273) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.common.xcontent.NamedXContentRegistry.parseNamedObject(NamedXContentRegistry.java:166) ~[opensearch-x-content-1.2.4.jar:1.2.4]
	at org.opensearch.common.xcontent.support.AbstractXContentParser.namedObject(AbstractXContentParser.java:435) ~[opensearch-x-content-1.2.4.jar:1.2.4]
	at org.opensearch.index.query.AbstractQueryBuilder.parseInnerQueryBuilder(AbstractQueryBuilder.java:331) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.index.query.BoolQueryBuilder.lambda$static$3(BoolQueryBuilder.java:295) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.common.xcontent.AbstractObjectParser.lambda$declareObjectArrayOrNull$15(AbstractObjectParser.java:326) ~[opensearch-x-content-1.2.4.jar:1.2.4]
	at org.opensearch.common.xcontent.AbstractObjectParser.parseArray(AbstractObjectParser.java:443) ~[opensearch-x-content-1.2.4.jar:1.2.4]
	at org.opensearch.common.xcontent.AbstractObjectParser.lambda$declareObjectArrayOrNull$16(AbstractObjectParser.java:326) ~[opensearch-x-content-1.2.4.jar:1.2.4]
	at org.opensearch.common.xcontent.ObjectParser.lambda$declareField$9(ObjectParser.java:414) ~[opensearch-x-content-1.2.4.jar:1.2.4]
	at org.opensearch.common.xcontent.ObjectParser.parseValue(ObjectParser.java:582) ~[opensearch-x-content-1.2.4.jar:1.2.4]
	at org.opensearch.common.xcontent.ObjectParser.parseArray(ObjectParser.java:576) ~[opensearch-x-content-1.2.4.jar:1.2.4]
	at org.opensearch.common.xcontent.ObjectParser.parseSub(ObjectParser.java:611) ~[opensearch-x-content-1.2.4.jar:1.2.4]
	at org.opensearch.common.xcontent.ObjectParser.parse(ObjectParser.java:352) ~[opensearch-x-content-1.2.4.jar:1.2.4]
	at org.opensearch.common.xcontent.ObjectParser.parse(ObjectParser.java:284) ~[opensearch-x-content-1.2.4.jar:1.2.4]
	at org.opensearch.index.query.BoolQueryBuilder.fromXContent(BoolQueryBuilder.java:320) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.search.SearchModule.lambda$registerQuery$17(SearchModule.java:1273) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.common.xcontent.NamedXContentRegistry.parseNamedObject(NamedXContentRegistry.java:166) ~[opensearch-x-content-1.2.4.jar:1.2.4]
	at org.opensearch.common.xcontent.support.AbstractXContentParser.namedObject(AbstractXContentParser.java:435) ~[opensearch-x-content-1.2.4.jar:1.2.4]
	at org.opensearch.index.query.AbstractQueryBuilder.parseInnerQueryBuilder(AbstractQueryBuilder.java:331) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.index.query.BoolQueryBuilder.lambda$static$3(BoolQueryBuilder.java:295) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.common.xcontent.AbstractObjectParser.lambda$declareObjectArrayOrNull$15(AbstractObjectParser.java:326) ~[opensearch-x-content-1.2.4.jar:1.2.4]
	at org.opensearch.common.xcontent.AbstractObjectParser.parseArray(AbstractObjectParser.java:443) ~[opensearch-x-content-1.2.4.jar:1.2.4]

	<<<stack repeated another 66 times more>>>

Questions:

  1. is there a way to identify what query caused this issue?
  2. shouldn’t the stackoverflow be handled gracefully and not crash the node?
  3. any workarounds other than increasing the stack size using -Xss JVM option to increase stack size?
2

@ronniepg the 1.2.x release branch is not supported for the long time, do you have an way to update to 1.3.x or, alternatively, check if this exception is reproducible in latest 1.3.x releases? Thank you.

3

Hi @reta , thanks for your response!
Unfortunately this issue occurred only once in a specific production cluster and hence not easily reproducible.

We believe this issue was caused by some deeply nested boolean query. Is there some specific debug/trace logging which can be enabled to print the search queries? once the query can be identified, the issue could possible be reproducible in test environments as well.

Thanks!