Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch: 2.5
Dashboard: 2.5
Ubuntu 22.04 LTS, OpenSearch running in docker
Describe the issue:
I created a rollup job. The rollup index gets created. When I try to use the index while creating a visualization, I see a null pointer exception in the logs (pasted below)
I don’t know if it’s related, but I also see this in the log output:
[2023-03-01T18:54:32,638][WARN ][o.o.i.u.JobSchedulerUtils] [opensearch] Could not release lock for job rp-metrics-rollup-1h
Configuration:
filebeat → logstash → opensearch. I can visualize/use the discover tools on the source data, I just can’t create a visualization of the rolled-up data.
Relevant Logs or Screenshots:
[2023-03-01T18:56:40,363][WARN ][r.suppressed ] [opensearch] path: /rp-metrics-rollup-1h/_search, params: {ignore_unavailable=true, preference=1677691632613, index=rp-metrics-rollup-1h, timeout=30000ms, track_total_hits=true}
org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:663) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:372) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:698) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:471) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:294) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:104) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:74) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:753) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.transport.TransportService$6.handleException(TransportService.java:794) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleException(SecurityInterceptor.java:312) [opensearch-security-2.5.0.0.jar:2.5.0.0]
at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1414) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1528) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1502) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.transport.TransportService.sendLocalRequest(TransportService.java:1026) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.transport.TransportService$3.sendRequest(TransportService.java:152) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:928) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.security.transport.SecurityInterceptor.sendRequestDecorate(SecurityInterceptor.java:206) [opensearch-security-2.5.0.0.jar:2.5.0.0]
at org.opensearch.security.OpenSearchSecurityPlugin$7$2.sendRequest(OpenSearchSecurityPlugin.java:673) [opensearch-security-2.5.0.0.jar:2.5.0.0]
at org.opensearch.transport.TransportService.sendRequest(TransportService.java:815) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.transport.TransportService.sendChildRequest(TransportService.java:890) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.transport.TransportService.sendChildRequest(TransportService.java:878) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:248) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:133) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:281) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:324) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:252) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:427) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:218) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1006) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:769) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.search.TransportSearchAction.lambda$executeRequest$3(TransportSearchAction.java:410) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:138) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:103) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:499) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:284) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:125) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.5.0.0.jar:2.5.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.5.0.0.jar:2.5.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:317) [opensearch-security-2.5.0.0.jar:2.5.0.0]
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.5.0.0.jar:2.5.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.rest.action.RestCancellableNodeClient.doExecute(RestCancellableNodeClient.java:106) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:461) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.rest.action.search.RestSearchAction.lambda$prepareRequest$2(RestSearchAction.java:135) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:125) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.security.filter.SecurityRestFilter$1.handleRequest(SecurityRestFilter.java:127) [opensearch-security-2.5.0.0.jar:2.5.0.0]
at org.opensearch.rest.RestController.dispatchRequest(RestController.java:312) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:398) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.rest.RestController.dispatchRequest(RestController.java:241) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [opensearch-security-2.5.0.0.jar:2.5.0.0]
at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:366) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:445) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:356) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:55) [transport-netty4-client-2.5.0.jar:2.5.0]
at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:41) [transport-netty4-client-2.5.0.jar:2.5.0]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:71) [transport-netty4-client-2.5.0.jar:2.5.0]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.86.Final.jar:4.1.86.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373) [netty-handler-4.1.86.Final.jar:4.1.86.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) [netty-handler-4.1.86.Final.jar:4.1.86.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) [netty-handler-4.1.86.Final.jar:4.1.86.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) [netty-codec-4.1.86.Final.jar:4.1.86.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) [netty-codec-4.1.86.Final.jar:4.1.86.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.86.Final.jar:4.1.86.Final]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.opensearch.OpenSearchException$1
at org.opensearch.OpenSearchException.guessRootCauses(OpenSearchException.java:686) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:370) [opensearch-2.5.0.jar:2.5.0]
… 114 more
Caused by: java.lang.NullPointerException
at org.opensearch.indexmanagement.util.IndexUtils$Companion.isConcreteIndex(IndexUtils.kt:234) ~[?:?]
at org.opensearch.indexmanagement.util.IndexUtils$Companion.getConcreteIndex(IndexUtils.kt:245) ~[?:?]
at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:95) ~[?:?]
at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:43) ~[?:?]
at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.transport.TransportService.sendLocalRequest(TransportService.java:990) ~[opensearch-2.5.0.jar:2.5.0]
… 102 more