Occurs intermittently this error o.o.OpenSearchStatusException: OpenSearch exception

slowup · January 11, 2024, 8:07am

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.9.0

Describe the issue:

||o.o.OpenSearchStatusException: OpenSearch exception [type=security_exception, reason=Unexpected exception indices:data/read/search]|
|---|---|
||at o.o.r.BytesRestResponse.errorFromXContent(BytesRestResponse.java:207)|
||at o.o.c.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2233)|
||at o.o.c.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2210)|
||at o.o.c.RestHighLevelClient$1.onFailure(RestHighLevelClient.java:2121)|
||at i.o.j.i.o.r.RestResponseListener.onFailure(RestResponseListener.java:47)|
||at o.o.c.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:707)|
||at o.o.c.RestClient$1.completed(RestClient.java:437)|
||at o.o.c.RestClient$1.completed(RestClient.java:421)|
||at i.o.j.i.a.ApacheHttpAsyncClientInstrumentation$WrappedFutureCallback.completeDelegate(ApacheHttpAsyncClientInstrumentation.java:234)|
||at i.o.j.i.a.ApacheHttpAsyncClientInstrumentation$WrappedFutureCallback.completed(ApacheHttpAsyncClientInstrumentation.java:183)|

This error occurs sometimes
The index being accessed is an index that has no problems with both read and write.
Is there anything I should check?

pablo · January 11, 2024, 9:18am

@slowup Have you noticed any preceding errors in the logs?

slowup · February 18, 2024, 12:13pm

hi!

error log!

[2024-02-18T11:34:53,029][INFO ][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [flex-prod-master-0] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_1: Next refresh cycle for metadata provider 'www.example.com/sso/saml/metadata' will occur on '2024-02-18T14:34:52.488Z' ('2024-02-18T14:34:52.488Z' local time)
[2024-02-18T11:35:05,520][WARN ][c.a.d.a.h.s.Saml2SettingsProvider] [flex-prod-master-0] The IdP does not provide a Single Logout Service. In order to ensure that users have to re-enter their password after logging out, OpenSearch Security will issue all SAML authentication requests with a mandatory password input (ForceAuthn=true)


[2024-02-18T11:35:05,520][WARN ][o.o.s.a.BackendRegistry  ] [flex-prod-master-0] No 'Authorization' header, send 401 and 'WWW-Authenticate Basic'


[2024-02-18T11:41:45,891][ERROR][o.o.s.f.SecurityFilter   ] [flex-prod-master-2] Unexpected exception java.util.ConcurrentModificationException
java.util.ConcurrentModificationException: null
	at java.util.HashMap$HashIterator.nextNode(HashMap.java:1597) ~[?:?]
	at java.util.HashMap$KeyIterator.next(HashMap.java:1620) ~[?:?]
	at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1054) ~[?:?]
	at com.google.common.collect.Sets$1$1.computeNext(Sets.java:745) ~[guava-32.1.1-jre.jar:?]
	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145) ~[guava-32.1.1-jre.jar:?]
	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140) ~[guava-32.1.1-jre.jar:?]
	at java.lang.String.join(String.java:3325) ~[?:?]
	at org.opensearch.security.privileges.PrivilegesEvaluator.setUserInfoInThreadContext(PrivilegesEvaluator.java:210) ~[opensearch-security-2.9.0.0.jar:2.9.0.0]
	at org.opensearch.security.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:263) ~[opensearch-security-2.9.0.0.jar:2.9.0.0]
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:377) [opensearch-security-2.9.0.0.jar:2.9.0.0]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) [opensearch-security-2.9.0.0.jar:2.9.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.rest.action.RestCancellableNodeClient.doExecute(RestCancellableNodeClient.java:106) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.rest.action.search.RestSearchAction.lambda$prepareRequest$2(RestSearchAction.java:135) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:127) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.security.filter.SecurityRestFilter$1.handleRequest(SecurityRestFilter.java:134) [opensearch-security-2.9.0.0.jar:2.9.0.0]
	at org.opensearch.rest.RestController.dispatchRequest(RestController.java:320) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:411) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.rest.RestController.dispatchRequest(RestController.java:249) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:366) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:445) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:356) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:55) [transport-netty4-client-2.9.0.jar:2.9.0]
	at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:41) [transport-netty4-client-2.9.0.jar:2.9.0]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:71) [transport-netty4-client-2.9.0.jar:2.9.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.94.Final.jar:4.1.94.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2024-02-18T11:41:45,893][WARN ][r.suppressed             ] [flex-prod-master-2] path: /pd-search/_search, params: {typed_keys=true, max_concurrent_shard_requests=5, ignore_unavailable=false, expand_wildcards=open, allow_no_indices=true, ignore_throttled=true, index=prod-search-user, request_cache=true, search_type=query_then_fetch, batched_reduce_size=512, ccs_minimize_roundtrips=true}
org.opensearch.OpenSearchSecurityException: Unexpected exception indices:data/read/search
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:474) [opensearch-security-2.9.0.0.jar:2.9.0.0]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) [opensearch-security-2.9.0.0.jar:2.9.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.rest.action.RestCancellableNodeClient.doExecute(RestCancellableNodeClient.java:106) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.rest.action.search.RestSearchAction.lambda$prepareRequest$2(RestSearchAction.java:135) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:127) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.security.filter.SecurityRestFilter$1.handleRequest(SecurityRestFilter.java:134) [opensearch-security-2.9.0.0.jar:2.9.0.0]
	at org.opensearch.rest.RestController.dispatchRequest(RestController.java:320) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:411) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.rest.RestController.dispatchRequest(RestController.java:249) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:366) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:445) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:356) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:55) [transport-netty4-client-2.9.0.jar:2.9.0]
	at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:41) [transport-netty4-client-2.9.0.jar:2.9.0]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:71) [transport-netty4-client-2.9.0.jar:2.9.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.94.Final.jar:4.1.94.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]

thank u to response!

pablo · February 29, 2024, 3:43pm

@slowup Could you have a look at the release notes of 2.10.0?

github.com

opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.10.0.md

OpenSearch and OpenSearch Dashboards 2.10.0 Release Notes

## Release Highlights

OpenSearch 2.10.0 includes a number of new features for search, security, and machine learning applications, as well as user interface improvements and new options for durable data storage. This release also includes new experimental tools to boost search performance and build conversational search applications.

### New Features

* Remote-backed storage is now generally available, offering a production-ready alternative to snapshots or replica copies for durable data backup.
* Segment replication is now fully integrated with remote-backed storage. This allows users to select remote-backed storage as the source for replication, with the potential to reduce compute resources on ingest.
* New hybrid query functionality and a built-in normalization processor offer a way to improve relevance for search results. Now, you can combine and normalize the relevance scores of lexical queries with natural language-based k-NN vector search queries within OpenSearch.
* You can now define custom log types with Security Analytics and use them as you would other log data to build detectors, create custom rules, provide additional mappings, and more.
* A new visual theme for OpenSearch Dashboards includes changes to typography, colors, and actions for light mode and dark mode designs, providing an updated user experience.
* Updates to the Discover tool in OpenSearch Dashboards include usability improvements designed to make the tool more intuitive and cohesive.
* A new IP2Geo processor accesses location data based on IP addresses using external databases, allowing OpenSearch to enrich data with up-to-date geographical location information.
* With  this release, OpenSearch Dashboards is upgraded to version 18 of Node.js, with backward-compatible support for Node.js versions 14 and 16. Users have the flexibility to choose from these versions to run the web application. As a result of this update, the Docker images for OpenSearch and OpenSearch Dashboards now use Amazon Linux 2023 as the base image, changed from Amazon Linux 2.

### Experimental Features

OpenSearch 2.10.0 includes the following experimental features. Experimental features are disabled by default. For instructions on how to enable them, refer to the documentation for the feature.

This file has been truncated. show original

And this bug.

github.com/opensearch-project/security

[BUG] java.util.ConcurrentModificationException with plugins.security.cache.ttl_minutes=0

opened 11:14AM - 16 Nov 22 UTC

closed 08:50PM - 07 Aug 23 UTC

olddanmer

bug triaged v2.10.0

**What is the bug?** With option `plugins.security.cache.ttl_minutes: 0` opense…arch sometimes raises java.util.ConcurrentModificationException error and logstash get 500 error at that moment. I use dashboards with OpenID integration, so option `plugins.security.cache.ttl_minutes: 0` is recommended by documentation. Without `plugins.security.cache.ttl_minutes: 0` dashboards doesn't work properly with OpenID (see bug https://github.com/opensearch-project/security-dashboards-plugin/issues/1138#issuecomment-1279635558) **How can one reproduce the bug?** Steps to reproduce the behavior: 1. Setup opensearch 2.3.0, logstash-oss 7.12.1 2. Set opensearch setting `plugins.security.cache.ttl_minutes=0` 3. logstash uses bulk write (?) 4. See errors in logs opensearch logs ``` Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: [2022-11-15T15:15:07,061][ERROR][o.o.s.f.SecurityFilter ] [es-logs-1.srvroom.labs.intellij.net] Unexpected exception java.util.ConcurrentModificationException Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: java.util.ConcurrentModificationException: null Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at java.util.HashMap$HashIterator.nextNode(HashMap.java:1597) ~[?:?] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at java.util.HashMap$KeyIterator.next(HashMap.java:1620) ~[?:?] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1054) ~[?:?] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at java.util.AbstractCollection.addAll(AbstractCollection.java:335) ~[?:?] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at java.util.HashSet.<init>(HashSet.java:121) ~[?:?] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.security.securityconf.ConfigModelV7$RoleMappingHolder.map(ConfigModelV7.java:1200) ~[opensearch-security-2.3.0.0.jar:2.3.0.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.security.securityconf.ConfigModelV7.mapSecurityRoles(ConfigModelV7.java:1268) ~[opensearch-security-2.3.0.0.jar:2.3.0.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.security.privileges.PrivilegesEvaluator.mapRoles(PrivilegesEvaluator.java:488) ~[opensearch-security-2.3.0.0.jar:2.3.0.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.security.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:220) ~[opensearch-security-2.3.0.0.jar:2.3.0.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:303) [opensearch-security-2.3.0.0.jar:2.3.0.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.3.0.0.jar:2.3.0.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:426) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.client.support.AbstractClient.bulk(AbstractClient.java:502) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.rest.action.document.RestBulkAction.lambda$prepareRequest$0(RestBulkAction.java:111) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:125) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.security.filter.SecurityRestFilter$1.handleRequest(SecurityRestFilter.java:127) [opensearch-security-2.3.0.0.jar:2.3.0.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.rest.RestController.dispatchRequest(RestController.java:312) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:398) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.rest.RestController.dispatchRequest(RestController.java:241) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [opensearch-security-2.3.0.0.jar:2.3.0.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:366) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:445) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:356) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:55) [transport-netty4-client-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:41) [transport-netty4-client-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:71) [transport-netty4-client-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) [netty-codec-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) [netty-codec-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373) [netty-handler-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) [netty-handler-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) [netty-handler-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) [netty-codec-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) [netty-codec-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) [netty-codec-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at java.lang.Thread.run(Thread.java:833) [?:?] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: [2022-11-15T15:15:07,067][WARN ][r.suppressed ] [es-logs-1.srvroom.domain.net] path: /_bulk, params: {} Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: org.opensearch.OpenSearchSecurityException: Unexpected exception indices:data/write/bulk Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:374) [opensearch-security-2.3.0.0.jar:2.3.0.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.3.0.0.jar:2.3.0.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:426) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.client.support.AbstractClient.bulk(AbstractClient.java:502) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.rest.action.document.RestBulkAction.lambda$prepareRequest$0(RestBulkAction.java:111) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:125) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.security.filter.SecurityRestFilter$1.handleRequest(SecurityRestFilter.java:127) [opensearch-security-2.3.0.0.jar:2.3.0.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.rest.RestController.dispatchRequest(RestController.java:312) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:398) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.rest.RestController.dispatchRequest(RestController.java:241) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [opensearch-security-2.3.0.0.jar:2.3.0.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:366) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:445) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:356) [opensearch-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:55) [transport-netty4-client-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:41) [transport-netty4-client-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:71) [transport-netty4-client-2.3.0.jar:2.3.0] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) [netty-codec-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) [netty-codec-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373) [netty-handler-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) [netty-handler-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) [netty-handler-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) [netty-codec-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) [netty-codec-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) [netty-codec-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.79.Final.jar:4.1.79.Final] Nov 15 15:15:07 srvroom-es-logs-1 docker[1142]: at java.lang.Thread.run(Thread.java:833) [?:?] ``` Logstash logs: ``` Nov 15 15:15:07 srvroom-es-logs-1 docker[6711]: [2022-11-15T15:15:07,069][ERROR][logstash.outputs.elasticsearch][main][1f047d8c251176835ae470d6323b5986e9ec9d34e8a425b30855ce135ee014a5] Encountered a retryable error. Will Retry with exponential backoff {:code=>500, :url=>"https://es-logs-1.srvroom.domain.net:9200/_bulk", :content_length=>79914} ``` **What is the expected behavior?** opensearch indexes data from logstash without errors **What is your host/environment?** - OS: Amazon Linux 2 - opensearch 2.3.0 - logstash-oss 7.12.1 - opensearch and logstash run in docker opensearch.yml ``` cluster.name: es-logs node.name: es-logs-1.srvroom.domain.net node.attr.node_type: hot node.master: true node.data: true network.bind_host: "0.0.0.0" network.publish_host: es-logs-1.srvroom.domain.net indices.memory.index_buffer_size: 24% indices.query.bool.max_clause_count: 2048 discovery.seed_hosts: ["es-logs-1.srvroom.domain.net","es-logs-2.srvroom.domain.net","es-logs-3.srvroom.domain.net","es-logs-4.srvroom.domain.net","es-logs-5.srvroom.domain.net","es-logs-warm-1.srvroom.domain.net"] path.data: /var/opensearch/data cluster.routing.allocation.disk.watermark.low: "30gb" cluster.routing.allocation.disk.watermark.high: "8gb" cluster.routing.allocation.disk.watermark.flood_stage: "4gb" plugins.security.ssl.transport.pemcert_filepath: es-logs-1.srvroom.domain.net.crt plugins.security.ssl.transport.pemkey_filepath: es-logs-1.srvroom.domain.net.key plugins.security.ssl.transport.pemtrustedcas_filepath: es-logs-1.srvroom.domain.net.crt plugins.security.ssl.transport.enforce_hostname_verification: false plugins.security.ssl.transport.enabled: true plugins.security.ssl.http.pemcert_filepath: es-logs-1.srvroom.domain.net.crt plugins.security.ssl.http.pemkey_filepath: es-logs-1.srvroom.domain.net.key plugins.security.ssl.http.pemtrustedcas_filepath: es-logs-1.srvroom.domain.net.crt plugins.security.ssl.http.enabled: true plugins.security.allow_unsafe_democertificates: false plugins.security.allow_default_init_securityindex: true plugins.security.authcz.admin_dn: - CN=es-logs-admin.srvroom.domain.net plugins.security.ssl_cert_reload_enabled: true plugins.security.nodes_dn: - "CN=es-logs-1.srvroom.domain.net" - "CN=es-logs-2.srvroom.domain.net" - "CN=es-logs-3.srvroom.domain.net" - "CN=es-logs-4.srvroom.domain.net" - "CN=es-logs-5.srvroom.domain.net" - "CN=es-logs-warm-1.srvroom.domain.net" plugins.security.enable_snapshot_restore_privilege: true plugins.security.check_snapshot_restore_write_privileges: true plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"] plugins.security.audit.type: internal_opensearch plugins.security.audit.config.disabled_rest_categories: NONE plugins.security.audit.config.disabled_transport_categories: NONE plugins.security.cache.ttl_minutes: 0 opendistro_security.audit.ignore_users: - kibanaserver - elastalert - logwriter - exporter compatibility.override_main_response_version: true ``` opensearch-security/config.yml ``` --- _meta: type: "config" config_version: 2 config: dynamic: http: anonymous_auth_enabled: true authc: basic_internal_auth_domain: http_enabled: true transport_enabled: false order: 0 http_authenticator: type: basic challenge: false authentication_backend: type: internal openid_auth_domain: http_enabled: true transport_enabled: false order: 1 http_authenticator: type: openid challenge: false config: subject_key: email openid_connect_url: https://accounts.google.com/.well-known/openid-configuration authentication_backend: type: noop ```

slowup · March 20, 2024, 4:58am

@pablo
thank u! pablo

So, does this exception no longer occur when searching for “/pd-search/_search”: org.opensearch.OpenSearch Security Exception: Unexpected exception indices:data/read/search? if version up to opensearch 2.10.0

Topic		Replies	Views
405 Errors from Java HighLevel Rest Client OpenSearch Client Libraries opensearch-java	6	527	September 22, 2022
Rest Client - Precondition RunTimeException - JDK 17 OpenSearch Client Libraries opensearch-java	5	468	October 9, 2023
Unable to Parse response body error OpenSearch Client Libraries opensearch-java	4	1810	August 15, 2023
Using opensearch java high level rest client in Android OpenSearch Client Libraries opensearch-java	2	349	September 15, 2022
Opensearch is down when getting errors OpenSearch troubleshoot	2	215	December 10, 2023

Occurs intermittently this error o.o.OpenSearchStatusException: OpenSearch exception

Related Topics