The percentiles type of aggregation fails with security_exception, but only when done through a remote cross-search configuration. When the same query is applied to local indices, it succeeds.
# curl --insecure -u admin:redacted -X GET "https://my-coordinating-cluster-1:9200/os3:my-index-family-000003/_search?pretty" -H 'Content-Type: application/json' -d'
{
"size": 0,
"aggs": {
"price_by_percentile": {
"percentiles": { "field": "params.latency_seconds"}
}
}
}
'
{
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "Unexpected exception indices:data/read/search"
}
],
"type" : "security_exception",
"reason" : "Unexpected exception indices:data/read/search"
},
"status" : 500
}
This exact same command succeeds when done against the cluster which my-index-family-* lives locally:
# curl --insecure -u admin:redacted -X GET "https://my-data-cluster-host-1:9200/my-index-family-000003/_search?pretty" -H 'Content-Type: application/json' -d'
{
"size": 0,
"aggs": {
"price_by_percentile": {
"percentiles": { "field": "params.latency_seconds"}
}
}
}
'
{
"took" : 3317,
"timed_out" : false,
"_shards" : {
"total" : 12,
"successful" : 12,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 10000,
"relation" : "gte"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"price_by_percentile" : {
"values" : {
"1.0" : 0.005121995169189605,
"5.0" : 0.027560057707115882,
"25.0" : 0.3060668743001799,
"50.0" : 0.6620858926084826,
"75.0" : 2.9884932599837266,
"95.0" : 28.044821776170306,
"99.0" : 237.3698258252764
}
}
}
}
Crazy, right? The issue only manifests when accessing the indices via remote cross-search configuration. When the indices live locally, no problem.
I have the audit logging turned all the way up, not omitting anything, and I only find evidence that indices:data/read/search is being successfully granted. No indications in security-auditlog that anything is being blocked. Yes, I removed the exclusions from audit.yml security config so everything would get logged to security auditlog. Nothing is logged there that would indicate which user is allegedly being denied this privilege. My ‘admin’ user is assigned the built-in all_access role.
In opensearch.log on the coordinating cluster, I see an error message like this which apparently correlates:
[my-coordinating-cluster-1-9200] path: /os3:my-index-family-*/_search, params: {pretty=, index=os3:my-index-family-*}
org.opensearch.transport.RemoteTransportException: [error while communicating with remote cluster [os3]]
Caused by: org.opensearch.transport.RemoteTransportException: [opensearch-host-11-9211][1892.168.0.11:9311][indices:data/read/search]
Caused by: org.opensearch.OpenSearchSecurityException: Unexpected exception indices:data/read/search
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:376) ~[?:?]
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:154) ~[?:?]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:169) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:95) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:91) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:139) ~[?:?]
at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceivedDecorate(SecuritySSLRequestHandler.java:193) ~[?:?]
at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:336) ~[?:?]
at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:153) ~[?:?]
at org.opensearch.security.OpenSearchSecurityPlugin$7$1.messageReceived(OpenSearchSecurityPlugin.java:647) ~[?:?]
at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:64) ~[?:?]
at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:91) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:268) [opensearch-1.2.4.jar:1.2.4]
at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:126) [opensearch-1.2.4.jar:1.2.4]
at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:108) [opensearch-1.2.4.jar:1.2.4]
at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:759) [opensearch-1.2.4.jar:1.2.4]
at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:170) [opensearch-1.2.4.jar:1.2.4]
at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:145) [opensearch-1.2.4.jar:1.2.4]
at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:110) [opensearch-1.2.4.jar:1.2.4]
at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:94) [transport-netty4-client-1.2.4.jar:1.2.4]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1371) [netty-handler-4.1.72.Final.jar:4.1.72.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234) [netty-handler-4.1.72.Final.jar:4.1.72.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283) [netty-handler-4.1.72.Final.jar:4.1.72.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.72.Final.jar:4.1.72.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.72.Final.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:832) [?:?]
Hjälp!