After some time passed, It really stopped working.
Kibana is displaying the “Create index pattern” page.
However, it definitely have several index patterns defined:
# curl "https://v161:9200/.kibana/_search?pretty=true&q=*:*" | jq . | grep index-pattern\:
"_id": "index-pattern:97bf27a0-7cfd-11ec-be29-85b221d38195",
"_id": "index-pattern:1d24cfb0-7cfb-11ec-be29-85b221d38195",
"_id": "index-pattern:a18ea0b0-7cfa-11ec-be29-85b221d38195",
"_id": "index-pattern:74c49ac0-7cfb-11ec-be29-85b221d38195",
"_id": "index-pattern:37a9e780-7cfb-11ec-be29-85b221d38195",
"_id": "index-pattern:f78951d0-7cfb-11ec-be29-85b221d38195",
"_id": "index-pattern:5f144610-7d07-11ec-be29-85b221d38195",
Kibana log error seems to be these lines:
{"type":"log","@timestamp":"2022-01-25T09:45:17Z","tags":["error","elasticsearch","data"],"pid":1,"message":"[security_exception]: no permissions for [indices:monitor/settings/get] and User [name=kibanaserver, backend_roles=[], requestedTenant=null]"}
{"type":"log","@timestamp":"2022-01-25T09:45:17Z","tags":["warning","plugins","securityOss"],"pid":1,"message":"Error encountered while checking cluster for user data: ResponseError: security_exception"}
Related Elasticsearch log file looks like this:
[2022-01-25T12:45:17,052][W ][c.a.o.s.h.HTTPBasicAuthenticator] [v161.company.com] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2022-01-25T12:45:17,052][W ][c.a.o.s.h.HTTPBasicAuthenticator] [v161.company.com] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2022-01-25T12:45:17,052][W ][stderr] [v161.company.com] java.lang.UnsupportedOperationException
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at java.base/java.util.Collections$UnmodifiableMap.put(Collections.java:1473)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at com.amazon.opendistroforelasticsearch.security.dlic.rest.api.PermissionsInfoAction$1.accept(PermissionsInfoAction.java:110)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at com.amazon.opendistroforelasticsearch.security.dlic.rest.api.PermissionsInfoAction$1.accept(PermissionsInfoAction.java:95)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:115)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at com.amazon.opendistroforelasticsearch.security.filter.OpenDistroSecurityRestFilter$1.handleRequest(OpenDistroSecurityRestFilter.java:116)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:258)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:340)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:191)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at com.amazon.opendistroforelasticsearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:319)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:384)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:309)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:28)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] at java.base/java.lang.Thread.run(Thread.java:832)
[2022-01-25T12:45:17,119][W ][c.a.o.s.h.HTTPBasicAuthenticator] [v161.company.com] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2022-01-25T12:45:17,119][W ][c.a.o.s.h.HTTPBasicAuthenticator] [v161.company.com] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2022-01-25T12:45:17,125][W ][c.a.o.s.h.HTTPBasicAuthenticator] [v161.company.com] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2022-01-25T12:45:17,125][W ][c.a.o.s.h.HTTPBasicAuthenticator] [v161.company.com] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2022-01-25T12:45:17,574][INFO ][c.a.o.s.p.PrivilegesEvaluator] [v161.company.com] No index-level perm match for User [name=kibanaserver, backend_roles=[], requestedTenant=null] Resolved [aliases=[*], allIndices=[*], types=[*], originalRequested=[*], remoteIndices=[]] [Action [indices:monitor/settings/get]] [RolesChecked [own_index, kibana_server]]
[2022-01-25T12:45:17,575][INFO ][c.a.o.s.p.PrivilegesEvaluator] [v161.company.com] No permissions for [indices:monitor/settings/get]
Dear All,
Any ideas how I would identify the issue?