Kibana stopped working after upgrade Opendistro 1.10.2 to 1.13.3

rlevitsky · December 17, 2021, 5:53pm

Hi All,
After upgrading Opendistro 1.10.2 to to 1.13.3, my Kibana forces me to choose tenant on every login.
Here are some log excerpts:
Elasticsearch log:

[2021-12-17T09:22:04,249][INFO ][c.a.o.s.p.PrivilegesEvaluator] [h161.company.com] No index-level perm match for User [name=kibanaserver, backend_roles=[], requestedTenant=null] Resolved [aliases=[ *], allIndices=[* ], types=[ *], originalRequested=[* ], remoteIndices=[]] [Action [indices:monitor/settings/get]] [RolesChecked [own_index, kibana_server]]
[2021-12-17T09:22:04,249][INFO ][c.a.o.s.p.PrivilegesEvaluator] [h161.company.com] No permissions for [indices:monitor/settings/get]

Here is exerpt from my ‘internal_users.yml’:

`kibanaserver:
hash: “$2y$12$K…”
reserved: true
description: “Kibanaserver user”

Here is an excerpt from my ‘roles_mapping.yml’:

kibana_server:
reserved: true
users:

“kibanaserver”

Here is an excerpt from my ‘roles.yml’

kibana_server:
cluster_permissions:

“cluster:*”

“indices:*”
index_permissions:

index_patterns:

“*”
allowed_actions:

“indices_all”

“indices:*”

Could you please advise on identifying the issue?

pablo · December 17, 2021, 11:04pm

@rlevitsky Could you share your kibana.yml and config.yml files?

rlevitsky · December 18, 2021, 12:29am

@pablo Thank you for your reply.
Here is my kibana.yml file:

---

server.name: "h161.company.com"
server.host: "0"
elasticsearch.hosts:
- https://h161.company.com:9200
elasticsearch.ssl.verificationMode: full
elasticsearch.ssl.certificateAuthorities: ["/usr/share/kibana/config/opendistroCA.crt"]
elasticsearch.username: kibanaserver
elasticsearch.password: *********
elasticsearch.requestHeadersWhitelist: ["securitytenant","Authorization"]
elasticsearch.requestTimeout: 30000

opendistro_security.multitenancy.enabled: true
opendistro_security.multitenancy.tenants.enable_private: false
opendistro_security.multitenancy.tenants.preferred: ["Global","Private"]
opendistro_security.readonly_mode.roles: ["kibana_read_only"]

uiSettings.overrides.defaultRoute: /app/discover

opendistro_security.auth.type: "saml"
server.xsrf.whitelist: ["/_opendistro/_security/saml/acs", "/_opendistro/_security/saml/logout"]

newsfeed.enabled: false
telemetry.optIn: false
telemetry.enabled: false

logging:
  root:
    appenders: [default]
    level: debug

Here is my config.yml:

---
_meta:
  type: "config"
  config_version: 2
config:
  dynamic:
    do_not_fail_on_forbidden: false
    kibana:
      multitenancy_enabled: true
      server_username: kibanaserver
      index: '.kibana'
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
    authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: false
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: internal
      saml_auth_domain:
        http_enabled: true
        transport_enabled: false
        order: 2
        http_authenticator:
          type: saml
          challenge: true
          config:
            idp:
              enable_ssl: true
              verify_hostnames: true
              metadata_url: https://login.company.com/auth/realms/company/protocol/saml/descriptor
              entity_id: https://login.company.com/auth/realms/company-uat
              pemtrustedcas_filepath: ca-bundle.crt
            sp:
              entity_id: ELK-STAGING
            roles_key: Role
            kibana_url: https://h161.company.com/
            exchange_key: hohnge9ujaiF1ooCei2zo9phizoYoo2f
        authentication_backend:
          type: noop
      ldap:
        description: "Authenticate via LDAP or Active Directory"
        http_enabled: true
        transport_enabled: false
        order: 1
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: ldap
          config:
            enable_ssl: true
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: true
            pemtrustedcas_filepath: ca-bundle.crt
            hosts:
            - ldap.company.com
            bind_dn: 'cn=lookup,ou=Special,dc=company,dc=com'
            password: '*********'
            userbase: 'ou=addressbook,dc=company,dc=com'
            usersearch: '(uid={0})'
            username_attribute: 'uid'
    authz:
      roles_from_myldap:
        description: "Authorize via LDAP or Active Directory"
        http_enabled: true
        transport_enabled: false
        authorization_backend:
          type: ldap
          config:
            enable_ssl: true
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: true
            pemtrustedcas_filepath: ca-bundle.crt
            hosts:
            - ldap.company.com
            bind_dn: 'cn=lookup,ou=Special,dc=company,dc=com'
            password: '**********'
            rolebase: 'ou=Groups,dc=company,dc=com'
            rolesearch: '(member={0})'
            userroleattribute: null
            userrolename: disabled
            rolename: cn
            resolve_nested_roles: true
            userbase: 'ou=addressbook,dc=company,dc=com'
            usersearch: '(uid={0})'
            skip_users:
            - admin
            - logstash
            - kibanaserver
            - zabbix
            - grafana
            - retention
            - curator

pablo · December 21, 2021, 4:45pm

@rlevitsky In ODFE you’ll need to select the option Remember my selection... to stop the tenant window appreaing.

In OpenSearch, this action is default and the tenant window will appear only once.
Please be aware that tenant selection is kept as a cookie, so if you run your browser in the private mode it will always ask you about the tenant.

rlevitsky · December 22, 2021, 5:23am

Thank you very much Pawel,

I didn’t notice that option. After setting it, Kibana is no longer pestering me to select a tenant.

However, I still see those “No permissions” messages at my elasticsearch logs.

Could you please advise on fixing it?

Best,
Roman.

pablo · December 22, 2021, 2:15pm

@rlevitsky That is only INFO level message. This is a service account and OpenDistro is treating it as a regular user. No functionality should be affected here.

I’ve checked OpenSearch and it seems to be fixed there.

rlevitsky · January 25, 2022, 10:00am

After some time passed, It really stopped working.
Kibana is displaying the “Create index pattern” page.

However, it definitely have several index patterns defined:

# curl "https://v161:9200/.kibana/_search?pretty=true&q=*:*" | jq . | grep index-pattern\:
        "_id": "index-pattern:97bf27a0-7cfd-11ec-be29-85b221d38195",
        "_id": "index-pattern:1d24cfb0-7cfb-11ec-be29-85b221d38195",
        "_id": "index-pattern:a18ea0b0-7cfa-11ec-be29-85b221d38195",
        "_id": "index-pattern:74c49ac0-7cfb-11ec-be29-85b221d38195",
        "_id": "index-pattern:37a9e780-7cfb-11ec-be29-85b221d38195",
        "_id": "index-pattern:f78951d0-7cfb-11ec-be29-85b221d38195",
        "_id": "index-pattern:5f144610-7d07-11ec-be29-85b221d38195",

Kibana log error seems to be these lines:

{"type":"log","@timestamp":"2022-01-25T09:45:17Z","tags":["error","elasticsearch","data"],"pid":1,"message":"[security_exception]: no permissions for [indices:monitor/settings/get] and User [name=kibanaserver, backend_roles=[], requestedTenant=null]"}
{"type":"log","@timestamp":"2022-01-25T09:45:17Z","tags":["warning","plugins","securityOss"],"pid":1,"message":"Error encountered while checking cluster for user data: ResponseError: security_exception"}

Related Elasticsearch log file looks like this:

[2022-01-25T12:45:17,052][W ][c.a.o.s.h.HTTPBasicAuthenticator] [v161.company.com] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2022-01-25T12:45:17,052][W ][c.a.o.s.h.HTTPBasicAuthenticator] [v161.company.com] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2022-01-25T12:45:17,052][W ][stderr] [v161.company.com] java.lang.UnsupportedOperationException
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at java.base/java.util.Collections$UnmodifiableMap.put(Collections.java:1473)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at com.amazon.opendistroforelasticsearch.security.dlic.rest.api.PermissionsInfoAction$1.accept(PermissionsInfoAction.java:110)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at com.amazon.opendistroforelasticsearch.security.dlic.rest.api.PermissionsInfoAction$1.accept(PermissionsInfoAction.java:95)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:115)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at com.amazon.opendistroforelasticsearch.security.filter.OpenDistroSecurityRestFilter$1.handleRequest(OpenDistroSecurityRestFilter.java:116)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:258)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:340)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:191)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at com.amazon.opendistroforelasticsearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:319)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:384)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:309)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:28)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[2022-01-25T12:45:17,053][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[2022-01-25T12:45:17,054][W ][stderr] [v161.company.com] 	at java.base/java.lang.Thread.run(Thread.java:832)
[2022-01-25T12:45:17,119][W ][c.a.o.s.h.HTTPBasicAuthenticator] [v161.company.com] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2022-01-25T12:45:17,119][W ][c.a.o.s.h.HTTPBasicAuthenticator] [v161.company.com] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2022-01-25T12:45:17,125][W ][c.a.o.s.h.HTTPBasicAuthenticator] [v161.company.com] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2022-01-25T12:45:17,125][W ][c.a.o.s.h.HTTPBasicAuthenticator] [v161.company.com] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2022-01-25T12:45:17,574][INFO ][c.a.o.s.p.PrivilegesEvaluator] [v161.company.com] No index-level perm match for User [name=kibanaserver, backend_roles=[], requestedTenant=null] Resolved [aliases=[*], allIndices=[*], types=[*], originalRequested=[*], remoteIndices=[]] [Action [indices:monitor/settings/get]] [RolesChecked [own_index, kibana_server]]
[2022-01-25T12:45:17,575][INFO ][c.a.o.s.p.PrivilegesEvaluator] [v161.company.com] No permissions for [indices:monitor/settings/get]

Dear All,
Any ideas how I would identify the issue?

pablo · January 25, 2022, 10:25am

@rlevitsky According to your kibana screenshot you don’t use admin user. Could you try to log in as the admin user and use Global Tenant?

Do you see .kibana index in results of GET _cat/indices?v&s=index ?

rlevitsky · January 25, 2022, 11:43am

Hi Pawel,
Thank you very much for your quick reply.

Re .kibana index - yes, it is here with 18 documents. I also have the .opendistro_security index and several data indices with increasing document number.

Re non-admin user, could you please teach me how you can tell if my account doesn’t powerful enough just by the screenshot (actually, my account is configured as the member of the ‘sa’ group which have a plenty of permissions set)?
Are there any way to debug the actual authorization my account has?

rlevitsky · January 25, 2022, 11:49am

When I click 'view roles and identities, I see this:

Roles (3)

Roles you are currently mapped to by your administrator.
own_index
cluster_admins
sa

Backend roles (32)

Backend roles you are currently mapped to by your administrator.
SA
ELK_EM
LDAP_admins
git
SonarQube
HQ VPN
JC_Asterisk_Administration
ELK_admins
JC_Sonarqube_Administration
Sonar-administrator
ELK_SA
Password_Manager
JC_Testlink_Administration
…

rlevitsky · January 25, 2022, 11:51am

And yes, my ‘Tenant’ is ‘Global’.
I’ve tried switching back and forth to other tenants - no luck.

pablo · January 27, 2022, 6:10pm

Hi @rlevitsky I was referring to the built-in admin account itself rather than the role.
Just saw the user icon in the top corner which starts with r and not a.

I was going to ask you if you can see index patterns using built-in admin user and at which tenant are they visible.

rlevitsky · January 28, 2022, 11:19am

After changing the opendistro_security.auth.type to basicauth an logging with local admin user, I see two * patterns, three applications-* patterns and several other patterns.

pablo · January 28, 2022, 10:10pm

@rlevitsky I’ve tested with Microsoft AD and LDAP. Index patterns should be visible with an internal user and LDAP user once they’re in the Global tenant.

Does any of those roles have access to .kibana*?
This looks like a lack of privileges for those indices.

rlevitsky · February 1, 2022, 8:31am

I’m pretty much sure they has access.
As I wrote before, there are several * patterns, several applications-* patterns etc.
This is because I am able to create an index pattern but unable to see them.

Is there any way to debug this?

pablo · February 1, 2022, 7:59pm

@rlevitsky Could you backup your security config and share roles.yml, roles_mapping.yml and config.yml?

rlevitsky · February 1, 2022, 10:28pm

@pablo here they are:
config.yml

---
_meta:
  type: "config"
  config_version: 2
config:
  dynamic:
    do_not_fail_on_forbidden: false
    kibana:
      multitenancy_enabled: true
      server_username: "kibanaserver"
      index: ".kibana"
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
        internalProxies: "192\\.168\\.0\\.10|192\\.168\\.0\\.11"
    authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: false
        order: 0
        http_authenticator:
          type: "basic"
          challenge: false
        authentication_backend:
          type: "internal"
      saml_auth_domain:
        http_enabled: true
        transport_enabled: false
        order: 2
        http_authenticator:
          type: "saml"
          challenge: true
          config:
            skip_users:
            - "admin"
            idp:
              enable_ssl: true
              verify_hostnames: true
              metadata_url: "https://login.company.com/auth/realms/company/protocol/saml/descriptor"
              entity_id: "https://login.company.com/auth/realms/company"
              pemtrustedcas_filepath: "ca-bundle.crt"
            sp:
              entity_id: "ELK-STAGING"
            roles_key: "Role"
            kibana_url: "https://v161.company.com/"
            exchange_key: "hohnge9ujaiF1ooCei2zo9phizoYoo2f"
        authentication_backend:
          type: "noop"
      ldap:
        description: "Authenticate via LDAP or Active Directory"
        http_enabled: true
        transport_enabled: false
        order: 1
        http_authenticator:
          type: "basic"
          challenge: false
        authentication_backend:
          type: "ldap"
          config:
            enable_ssl: true
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: true
            pemtrustedcas_filepath: "ca-bundle.crt"
            hosts:
            - "ldap.company.com"
            bind_dn: "cn=access,dc=company,dc=com"
            password: "********"
            userbase: "ou=addressbook,dc=company,dc=com"
            usersearch: "(uid={0})"
            username_attribute: "uid"
    authz:
      roles_from_myldap:
        description: "Authorize via LDAP or Active Directory"
        http_enabled: true
        transport_enabled: false
        authorization_backend:
          type: "ldap"
          config:
            enable_ssl: true
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: true
            pemtrustedcas_filepath: "ca-bundle.crt"
            hosts:
            - "ldap.company.com"
            bind_dn: "cn=access,dc=company,dc=com"
            password: "*******"
            rolebase: "ou=Groups,dc=company,dc=com"
            rolesearch: "(member={0})"
            userroleattribute: null
            userrolename: "disabled"
            rolename: "cn"
            resolve_nested_roles: true
            userbase: "ou=addressbook,dc=company,dc=com"
            usersearch: "(uid={0})"
            skip_users:
            - "admin"
            - "logstash"
            - "kibanaserver"
            - "zabbix"
            - "grafana"
            - "retention"
            - "curator"

roles.yml

---
_meta:
  type: "roles"
  config_version: 2
logstash_indexer:
  cluster_permissions:
  - "cluster_monitor"
  - "cluster_composite_ops"
  - "indices:admin/template/get"
  - "indices:admin/template/put"
  - "cluster:admin/ingest/pipeline/put"
  - "cluster:admin/ingest/pipeline/get"
  - "indices:admin/create"
  - "indices:data/write/index"
  index_permissions:
  - index_patterns:
    - "*"
    allowed_actions:
    - "indices:admin/settings/update"
    - "crud"
    - "create_index"
zabbix:
  cluster_permissions:
  - "cluster_monitor"
  index_permissions:
  - index_patterns:
    - "*"
    allowed_actions:
    - "indices:monitor/stats"
grafana:
  index_permissions:
  - index_patterns:
    - "netflow-*"
    - "iptnetflow-*"
    allowed_actions:
    - "read"
retention:
  cluster_permissions:
  - "cluster:monitor/state"
  - "cluster:monitor/health"
  index_permissions:
  - index_patterns:
    - "*"
    allowed_actions:
    - "indices:monitor/stats"
    - "indices:monitor/settings/get"
    - "indices:admin/delete"
curator:
  cluster_permissions:
  - "cluster:admin/snapshot/create"
  - "cluster:monitor/main"
  - "cluster:monitor/state"
  - "cluster:admin/repository/get"
  - "cluster:admin/repository/verify"
  - "cluster:admin/snapshot/status"
  - "cluster:admin/snapshot/get"
  - "cluster:admin/repository/put"
  - "cluster:monitor/nodes/info"
  - "cluster:admin/snapshot/restore"
  - "cluster:monitor/tasks/lists"
  - "cluster:admin/snapshot/delete"
  - "cluster:admin/snapshot/status[nodes]"
  index_permissions:
  - index_patterns:
    - "*"
    allowed_actions:
    - "indices:admin/delete"
    - "indices:monitor/settings/get"
    - "indices:monitor/stats"
    - "indices:admin/create"
    - "indices:monitor/recovery"
    - "indices:admin/synced_flush"
    - "indices:admin/close"
    - "indices:admin/open"
    - "indices:admin/close[s]"
sa:
  reserved: false
  hidden: false
  index_permissions:
  - index_patterns:
    - "*"
    allowed_actions:
    - "read"
    - "indices:monitor/stats"
    - "indices:monitor/settings/get"
    - "indices:admin/mappings/get"
    - "indices:admin/aliases/get"
  - index_patterns:
    - ".kibana_*"
    allowed_actions:
    - "crud"
  - index_patterns:
    - ".opendistro-alerting-config"
    allowed_actions:
    - "indices:data/write/index"
    - "indices:admin/mapping/put"
    - "indices:data/write/bulk[s]"
    - "indices:data/write/delete"
  - index_patterns:
    - ".opendistro-alerting-alerts"
    - ".opendistro-alerting-alert-history-*"
    allowed_actions:
    - "indices:data/write/update"
    - "indices:data/write/bulk[s]"
    - "indices:admin/mapping/put"
  tenant_permissions:
  - tenant_patterns:
    - "global_tenant"
    allowed_actions:
    - "kibana_all_write"
  cluster_permissions:
  - "cluster:admin/alerting/monitor/search"
  - "cluster:admin/alerting/destination/email_account/search"
  - "cluster:admin/alerting/monitor/get"
  - "cluster:monitor/state"
  - "cluster:monitor/health"
  - "cluster:monitor/nodes/info"
  - "cluster:admin/alerting/monitor/execute"
  - "cluster:admin/alerting/monitor/write"
  - "cluster:admin/alerting/monitor/delete"
  - "cluster:admin/alerting/destination/email_account/write"
  - "cluster:admin/alerting/destination/write"
  - "cluster:admin/alerting/alerts/ack"
  - "cluster:admin/alerting/destination/email_group/search"
  - "cluster:admin/alerting/destination/email_account/get"
cluster_admins:
  cluster_permissions:
  - "*"
  index_permissions:
  - index_patterns:
    - "*"
    allowed_actions:
    - "*"
  tenant_permissions:
  - tenant_patterns:
    - "*"
    allowed_actions:
    - "kibana_all_write"
ELK_tms:
  reserved: false
  hidden: false
  index_permissions:
  - index_patterns:
    - "applications-*"
    dls: "{ \"match_phrase\": { \"host\": \"v2.company.com\" } }"
    allowed_actions:
    - "read"
  - index_patterns:
    - ".kibana_*_elktms*"
    allowed_actions:
    - "crud"
  - index_patterns:
    - "kibana_sample_*"
    allowed_actions:
    - "read"
  tenant_permissions:
  - tenant_patterns:
    - "ELK_tms"
    allowed_actions:
    - "kibana_all_write"
  static: false
ELK_kc:
  reserved: false
  hidden: false
  index_permissions:
  - index_patterns:
    - "keycloak-*"
    dls: "{ \"bool\": { \"must\": [], \"filter\": [ { \"bool\": { \"should\": [  {\
      \ \"bool\": { \"should\": [ { \"match_phrase\": { \"host\": \"v1.company.com\"\
      \ } } ], \"minimum_should_match\": 1 } }, { \"bool\": { \"should\": [ { \"match_phrase\"\
      : { \"host\": \"v1.company.com\" } } ], \"minimum_should_match\": 1 }\
      \ }, { \"bool\": { \"should\": [ { \"match_phrase\": { \"host\": \"v1.company.com\"\
      \ } } ], \"minimum_should_match\": 1 } }, { \"bool\": { \"should\": [ { \"match_phrase\"\
      : { \"host\": \"v3.company.com\" } } ], \"minimum_should_match\": 1 } },\
      \ { \"bool\": { \"should\": [ { \"match_phrase\": { \"host\": \"v4.company.com\"\
      \ } } ], \"minimum_should_match\": 1 } }], \"minimum_should_match\": 1 } } ]\
      \ } }"
    allowed_actions:
    - "read"
  - index_patterns:
    - ".kibana_*_elkkc*"
    allowed_actions:
    - "crud"
  - index_patterns:
    - "kibana_sample_*"
    allowed_actions:
    - "read"
  tenant_permissions:
  - tenant_patterns:
    - "ELK_kc"
    allowed_actions:
    - "kibana_all_write"
  static: false
ELK_nginx:
  reserved: false
  hidden: false
  index_permissions:
  - index_patterns:
    - "applications-*"
    dls: "{ \"bool\": { \"must\": [], \"filter\": [ { \"bool\": { \"should\": [  {\
      \ \"bool\": { \"should\": [ { \"match_phrase\": { \"host\": \"v1.company.com\"\
      \ } } ], \"minimum_should_match\": 1 } }, { \"bool\": { \"should\": [ { \"match_phrase\"\
      : { \"host\": \"v1.company.com\" } } ], \"minimum_should_match\": 1 }\
      \ }], \"minimum_should_match\": 1 } } ] } }"
    allowed_actions:
    - "read"
  - index_patterns:
    - ".kibana_*_elknginx*"
    allowed_actions:
    - "crud"
  - index_patterns:
    - "kibana_sample_*"
    allowed_actions:
    - "read"
  tenant_permissions:
  - tenant_patterns:
    - "ELK_nginx"
    allowed_actions:
    - "kibana_all_write"
  static: false
ELK_rvision_wtn_uat:
  reserved: false
  hidden: false
  index_permissions:
  - index_patterns:
    - "applications-*"
    dls: "{ \"match_phrase\": { \"host\": \"v656.company.com\" } }"
    allowed_actions:
    - "read"
  - index_patterns:
    - ".kibana_*_elkrvisionwtnuat*"
    allowed_actions:
    - "crud"
  - index_patterns:
    - "kibana_sample_*"
    allowed_actions:
    - "read"
  tenant_permissions:
  - tenant_patterns:
    - "ELK_rvision_wtn_uat"
    allowed_actions:
    - "kibana_all_write"
  static: false
ELK_rVision_flash:
  reserved: false
  hidden: false
  index_permissions:
  - index_patterns:
    - "applications-*"
    dls: "{ \"match_phrase\": { \"host\": \"v683.company.com\" } }"
    allowed_actions:
    - "read"
  - index_patterns:
    - ".kibana_*_elkrvisionflash*"
    allowed_actions:
    - "crud"
  - index_patterns:
    - "kibana_sample_*"
    allowed_actions:
    - "read"
  tenant_permissions:
  - tenant_patterns:
    - "ELK_rVision_flash"
    allowed_actions:
    - "kibana_all_write"
  static: false
ELK_greenhouse:
  reserved: false
  hidden: false
  index_permissions:
  - index_patterns:
    - "applications-*"
    dls: "{ \"match_phrase\": { \"host\": \"v1.company.com\" } }"
    allowed_actions:
    - "read"
  - index_patterns:
    - ".kibana_*_elkgreenhouse*"
    allowed_actions:
    - "crud"
  - index_patterns:
    - "kibana_sample_*"
    allowed_actions:
    - "read"
  tenant_permissions:
  - tenant_patterns:
    - "ELK_greenhouse"
    allowed_actions:
    - "kibana_all_write"
  static: false
ELK_rvision_uat:
  reserved: false
  hidden: false
  index_permissions:
  - index_patterns:
    - "applications-*"
    dls: "{ \"bool\": { \"must\": [], \"filter\": [ { \"bool\": { \"should\": [  {\
      \ \"bool\": { \"should\": [ { \"match_phrase\": { \"host\": \"v1.company.com\"\
      \ } } ], \"minimum_should_match\": 1 } }, { \"bool\": { \"should\": [ { \"match_phrase\"\
      : { \"host\": \"v012.company.com\" } } ], \"minimum_should_match\": 1 } },\
      \ { \"bool\": { \"should\": [ { \"match_phrase\": { \"host\": \"v421.company.com\"\
      \ } } ], \"minimum_should_match\": 1 } }, { \"bool\": { \"should\": [ { \"match_phrase\"\
      : { \"host\": \"v158.company.com\" } } ], \"minimum_should_match\": 1 } },\
      \ { \"bool\": { \"should\": [ { \"match_phrase\": { \"host\": \"v1.company.com\"\
      \ } } ], \"minimum_should_match\": 1 } }], \"minimum_should_match\": 1 } } ]\
      \ } }"
    allowed_actions:
    - "read"
  - index_patterns:
    - ".kibana_*_elkrvisionuat*"
    allowed_actions:
    - "crud"
  - index_patterns:
    - "kibana_sample_*"
    allowed_actions:
    - "read"
  tenant_permissions:
  - tenant_patterns:
    - "ELK_rvision_uat"
    allowed_actions:
    - "kibana_all_write"
  static: false
ELK_EEAS:
  reserved: false
  hidden: false
  index_permissions:
  - index_patterns:
    - "applications-*"
    dls: "{ \"match_phrase\": { \"host\": \"v1.company.com\" } }"
    allowed_actions:
    - "read"
  - index_patterns:
    - ".kibana_*_elkeeas*"
    allowed_actions:
    - "crud"
  - index_patterns:
    - "kibana_sample_*"
    allowed_actions:
    - "read"
  tenant_permissions:
  - tenant_patterns:
    - "ELK_EEAS"
    allowed_actions:
    - "kibana_all_write"
  static: false
ELK_rvision_badge:
  reserved: false
  hidden: false
  index_permissions:
  - index_patterns:
    - "applications-*"
    dls: "{ \"match_phrase\": { \"host\": \"v1.company.com\" } }"
    allowed_actions:
    - "read"
  - index_patterns:
    - ".kibana_*_elkrvisionbadge*"
    allowed_actions:
    - "crud"
  - index_patterns:
    - "kibana_sample_*"
    allowed_actions:
    - "read"
  tenant_permissions:
  - tenant_patterns:
    - "ELK_rvision_badge"
    allowed_actions:
    - "kibana_all_write"
  static: false
ELK_rVision_reWrite_DEV:
  reserved: false
  hidden: false
  index_permissions:
  - index_patterns:
    - "applications-*"
    dls: "{ \"match_phrase\": { \"host\": \"v425.company.com\" } }"
    allowed_actions:
    - "read"
  - index_patterns:
    - ".kibana_*_elkrvisionrewritedev*"
    allowed_actions:
    - "crud"
  - index_patterns:
    - "kibana_sample_*"
    allowed_actions:
    - "read"
  tenant_permissions:
  - tenant_patterns:
    - "ELK_rVision_reWrite_DEV"
    allowed_actions:
    - "kibana_all_write"
  static: false
ELK_rVision_reWrite_PROD:
  reserved: false
  hidden: false
  index_permissions:
  - index_patterns:
    - "applications-*"
    dls: "{ \"match_phrase\": { \"host\": \"v1.company.com\" } }"
    allowed_actions:
    - "read"
  - index_patterns:
    - ".kibana_*_elkrvisionrewriteprod*"
    allowed_actions:
    - "crud"
  - index_patterns:
    - "kibana_sample_*"
    allowed_actions:
    - "read"
  tenant_permissions:
  - tenant_patterns:
    - "ELK_rVision_reWrite_PROD"
    allowed_actions:
    - "kibana_all_write"
  static: false
ELK_rvision_prod:
  reserved: false
  hidden: false
  index_permissions:
  - index_patterns:
    - "applications-*"
    dls: "{ \"bool\": { \"must\": [], \"filter\": [ { \"bool\": { \"should\": [  {\
      \ \"bool\": { \"should\": [ { \"match_phrase\": { \"host\": \"v1.company.com\"\
      \ } } ], \"minimum_should_match\": 1 } }, { \"bool\": { \"should\": [ { \"match_phrase\"\
      : { \"host\": \"v1.company.com\" } } ], \"minimum_should_match\": 1 }\
      \ }], \"minimum_should_match\": 1 } } ] } }"
    allowed_actions:
    - "read"
  - index_patterns:
    - ".kibana_*_elkrvisionprod*"
    allowed_actions:
    - "crud"
  - index_patterns:
    - "kibana_sample_*"
    allowed_actions:
    - "read"
  tenant_permissions:
  - tenant_patterns:
    - "ELK_rvision_prod"
    allowed_actions:
    - "kibana_all_write"
  static: false

roles_mapping.yml

---
_meta:
  type: "rolesmapping"
  config_version: 2
all_access:
  reserved: true
  backend_roles:
  - "admin"
  description: "Maps admin to all_access"
own_index:
  reserved: false
  users:
  - "*"
  description: "Allow full access to an index named like the username"
logstash:
  reserved: false
  backend_roles:
  - "logstash"
zabbix:
  reserved: false
  backend_roles:
  - "zabbix"
retention:
  reserved: false
  backend_roles:
  - "retention"
curator:
  reserved: false
  backend_roles:
  - "curator"
grafana:
  reserved: false
  backend_roles:
  - "grafana"
logstash_indexer:
  reserved: false
  backend_roles:
  - "logstash_indexer"
kibana_user:
  reserved: false
  backend_roles:
  - "kibanauser"
  description: "Maps kibanauser to kibana_user"
readall:
  reserved: true
  backend_roles:
  - "readall"
manage_snapshots:
  reserved: true
  backend_roles:
  - "snapshotrestore"
kibana_server:
  reserved: true
  users:
  - "kibanaserver"
cluster_admins:
  reserved: false
  backend_roles:
  - "ELK_admins"
  - "elk_admin"
  description: "Maps ELK_admin to cluster admins"
sa:
  reserved: false
  backend_roles:
  - "ELK_SA"
  description: "Maps SA to admins"
ELK_tms:
  reserved: false
  backend_roles:
  - "ELK_tms"
  description: "TS-69264 - TMS Prod logs"
ELK_kc:
  reserved: false
  backend_roles:
  - "ELK_kc"
  description: "TS-77808 dev keycloak"
ELK_nginx:
  reserved: false
  backend_roles:
  - "ELK_nginx"
  description: "TS-66316 fshekhter"
ELK_rvision_wtn_uat:
  reserved: false
  backend_roles:
  - "ELK_rvision_wtn_uat"
  description: "Tenant for Webtime DEV, TS-67124, TS-67620"
ELK_rVision_flash:
  reserved: false
  backend_roles:
  - "ELK_rVision_flash"
  description: "TS-78549 - rvision flash"
ELK_greenhouse:
  reserved: false
  backend_roles:
  - "ELK_greenhouse"
  description: "TS-83218 greenhouse"
ELK_rvision_uat:
  reserved: false
  backend_roles:
  - "ELK_rvision_uat"
  description: "Tenant for Rvision UAT TS-45414"
ELK_EEAS:
  reserved: false
  backend_roles:
  - "ELK_EEAS"
  description: "TS-77380 EEAS Prod"
ELK_rvision_badge:
  reserved: false
  backend_roles:
  - "ELK_rvision_badge"
  description: "TS-81892 for shuryn"
ELK_rVision_reWrite_DEV:
  reserved: false
  backend_roles:
  - "ELK_rVision_reWrite_DEV"
  description: "Rvision ReWrite TS-55975"
ELK_rVision_reWrite_PROD:
  reserved: false
  backend_roles:
  - "ELK_rVision_reWrite_PROD"
  description: "Access to nginx log of rVisionrWrite PROD TS-57147"
ELK_rvision_prod:
  reserved: false
  backend_roles:
  - "ELK_rvision_prod"
  description: "Tenant for Rvision Prod TS-45537"

rlevitsky · February 14, 2022, 9:05am

After some while it just returned to live.
Cannot say for sure what caused this, probably one of the subsequent reinstalls…

Topic		Replies	Views
BUG Opendistro(1.8) kibana_read_only role not working in elastic 7.7 Security	7	2773	June 15, 2021
No index-level perm match for User name=kibanaserver Security troubleshoot	2	1519	April 21, 2022
Plugin:opendistro_security@7.4.2 Tenant indices migration failed Open Source Elasticsearch and Kibana	3	1147	December 21, 2020
Setting index level permissions makes the kibana discover/dashboard/visualize pages blank Security troubleshoot	6	2440	July 25, 2022
When kibana is enabled with multitenancy then after upgrade kibana tenant index gets migrated and after that it is not working properly Security	1	452	April 12, 2021

Kibana stopped working after upgrade Opendistro 1.10.2 to 1.13.3

Roles (3)

Backend roles (32)

Related topics