Kibana Multi-Tenant "Tenant Indices migration failed"

I am implementing Multi-Tenant in Kibana. I am following below document for achieving the purpose.

Open Distro Version of Security Plugin - 1.9.0.0
ELK version - 7.8.0

However after following all steps I am getting below error.

“Tenant Indices migration failed”

kibana.yml

kibana.index: “.kibana”
elasticsearch.username: “admin”
elasticsearch.password: “admin”
elasticsearch.requestHeadersWhitelist: [“securitytenant”,“Authorization”]
opendistro_security.multitenancy.enabled: true
opendistro_security.multitenancy.tenants.enable_global: true
opendistro_security.multitenancy.tenants.enable_private: true
opendistro_security.multitenancy.tenants.preferred: [“Private”, “Global”]
opendistro_security.multitenancy.enable_filter: false

elasticsearch.yml

opendistro_security.ssl.transport.pemcert_filepath: esnode.pem
opendistro_security.ssl.transport.pemkey_filepath: esnode-key.pem
opendistro_security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
opendistro_security.ssl.transport.enforce_hostname_verification: false
opendistro_security.ssl.http.enabled: false
opendistro_security.ssl.http.pemcert_filepath: esnode.pem
opendistro_security.ssl.http.pemkey_filepath: esnode-key.pem
opendistro_security.ssl.http.pemtrustedcas_filepath: root-ca.pem
opendistro_security.allow_unsafe_democertificates: true
opendistro_security.allow_default_init_securityindex: true
opendistro_security.authcz.admin_dn:

  • CN=kirk,OU=client,O=client,L=test, C=de

opendistro_security.audit.type: internal_elasticsearch
opendistro_security.enable_snapshot_restore_privilege: true
opendistro_security.check_snapshot_restore_write_privileges: true
opendistro_security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
cluster.routing.allocation.disk.threshold_enabled: false
node.max_local_storage_nodes: 3

config.yml of open-distro security

_meta:
type: “config”
config_version: 2

config:
dynamic:
#filtered_alias_mode: warn
do_not_fail_on_forbidden: false
kibana:
multitenancy_enabled: true
server_username: admin
index: ‘.kibana’
http:
anonymous_auth_enabled: false
xff:
enabled: false
internalProxies: ‘192.168.0.10|192.168.0.11’
authc:
kerberos_auth_domain:
http_enabled: false
transport_enabled: false
order: 6
http_authenticator:
type: kerberos
challenge: true
config:
krb_debug: false
strip_realm_from_principal: true
authentication_backend:
type: noop
basic_internal_auth_domain:
description: “Authenticate via HTTP Basic against internal users database”
http_enabled: true
transport_enabled: false
order: 4
http_authenticator:
type: basic
challenge: true
authentication_backend:
type: intern
proxy_auth_domain:
description: “Authenticate via proxy”
http_enabled: false
transport_enabled: false
order: 3
http_authenticator:
type: proxy
challenge: false
config:
user_header: “x-proxy-user”
roles_header: “x-proxy-roles”
authentication_backend:
type: noop
jwt_auth_domain:
description: “Authenticate via Json Web Token”
http_enabled: false
transport_enabled: false
order: 0
http_authenticator:
type: jwt
challenge: false
config:
signing_key: “base64 encoded HMAC key or public RSA/ECDSA pem key”
jwt_header: “Authorization”
jwt_url_parameter: null
roles_key: null
subject_key: null
authentication_backend:
type: noop
clientcert_auth_domain:
description: “Authenticate via SSL client certificates”
http_enabled: false
transport_enabled: false
order: 2
http_authenticator:
type: clientcert
config:
username_attribute: cn
challenge: false
authentication_backend:
type: noop
ldap:
description: “Authenticate via LDAP or Active Directory”
http_enabled: false
transport_enabled: false
order: 5
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: ldap
config:
enable_ssl: false
enable_start_tls: false
enable_ssl_client_auth: false
verify_hostnames: true
hosts:
- localhost:8389
bind_dn: null
password: null
userbase: ‘ou=people,dc=example,dc=com’
usersearch: ‘(sAMAccountName={0})’
username_attribute: null
authz:
roles_from_myldap:
description: “Authorize via LDAP or Active Directory”
http_enabled: false
transport_enabled: false
authorization_backend:
type: ldap
config:
enable_ssl: false
enable_start_tls: false
enable_ssl_client_auth: false
verify_hostnames: true
hosts:
- localhost:8389
bind_dn: null
password: null
rolebase: ‘ou=groups,dc=example,dc=com’
rolesearch: ‘(member={0})’
userroleattribute: null
userrolename: disabled
rolename: cn
resolve_nested_roles: true
userbase: ‘ou=people,dc=example,dc=com’
usersearch: ‘(uid={0})’
roles_from_another_ldap:
description: “Authorize via another Active Directory”
http_enabled: false
transport_enabled: false
authorization_backend:
type: ldap

Please tell me if there is anything wrong in configuration.

Hello !
Maybe you can check the logs ? Kibana and elasticsearch, sometimes you have information.
When I usually get “Tenant indices migration failed”, it often states some index is already created and can’t be replaced so it fails. In the logs, it says to delete one of the elastic index and then all fixed.
Do you have multiple nodes with kibana?

Thi

Thank u @ThibaudF for the quick reply.

Here is the kibana.log

{"type":"log","@timestamp":"2020-11-03T12:41:36Z","tags":["warning","plugins-discovery"],"pid":15744,"message":"Expect plugin \"id\" in camelCase, but found: apm_oss"}
{"type":"log","@timestamp":"2020-11-03T12:41:55Z","tags":["info","plugins-service"],"pid":15744,"message":"Plugin \"visTypeXy\" is disabled."}
{"type":"log","@timestamp":"2020-11-03T12:42:38Z","tags":["warning","legacy-service"],"pid":15744,"message":"Some installed third party plugin(s) [opendistro-alerting, opendistro_index_management_kibana, opendistro_security, console_legacy, apm_oss, elasticsearch, region_map, newsfeed, status_page, tile_map, ui_metric, timelion, kibana] are using the legacy plugin format and will no longer work in a future Kibana release. Please refer to https://ela.st/kibana-breaking-changes-8-0 for a list of breaking changes and https://ela.st/kibana-platform-migration for documentation on how to migrate legacy plugins."}
{"type":"log","@timestamp":"2020-11-03T12:42:38Z","tags":["info","plugins-system"],"pid":15744,"message":"Setting up [42] plugins: [usageCollection,telemetryCollectionManager,telemetry,kibanaLegacy,devTools,uiActions,kibanaUtils,statusPage,share,newsfeed,mapsLegacy,kibanaReact,inspector,embeddable,indexPatternManagement,esUiShared,charts,discover,bfetch,expressions,data,home,console,apm_oss,management,advancedSettings,telemetryManagementSection,visualizations,visTypeVega,visTypeTimeseries,visTypeVislib,visTypeTagcloud,visTypeMetric,visTypeTimelion,visTypeMarkdown,visTypeTable,inputControlVis,savedObjects,navigation,visualize,dashboard,savedObjectsManagement]"}
{"type":"log","@timestamp":"2020-11-03T12:42:43Z","tags":["info","savedobjects-service"],"pid":15744,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2020-11-03T12:42:44Z","tags":["info","savedobjects-service"],"pid":15744,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2020-11-03T12:42:44Z","tags":["info","plugins-system"],"pid":15744,"message":"Starting [27] plugins: [usageCollection,telemetryCollectionManager,telemetry,kibanaLegacy,share,discover,bfetch,expressions,data,home,console,apm_oss,management,advancedSettings,visualizations,visTypeVega,visTypeTimeseries,visTypeVislib,visTypeTagcloud,visTypeMetric,visTypeTimelion,visTypeMarkdown,visTypeTable,inputControlVis,visualize,dashboard,savedObjectsManagement]"}
{"type":"log","@timestamp":"2020-11-03T12:43:43Z","tags":["status","plugin:kibana@7.8.0","info"],"pid":15744,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2020-11-03T12:43:45Z","tags":["info","http","server","Kibana"],"pid":15744,"message":"http server running at http://0.0.0.0:5601"}
{"type":"log","@timestamp":"2020-11-03T12:43:43Z","tags":["status","plugin:elasticsearch@7.8.0","info"],"pid":15744,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2020-11-03T12:43:43Z","tags":["status","plugin:elasticsearch@7.8.0","info"],"pid":15744,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2020-11-03T12:43:43Z","tags":["status","plugin:opendistro-alerting@1.9.0.0","info"],"pid":15744,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2020-11-03T12:43:43Z","tags":["status","plugin:opendistro_index_management_kibana@1.9.0.1","info"],"pid":15744,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2020-11-03T12:43:44Z","tags":["status","plugin:opendistro_security@1.9.0.0","info"],"pid":15744,"state":"yellow","message":"Status changed from uninitialized to yellow - Initialising Security authentication plugin.","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2020-11-03T12:43:44Z","tags":["status","plugin:opendistro_security@1.9.0.0","info"],"pid":15744,"state":"yellow","message":"Status changed from yellow to yellow - Default cookie password detected, please set a password in kibana.yml by setting 'opendistro_security.cookie.password' (min. 32 characters).","prevState":"yellow","prevMsg":"Initialising Security authentication plugin."}
{"type":"log","@timestamp":"2020-11-03T12:43:44Z","tags":["status","plugin:opendistro_security@1.9.0.0","info"],"pid":15744,"state":"yellow","message":"Status changed from yellow to yellow - 'opendistro_security.cookie.secure' is set to false, cookies are transmitted over unsecure HTTP connection. Consider using HTTPS and set this key to 'true'","prevState":"yellow","prevMsg":"Default cookie password detected, please set a password in kibana.yml by setting 'opendistro_security.cookie.password' (min. 32 characters)."}
{"type":"log","@timestamp":"2020-11-03T12:43:44Z","tags":["status","plugin:opendistro_security@1.9.0.0","info"],"pid":15744,"state":"yellow","message":"Status changed from yellow to yellow - Security session management enabled.","prevState":"yellow","prevMsg":"'opendistro_security.cookie.secure' is set to false, cookies are transmitted over unsecure HTTP connection. Consider using HTTPS and set this key to 'true'"}
{"type":"log","@timestamp":"2020-11-03T12:43:44Z","tags":["status","plugin:opendistro_security@1.9.0.0","info"],"pid":15744,"state":"yellow","message":"Status changed from yellow to yellow - Security copy JWT params disabled","prevState":"yellow","prevMsg":"Security session management enabled."}
{"type":"log","@timestamp":"2020-11-03T12:43:44Z","tags":["status","plugin:opendistro_security@1.9.0.0","info"],"pid":15744,"state":"yellow","message":"Status changed from yellow to yellow - Security multitenancy registered.","prevState":"yellow","prevMsg":"Security copy JWT params disabled"}
{"type":"log","@timestamp":"2020-11-03T12:43:44Z","tags":["status","plugin:opendistro_security@1.9.0.0","info"],"pid":15744,"state":"yellow","message":"Status changed from yellow to yellow - Routes for Security configuration GUI registered.","prevState":"yellow","prevMsg":"Security multitenancy registered."}
{"type":"log","@timestamp":"2020-11-03T12:43:44Z","tags":["status","plugin:console_legacy@7.8.0","info"],"pid":15744,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2020-11-03T12:43:44Z","tags":["status","plugin:apm_oss@7.8.0","info"],"pid":15744,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2020-11-03T12:43:44Z","tags":["status","plugin:region_map@7.8.0","info"],"pid":15744,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2020-11-03T12:43:44Z","tags":["status","plugin:ui_metric@7.8.0","info"],"pid":15744,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2020-11-03T12:43:45Z","tags":["listening","info"],"pid":15744,"message":"Server running at http://0.0.0.0:5601"}
{"type":"log","@timestamp":"2020-11-03T12:43:45Z","tags":["status","plugin:opendistro_security@1.9.0.0","info"],"pid":15744,"state":"yellow","message":"Status changed from yellow to yellow - Setting up index template.","prevState":"yellow","prevMsg":"Routes for Security configuration GUI registered."}
{"type":"log","@timestamp":"2020-11-03T12:43:45Z","tags":["info","OpenDistro Security Migration"],"pid":15744,"message":"Starting tenant migration"}
{"type":"error","@timestamp":"2020-11-03T12:43:45Z","tags":["error","migration"],"pid":15744,"level":"error","error":{"message":"Authorization Exception: Authorization Exception","name":"Error","stack":"Error: Authorization Exception: Authorization Exception\n    at respond (C:\\Users\\dhruv\\Desktop\\tools\\elk\\kibana-7.8.0-windows-x86_64\\node_modules\\elasticsearch\\src\\lib\\transport.js:349:15)\n    at checkRespForFailure (C:\\Users\\dhruv\\Desktop\\tools\\elk\\kibana-7.8.0-windows-x86_64\\node_modules\\elasticsearch\\src\\lib\\transport.js:306:7)\n    at HttpConnector.<anonymous> (C:\\Users\\dhruv\\Desktop\\tools\\elk\\kibana-7.8.0-windows-x86_64\\node_modules\\elasticsearch\\src\\lib\\connectors\\http.js:173:7)\n    at IncomingMessage.wrapper (C:\\Users\\dhruv\\Desktop\\tools\\elk\\kibana-7.8.0-windows-x86_64\\node_modules\\elasticsearch\\node_modules\\lodash\\lodash.js:4929:19)\n    at IncomingMessage.emit (events.js:203:15)\n    at endReadableNT (_stream_readable.js:1145:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"Authorization Exception: Authorization Exception"}
{"type":"log","@timestamp":"2020-11-03T12:43:45Z","tags":["status","plugin:opendistro_security@1.9.0.0","info"],"pid":15744,"state":"yellow","message":"Status changed from yellow to yellow - Tenant indices migration failed","prevState":"yellow","prevMsg":"Setting up index template."}

This is elasticsearch.log

[2020-11-03T18:11:10,598][INFO ][o.e.n.Node ] [DESKTOP-8P20EIO] version[7.8.0], pid[4816], build[oss/zip/757314695644ea9a1dc2fecd26d1a43856725e65/2020-06-14T19:35:50.234439Z], OS[Windows 10/10.0/amd64], JVM[Oracle Corporation/Java HotSpot™ 64-Bit Server VM/14.0.2/14.0.2+12-46]
[2020-11-03T18:11:10,621][INFO ][o.e.n.Node ] [DESKTOP-8P20EIO] JVM home [C:\Program Files\Java\jdk-14.0.2]
[2020-11-03T18:11:10,652][INFO ][o.e.n.Node ] [DESKTOP-8P20EIO] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=C:\Users\dhruv\AppData\Local\Temp\elasticsearch, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Delasticsearch, -Des.path.home=C:\Users\dhruv\Desktop\tools\elk\elasticsearch-7.8.0, -Des.path.conf=C:\Users\dhruv\Desktop\tools\elk\elasticsearch-7.8.0\config, -Des.distribution.flavor=oss, -Des.distribution.type=zip, -Des.bundled_jdk=true]
[2020-11-03T18:11:26,433][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [DESKTOP-8P20EIO] ES Config path is C:\Users\dhruv\Desktop\tools\elk\elasticsearch-7.8.0\config
[2020-11-03T18:11:31,544][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [DESKTOP-8P20EIO] JVM supports TLSv1.3
[2020-11-03T18:11:31,567][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [DESKTOP-8P20EIO] Config directory is C:\Users\dhruv\Desktop\tools\elk\elasticsearch-7.8.0\config/, from there the key- and truststore files are resolved relatively
[2020-11-03T18:11:34,205][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [DESKTOP-8P20EIO] TLS Transport Client Provider : JDK
[2020-11-03T18:11:34,207][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [DESKTOP-8P20EIO] TLS Transport Server Provider : JDK
[2020-11-03T18:11:34,208][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [DESKTOP-8P20EIO] TLS HTTP Provider : null
[2020-11-03T18:11:34,209][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [DESKTOP-8P20EIO] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2, TLSv1.1]
[2020-11-03T18:11:34,211][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [DESKTOP-8P20EIO] Enabled TLS protocols for HTTP layer :
[2020-11-03T18:11:35,930][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [DESKTOP-8P20EIO] Clustername: elasticsearch
[2020-11-03T18:11:37,716][INFO ][c.a.o.j.JobSchedulerPlugin] [DESKTOP-8P20EIO] Loaded scheduler extension: opendistro-managed-index, index: .opendistro-ism-config
[2020-11-03T18:11:37,749][INFO ][o.e.p.PluginsService ] [DESKTOP-8P20EIO] loaded module [aggs-matrix-stats]
[2020-11-03T18:11:37,752][INFO ][o.e.p.PluginsService ] [DESKTOP-8P20EIO] loaded module [analysis-common]
[2020-11-03T18:11:37,754][INFO ][o.e.p.PluginsService ] [DESKTOP-8P20EIO] loaded module [geo]
[2020-11-03T18:11:37,756][INFO ][o.e.p.PluginsService ] [DESKTOP-8P20EIO] loaded module [ingest-common]
[2020-11-03T18:11:37,759][INFO ][o.e.p.PluginsService ] [DESKTOP-8P20EIO] loaded module [ingest-geoip]
[2020-11-03T18:11:37,761][INFO ][o.e.p.PluginsService ] [DESKTOP-8P20EIO] loaded module [ingest-user-agent]
[2020-11-03T18:11:37,764][INFO ][o.e.p.PluginsService ] [DESKTOP-8P20EIO] loaded module [kibana]
[2020-11-03T18:11:37,769][INFO ][o.e.p.PluginsService ] [DESKTOP-8P20EIO] loaded module [lang-expression]
[2020-11-03T18:11:37,771][INFO ][o.e.p.PluginsService ] [DESKTOP-8P20EIO] loaded module [lang-mustache]
[2020-11-03T18:11:37,773][INFO ][o.e.p.PluginsService ] [DESKTOP-8P20EIO] loaded module [lang-painless]
[2020-11-03T18:11:37,775][INFO ][o.e.p.PluginsService ] [DESKTOP-8P20EIO] loaded module [mapper-extras]
[2020-11-03T18:11:37,780][INFO ][o.e.p.PluginsService ] [DESKTOP-8P20EIO] loaded module [parent-join]
[2020-11-03T18:11:37,785][INFO ][o.e.p.PluginsService ] [DESKTOP-8P20EIO] loaded module [percolator]
[2020-11-03T18:11:37,788][INFO ][o.e.p.PluginsService ] [DESKTOP-8P20EIO] loaded module [rank-eval]
[2020-11-03T18:11:37,790][INFO ][o.e.p.PluginsService ] [DESKTOP-8P20EIO] loaded module [reindex]
[2020-11-03T18:11:37,793][INFO ][o.e.p.PluginsService ] [DESKTOP-8P20EIO] loaded module [repository-url]
[2020-11-03T18:11:37,795][INFO ][o.e.p.PluginsService ] [DESKTOP-8P20EIO] loaded module [tasks]
[2020-11-03T18:11:37,797][INFO ][o.e.p.PluginsService ] [DESKTOP-8P20EIO] loaded module [transport-netty4]
[2020-11-03T18:11:37,800][INFO ][o.e.p.PluginsService ] [DESKTOP-8P20EIO] loaded plugin [opendistro-job-scheduler]
[2020-11-03T18:11:37,804][INFO ][o.e.p.PluginsService ] [DESKTOP-8P20EIO] loaded plugin [opendistro_index_management]
[2020-11-03T18:11:37,806][INFO ][o.e.p.PluginsService ] [DESKTOP-8P20EIO] loaded plugin [opendistro_security]
[2020-11-03T18:11:38,140][INFO ][o.e.e.NodeEnvironment ] [DESKTOP-8P20EIO] using [1] data paths, mounts [[OS (C:)]], net usable_space [642.3gb], net total_space [1.6tb], types [NTFS]
[2020-11-03T18:11:38,145][INFO ][o.e.e.NodeEnvironment ] [DESKTOP-8P20EIO] heap size [1gb], compressed ordinary object pointers [true]
[2020-11-03T18:11:41,720][INFO ][o.e.n.Node ] [DESKTOP-8P20EIO] node name [DESKTOP-8P20EIO], node ID [O48TsiO9TGCaOqJqiJhgDQ], cluster name [elasticsearch]
[2020-11-03T18:11:53,028][WARN ][c.a.o.s.c.Salt ] [DESKTOP-8P20EIO] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2020-11-03T18:11:53,326][INFO ][c.a.o.s.a.i.AuditLogImpl ] [DESKTOP-8P20EIO] Message routing enabled: true
[2020-11-03T18:11:54,347][INFO ][c.a.o.s.f.OpenDistroSecurityFilter] [DESKTOP-8P20EIO] indices are made immutable.
[2020-11-03T18:11:56,351][INFO ][o.e.d.DiscoveryModule ] [DESKTOP-8P20EIO] using discovery type [zen] and seed hosts providers [settings]
[2020-11-03T18:11:58,101][INFO ][o.e.n.Node ] [DESKTOP-8P20EIO] initialized
[2020-11-03T18:11:58,102][INFO ][o.e.n.Node ] [DESKTOP-8P20EIO] starting …
[2020-11-03T18:11:59,602][INFO ][o.e.t.TransportService ] [DESKTOP-8P20EIO] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2020-11-03T18:12:01,773][WARN ][o.e.b.BootstrapChecks ] [DESKTOP-8P20EIO] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2020-11-03T18:12:01,784][INFO ][o.e.c.c.Coordinator ] [DESKTOP-8P20EIO] cluster UUID [bIoP5N_uRqiBkyWQsX_HfA]
[2020-11-03T18:12:01,889][INFO ][o.e.c.c.ClusterBootstrapService] [DESKTOP-8P20EIO] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2020-11-03T18:12:02,488][INFO ][o.e.c.s.MasterService ] [DESKTOP-8P20EIO] elected-as-master ([1] nodes joined)[{DESKTOP-8P20EIO}{O48TsiO9TGCaOqJqiJhgDQ}{mI8TjNpGQ3iHsz5jKj9PfQ}{127.0.0.1}{127.0.0.1:9300}{dimr} elect leader, BECOME_MASTER_TASK, FINISH_ELECTION], term: 13, version: 258, delta: master node changed {previous , current [{DESKTOP-8P20EIO}{O48TsiO9TGCaOqJqiJhgDQ}{mI8TjNpGQ3iHsz5jKj9PfQ}{127.0.0.1}{127.0.0.1:9300}{dimr}]}
[2020-11-03T18:12:03,071][INFO ][o.e.c.s.ClusterApplierService] [DESKTOP-8P20EIO] master node changed {previous , current [{DESKTOP-8P20EIO}{O48TsiO9TGCaOqJqiJhgDQ}{mI8TjNpGQ3iHsz5jKj9PfQ}{127.0.0.1}{127.0.0.1:9300}{dimr}]}, term: 13, version: 258, reason: Publication{term=13, version=258}
[2020-11-03T18:12:03,893][INFO ][o.e.h.AbstractHttpServerTransport] [DESKTOP-8P20EIO] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
[2020-11-03T18:12:03,895][INFO ][o.e.n.Node ] [DESKTOP-8P20EIO] started
[2020-11-03T18:12:03,898][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [DESKTOP-8P20EIO] Node started
[2020-11-03T18:12:03,900][INFO ][c.a.o.s.c.ConfigurationRepository] [DESKTOP-8P20EIO] Will attempt to create index .opendistro_security and default configs if they are absent
[2020-11-03T18:12:03,904][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [DESKTOP-8P20EIO] 3 Open Distro Security modules loaded so far: [Module [type=REST_MANAGEMENT_API, implementing class=com.amazon.opendistroforelasticsearch.security.dlic.rest.api.OpenDistroSecurityRestApiActions], Module [type=MULTITENANCY, implementing class=com.amazon.opendistroforelasticsearch.security.configuration.PrivilegesInterceptorImpl], Module [type=AUDITLOG, implementing class=com.amazon.opendistroforelasticsearch.security.auditlog.impl.AuditLogImpl]]
[2020-11-03T18:12:03,904][INFO ][c.a.o.s.c.ConfigurationRepository] [DESKTOP-8P20EIO] Background init thread started. Install default config?: true
[2020-11-03T18:12:03,937][INFO ][o.e.g.GatewayService ] [DESKTOP-8P20EIO] recovered [17] indices into cluster_state
[2020-11-03T18:12:06,699][INFO ][c.a.o.s.c.ConfigurationRepository] [DESKTOP-8P20EIO] Index .opendistro_security already exists
[2020-11-03T18:12:16,068][ERROR][c.a.o.s.a.BackendRegistry] [DESKTOP-8P20EIO] Not yet initialized (you may need to run securityadmin)
[2020-11-03T18:12:25,432][INFO ][stdout ] [DESKTOP-8P20EIO] [FINE] No subscribers registered for event class com.amazon.opendistroforelasticsearch.security.securityconf.DynamicConfigFactory$NodesDnModelImpl
[2020-11-03T18:12:25,486][INFO ][stdout ] [DESKTOP-8P20EIO] [FINE] No subscribers registered for event class org.greenrobot.eventbus.NoSubscriberEvent
[2020-11-03T18:12:25,515][INFO ][c.a.o.s.c.ConfigurationRepository] [DESKTOP-8P20EIO] Hot-reloading of audit configuration is disabled. Using configuration with defaults from elasticsearch settings. Populate the configuration in index using audit.yml or securityadmin to enable it.
[2020-11-03T18:12:25,523][INFO ][c.a.o.s.a.i.AuditLogImpl ] [DESKTOP-8P20EIO] Auditing on REST API is enabled.
[2020-11-03T18:12:25,532][INFO ][c.a.o.s.a.i.AuditLogImpl ] [DESKTOP-8P20EIO] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing.
[2020-11-03T18:12:25,535][INFO ][c.a.o.s.a.i.AuditLogImpl ] [DESKTOP-8P20EIO] Auditing on Transport API is enabled.
[2020-11-03T18:12:25,543][INFO ][c.a.o.s.a.i.AuditLogImpl ] [DESKTOP-8P20EIO] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing.
[2020-11-03T18:12:25,552][INFO ][c.a.o.s.a.i.AuditLogImpl ] [DESKTOP-8P20EIO] Auditing of request body is enabled.
[2020-11-03T18:12:25,556][INFO ][c.a.o.s.a.i.AuditLogImpl ] [DESKTOP-8P20EIO] Bulk requests resolution is disabled during request auditing.
[2020-11-03T18:12:25,559][INFO ][c.a.o.s.a.i.AuditLogImpl ] [DESKTOP-8P20EIO] Index resolution is enabled during request auditing.
[2020-11-03T18:12:25,564][INFO ][c.a.o.s.a.i.AuditLogImpl ] [DESKTOP-8P20EIO] Sensitive headers auditing is enabled.
[2020-11-03T18:12:25,569][INFO ][c.a.o.s.a.i.AuditLogImpl ] [DESKTOP-8P20EIO] Auditing requests from kibanaserver users is disabled.
[2020-11-03T18:12:25,595][WARN ][c.a.o.s.a.r.AuditMessageRouter] [DESKTOP-8P20EIO] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTERNAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint
[2020-11-03T18:12:25,597][INFO ][c.a.o.s.a.i.AuditLogImpl ] [DESKTOP-8P20EIO] Auditing of external configuration is disabled.
[2020-11-03T18:12:25,605][INFO ][c.a.o.s.a.i.AuditLogImpl ] [DESKTOP-8P20EIO] Auditing of internal configuration is disabled.
[2020-11-03T18:12:25,607][INFO ][c.a.o.s.a.i.AuditLogImpl ] [DESKTOP-8P20EIO] Auditing only metadata information for read request is disabled.
[2020-11-03T18:12:25,609][INFO ][c.a.o.s.a.i.AuditLogImpl ] [DESKTOP-8P20EIO] Auditing will watch {} for read requests.
[2020-11-03T18:12:25,612][INFO ][c.a.o.s.a.i.AuditLogImpl ] [DESKTOP-8P20EIO] Auditing read operation requests from kibanaserver users is disabled.
[2020-11-03T18:12:25,616][INFO ][c.a.o.s.a.i.AuditLogImpl ] [DESKTOP-8P20EIO] Auditing only metadata information for write request is disabled.
[2020-11-03T18:12:25,618][INFO ][c.a.o.s.a.i.AuditLogImpl ] [DESKTOP-8P20EIO] Auditing diffs for write requests is disabled.
[2020-11-03T18:12:25,622][INFO ][c.a.o.s.a.i.AuditLogImpl ] [DESKTOP-8P20EIO] Auditing write operation requests from kibanaserver users is disabled.
[2020-11-03T18:12:25,626][INFO ][c.a.o.s.a.i.AuditLogImpl ] [DESKTOP-8P20EIO] Auditing will watch for write requests.
[2020-11-03T18:12:25,628][INFO ][c.a.o.s.a.i.AuditLogImpl ] [DESKTOP-8P20EIO] .opendistro_security is used as internal security index.
[2020-11-03T18:12:25,638][INFO ][c.a.o.s.a.i.AuditLogImpl ] [DESKTOP-8P20EIO] Internal index used for posting audit logs is null
[2020-11-03T18:12:25,639][INFO ][c.a.o.s.c.ConfigurationRepository] [DESKTOP-8P20EIO] Node ‘DESKTOP-8P20EIO’ initialized

This are the logs. In elasticsearch, I have run securityadmin_demo.sh file but still it is giving same error.
Not yet initialized (you may need to run securityadmin)

Seems like you have permissions problems ?
Might check on this?
Maybe you should give more information about your infrastructure so it’s easier to understand.

@ThibaudF I am not sure what exact details you want. I have shared both ymls of elasticsearch and kibana. Is the permission problem in Folder Structure of installation files or Users created in Kibana ?

Like number of nodes ?
And it’s Windows I never tried Windows cluster.

You should check folder permissions yes?
In linux, I would give owner rights to elasticsearch for the folders associated to els, kibana to kibana folders.

I have single node architecture. I am using Windows 10. I was able to solve it by changing below line in config.yml

server_username: kibanaserver

I created users and tenants as per below document

But after logging from the created users, I am not able to access elasticsearch data.
It is showing “No elasticsearch data found”.

Great you found the source of your problem, sorry I didn’t touch config file, I don’t know what is server_username.

Are you sure you have data?

Check this ?
https://elasticsearch_url:9200/_cat/indices?v

My investigation after a long hunt with this issue (still using OD security 0.10) tells me that the migration_tenants.js in OD security for kibana plugin is using an internal user to access _opendistro/_security/tenantinfo.

This internal user is nothing more and nothing less on what you declare as elasticsearch.username in your kibana.yml. This user should map into a role (e.g. kibana_server) in your roles.yml and rolesmapping.yml for the elasticsearch opendistro security plugin.

In addition this role needs to be enabled in your elasticsearch.yml to get access to the opendistro security API via:
opendistro_security.restapi.roles_enabled: ["kibana_server"]

And this worked for me.

In summary, the access control of kibana to the OpenDistro Security API is split in three places:

  • The user used to access the API in elasticsearch.username in kibana.yml
  • The roles and roles mapping for this user in roles.yml and rolesmapping.yml on elasticsearch
  • The access control of the user’s role to in opendistro_security.restapi.roles_enabled in elasticsearch.yml

HI @dhruvil7doshi Did you manage to fully resolve your issue?