Cannot Initialize the Cluster with Default Settings

bugravibes · September 7, 2024, 6:34pm

Versions : OpenSearch 2.12.0 & RHEL 8.8

Describe the issue: I am trying to initialize my 4 node cluster (1 for coordinating, 1 for manager and 2 for data as described in documentation) and after starting opensearch.service for all of my nodes, the logs say “No data for rolesmapping while retrieving configuration”. The logs:

[2024-09-07T21:26:04,248][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [DATANODE01] No data for internalusers while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT]  (index=.opendistro_security)
[2024-09-07T21:26:04,249][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [DATANODE01] No data for actiongroups while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT]  (index=.opendistro_security)
[2024-09-07T21:26:04,249][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [DATANODE01] No data for config while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT]  (index=.opendistro_security)
[2024-09-07T21:26:04,249][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [DATANODE01] No data for roles while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT]  (index=.opendistro_security)
[2024-09-07T21:26:04,249][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [DATANODE01] No data for rolesmapping while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT]  (index=.opendistro_security)
[2024-09-07T21:26:04,249][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [DATANODE01] No data for tenants while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT]  (index=.opendistro_security)

and when I try to run the securityadmin.sh in my master node with:

/path/to/securityadmin.sh -cd /path/to/opensearch-security/ -nhnv -cacert /path/to/root-ca.pem -cert /path/to/admin.pem -key /path/to/admin-key.pem -h MASTERNODEIP -p 9200 -cn test

I get the output:

**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to MASTERNODEIP:9200 ... done
Connected as "CN=admin,O=***,L=***,C=***"
OpenSearch Version: 2.12.0
Contacting opensearch cluster 'test' and wait for YELLOW clusterstate ...
Clustername: test
Clusterstate: GREEN
Number of nodes: 4
Number of data nodes: 2
.opendistro_security index already exists, so we do not need to create one.
Populate config from /opt/opensearch/opensearch-2.12.0/config/opensearch-security/
Will update '/config' with /opt/opensearch/opensearch-2.12.0/config/opensearch-security/config.yml
   FAIL: Configuration for 'config' failed because of OpenSearchStatusException[OpenSearch exception [type=security_exception, reason=OpenSearch Security not initialized for indices:admin/mapping/auto_put]]
Will update '/roles' with /opt/opensearch/opensearch-2.12.0/config/opensearch-security/roles.yml
   FAIL: Configuration for 'roles' failed because of OpenSearchStatusException[OpenSearch exception [type=security_exception, reason=OpenSearch Security not initialized for indices:admin/mapping/auto_put]]
Will update '/rolesmapping' with /opt/opensearch/opensearch-2.12.0/config/opensearch-security/roles_mapping.yml
   FAIL: Configuration for 'rolesmapping' failed because of OpenSearchStatusException[OpenSearch exception [type=security_exception, reason=OpenSearch Security not initialized for indices:admin/mapping/auto_put]]
Will update '/internalusers' with /opt/opensearch/opensearch-2.12.0/config/opensearch-security/internal_users.yml
   FAIL: Configuration for 'internalusers' failed because of OpenSearchStatusException[OpenSearch exception [type=security_exception, reason=OpenSearch Security not initialized for indices:admin/mapping/auto_put]]
Will update '/actiongroups' with /opt/opensearch/opensearch-2.12.0/config/opensearch-security/action_groups.yml
   FAIL: Configuration for 'actiongroups' failed because of OpenSearchStatusException[OpenSearch exception [type=security_exception, reason=OpenSearch Security not initialized for indices:admin/mapping/auto_put]]
Will update '/tenants' with /opt/opensearch/opensearch-2.12.0/config/opensearch-security/tenants.yml
   FAIL: Configuration for 'tenants' failed because of OpenSearchStatusException[OpenSearch exception [type=security_exception, reason=OpenSearch Security not initialized for indices:admin/mapping/auto_put]]
Will update '/nodesdn' with /opt/opensearch/opensearch-2.12.0/config/opensearch-security/nodes_dn.yml
   FAIL: Configuration for 'nodesdn' failed because of OpenSearchStatusException[OpenSearch exception [type=security_exception, reason=OpenSearch Security not initialized for indices:admin/mapping/auto_put]]
Will update '/whitelist' with /opt/opensearch/opensearch-2.12.0/config/opensearch-security/whitelist.yml
   FAIL: Configuration for 'whitelist' failed because of OpenSearchStatusException[OpenSearch exception [type=security_exception, reason=OpenSearch Security not initialized for indices:admin/mapping/auto_put]]
Will update '/audit' with /opt/opensearch/opensearch-2.12.0/config/opensearch-security/audit.yml
   FAIL: Configuration for 'audit' failed because of OpenSearchStatusException[OpenSearch exception [type=security_exception, reason=OpenSearch Security not initialized for indices:admin/mapping/auto_put]]
Will update '/allowlist' with /opt/opensearch/opensearch-2.12.0/config/opensearch-security/allowlist.yml
   FAIL: Configuration for 'allowlist' failed because of OpenSearchStatusException[OpenSearch exception [type=security_exception, reason=OpenSearch Security not initialized for indices:admin/mapping/auto_put]]
ERR: cannot upload configuration, see errors above

Any idea is appreciated,
Thanks.

Configuration:
opensearch.yml for master node:

cluster.name: test
node.name: MASTERNODE

node.roles: [ cluster_manager ]

network.bind_host: MASTERNODE.IP
network.host: MASTERNODE.IP
network.publish_host: MASTERNODE.IP

path.data: "/var/lib/opensearch/data"

http.port: 9200
transport.port: 9300


cluster.initial_cluster_manager_nodes: ["masternode"]

discovery.seed_hosts: ["COORDINATINGNODE.IP","DATANODE01.IP","DATANODE02.IP"]

plugins.security.ssl.transport.pemcert_filepath: certs/masternode.pem
plugins.security.ssl.transport.pemkey_filepath: certs/masternode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: certs/root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false


plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: certs/masternode.pem
plugins.security.ssl.http.pemkey_filepath: certs/masternode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: certs/root-ca.pem

plugins.security.authcz.admin_dn:
 - 'CN=admin,O=***,L=***,C=***'


plugins.security.nodes_dn:
 - 'CN=COORDINATINGNODE,O=***,L=***,C=***'
 - 'CN=MASTERNODE,O=***,L=***,C=***'
 - 'CN=DATANODE01,O=***,L=***,C=***'
 - 'CN=DATANODE02,O=***,L=***,C=***'

other configs are almost the same except the IP’s and roles ofc.

merlinz01 · September 7, 2024, 8:43pm

Do you have the security plugin installed on all nodes?

bugravibes · September 7, 2024, 9:50pm

I might be considered as a newbie. Can you give more detail. I am taking action according to the official documentation and this forum’s replies but as I understand I build opensearch tar’s on each node with opensearch.yml and only implement securityadmin.sh for master eligible node and it is also effective for other nodes as I understand. Can you bring light to this topic?

Mantas · September 9, 2024, 11:17am

HI @bugravibes,

Can you set the below in your opensearch.yml and see if it fixes it:

plugins.security.allow_default_init_securityindex: true

Best,
mj

bugravibes · September 10, 2024, 7:15am

Hi @Mantas ,
Thanks for the solution, it works fine now.
Should I add it to all of my nodes? I mean it worked when I only added this to my master node.

Mantas · September 10, 2024, 9:45am

@bugravibes, you are welcome, glad to hear it worked. You can set it to all nodes here is more info: Security settings - OpenSearch Documentation

Best,
mj

Topic		Replies	Views
Opensearch configuration OpenSearch configure , install , security-issue	4	1060	June 16, 2023
OpenSearch Security not initialized error in master node Security troubleshoot , configure , install	20	1646	June 7, 2024
Opensearch launch failed on cluster manager OpenSearch	4	1553	October 4, 2023
Trouble configuring OpenDistro Security Index Security	1	1866	May 21, 2021
Setting up security for OpenSearch Security configure , install	3	1573	March 2, 2022

Cannot Initialize the Cluster with Default Settings

Related topics