Opensearch cluster : OpenSearch Security not initialized

Security
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OS : Rocky Linux 9
Version : Opensearch 2.18
Method : rpm

IP Cluster : 10.251.0.207
IP Node 1 : 10.251.0.137

Describe the issue:
Hello,

I try to create an opensearch cluster. Everything seems to be ok, except while i browse : https://opensearch-cluster.local:9200 i get :
OpenSearch Security not initialized

My opensearch.yml :

#### cluster part ####
cluster.name: opensearch-cluster
node.name: opensearch-cluster_manager
node.roles: ["master"]
#node.roles: [ cluster_manager ]
cluster.initial_cluster_manager_nodes: ["opensearch-cluster_manager"]
discovery.seed_hosts: ["10.251.0.130"]
#### standard part ####
network.bind_host: 10.251.0.207
network.host: 0.0.0.0
  #discovery.type: single-node
plugins.security.disabled: false
plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/node1.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/node1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: /etc/opensearch/node1.pem
plugins.security.ssl.http.pemkey_filepath: /etc/opensearch/node1-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - 'CN=ADMIN,OU=AXI,O=AXI,L=PARIS,ST=PARIS,C=FR'
plugins.security.nodes_dn:
  #  - 'CN=opensearch-cluster.local,OU=AXIGATE,O=AXI,L=PARIS,ST=PARIS,C=FR'
  - 'CN=10.251.0.130,OU=AXI,O=AXI,L=PARIS,ST=PARIS,C=FR'
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]

I usually install only one VM with opensearch, opensearch-dashboards, logstash.
It’ok with the default installation and the “#standard part” in opensearch.yml

Actually, the log says : [opensearch-cluster_manager] Not yet initialized (you may need to run securityadmin)
But i can’t because when i try to user securityadmin i get this error :

/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -cd /etc/opensearch/opensearch-security -rev -icl -nhnv    -cacert /etc/opensearch/root-ca.pem -cert /etc/opensearch/admin.pem -key /etc/opensearch/admin-key.pem -h opensearch-cluster.local -p 9200 --accept-red-cluster
Security Admin v7
Will connect to opensearch-cluster.local:9200 ... done
Connected as "CN=ADMIN,OU=AXI,O=AXI,L=PARIS,ST=PARIS,C=FR"
OpenSearch Version: 2.18.0
Contacting opensearch cluster 'opensearch' ...
Clustername: opensearch-cluster
Clusterstate: RED
Number of nodes: 1
Number of data nodes: 0
.opendistro_security index already exists, so we do not need to create one.
ERR: .opendistro_security index state is RED.
Populate config from /etc/opensearch/opensearch-security/
Will update '/config' with /etc/opensearch/opensearch-security/config.yml
   FAIL: Configuration for 'config' failed because of java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-6 [ACTIVE]

Any ideas ?

Nico

2

Hi @NicoLef ,

How many OpenSearch nodes do you have? Where did you run the securityadmin tool? Have you checked if the IP addresses are valid in discovery.seed_hosts and network.bind_host?

3

Hello @Eugene7,

I admit i’m pretty new to opensearch.

How many OpenSearch nodes do you have?
2
One as cluster, and one as data.
The data’s node is stopped for the moment.

Where did you run the securityadmin tool?
I ran securityadmin tool logged as root on directly on the cluster’s console (ssh 10.251.0.207).

Have you checked if the IP addresses are valid in discovery.seed_hosts and network.bind_host ?
Yes
discovery.seed_hosts contains the ip address of the node’s data (10.251.0.130)
network.bind_host contains the ip address of the cluster

I don’t really understand this initialisation error on the cluster.

Do i need to do some operations with the certificates between cluster and node data like copying certificates on each host.
I didn’t find real informations about it in the documentation.

4

Hey NicoLef, if this is a fresh, one-node cluster for testing can you remove the data/ directory, set plugins.security.allow_default_init_securityindex to false and then start the node again?

Once the node is started you can run securityadmin to seed the security index.

Alternatively, if you have plugins.security.allow_default_init_securityindex set to true then you can put the initial security configuration yaml files in config/opensearch-security and they will be automatically sourced into the security index.

Please ensure the the host has sufficient storage. If the host is above the high watermark then there will be a Write block on the cluster which could prevent index creation for the security index or fail to source the initial config into the index from yaml files.

5

Hello @cwperks,

I follow your rules and it fail while trying to update :

[root@osearch-cluster rocky]# /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -cd /etc/opensearch/opensearch-security -rev -icl -nhnv    -cacert /etc/opensearch/root-ca.pem -cert /etc/opensearch/admin.pem -key /etc/opensearch/admin-key.pem -h 10.251.0.207 -p 9200 --accept-red-cluster
Security Admin v7
Will connect to 10.251.0.207:9200 ... done
Connected as "CN=ROOT,OU=AXI,O=AXI,L=PARIS,ST=PARIS,C=FR"
OpenSearch Version: 2.18.0
Contacting opensearch cluster 'opensearch' ...
Clustername: opensearch-cluster
Clusterstate: RED
Number of nodes: 1
Number of data nodes: 0
.opendistro_security index does not exists, attempt to create it ... done (0-all replicas)
Populate config from /etc/opensearch/opensearch-security/
Will update '/config' with /etc/opensearch/opensearch-security/config.yml
   FAIL: Configuration for 'config' failed because of java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-6 [ACTIVE]

Sample logs during the command (The data’s node is disable, too many errors about certificates) :

[2024-12-07T01:22:52,452][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster_manager] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@4adb3bcb] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2024-12-07T01:22:52,452][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster_manager] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@4adb3bcb] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2024-12-07T01:22:57,393][WARN ][r.suppressed             ] [opensearch-cluster_manager] path: /.opendistro_security/_doc/whitelist, params: {refresh=true, index=.opendistro_security, id=whitelist, timeout=1m}
org.opensearch.action.UnavailableShardsException: [.opendistro_security][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.opendistro_security][0]] containing [index {[.opendistro_security][whitelist], source[{"whitelist":"eyJfbWV0YSI6eyJ0eXBlIjoid2hpdGVsaXN0IiwiY29uZmlnX3ZlcnNpb24iOjJ9LCJjb25maWciOnsiZW5hYmxlZCI6ZmFsc2UsInJlcXVlc3RzIjp7Ii9fY2x1c3Rlci9zZXR0aW5ncyI6WyJHRVQiXSwiL19jYXQvbm9kZXMiOlsiR0VUIl19fX0="}]}] and a refresh]
        at org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase.retryBecauseUnavailable(TransportReplicationAction.java:1249) [opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:1040) [opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onTimeout(TransportReplicationAction.java:1204) [opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394) [opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294) [opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:742) [opensearch-2.18.0.jar:2.18.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:946) [opensearch-2.18.0.jar:2.18.0]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
[2024-12-07T01:23:05,454][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster_manager] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@49ee983] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2024-12-07T01:23:05,454][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster_manager] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@49ee983] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)

But, everything is fine if i disable the cluster’s configuration.

45 G of disk space are available.

6

Ok i found the problem.
According to this post :

the problem is it’s not possible to start a node with only the role cluster_manager.
In definitive, role need to be : [cluster_manager, data]

Here is my opensearch.yml
Cluster manager => 192.168.3.101

cluster.name: opensearch-cluster
node.name: cluster
network.bind_host: 192.168.3.101
network.publish_host: 192.168.3.101
network.host: 192.168.3.101
http.port: 9200
node.roles: [ cluster_manager, data ]
cluster.initial_cluster_manager_nodes: ["cluster"]
discovery.seed_hosts: ["192.168.3.106:9200", "192.168.3.101:9200"]
      #discovery.type: single-node
path.data: /srv/data
#
# Path to log files:
#
path.logs: /srv/logs
plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/node1_ip.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/node1-key_ip.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: /etc/opensearch/node1_ip.pem
plugins.security.ssl.http.pemkey_filepath: /etc/opensearch/node1-key_ip.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - 'CN=A,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
plugins.security.nodes_dn:
  - 'CN=192.168.3.101,OU=AXI,O=AXI,L=PARIS,ST=FRANCE,C=FR'
  - 'CN=192.168.3.106,OU=AXI,O=AXI,L=PARIS,ST=FRANCE,C=FR'
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]

Data node (192.168.3.106)

cluster.name: opensearch-cluster
node.name: node1
network.bind_host: 192.168.3.106
network.host: 192.168.3.106
http.port: 9200
node.roles: [ data ]
cluster.initial_cluster_manager_nodes: ["cluster"]
discovery.seed_hosts: ["192.168.3.101", "192.168.3.106"]
    #discovery.type: single-node
path.data: /srv/data
#
# Path to log files:
#
path.logs: /srv/logs
plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/node1_ip.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/node1-key_ip.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: /etc/opensearch/node1_ip.pem
plugins.security.ssl.http.pemkey_filepath: /etc/opensearch/node1-key_ip.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - 'CN=A,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
plugins.security.nodes_dn:
  - 'CN=192.168.3.101,OU=AXI,O=AXI,L=PARIS,ST=FRANCE,C=FR'
  - 'CN=192.168.3.106,OU=AXI,O=AXI,L=PARIS,ST=FRANCE,C=FR'
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]

There’s probably mistakes, but at the moment it’s the only method to get all nodes seeing each other.

[root@localhost opensearch]# curl -XGET https://192.168.3.101:9200/_cat/nodes?v -u 'admin:******' --insecure
ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role node.roles           cluster_manager name
192.168.3.101           15          34   0    0.15    0.07     0.07 dm        cluster_manager,data *               cluster
192.168.3.106           13          67   0    0.03    0.07     0.10 d         data                 -               node1
[root@localhost opensearch]#