OpenSearch tarball Installation | securityadmin.sh | UnavailableShardsException[[.opendistro_security][0] primary shard is not active Timeout

Security
1

Hello,
We installed OpenSearch on 4 VMs(1 coordinating node, 1 master node and 2 data nodes) and according to documentation Cluster formation - OpenSearch documentation

when we login to OpenSearch URL or via curl, we are getting following msg:

e.g.
[apm@IR-APM-DEV-MN1 config]$ curl -XGET https:// :9200/_cat/plugins?v -u ‘admin:admin’ --insecure

OpenSearch Security not initialized.

According to it and msg we saw “ [opensearch-master] Not yet initialized (you may need to run securityadmin) " , we executed securityadmin script as follows:

./securityadmin.sh -cd …/securityconfig/ -nhnv -cacert …/…/…/config/root-ca.pem -cert …/…/…/config/kirk.pem -key …/…/…/config/kirk-key.pem -h -cn apm-cluster-1 -arc -diagnose

And got following error msg for example:

Will update ‘_doc/config’ with …/securityconfig/config.yml FAIL: Configuration for ‘config’ failed because of UnavailableShardsException[[.opendistro_security][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.opendistro_security][0]] containing [index {[.opendistro_security][_doc][config], source[n/a, actual length: [3.7kb], max length: 2kb]}] and a refresh]] …

Can someone advise if any suggestions to overcome those errors? (primary shard is not active Timeout / increase max length )

Thanks,
Noam

2

@noamsh88
Could you share your config.yml file?

3

Thanks Pablo,
sharing opensearch.yml from each node, highlighting changes made on each yaml

Master node opensearch.yml:

cluster.name: apm-cluster-1
node.name: opensearch-master
node.master: true
node.data: false
node.ingest: false
network.host:
discovery.seed_hosts: [“<Data Node 1 IP>”, “<Data Node 2 IP>”, “<Coordinating Node 2 IP>”]

plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:

  • CN=kirk,OU=client,O=client,L=test, C=de

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [“.opendistro-alerting-config”, “.opendistro-alerting-alert*”, “.opendistro-anomaly-results*”, “.opendistro-anomaly-detector*”, “.opendistro-anomaly-checkpoints”, “.opendistro-anomaly-detection-state”, “.opendistro-reports-", ".opendistro-notifications-”, “.opendistro-notebooks”, “.opendistro-asynchronous-search-response*”, “.replication-metadata-store”]
node.max_local_storage_nodes: 3

plugins.security.disabled: true

Coordinating node opensearch.yml:

cluster.name: apm-cluster-1
node.name: opensearch-c1
node.master: false
node.data: false
node.ingest: false
network.host: <Coordinating Node 2 IP>
discovery.seed_hosts: [“<Data Node 1 IP>”, “<Data Node 2 IP>”, “”]

plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:

  • CN=kirk,OU=client,O=client,L=test, C=de

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [“.opendistro-alerting-config”, “.opendistro-alerting-alert*”, “.opendistro-anomaly-results*”, “.opendistro-anomaly-detector*”, “.opendistro-anomaly-checkpoints”, “.opendistro-anomaly-detection-state”, “.opendistro-reports-", ".opendistro-notifications-”, “.opendistro-notebooks”, “.opendistro-asynchronous-search-response*”, “.replication-metadata-store”]
node.max_local_storage_nodes: 3

plugins.security.disabled: true

Data node 1 opensearch.yml:

cluster.name: apm-cluster-1
node.name: opensearch-d1
node.master: true
node.data: true
node.ingest: true
network.host: <Data Node 1 IP>
discovery.seed_hosts: [“”, “<Data Node 2 IP>”, “<Coordinating Node 2 IP>”]

plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:

  • CN=kirk,OU=client,O=client,L=test, C=de

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [“.opendistro-alerting-config”, “.opendistro-alerting-alert*”, “.opendistro-anomaly-results*”, “.opendistro-anomaly-detector*”, “.opendistro-anomaly-checkpoints”, “.opendistro-anomaly-detection-state”, “.opendistro-reports-", ".opendistro-notifications-”, “.opendistro-notebooks”, “.opendistro-asynchronous-search-response*”, “.replication-metadata-store”]
node.max_local_storage_nodes: 3

plugins.security.disabled: true

Data node 2 opensearch.yml:

cluster.name: apm-cluster-1
node.name: opensearch-d2
node.master: true
node.data: true
node.ingest: true
network.host: <Data Node 2 IP>
discovery.seed_hosts: [“<Data Node 1 IP>”, “”, “<Coordinating Node 2 IP>”]

plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:

  • CN=kirk,OU=client,O=client,L=test, C=de

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [“.opendistro-alerting-config”, “.opendistro-alerting-alert*”, “.opendistro-anomaly-results*”, “.opendistro-anomaly-detector*”, “.opendistro-anomaly-checkpoints”, “.opendistro-anomaly-detection-state”, “.opendistro-reports-", ".opendistro-notifications-”, “.opendistro-notebooks”, “.opendistro-asynchronous-search-response*”, “.replication-metadata-store”]
node.max_local_storage_nodes: 3

plugins.security.disabled: true

4

@noamsh88
Thanks for sharing the opensearch.yml file. However, I’m still missing the config.yml file.

5

@noamsh88

Could you also follow the steps below and share the config.yml file from the results?


1. Create backup folder in /usr/share/opensearch/ folder

        mkdir /usr/share/opensearch/backup

2. Run the following command to retrieve the current security configuration to the backup folder. 

        "/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh" -h localhost -icl -key "/usr/share/opensearch/config/kirk-key.pem" -cert "/usr/share/opensearch/config/kirk.pem" -cacert "/usr/share/opensearch/config/root-ca.pem" -cd "/usr/share/opensearch/backup" --retrieve
6

Thanks Pablo, following is our ~/opensearch-1.1.0/plugins/opensearch-security/securityconfig/config.yml file, please note that we didn’t make changes in it

_meta:
type: “config”
config_version: 2

config:
dynamic:
http:
anonymous_auth_enabled: false
xff:
enabled: false
internalProxies: ‘192.168.0.10|192.168.0.11’ # regex pattern
authc:
kerberos_auth_domain:
http_enabled: false
transport_enabled: false
order: 6
http_authenticator:
type: kerberos
challenge: true
config:
krb_debug: false
strip_realm_from_principal: true
authentication_backend:
type: noop
basic_internal_auth_domain:
description: “Authenticate via HTTP Basic against internal users database”
http_enabled: true
transport_enabled: true
order: 4
http_authenticator:
type: basic
challenge: true
authentication_backend:
type: intern
proxy_auth_domain:
description: “Authenticate via proxy”
http_enabled: false
transport_enabled: false
order: 3
http_authenticator:
type: proxy
challenge: false
config:
user_header: “x-proxy-user”
roles_header: “x-proxy-roles”
authentication_backend:
type: noop
jwt_auth_domain:
description: “Authenticate via Json Web Token”
http_enabled: false
transport_enabled: false
order: 0
http_authenticator:
type: jwt
challenge: false
config:
signing_key: “base64 encoded HMAC key or public RSA/ECDSA pem key”
jwt_header: “Authorization”
jwt_url_parameter: null
roles_key: null
subject_key: null
authentication_backend:
type: noop
clientcert_auth_domain:
description: “Authenticate via SSL client certificates”
http_enabled: false
transport_enabled: false
order: 2
http_authenticator:
type: clientcert
config:
username_attribute: cn
challenge: false
authentication_backend:
type: noop
ldap:
description: “Authenticate via LDAP or Active Directory”
http_enabled: false
transport_enabled: false
order: 5
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: ldap
config:
enable_ssl: false
enable_start_tls: false
enable_ssl_client_auth: false
verify_hostnames: true
hosts:
- :8389
bind_dn: null
password: null
userbase: ‘ou=people,dc=example,dc=com’
usersearch: ‘(sAMAccountName={0})’
username_attribute: null
authz:
roles_from_myldap:
description: “Authorize via LDAP or Active Directory”
http_enabled: false
transport_enabled: false
authorization_backend:
type: ldap
config:
enable_ssl: false
enable_start_tls: false
enable_ssl_client_auth: false
verify_hostnames: true
hosts:
- :8389
bind_dn: null
password: null
rolebase: ‘ou=groups,dc=example,dc=com’
rolesearch: ‘(member={0})’
userroleattribute: null
userrolename: disabled
rolename: cn
resolve_nested_roles: true
userbase: ‘ou=people,dc=example,dc=com’
usersearch: ‘(uid={0})’
roles_from_another_ldap:
description: “Authorize via another Active Directory”
http_enabled: false
transport_enabled: false
authorization_backend:
type: ldap

7

@noamsh88 do you get the same behaviour on all nodes?

8

Thanks for the advise, when executing security admin script to retrieve configuration, we are getting:
following msg in script execution:
[apm@<Master Node Host Name tools]$ ./securityadmin.sh -h -key …/…/…/config/kirk-key.pem -cert …/…/…/config/kirk.pem -cacert …/…/…/config/root-ca.pem -cd “/home/apm/opensearch-1.1.0/plugins/opensearch-security/tools/backup” --retrieve
Security Admin v7
Will connect to <Master Node IP:9300 … done
11:28:54.187 [opensearch[client][transport_worker][T#1]] ERROR org.opensearch.security.ssl.transport.SecuritySSLNettyTransport - Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching <Master Node Host Name found.
javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching <Master Node Host Name found.

and following error msg in OpenSearch log:

[2021-10-31T11:25:42,321][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [opensearch-master] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

9

Hi Pablo,
Updating that was able to execute successfully the security admin script and successfully install opensearch and opensearch dashboard.

I still didn’t understand what root cause for errors mentioned above, but possible reason i can think of that it failed to execute is that since securityadmin.sh script is deploying security configuration using APIs, and when i execute securityadmin script the opensearch was not running (cluster was down)

Anyway, many thanks for your advises and help

1 Like
10

@noamsh88

Regarding the configuration upload. It has to be executed when OpenSearch is fully running as the security plug-in configuration is written in the security index.

Your last ./securityadmi.sh execution with –retrieve option was missing -nhnv option. It ignores hostname verification with node SSL certificate. -h option didn’t have any hostname defined so securityadmin.sh script took localhost as the default option. When securityadmin.sh is matching the hostname it will use the node certificate’s SAN values.