# Securityadmin error when initializing the cluster **URL:** \ **Category:** Security\ **Tags:** discuss, troubleshoot, configure, security-issue\ **Created:** [November 7, 2024, 1:56am UTC](https://forum.opensearch.org/t/securityadmin-error-when-initializing-the-cluster/22324 "2024-11-07T01:56:07Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![yeonghyeonKo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/yeonghyeonko/32/8202_2.png) [@yeonghyeonKo](https://forum.opensearch.org/u/yeonghyeonKo)\ **Post date:** [November 7, 2024, 1:56am UTC](https://forum.opensearch.org/t/securityadmin-error-when-initializing-the-cluster/22324/1 "2024-11-07T01:56:07Z")
**Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser): - [opensearch-operator-2.5.1](https://artifacthub.io/packages/helm/opensearch-operator/opensearch-operator) (helm chart) - [opensearch-cluster-2.6.0](https://artifacthub.io/packages/helm/opensearch-operator/opensearch-cluster) (helm chart) - [opensearch-2.15.0](https://hub.docker.com/r/opensearchproject/opensearch) (docker image used by opensearch-cluster-2.6.0) - [cert-manager-1.6.3](https://artifacthub.io/packages/helm/cert-manager/cert-manager/1.6.3) (helm chart) ## PurPose: - I want to replace auto-generated certificates(CA, admin, transport, http) to self-signed certificates made by **[cert-manager](https://artifacthub.io/packages/helm/cert-manager/cert-manager/1.6.3)**. Since I tried to copy the architecture/contents of previous certificates, Subject(OU, CN) and DNS in SAN are exactly same. - I assumed that if all contents of certificates between auto-generated and self-seigned have no difference, we don’t have to change **opensearch.yml**. ## What I’ve Done: ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/f/f5ac16e3f437a3724ddcbf75c4549b084916345c.png) - With manifests for Issuers/Certificates CRDs of cert-manager, the below Secrets had been created. ```auto $ kubectl get certificates -n test-opensearch-cluster-1 -o wide NAME READY SECRET ISSUER STATUS AGE test-opensearch-cluster-1-admin-certificate True test-opensearch-cluster-1-admin-cert ca-issuer Certificate is up to date and has not expired 17h test-opensearch-cluster-1-ca-certificate True test-opensearch-cluster-1-ca self-signed-issuer Certificate is up to date and has not expired 17h test-opensearch-cluster-1-transport-certificate True test-opensearch-cluster-1-transport-cert ca-issuer Certificate is up to date and has not expired 17h test-opensearch-cluster-1-http-certificate True test-opensearch-cluster-1-http-cert ca-issuer Certificate is up to date and has not expired 17h ``` ```yaml # k get secret test-opensearch-cluster-1-admin-cert -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -text -noout Certificate: Data: Version: 3 (0x2) Serial Number: ... Signature Algorithm: sha256WithRSAEncryption Issuer: CN = test-opensearch-cluster-1 Validity Not Before: Nov 6 08:26:58 2024 GMT Not After : Nov 4 08:26:58 2034 GMT Subject: OU = test-opensearch-cluster-1, CN = admin Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public-Key: (2048 bit) Modulus: ... Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Basic Constraints: critical CA:FALSE X509v3 Authority Key Identifier: keyid:[CA's keyId] ``` ```yaml # k get secret test-opensearch-cluster-1-transport-cert -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -text -noout Certificate: Data: Version: 3 (0x2) Serial Number: ... Signature Algorithm: sha256WithRSAEncryption Issuer: CN = test-opensearch-cluster-1 Validity Not Before: Nov 6 08:26:58 2024 GMT Not After : Nov 4 08:26:58 2034 GMT Subject: OU = test-opensearch-cluster-1, CN = test-opensearch-cluster-1 Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public-Key: (2048 bit) Modulus: ... Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication X509v3 Basic Constraints: critical CA:FALSE X509v3 Authority Key Identifier: keyid:[CA's keyId] X509v3 Subject Alternative Name: DNS:test-opensearch-cluster-1, DNS:test-opensearch-cluster-1.test-opensearch-cluster-1, DNS:test-opensearch-cluster-1.test-opensearch-cluster-1.svc, DNS:test-opensearch-cluster-1.test-opensearch-cluster-1.svc.cluster.local ``` ```yaml # k get secret test-opensearch-cluster-1-http-cert -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -text -noout Certificate: Data: Version: 3 (0x2) Serial Number: ... Signature Algorithm: sha256WithRSAEncryption Issuer: CN = test-opensearch-cluster-1 Validity Not Before: Nov 6 08:26:58 2024 GMT Not After : Nov 4 08:26:58 2034 GMT Subject: OU = test-opensearch-cluster-1, CN = test-opensearch-cluster-1 Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public-Key: (2048 bit) Modulus: ... Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication X509v3 Basic Constraints: critical CA:FALSE X509v3 Authority Key Identifier: keyid:[CA's keyId] X509v3 Subject Alternative Name: DNS:test-opensearch-cluster-1, DNS:test-opensearch-cluster-1-discovery, DNS:test-opensearch-cluster-1.test-opensearch-cluster-1, DNS:test-opensearch-cluster-1.test-opensearch-cluster-1.svc, DNS:test-opensearch-cluster-1.test-opensearch-cluster-1.svc.cluster.local ``` ## Problems - K8s Resources returned errors related with ```bash # test-opensearch-cluster-1-bootstrap-0 [ERROR][o.o.s.a.BackendRegistry] [test-opensearch-cluster-1-bootstrap-0] Not yet initialized (you may need to run securityadmin) [ERROR][o.o.s.t.SecurityRequestHandler] [test-opensearch-cluster-1-bootstrap-0] OpenSearchException[Transport client authentication no longer supported.] # test-opensearch-cluster-1-master-0 [WARN][o.o.d.HandshakingTransportAddressConnector] [test-opensearch-cluster-1-master-0] handshake failed for [connectToRemoteMasterNode[10.251.197.154:9300]] org.opensearch.transport.RemoteTransportException: [test-opensearch-cluster-1-ml-0][10.251.197.154:9300][internal:transport/handshake] Caused by: org.opensearch.OpenSearchException: Transport client authentication no longer supported. at org.opensearch.security.ssl.util.ExceptionUtils.createTransportClientNoLongerSupportedException(ExceptionUtils.java:80) ~[?:?] ... [ERROR][o.o.s.t.SecurityRequestHandler] [test-opensearch-cluster-1-master-0] OpenSearchException[Transport client authentication no longer supported.] # test-opensearch-cluster-1-securityconfig-update Waiting to connect to the cluster OpenSearch Security not initialized. ************************************************************************** **This tool will be deprecated in the next major release of OpenSearch** **https://github.com/opensearch-project/security/issues/1755** ************************************************************************** Security Admin v7 Will connect to test-opensearch-cluster-1.test-opensearch-cluster-1.svc.cluster.local:9200 ... done Connected as "CN=admin,OU=test-opensearch-cluster-1" ERR: "CN=admin,OU=test-opensearch-cluster-1" is not an admin user Seems you use a client certificate but this one is not registered as admin_dn Make sure opensearch.yml on all nodes contains: plugins.security.authcz.admin_dn: - ""CN=admin,OU=test-opensearch-cluster-1"" ``` - **opensearch.yml** from each Nodes (This is auto-genereated somehow… don’t know why) ```yaml plugins.security.audit.type: internal_opensearch plugins.security.authcz.admin_dn: ["CN=admin","OU=test-opensearch-cluster-1"] plugins.security.check_snapshot_restore_write_privileges: true plugins.security.enable_snapshot_restore_privilege: true plugins.security.nodes_dn: ["CN=test-opensearch-cluster-1","OU=test-opensearch-cluster-1"] plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"] plugins.security.ssl.http.enabled: true plugins.security.ssl.http.pemcert_filepath: tls-http/tls.crt plugins.security.ssl.http.pemkey_filepath: tls-http/tls.key plugins.security.ssl.http.pemtrustedcas_filepath: tls-http/ca.crt plugins.security.ssl.transport.enforce_hostname_verification: false plugins.security.ssl.transport.pemcert_filepath: tls-transport/tls.crt plugins.security.ssl.transport.pemkey_filepath: tls-transport/tls.key plugins.security.ssl.transport.pemtrustedcas_filepath: tls-transport/ca.crt plugins.security.system_indices.enabled: true plugins.security.system_indices.indices: [".opendistro-alerting-config",".opendistro-alerting-alert*",".opendistro-anomaly-results*",".opendistro-anomaly-detector*",".opendistro-anomaly-checkpoints",".opendistro-anomaly-detection-state",".opendistro-reports-*",".opendistro-notifications-*",".opendistro-notebooks",".opensearch-observability",".opendistro-asynchronous-search-response*",".replication-metadata-store"] ``` ---
**Author:** ![yeonghyeonKo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/yeonghyeonko/32/8202_2.png) [@yeonghyeonKo](https://forum.opensearch.org/u/yeonghyeonKo)\ **Post date:** [November 7, 2024, 5:51am UTC](https://forum.opensearch.org/t/securityadmin-error-when-initializing-the-cluster/22324/2 "2024-11-07T05:51:31Z")
## Self-Answering - By comparing two opensearch.yml (Cluster **A** ’s certificates are created by cert-manager and the other Cluster **B** ’s are by operator), I found that `authcz.admin_dn` and `nodes_dn` settings are different. ```yaml # Cluster A : test-opensearch-cluster-1 security: tls: transport: generate: false perNode: false secret: name: test-opensearch-cluster-1-transport-cert caSecret: name: test-opensearch-cluster-1-ca nodesDn: [CN=test-opensearch-cluster-1,OU=test-opensearch-cluster-1] adminDn: [CN=admin,OU=test-opensearch-cluster-1] http: generate: false secret: name: test-opensearch-cluster-1-http-cert ``` ```yaml # Cluster B : test-opensearch-cluster-2 security: tls: transport: generate: true perNode: true http: generate: true ``` * * * As printing two opensearch.yml, DNs for the former have been splitted by double-quotes but the latter’s have been combined with a single double-quote. (See the below:) ```auto $ k get cm test-opensearch-cluster-1-config -o yaml | grep plugins.security.nodes_dn --context=3 plugins.security.authcz.admin_dn: ["CN=admin","OU=test-opensearch-cluster-1"] plugins.security.check_snapshot_restore_write_privileges: true plugins.security.enable_snapshot_restore_privilege: true plugins.security.nodes_dn: ["CN=test-opensearch-cluster-1","OU=test-opensearch-cluster-1"] plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"] plugins.security.ssl.http.enabled: true plugins.security.ssl.http.pemcert_filepath: tls-http/tls.crt $ k get cm test-opensearch-cluster-2-config -o yaml -n test-opensearch-cluster-2 | grep plugins.security.nodes_dn --context=3 plugins.security.authcz.admin_dn: ["CN=admin,OU=test-opensearch-cluster-2"] plugins.security.check_snapshot_restore_write_privileges: true plugins.security.enable_snapshot_restore_privilege: true plugins.security.nodes_dn: ["CN=test-opensearch-cluster-2,OU=test-opensearch-cluster-2"] plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"] plugins.security.ssl.http.enabled: true plugins.security.ssl.http.pemcert_filepath: tls-http/tls.crt ``` ### Today’s Lesson - Watch OUT **Array** type and **Double Quot** e in yaml (Especially, If you use a single transport certificate across nodes.)