Securityadmin error when initializing the cluster

Security
, , ,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

PurPose:

  • I want to replace auto-generated certificates(CA, admin, transport, http) to self-signed certificates made by cert-manager. Since I tried to copy the architecture/contents of previous certificates, Subject(OU, CN) and DNS in SAN are exactly same.
  • I assumed that if all contents of certificates between auto-generated and self-seigned have no difference, we don’t have to change opensearch.yml.

What I’ve Done:

image
image1237×614 47.8 KB

  • With manifests for Issuers/Certificates CRDs of cert-manager, the below Secrets had been created.
$ kubectl get certificates -n test-opensearch-cluster-1 -o wide

NAME                                             READY   SECRET                                     ISSUER               STATUS                                          AGE
test-opensearch-cluster-1-admin-certificate      True    test-opensearch-cluster-1-admin-cert       ca-issuer            Certificate is up to date and has not expired   17h
test-opensearch-cluster-1-ca-certificate         True    test-opensearch-cluster-1-ca               self-signed-issuer   Certificate is up to date and has not expired   17h
test-opensearch-cluster-1-transport-certificate  True    test-opensearch-cluster-1-transport-cert   ca-issuer            Certificate is up to date and has not expired   17h
test-opensearch-cluster-1-http-certificate       True    test-opensearch-cluster-1-http-cert        ca-issuer            Certificate is up to date and has not expired   17h

#  k get secret test-opensearch-cluster-1-admin-cert -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            ...
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = test-opensearch-cluster-1
        Validity
            Not Before: Nov  6 08:26:58 2024 GMT
            Not After : Nov  4 08:26:58 2034 GMT
        Subject: OU = test-opensearch-cluster-1, CN = admin
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    ...
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier:
                keyid:[CA's keyId]

#  k get secret test-opensearch-cluster-1-transport-cert -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            ...
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = test-opensearch-cluster-1
        Validity
            Not Before: Nov  6 08:26:58 2024 GMT
            Not After : Nov  4 08:26:58 2034 GMT
        Subject: OU = test-opensearch-cluster-1, CN = test-opensearch-cluster-1
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    ...
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier:
                keyid:[CA's keyId]

            X509v3 Subject Alternative Name:
                DNS:test-opensearch-cluster-1, DNS:test-opensearch-cluster-1.test-opensearch-cluster-1, DNS:test-opensearch-cluster-1.test-opensearch-cluster-1.svc, DNS:test-opensearch-cluster-1.test-opensearch-cluster-1.svc.cluster.local

#  k get secret test-opensearch-cluster-1-http-cert -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            ...
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = test-opensearch-cluster-1
        Validity
            Not Before: Nov  6 08:26:58 2024 GMT
            Not After : Nov  4 08:26:58 2034 GMT
        Subject: OU = test-opensearch-cluster-1, CN = test-opensearch-cluster-1
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    ...
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier:
                keyid:[CA's keyId]

            X509v3 Subject Alternative Name:
                DNS:test-opensearch-cluster-1, DNS:test-opensearch-cluster-1-discovery, DNS:test-opensearch-cluster-1.test-opensearch-cluster-1, DNS:test-opensearch-cluster-1.test-opensearch-cluster-1.svc, DNS:test-opensearch-cluster-1.test-opensearch-cluster-1.svc.cluster.local

Problems

  • K8s Resources returned errors related with
# test-opensearch-cluster-1-bootstrap-0
[ERROR][o.o.s.a.BackendRegistry  ] [test-opensearch-cluster-1-bootstrap-0] Not yet initialized (you may need to run securityadmin)
[ERROR][o.o.s.t.SecurityRequestHandler] [test-opensearch-cluster-1-bootstrap-0] OpenSearchException[Transport client authentication no longer supported.]


# test-opensearch-cluster-1-master-0
[WARN ][o.o.d.HandshakingTransportAddressConnector] [test-opensearch-cluster-1-master-0] handshake failed for [connectToRemoteMasterNode[10.251.197.154:9300]]
org.opensearch.transport.RemoteTransportException: [test-opensearch-cluster-1-ml-0][10.251.197.154:9300][internal:transport/handshake]
Caused by: org.opensearch.OpenSearchException: Transport client authentication no longer supported.
	at org.opensearch.security.ssl.util.ExceptionUtils.createTransportClientNoLongerSupportedException(ExceptionUtils.java:80) ~[?:?]
    ...
[ERROR][o.o.s.t.SecurityRequestHandler] [test-opensearch-cluster-1-master-0] OpenSearchException[Transport client authentication no longer supported.]


# test-opensearch-cluster-1-securityconfig-update
Waiting to connect to the cluster
OpenSearch Security not initialized.**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to test-opensearch-cluster-1.test-opensearch-cluster-1.svc.cluster.local:9200 ... done
Connected as "CN=admin,OU=test-opensearch-cluster-1"
ERR: "CN=admin,OU=test-opensearch-cluster-1" is not an admin user
Seems you use a client certificate but this one is not registered as admin_dn
Make sure opensearch.yml on all nodes contains:
plugins.security.authcz.admin_dn:
  - ""CN=admin,OU=test-opensearch-cluster-1""

  • opensearch.yml from each Nodes (This is auto-genereated somehow… don’t know why)
plugins.security.audit.type: internal_opensearch
plugins.security.authcz.admin_dn: ["CN=admin","OU=test-opensearch-cluster-1"]
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.nodes_dn: ["CN=test-opensearch-cluster-1","OU=test-opensearch-cluster-1"]
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: tls-http/tls.crt
plugins.security.ssl.http.pemkey_filepath: tls-http/tls.key
plugins.security.ssl.http.pemtrustedcas_filepath: tls-http/ca.crt
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.pemcert_filepath: tls-transport/tls.crt
plugins.security.ssl.transport.pemkey_filepath: tls-transport/tls.key
plugins.security.ssl.transport.pemtrustedcas_filepath: tls-transport/ca.crt
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config",".opendistro-alerting-alert*",".opendistro-anomaly-results*",".opendistro-anomaly-detector*",".opendistro-anomaly-checkpoints",".opendistro-anomaly-detection-state",".opendistro-reports-*",".opendistro-notifications-*",".opendistro-notebooks",".opensearch-observability",".opendistro-asynchronous-search-response*",".replication-metadata-store"]
2

Self-Answering

  • By comparing two opensearch.yml (Cluster A’s certificates are created by cert-manager and the other Cluster B’s are by operator), I found that authcz.admin_dn and nodes_dn settings are different.
# Cluster A : test-opensearch-cluster-1
security:
    tls:
      transport:
        generate: false
        perNode: false
        secret: 
          name: test-opensearch-cluster-1-transport-cert
        caSecret:
          name: test-opensearch-cluster-1-ca
        nodesDn: [CN=test-opensearch-cluster-1,OU=test-opensearch-cluster-1]
        adminDn: [CN=admin,OU=test-opensearch-cluster-1]
      http:
        generate: false
        secret:
          name: test-opensearch-cluster-1-http-cert

# Cluster B : test-opensearch-cluster-2
security:
    tls:
      transport:
        generate: true
        perNode: true
      http:
        generate: true

As printing two opensearch.yml, DNs for the former have been splitted by double-quotes but the latter’s have been combined with a single double-quote. (See the below:)

$ k get cm test-opensearch-cluster-1-config -o yaml | grep plugins.security.nodes_dn --context=3
    plugins.security.authcz.admin_dn: ["CN=admin","OU=test-opensearch-cluster-1"]
    plugins.security.check_snapshot_restore_write_privileges: true
    plugins.security.enable_snapshot_restore_privilege: true
    plugins.security.nodes_dn: ["CN=test-opensearch-cluster-1","OU=test-opensearch-cluster-1"]
    plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
    plugins.security.ssl.http.enabled: true
    plugins.security.ssl.http.pemcert_filepath: tls-http/tls.crt

$ k get cm test-opensearch-cluster-2-config -o yaml -n test-opensearch-cluster-2 | grep plugins.security.nodes_dn --context=3
    plugins.security.authcz.admin_dn: ["CN=admin,OU=test-opensearch-cluster-2"]
    plugins.security.check_snapshot_restore_write_privileges: true
    plugins.security.enable_snapshot_restore_privilege: true
    plugins.security.nodes_dn: ["CN=test-opensearch-cluster-2,OU=test-opensearch-cluster-2"]
    plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
    plugins.security.ssl.http.enabled: true
    plugins.security.ssl.http.pemcert_filepath: tls-http/tls.crt

Today’s Lesson

  • Watch OUT Array type and Double Quote in yaml (Especially, If you use a single transport certificate across nodes.)
2 Likes