SSLconfigException Error

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Describe the issue:

Hi Team,
Am integrating opensearch and keycloak with openid.

Opensearch throws an error like unable to load the pemtrustedcas_filepath: /usr/share/opensearch/config/data/openid-certs.

Configuration:

authc:
basic_internal_auth_domain:
http_enabled: true
transport_enabled: true
order: 0
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: internal
openid_auth_domain:
http_enabled: true
transport_enabled: true
order: 1
http_authenticator:
type: openid
challenge: false
config:
enable_ssl: true
verify_hostnames: false
enable_ssl_client_auth: true
pemtrustedcas_filepath: /usr/share/opensearch/config/certs/openid-certs
openid_connect_url: https://<https://195.xxx.xxx.xxx.sslip.io/auth/realms/realm/.well-known/openid-configuration
subject_key: preferred_username
roles_key: roles
authentication_backend:
type: noop
Relevant Logs or Screenshots:
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.security.cert.CertificateException: No certificate data found
at sun.security.provider.X509Factory.parseX509orPKCS7Cert(X509Factory.java:456) ~[?:?]
at sun.security.provider.X509Factory.engineGenerateCertificates(X509Factory.java:356) ~[?:?]
at java.security.cert.CertificateFactory.generateCertificates(CertificateFactory.java:480) ~[?:?]
at org.opensearch.security.support.PemKeyReader.loadCertificatesFromFile(PemKeyReader.java:279) ~[opensearch-security-2.8.0.0.jar:2.8.0.0]
at com.amazon.dlic.util.SettingsBasedSSLConfigurator.initFromPem(SettingsBasedSSLConfigurator.java:236) ~[opensearch-security-2.8.0.0.jar:2.8.0.0]
… 22 more
[2023-09-05T09:25:56,139][WARN ][o.o.s.s.ReflectionHelper ] [my-cluster1-masters-0] Unable to enable ‘com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator’ due to java.lang.reflect.InvocationTargetException
[2023-09-05T09:25:56,147][ERROR][o.o.s.s.DynamicConfigModelV7] [my-cluster1-masters-0] Unable to initialize auth domain openid_auth_domain=AuthcDomain [http_enabled=true, transport_enabled=true, order=1, http_authenticator=HttpAuthenticator challenge=false, type=openid, config={openid_connect_idp={enable_ssl=true, verify_hostnames=false, pemtrustedcas_filepath=/usr/share/opensearch/config/certs/openid-certs}, subject_key=preferred_username, roles_key=roles, openid_connect_url=[https://xxx.xxx.xxx.xxx.sslip.io/auth/realms/nha/.well-known/openid-configuration}], authentication_backend=AuthcBackend [type=noop, config={}], description=null] due to OpenSearchException[java.lang.reflect.InvocationTargetException]; nested: InvocationTargetException; nested: RuntimeException[com.amazon.dlic.util.SettingsBasedSSLConfigurator$SSLConfigException: Error loading PEM from /usr/share/opensearch/config/certs/openid-certs (openid_connect_idp.pemtrustedcas_filepath) for openid_connect_idp.]; nested: SSLConfigException[Error loading PEM from /usr/share/opensearch/config/certs/openid-certs (openid_connect_idp.pemtrustedcas_filepath) for openid_connect_idp.]; nested: CertificateException[No certificate data found];
org.opensearch.OpenSearchException: java.lang.reflect.InvocationTargetException
at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:73) ~[opensearch-security-2.8.0.0.jar:2.8.0.0]

You must provide a file name and not a folder name. If you have more than one CA then you must concatenate them in one single file.

Hi @pablo

The file path ending with openid-certs is a file which has PEM, Certificate and keys which is concatenated into one file. Still the issue persists.

Note: Am using opensearch operator in the k8s environment.

@Tamilkumar How did you mount that certificate?

My understanding is that ‘pemtrustedcas_filepath’ only needs to contain your trusted CA cert (or concatenate more than one if you have many), but you don’t need to concatenate any other certs or keys into it.

Hi @pablo

Openid-certs mentioned in the file path has been fetched from the keycloak .(wget https://xxx.xxx.xxx.xxx.sslip.io/auth/realms//protocol/openid-connect/certs) Since its a n operator i have created a configmap and appended as additional volume sections in opensearch.yaml file.

config.yaml
apiVersion: v1
data:
openid-certs: '{“keys”:[{“kid”:“1qGlXsK7PC-H6-EFQ8fP69uJMpfXh_EP1TTP4xtuIrw”, “kty”: “RSA”, “alg”: “RSA-OAEP”, “use”: “enc”, “n”: “rykU_GBbcZ7P JM5hHviq3Modu-7iuRImn2aubVp_l2shMBLWC6E91DVVLSTD73V37FCHVxPEn980cUmHME-CK80JXrYHK4dPQ1Ho0e9WFgQL5GUKTfIqZabP1rnFvncYHvnHJGh6Jyts 98PNP4tBU pJEHbuE7Z5f2tBLrF6dkx5ivp7NEXps9JsD6VTwiTm2A_cfEWPEZBGgtUpcAeNIAZNqrixq4NiGPtX1j ink ims QCS BQmvhLtHaOPL jonkkPAXWD3Vtq-wdf-JRkoP1mBZS 34EU31T s1tDFKAETX6pK-lqbPSib-bvTmGSqRnjHttEUAoAtaJUU0ac6e2nxQ4w”, “e”: “AQAB”, “x5c”: [“MIICLTCCAX0CBgGItGYiljANBgkqhkiG9w0BAQS FADAOMQWWCgYDVQQDDANu aGEwHhcNMjMwNjEzMTA1NDMyWhcNMzMwNjEzMTA1NjEyWjAOMQwwCgYDVQQDDANuaGEwggE iMA0GCSqGSIb3DQEBAQUAA4IBDWAwggEKAoIBAQCVKRT8YFtxns 8kzmEe+Krcyh277
UK5EiafZq5tWn+XZKEWE VALOT3UNW8tJMPvdXfsUIe/E85f3w5XSYcwT4IrzQnGtgcrh09DUejR71YWBAVkZQpN8 iplps/WucW+dxge+cckaHonK2z3w80/10FSkkQdu4Ttnl/a0E us Xp2THmK+ns@Remz @mwPpVPCJ0bYD9X8RY8RkEaC1slwB40gBk2quLGrg21Y+1fWOKeSKaxAKWFCa+Eu0do48uM6e5Q8DFYPdW2r7B1/41GSg/WYFmzfgRTEVOZWOMUOARNfqkr6 Wps9KJv5u90YZKpGeMe20RQCgC10LRTRpzp7afFDjAgMBAAEWDQYJKoZIhvcNAQEL BQADggEBAEo+t06vbFXR98f0xj FZRD6RLQFpup imKzGlSGOKjjcPgTzNTADOwm1PZ9lb/BXC
wS4JJHJHhjsdjIgQIrnS3Y7tYbIgD68/8cFcbHSCqYo8gSooGp5cGgJdpSS/bbsb700edTGZFQTlLKHBUjseRbZ9orjMlIMalaTN2dSbFcr53H5b1NngxE3ZTJ2YgRBL5wbme3kVT Rww/2v/gCJ5HRytefLE4YY6Uz2LBd+4Q+uUoIzuzIz6hbJ+IeIxYFd3pF6x7ln0wIjxgiv+UDjsxqdamdmHRDeZcrFzKn2MZXSwxaJXiYe0Df+o/kxxkbQxdj CymPKOUJF762mQm+ F1hKA=”],“x5t”:“35laLNBNqn-cHwv3fxiLCKW-zj0”, “x5t#S256”: “R9zN7Uz7tCkCHtX470jnyiopcf8K fnBY1wKwDBCP5nk”}, { “kid”:“UAGGhgB005pY649QTr-0J8XVqn 1YKBONE 31bsP8dVhc”, “kty”: “RSA”, “alg”: “RS256”, “use”: “sig”, “n”: “rgMc Vs XunPi2fKiwetaAKbvslTg6zhyiINleMOzYni82ows OfrcIGLIMhdH2qFib2dvNg81aJwT CPXK8WdUS6362q2hj HuYihBvwZ4PAJDIFXQG-DOO_KVtnGhA2qbaN3nkLtPOCdjuQWuh8eEus-Dp422HLxrP70UW2PSBguVjr5zjS4IQR7ipLpUpxdzoulik JQZYGXQh9NPutQZ_6 3qaZF_3di_W6UFmlTEzCjg0G_uANy7qIdDxAPFj-_tiMs 9ozT-ODIAIg_Q5ekDevTv59RFkIqh27ofWpKH1g8f0KhEKbгL3JDpa1roLBQnWoodwelv1HXGcVJLEU7Ziv9w”, “e”:" AQAB",“x5”:[“MIICLTCCAX0CBgGItGYhIzANBgkqhkiG9w0BAQS FADAOMQwwCgYDVQQDDANuaGEwHhcNMjMwNjEzMTA1NDMyWhcNMzMwNjEzMTA1NjEyWjAOMQwwCgYDVQQDDAN
uaGEwggE iMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuAxxWxe6c+LZ8qJZ610Apu9KVODгOHKIg2V4w7Nielzahaw5+twgaIgyFofaoWJvZ282Dz
VonBMI/ErXZ1RLrfrar
aGMe5 LKEG/Bng8AkMh9dAb4M7T+RW2caEDapto3eeQu084J205Ba6Hx4S6z40njbYcvGs/vRRbY9IGC5W0vnONLghBHukkulSnF1k5SWKQlDNgbFCH00+61Bn/reppkX/d2L9bpQw aVMTMK0DQb+4A3Luoh@PEA8WP7+21yz2jNP7QMgAiD9Dl6QN690/n1EWQiqHbuh9ak ofWDx84qEQpus vck0lrwugs FCdag51bSW/UfEZXUks57tmk/3AgMBAAEWDQYJKoZIhvcNAQ ELBQADggEBAAVCYoHPcSb6J1T7mZqw+x7xbcA6HLYOj Ad1wQgpts SkAf0+n+ovp/T58tpAfaV5YipoZFto5fyXzKf6kMmQj vknVrr Fh+F6UTImTKiRflrQ6K6cHkghSyZ9lENOq5j 31mUyD7yqAANvaIQPCNZVpW0Xb2fqz7tdXSaanTkJ1i/Tuk0Y14mJqjYXRZYVKA 1Toz dUaBcDyt23ELHixBs 28Wl1CnQVHLoJj0jYrqt0wbYpeAXdI ic5c+0dVt1C+kM28rwmTOwu m7sgNUseW53cSZJnbVCLHPPCydJ+iLTdUnNevBfLEBg0nshGj7iwJipUaUp9mF7Wegk6wtVoIE3fT74=”],“x5t”: “u7seUrRxe6txc5gkMZiyas qviCI”, “x5t#S256”:“NaOiGy
XXynLZbNz7ieb_FKrnregWujbdhvgo561SbTU”}]}
kind: ConfigMap
metadata:
name: openid-certs

opensearch.yaml

kind: OpenSearchCluster
metadata:
name: my-cluster1
namespace: opensearch
spec:
security:
config:
adminCredentialsSecret:
name: a-admin-credentials-secret
securityConfigSecret:
tls:
name: a-securityconfig-secret
transport:
generate: true
http:
generate: true
general:
serviceName: my-cluster1 version: 2.8.0
pluginsList: [“repository-s3”] drainDataNodes: true
setVMMaxMapCount: true
additional volumes:

  • name: openid-certs
    path: /usr/share/opensearch/config/certs/ configMap:
    name: openid-certs
    restartPods: true

@Tamilkumar The mapping in the OpenSearch cluster object looks correct.
However, your configmap is not.

I’ve tested Opster deployment with configmap containing the keycloak certificate and all worked.

Your configmap should look similar to the below one.

kubectl get configmap keycloak-cert -o yaml
apiVersion: v1
data:
  keycloak.crt: |
    -----BEGIN CERTIFICATE-----
    MIIDoTCCAomgAwIBAgIEOa2AUjANBgkqhkiG9w0BAQsFADBtMQswCQYDVQQGEwJJ
    RTENMAsGA1UECBMEQ29yazENMAsGA1UEBxMEQ29yazENMAsGA1UEChMEQ29yazER
    ...
    U/JlUrz+Ht8MOSsJtIeo/bntvoHqSYJvV5GNSzRZs19VMeIgip3+qLM6sH9i70lU
    EW0ylLohWz3rwwF0U5D5wEBfqZnv
    -----END CERTIFICATE-----
kind: ConfigMap
metadata:
  creationTimestamp: "2023-09-06T22:01:42Z"
  name: keycloak-cert
  namespace: default
  resourceVersion: "125660"
  uid: 1f06d842-9ede-4f01-bd7b-a631b95f046a

I’ve created my configmap with the following command.

kubectl create configmap keycloak-cert --from-file ./keycloak.crt

This is my OpenSearch cluster yaml.

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: my-cluster1
  namespace: default
spec:
  security:
    config:
      adminCredentialsSecret:
        name: a-admin-credentials-secret
      securityConfigSecret:
        name: a-securityconfig-secret
    tls:
      transport:
        generate: true
      http:
        generate: true
  general:
    serviceName: my-cluster1
    version: 2.9.0
    pluginsList: ["repository-s3"]
    drainDataNodes: true
    setVMMaxMapCount: true
    additionalVolumes:
    - name: keycloak-cert-os-secret
      path: /usr/share/opensearch/config/certs/
      configMap:
         name: keycloak-cert

@Tamilkumar pemtrustedcas_filepath in openid configuration in config.yml must contain either signing certificate of the IDP (keycloak) or IDP certificate.

@pablo

Even my folder structure certs worked out the thing i missed is the mapper section. which sorted out the issue.

@pablo ,

I wanted my opensearch dashboard to configure in ssl mode authentication . To achieve this i’ve add the tls configs in the dashboard.yml but its not working as expected.

opensearch-keycloak authentication works out but from keycloak to opensearch dashboard its not working out. pls help what am i missing?

I’ve mentioned th e yaml file below:

dashboards:
additionalConfig:
logging.verbose: “true”
opensearch_security.auth.type: ‘[“basicauth”, “openid”]’
opensearch_security.auth.multiple_auth_enabled: “True”
opensearch_security.openid.connect_url: https://xx.xxx.xxx.xx.sslip.io/auth/realms/xxx/.well-known/openid-configuration opensearch_security.openid.base_redirect_url:https://xxx.dev26.xxx.com/
opensearch_security.openid.client_id: grafana
opensearch_security.openid.scope: openid profile email
opensearch_security.openid.client_secret:
opensearch_security.openid.header: Authorization
opensearch_security.openid.trust_dynamic_headers: “true”
opensearch.optimizedHealthcheckId: “my-cluster1”
opensearch_security.openid.verify_hostnames : “false”
opensearch.ssl.verificationMode: none
opensearch_security.cookie.secure: “false”
opensearch.requestHeaders Whitelist: |
[“securitytenant”, “Authorization”, “security_tenant”]
opensearch_security.readonly_mode.roles: [ “kibana_user”, “readall” ]’
opensearchCredentialsSecret:
name: a-admin-credentials-secret
enable: true
tls:
enable: true
generate: true
version: 2.8.0
replicas: 1

Hi @pablo

Your suggestion will help me to move fwd.

@Tamilkumar What exactly do you see? Are you getting redirected to the Keycloak’s login page?

You need to point OpenSearch Dashboards to Keycloak’s self-signed certificate in opensearch_dashboards.yml

opensearch_security.openid.root_ca: "/usr/share/opensearch-dashboards/config/keycloak.crt"

HI @pablo ,

Would like to thank you for the solutions and discussions provided to me during my deployments.

Am trying to setup key Cloak saml federation with OpenID authorization in the OpenSearch level but am facing an issue at the opensearch dashboard end like

Error Failed to obtain the endpoints from IDP.

Set up details
Key Cloak federation setup:
I’ve two key Cloak set it up locally named as central-key Cloak and local-key Cloak.

central-key Cloak is the one which needs to authorize the user when an user access from local-Key Cloak

The parameters required to achieve the setup is IDP at key Cloak-local end and Client id at key Cloak-central end. with saml authentication.

The parameters needed for the above task is IDP URL, ClientID and OpenSearch dashboard url for redirection.

dashboard config:
dashboards:
additionalConfig:
logging.verbose: “true”
opensearch_security.auth.type: ‘[“basicauth”, “openid”]’
opensearch_security.auth.multiple_auth_enabled: “True”
opensearch_security.openid.connect_url: https://xx.xxx.xxx.xx.sslip.io/auth/realms/xxx/.well-known/openid-configuration opensearch_security.openid.base_redirect_url:https://xxx.dev26.xxx.com/
opensearch_security.openid.client_id: grafana
opensearch_security.openid.scope: openid profile email
opensearch_security.openid.client_secret:
opensearch_security.openid.header: Authorization
opensearch_security.openid.trust_dynamic_headers: “true”
opensearch.optimizedHealthcheckId: “my-cluster1”
opensearch_security.openid.verify_hostnames : “false”
opensearch.ssl.verificationMode: none
opensearch_security.cookie.secure: “false”
opensearch.requestHeaders Whitelist: |
[“securitytenant”, “Authorization”, “security_tenant”]
opensearch_security.readonly_mode.roles: [ “kibana_user”, “readall” ]’
opensearchCredentialsSecret:
name: a-admin-credentials-secret
enable: true
tls:
enable: true
generate: true
version: 2.8.0
replicas: 1

Please share me the documents or configs or suggestions from ur end on how to achieve this or solve the dasjboard error(unable to find endpoint from idp)

@Tamilkumar Just to clarify.
You’d like to authenticate with SAML and authorize with OpenID using the OpenSearch security plugin, is that correct?

HI @pablo ,

Exactly the same.

@pablo

say for example in my scenario when i access the OpenSearch dashboard it should redirect to key Cloak-local first(idp)for authentication and then it should redirect to key Cloak-central (Client-id)for authorization.

Meaning two single sign on redirections has to happen

@Tamilkumar Unfortunately OpenSearch and OpenSearch Dashboards don’t have such functionality.
You can configure multiple authentication types with OpenSearch and OpenSearch Dashboards but this only allows you to use one authentication type at a time.

@pablo ,

I tried configuring multiple sign on with the order 0 order 1 and so on but its not meeting my requirements.

Thats bad luck for me that OpenSearch doesn’t have this functionality.

@Tamilkumar I can only suggest opening a feature request in the OpenSearch security GitHub.

@pablo
Thanks for your suggestion and Support.