OpenID connect integration with Opensearch

Hello @cwperks / All,

I am trying to integrate OpenID connect with Opensearch with fresh installation of single node cluster.

I was getting the error message as follows.

.opendistro_security index already exists, so we do not need to create one.
Legacy index '.opendistro_security' (ES 6) detected (or forced). You should migrate the configuration!

As I get to know that there is bug has been reported for the same it has been fixed as well…

[Backport 2.x] Fix legacy check in SecurityAdmin by opensearch-trigger-bot[bot] · Pull Request #2075 · opensearch-project/security (github.com)

This morning I tried to execute the securityadmin.sh script again with latest Opensearch version (2.2.1) and I am still getting that message…

Can you please assist me on this?

bash-4.2# ./securityadmin.sh -cd /usr/share/opensearch/config/opensearch-security -icl -nhnv \
>  -key /usr/share/opensearch/config/kirk-key.pem \
>  -cert /usr/share/opensearch/config/kirk.pem \
>  -cacert /usr/share/opensearch/config/root-ca.pem
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to localhost:9200 ... done
Connected as "CN=kirk,OU=client,O=client,L=test,C=de"
OpenSearch Version: 2.2.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: opensearch-cluster
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Legacy index '.opendistro_security' (ES 6) detected (or forced). You should migrate the configuration!
Populate config from /usr/share/opensearch/config/opensearch-security/
ERR: Seems /usr/share/opensearch/config/opensearch-security/config.yml is not in legacy format: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot construct instance of `org.opensearch.security.securityconf.impl.v7.ConfigV7$AuthcDomain` (although at least one Creator exists): no boolean/Boolean-argument constructor/factory method to deserialize from boolean value (true)

Here is my config.yaml

config:
  dynamic:
    authc:
      openid_auth_domain:
      http_enabled: true
      transport_enabled: true
      order: 0
      http_authenticator:
        type: openid
        challenge: false
        config:
          subject_key: preferred_username
          roles_key: admin_role
        openid_connect_idp:
          enable_ssl: true
          verify_hostnames: false
          openid_connect_url: https://idp-xxxxxxxxxx/.well-known/openid-configuration
          openid_connect_idp.pemtrustedcas_filepath: "/etc/ssl/certs/admin.pem"
      authentication_backend:
        type: noop

Here is my docker-compose file.

version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - plugins.security.ssl.transport.pemcert_filepath=/etc/ssl/certs/admin.pem
      - plugins.security.ssl.transport.pemkey_filepath=/etc/ssl/certs/admin-key.pem
      - plugins.security.ssl.transport.pemtrustedcas_filepath=/etc/ssl/certs/root-ca.pem
      - plugins.security.ssl.transport.enforce_hostname_verification=false
      - plugins.security.ssl.http.enabled=true
      - plugins.security.ssl.http.pemcert_filepath=/etc/ssl/certs/admin.pem
      - plugins.security.ssl.http.pemkey_filepath=/etc/ssl/certs/admin-key.pem
      - plugins.security.ssl.http.pemtrustedcas_filepath=/etc/ssl/certs/root-ca.pem
      - plugins.security.authcz.admin_dn=CN=xyz.com,OU=xxxxxx,O=VW,L=wolfsburg,ST=xxxxxxxx,C=IN
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    # - "DISABLE_INSTALL_DEMO_CONFIG=true" # disables execution of install_demo_configuration.sh bundled with security plugin, which installs demo certificates and security configurations to OpenSearch
#     - "DISABLE_SECURITY_PLUGIN=true" # disables security plugin entirely in OpenSearch by setting plugins.security.disabled: true in opensearch.yml
      - "discovery.type=single-node" # disables bootstrap checks that are enabled when network.host is set to a non-loopback address
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
      - ./config.yml:/usr/share/opensearch/config/opensearch-security/config.yml
      - /tmp/opensearch/:/etc/ssl/certs/
    ports:
      - 0.0.0.0:9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net

  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    ports:
      - 0.0.0.0:5601:5601
    expose:
      - "5601"
    environment:
      - 'OPENSEARCH_HOSTS=["https://localhost:9200"]'
      - "opensearch_security.auth.type=openid"
      - "opensearch_security.openid.connect_url=https://xxxxxx.well-known/openid-configuration"
      - "opensearch_security.openid.client_id="XXXXXXXXXXXXXXXXXXX"
      - "opensearch_security.openid.client_secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
      - "opensearch_security.openid.base_redirect_url=https://localhost:5601"
      - "opensearch.requestHeadersWhitelist=[ authorization,securitytenant, security_tenant]"
      - "opensearch_security.openid.scope=openid profile email address phone offline_access"
      - "opensearch_security.openid.verify_hostnames:false"
      - "opensearch_security.openid.refresh_tokens:false"
#     - "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true" # disables security dashboards plugin in OpenSearch Dashboards
      - "SERVER_BASEPATH=/opensearch"
      - "SERVER_REWRITEBASEPATH=true"
    volumes:
      - /var/lib/docker/volumes/containers:/var/lib/docker/volumes/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - opensearch-net
volumes:
  opensearch-data1:

networks:
  opensearch-net:

Any help/suggestion would be highly appreciated…

Thanks.

Could it be that your code isnt indented correctly?

      openid_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 5
        http_authenticator:
          type: openid
          challenge: false
          config:
            subject_key: preferred_username
            roles_key: roles...

Hello @cwperks,

I am referring to the below BUG and according to it I should not get the following error while executing the securityadmin.sh.

Please correct if I am wrong and kindly suggest me how to fix it and proceed with OpenID connect integration with Opensearch…

Thank you in advance.

.opendistro_security index already exists, so we do not need to create one.
Legacy index '.opendistro_security' (ES 6) detected (or forced). You should migrate the configuration!

Fix legacy check in SecurityAdmin by cwperks · Pull Request #2052 · opensearch-project/security (github.com)

Thanks.

Hey @sabil, the fix you linked to is scheduled to be included in the 2.3.0 release at the end of this week (9/15/22). This bug impacted 2.0, 2.1 and 2.2.

It sounds like the docker volume you are using may have been referenced elsewhere and already contains some data. Since this is a fresh installation, would it be ok to remove the existing volume or use a different named volume? i.e. instead of using opensearch-data1 you could name this something different so it creates a new volume, something like opensearch-data-node1.

See the documentation website on creating a cluster here: Docker - OpenSearch documentation - The documentation recommends using docker-compose down -v to remove volumes when spinning down a cluster.

Hello @cwperks,

Thank you for the information.

Okay sure, I will change the volume to opensearch-data-node1 and then will try after 15th Sep 2022.

Just FYI… I am using single node cluster for my opensearch.

Thanks.

Hi @sabil using a different volume is something you can do now. The bugfix linked above is for applying changes to an existing security index, but if you change the volume name it will create the security index on startup since it won’t exist on a new volume.

Hello @cwperks,

I have used 2.3.0 version and executed the securityadmin.sh… Still getting the errors

The Legacy index error has disappeared.

**Legacy index '.opendistro_security' (ES 6) detected (or forced). You should migrate the configuration!

However, still getting the following error message.

Security Admin v7
Will connect to localhost:9200 ... done
Connected as "CN=kirk,OU=client,O=client,L=test,C=de"
OpenSearch Version: 2.3.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: opensearch-cluster
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Populate config from /usr/share/opensearch/config/opensearch-security/
ERR: Seems /usr/share/opensearch/config/opensearch-security/config.yml is not in OpenSearch Security 7 format: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "http_enabled" (class org.opensearch.security.securityconf.impl.v7.ConfigV7), not marked as ignorable (one known property: "dynamic"])
 at [Source: (String)"{"openid_auth_domain":{"http_enabled":true,"transport_enab

Kindly suggest/assist on this…

Thanks.

Hey @sabil, make sure everything under openid_auth_domain: is indented to the right.

The nodes successfully startup for me with the configuration you posted and the section under openid_auth_domain indented.

_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    authc:
      openid_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: openid
          challenge: false
          config:
            subject_key: preferred_username
            roles_key: admin_role
            openid_connect_idp:
              enable_ssl: true
              verify_hostnames: false
              openid_connect_url: https://xxxxxxxx/kums/.well-known/openid-configuration
              #pemtrustedcas_filepath: /usr/share/opensearch/config/root-ca.pem
        authentication_backend:
          type: noop

Hey @cwperks,

Thank you for your assistance…

I have corrected the indentation. :slight_smile: Now when I try to execute the securityadmin.sh getting the following error message.

Security Admin v7
Will connect to localhost:9200 ... done
ERR: An unexpected ResponseException occured: method [GET], host [https://localhost:9200], URI [/_plugins/_security/whoami], status line [HTTP/1.1 403 Forbidden]
{"error":"GET /_plugins/_security/whoami API not whitelisted","status":"FORBIDDEN"}
Trace:
org.opensearch.client.ResponseException: method [GET], host [https://localhost:9200], URI [/_plugins/_security/whoami], status line [HTTP/1.1 403 Forbidden]
{"error":"GET /_plugins/_security/whoami API not whitelisted","status":"FORBIDDEN"}
        at org.opensearch.client.RestClient.convertResponse(RestClient.java:375)
        at org.opensearch.client.RestClient.performRequest(RestClient.java:345)
        at org.opensearch.client.RestClient.performRequest(RestClient.java:320)
        at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:462)
        at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:159)

I have found one Bug related to whiltelist.yml. I guess we need to keep both the files whitelist.yml and allowlist.yml until the opensearch 3.0 comes.

I have made the changes to whitelist.yml and allowlist.yml and added the following entry to it.

---
_meta:
  type: "allowlist"
  config_version: 2

# Description:
# enabled - feature flag.
# if enabled is false, all endpoints are accessible.
# if enabled is true, all users except the SuperAdmin can only submit the allowed requests to the specified endpoints.
# SuperAdmin can access all APIs.
# SuperAdmin is defined by the SuperAdmin certificate, which is configured with the opensearch.yml setting plugins.security.authcz.admin_dn:
# Refer to the example setting in opensearch.yml to learn more about configuring SuperAdmin.
#
# requests - map of allow listed endpoints and HTTP requests

#this name must be config
config:
  enabled: true
  requests:
    /_cluster/settings:
      - GET
    /_cat/nodes:
      - GET
    /_plugins/_security/whoami:
      - GET

Sorry to bother you again and again. I am very new to it. That’s the reason struggling a lot to do the integration of OpenID connect with Opensearch.

Any kind of help or suggestion is highly appreciated.

Thanks.

@sabil Make sure the subject line of the certificates you are using to run securityadmin with match the plugins.security.authcz.admin_dn line in your docker-compose to connect as the super admin and bypass the check.

Alternatively, as was mentioned earlier in this thread you can either delete the docker volume or use a different volume so that the security index is initialized on startup. You can use this option if you are starting from scratch to avoid needing to run the securityadmin script.

Hello @cwperks ,

Thank you for your response and suggestion.

I have included the value for plugins.security.authcz.admin_dn in docker-compose and I am getting following error.

Security Admin v7
Will connect to localhost:9200 ... done
Connected as "CN=kirk,OU=client,O=client,L=test,C=de"
ERR: "CN=kirk,OU=client,O=client,L=test,C=de" is not an admin user
Seems you use a client certificate but this one is not registered as admin_dn
Make sure opensearch.yml on all nodes contains:
plugins.security.authcz.admin_dn:
  - ""CN=kirk,OU=client,O=client,L=test,C=de""
bash-4.2#

Just wanted to know do we need to add admin_dn.yml as well for the security to enable.

One more thing I would like to know that after successful execution of securityadmin.sh. I should be able to login opensearch-dashbaord through OpenID connect or is there anything I need to do more in order to achieve this.

Thank you in advance for you all your help and support.

Thanks.

@sabil Any plugin.security... settings should be in the opensearch.yml file. See YAML files - OpenSearch documentation

One of the security settings in this file is plugins.security.authcz.admin_dn which takes a list of distinguished named. There is no admin_dn.yml, I think you may be mixing that up with nodes_dn.yml

If you create a custom custom_opensearch.yml you can mount it into the node container using

volumes:
   - /path/to/custom_opensearch.yml:/usr/share/opensearch/config/opensearch.yml

Hope that helps. If OIDC is successfully configured then you should be able to login to the dashboards through OpenID connect.

Again with securityadmin, it is not necessary to run securityadmin if you are starting a fresh cluster without any pre-existing data. Volumes in docker are a way to mount a local filesystem to that of a running container and are used to persist data in the event that a container is stopped and restarted. If you don’t reference a volume on the node in the docker-compose.yml file or remove the volume in between starting and stopping the cluster while troubleshooting the OIDC config then you will not need to run securityadmin.

Hey @cwperks,

Thank you so much for your kind responses and support. Really appreciate it. :slight_smile:

Now I have couple of queries. Hopefully the last ones :smiley:

plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=kirk,OU=client,O=client,L=test, C=de

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.cache.ttl_minutes: 60
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opendistro-asynchronous-search-response*"]
node.max_local_storage_nodes: 3

In the above variables I can see file path for the certificate. Currently I am using the default certificate “Kirk” one. I guess those are already exist for testing.

  1. Should I just use the names kirk.pe, kirk.key and root-ca.pem something like that for the above parameter wherever it is asking for path.

  2. In Future if we wanted to use the our own certificates then it should from IDP. Am I right?

Meanwhile I will try to remove the volume and try to execute the docker-compose.yml.

Thanks,
Sabil.

I have not mounted the volume and use the above default values in docker-compose.yml.

However, still getting an error message as follows.

opensearch-node1         | [2022-09-16T16:03:56,561][INFO ][o.o.i.i.m.ISMTemplateService] [opensearch-node1] Failure experienced when migrating ISM Template and update ISM policies: {}
opensearch-node1         | [2022-09-16T16:03:56,604][INFO ][o.o.c.s.ClusterSettings  ] [opensearch-node1] updating [plugins.index_state_management.template_migration.control] from [0] to [-1]
opensearch-node1         | [2022-09-16T16:03:56,605][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch-node1] Detected cluster change event for destination migration
opensearch-node1         | [2022-09-16T16:03:56,608][INFO ][o.o.i.i.m.ISMTemplateService] [opensearch-node1] Successfully update template migration setting
opensearch-dashboards    | {"type":"log","@timestamp":"2022-09-16T16:04:49Z","tags":["error","opensearch","data"],"pid":1,"message":"[TimeoutError]: Request timed out"}
opensearch-dashboards    | {"type":"log","@timestamp":"2022-09-16T16:04:49Z","tags":["error","savedobjects-service"],"pid":1,"message":"Unable to retrieve version information from OpenSearch nodes."}
opensearch-node1         | [2022-09-16T16:04:56,553][INFO ][o.o.i.i.ManagedIndexCoordinator] [opensearch-node1] Cancel background move metadata process.
opensearch-node1         | [2022-09-16T16:04:56,554][INFO ][o.o.i.i.ManagedIndexCoordinator] [opensearch-node1] Performing move cluster state metadata.
opensearch-node1         | [2022-09-16T16:04:56,554][INFO ][o.o.i.i.MetadataService  ] [opensearch-node1] Move metadata has finished.

When I try to access the opensearch-dashboard getting an error that dashboard is not yet ready.

:expressionless: :neutral_face:

@sabil, kirk.pem is the demo admin when the demo certificates are installed. The paths there are relative to the $OPENSEARCH_HOME/config directory. In the docker images on docker hub, this path is /usr/share/opensearch/config on the nodes. When referencing the admin cert for running security admin you can use absolute paths or relative paths, relative to the directory securityadmin.sh is invoked from.

These are demo certificates and should not be used in production.

You can setup enabled TLS for when Opensearch connects to the IdP and can find the documentation for it here: OpenID Connect - OpenSearch documentation

Can you provide more output from opensearch-node1 for the error you are receiving on startup?

Hi @nateynate. Could you move this case to the Security category?

@sabil The certificate used in the config.yml is incorrect.
The kirk certificate is an admin certificate and it is used to manage the security plugin configuration in the OpenSearch cluster.

The pemtrustedcas_content requires an IDP CA certificate (not a key) only when it’s self-signed.

What is your OpenID IDP?

@pablo Thank you for the information.

I am using our own IDP server for the open id connect authentication.

If I want use our own, do we need to have admin certificate and IDP CA certificate in place then only the authentication will work?

I was thinking to use default for testing purpose.

Kindly give some insights on the certificate… Thank you in advance! :slight_smile:

@sabil Run the below command and check the certificate in the response.
This is the only certificate you should have in your OpenID config. Don’t add admin cert.

openssl s_client -connect <IDP_IP_or_FQDN>:<port_number>

@sabil In this scenario the below config is correct as you didn’t configure a custom service account in config.yml. In this case, the OpenSearch security plugin will use the kibanaserver user by default.

      - "opensearch.username: kibanaserver"
      - "opensearch.password: kibanaserver"

Is there any need to use SERVER_BASEPATH? Do you have a reverse proxy in front of OpenSearch Dashboards?

Why did you disable certificates for the transport layer in opensearch.yml? Secured connection is mandatory in the transport layer.

Hello @pablo,

Thank you for providing the support…

Answer to your question.

  1. We are using the “kibanaserver:kibanaserver” as a user id and password.
  2. Yes, we are using the revers proxy for opensearch-dashbaord connect.
  3. We have enable the tranport layer related field.

I am very much confused here. Can you please help me to understand the process of OpenID connect integration with opensearch.

We have our own IDP setup/server which can be use to login to the opensearch-dashboard using our organization credentials.

We have the following information about the IDP.

  1. openid.connect_url
  2. openid.client.id
  3. openid.secret.id
  4. we have certificate information.

As per my understanding I should include the above information in config.yml to authenticate using IDP and login through our own organization credantials…

The confusion here is there few certificate also needs to be added into container for openid connect intergration.

1 Admin certificate
2 esnode certificate.

We just wanted to know what all the required configuration is needs to do the integration.

We have also created the below files from the following url.

YAML files - OpenSearch documentation

allowlist.yml
roles.yml
roles_mapping.yml
action_groups.yml
tenants.yml
nodes_dn.yml
whitelist.yml
internal_users.yml
config.yml

I would really like to know what exactly is the process of doing the integration.

It would be really appreciate if you can just give me some brief so that I can do the integration smoothly. Its been very long pending issue for me where i am trying to integrate the opensearch with IDP…

today I tried run the securityadmin.sh as I was doing the fresh installation of opensearch getting the following error.

ERR: An unexpected SSLHandshakeException occured: Extended key usage does not permit use for TLS server authentication
See https://opensearch.org/docs/latest/clients/java-rest-high-level/ for troubleshooting.
Trace:
javax.net.ssl.SSLHandshakeException: Extended key usage does not permit use for TLS server authentication
See https://opensearch.org/docs/latest/clients/java-rest-high-level/ for troubleshooting.
        at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:947)
        at org.opensearch.client.RestClient.performRequest(RestClient.java:332)
        at org.opensearch.client.RestClient.performRequest(RestClient.java:320)
        at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:462)
        at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:159)

Thank you in advance.