Locked out of .opendistro_security

Hi all.

I successfully managed to set up an OpenSearch Cluster with SAML authc and LDAP authz.
Setup: OS 1.2.3 tarball installation on Debian buster.

config/opensearch.yml:

cluster.name: log
node.name: log1
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
path.repo: ["/var/lib/opensearch-snapshot"]
network.host: 0.0.0.0
discovery.seed_hosts: ['10.xx.3', '10.xx.4', '10.xx.5']
cluster.initial_master_nodes: ['log1', 'log2', 'log3']

plugins.security.ssl.transport.pemcert_filepath: my-esnode.pem
plugins.security.ssl.transport.pemkey_filepath: my-esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: my-root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.truststore_filepath: cacerts
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: my-esnode.pem
plugins.security.ssl.http.pemkey_filepath: my-esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: my-root-ca.pem
plugins.security.allow_unsafe_democertificates: false
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - "CN=log...,C=DE"
plugins.security.nodes_dn:
  - "CN=log2...,C=DE"
  - "CN=log3...,C=DE"

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]
node.max_storage_nodes: 3

plugins/opensearch-security/securityconfig/config.yml:

_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: intern
      saml_auth_domain:
        http_enabled: true
        transport_enabled: false
        order: 1
        http_authenticator:
          type: saml
          challenge: true
          config:
            idp:
              metadata_file: saml_idp.xml
              entity_id: "https://oauth.xxx"
            sp:
              entity_id: "https://log1.xxx"
              signature_private_key_filepath: saml_sign.pem
              signature_algorithm: "http://www.w3.org/2001/04/xmldsig-more#rsa-sha512"
            kibana_url: "https://kibana.xxx:5601"
            roles_key: Role
            exchange_key: "xxx"
        authentication_backend:
          type: noop
    authz:
      roles_from_myldap:
        description: "Authorize via LDAP or Active Directory"
        http_enabled: true
        transport_enabled: true
        authorization_backend:
          # LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)
          type: ldap
          config:
            # enable ldaps
            enable_ssl: true
            # enable start tls, enable_ssl should be false
            enable_start_tls: false
            # send client certificate
            enable_ssl_client_auth: false
            # verify ldap hostname
            verify_hostnames: true
            hosts:
            - ldap....:636
            bind_dn: cn=...,dc=com
            password: xxx
            rolebase: 'ou=...,dc=com'
            # Filter to search for roles (currently in the whole subtree beneath rolebase)
            # {0} is substituted with the DN of the user
            # {1} is substituted with the username
            # {2} is substituted with an attribute value from user's directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute
            rolesearch: '(member={0})'
            # Roles as an attribute of the user entry
            userrolename: memberOf
            # The attribute in a role entry containing the name of that role, Default is "name".
            # Can also be "dn" to use the full DN as rolename.
            rolename: cn
            # Resolve nested roles transitive (roles which are members of other roles and so on ...)
            resolve_nested_roles: false
            userbase: 'ou=xxx,dc=com'
            # Filter to search for users (currently in the whole subtree beneath userbase)
            # {0} is substituted with the username
            usersearch: '(sAMAccountName={0})'
            # Skip users matching a user name, a wildcard or a regex pattern
            skip_users:
              - kibanaserver

Everything was working. I was able to update the security config with my own signed certificates.
Somehow I locked myself out. Today I tried to change the LDAP bind user, but I am not able to write the config any more:

$ ./securityadmin.sh
Security Admin v7
Will connect to localhost:9300 ... done
Connected as CN=log...,C=DE
OpenSearch Version: 1.2.3
OpenSearch Security Version: 1.2.3.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Cannot retrieve cluster state due to: no permissions for [cluster:monitor/health] and User [name=CN=log...,C=DE, backend_roles=[], requestedTenant=null]. This is not an error, will keep on trying ...
  Root cause: OpenSearchSecurityException[no permissions for [cluster:monitor/health] and User [name=CN=log...,C=DE, backend_roles=[], requestedTenant=null]] (org.opensearch.OpenSearchSecurityException/org.opensearch.OpenSearchSecurityException)
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.

Dashboards & Openserach is still working.

$ curl -XGET https://log1:9200/_cluster/health -u 'admin' --insecure
Enter host password for user 'admin':
{"cluster_name":"log","status":"green","timed_out":false,"number_of_nodes":3,"number_of_data_nodes":3,"discovered_master":true,"active_primary_shards":21,"active_shards":44,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}

How can I update my security config now?

Best regards,
Sebastian

@eskorte You need an admin certificate to use ./securityadmin.sh. Please take a look at the syntax in the OpenSearch documentation.

@pablo This is what I did before. The admins certificate dn is placed under plugins.security.authcz.admin_dn and I am connected with it:

Connected as CN=log…,C=DE

I used this admin certificates to write .opendistro_security with securityadmin.sh like 20 times before successfully. The securityconfig/config.yml above is my working config, but I cant change it any more. The config/opensearch.yml has not changed as well as the certificates.

Solved.

Here is what I did, if anyone is interested:
As I had still access to Dashboards I was able to map the role all_access to all users, because the login with the admin cert was valid but had no rights:


Now I was able to execute the script again:

$ ./securityadmin.sh
Security Admin v7
Will connect to localhost:9300 ... done
Connected as CN=log...,C=DE
OpenSearch Version: 1.2.3
OpenSearch Security Version: 1.2.3.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: log
Clusterstate: YELLOW
Number of nodes: 2
Number of data nodes: 2
.opendistro_security index already exists, so we do not need to create one.
Populate config from /usr/share/opensearch/plugins/opensearch-security/securityconfig/
Will update '_doc/config' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml
   SUCC: Configuration for 'config' created or updated
Will update '_doc/roles' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml
   SUCC: Configuration for 'roles' created or updated
Will update '_doc/rolesmapping' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '_doc/internalusers' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update '_doc/actiongroups' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Will update '_doc/tenants' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/tenants.yml
   SUCC: Configuration for 'tenants' created or updated
Will update '_doc/nodesdn' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/nodes_dn.yml
   SUCC: Configuration for 'nodesdn' created or updated
Will update '_doc/whitelist' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/whitelist.yml
   SUCC: Configuration for 'whitelist' created or updated
Will update '_doc/audit' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/audit.yml
   SUCC: Configuration for 'audit' created or updated
Done with success

As I uploaded all files the “all_access” mapping is gone.

Edit: This command was helping:

$ sudo curl --insecure --cert ./config/my-kirk.pem --key ./config/my-kirk-key.pem -XGET "https://log1:9200/_opendistro/_security/authinfo?pretty"
{
  "user" : "User [name=CN=log...,C=DE, backend_roles=[], requestedTenant=null]",
  "user_name" : "CN=log...,C=DE",
  "user_requested_tenant" : null,
  "remote_address" : null,
  "backend_roles" : [ ],
  "custom_attribute_names" : [ ],
  "roles" : [
    "own_index"
  ],
  "tenants" : {
    "CN=log...,C=DE" : true
  },
  "principal" : "CN=log...,C=DE",
  "peer_certificates" : "4",
  "sso_logout_url" : null
}

You can see that the certificates are working but have only the own_index role. That lead me to the idea to open the all_access role temporary for all users.

2 Likes

thank you for coming back to the forum and posting the answer @eskorte - I’m sure this will be useful to other community members in the future!

1 Like