Failed to connect 2.14.0

[nap608]

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Describe the issue:
Hello.
I’m testing upgrade from 2.4.0 to 2.14.0
I have two windows environments one with 2.4.0 and one with 2.14.0
in the 2.4.0 it works all good, in the 2.14.0 dashboard is down and logs errors shows “Response error”

The machines are identical, and the yaml settings are identical, so its something about the package that is different.

I did the same exact test in linux and it worked for me, so issue is windows only.

I’ve also changed opensearch-dashboards.bat
I added quotes around the call for ‘use_node.bat’
because it just didn’t run at all without them the double quotes.

Configuration:
opensearch.yml:

######## Start OpenSearch Security Demo Configuration ######## 
# WARNING: revise all the lines below before you go into production 
plugins.security.ssl.transport.pemcert_filepath: certificates/elk-transport-crt.pem
plugins.security.ssl.transport.pemkey_filepath: certificates/elk-transport-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: certificates/elastic_ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false 
plugins.security.ssl.http.enabled: true 
plugins.security.ssl.http.pemcert_filepath: certificates/elk-node-crt.pem
plugins.security.ssl.http.pemkey_filepath: certificates/elk-node-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: certificates/elastic_ca.pem
plugins.security.allow_unsafe_democertificates: true 
plugins.security.allow_default_init_securityindex: true 
plugins.security.authcz.admin_dn: 
  - 'CN=CONTROL-M_EM_ES_admin,O=ChangeMe,L=ChangeMeL,ST=ChangeMeST,C=CM'

 
#plugins.security.audit.type: internal_opensearch 
plugins.security.enable_snapshot_restore_privilege: true 
plugins.security.check_snapshot_restore_write_privileges: true 
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"] 
plugins.security.system_indices.enabled: true 
plugins.security.system_indices.indices: [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"] 
node.max_local_storage_nodes: 3 
######## End OpenSearch Security Demo Configuration ######## 
path.logs: C:\Program Files\BMC Software\Control-M EM\Default\log\services\workflow_insights
cluster.name: workflow_insights_cluster
network.host: 0
node.name: dba-tlv-wpbu0y
node.roles: ["initial_master","data","ingest","master","remote_cluster_client"]
action.auto_create_index: false
bootstrap.memory_lock: true
http.port: 19200
transport.port: 19300
cluster.initial_master_nodes: ['dba-tlv-wpbu0y']
discovery.seed_hosts: ['dba-tlv-wpbu0y']
plugins.security.nodes_dn: 
  - 'CN=CONTROL-M_EM_ES_transport,O=ChangeMe,L=ChangeMeL,ST=ChangeMeST,C=CM'

opensearch-dashboards.yml

---
server.port: 15061
server.host: '0.0.0.0'
opensearch.hosts: ['https://localhost:19200']
opensearch.ssl.verificationMode: certificate
opensearch.requestHeadersWhitelist: ["securitytenant","Authorization"]

opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: ["Global"]
opensearch_security.multitenancy.tenants.enable_private: false
opensearch_security.readonly_mode.roles: ["kibana_read_only","kibana_read"]
opensearch_security.cookie.secure: false

# Use this setting if you are running opensearch-dashboards without https
#server.rewriteBasePath: true
server.basePath: "/services-proxy/kibana"
csp.warnLegacyBrowsers: false
csp.rules:
 - "script-src 'unsafe-eval' 'self' 'unsafe-inline'"
# Options for the logging rotate feature
logging.rotate:
   enabled: true
   everyBytes: 10000000
   keepFiles: 5
   usePolling: true
   pollingInterval: 10000

#logging.quiet : true
logging.verbose: true
opensearch.ssl.certificateAuthorities: ['C:\\Program Files\\BMC Software\\Control-M EM\\Default\\ini\\ssl\\elastic_ca.pem']
#logging.dest: C:\Program Files\BMC Software\Control-M EM\Default\log\services\workflow_insights\opensearch-dashboards.log
server.ssl.enabled: true
server.ssl.key: C:\Program Files\BMC Software\Control-M EM\Default\opensearch-dashboards\config\certificates\kibana-node-key.pem
server.ssl.certificate: C:\Program Files\BMC Software\Control-M EM\Default\opensearch-dashboards\config\certificates\kibana-node-crt.pem

opensearch.ssl.key: C:\Program Files\BMC Software\Control-M EM\Default\opensearch-dashboards\config\certificates\kibana-node-key.pem
opensearch.ssl.certificate: C:\Program Files\BMC Software\Control-M EM\Default\opensearch-dashboards\config\certificates\kibana-node-crt.pem

Relevant Logs or Screenshots:

[pablo]

@nap608 What do you see in the OpenSearch logs?
How many nodes do you have in the cluster?

Do you get any response when running the following API GET call?

curl --insecure -u admin:<password> -XGET https://localhost:19200

[nap608]

Hi @pablo

Blockquote What do you see in the OpenSearch logs?
nothing out of the ordinary, clean log.

Blockquote How many nodes do you have in the cluster?
1

Blockquote Do you get any response when running the following API GET call?
Yes, I can access the API.
what do you want to run?

[pablo]

@nap608 Could you share the output of that API?

curl --insecure -u admin:<password> -XGET https://localhost:19200

[nap608]

curl command output

opensearch.yml

opensearch dashboards.yml

[nap608]

@pablo could it be related to the way i define the users and roles?

[nap608]

Hi @pablo

When I’ve tried to add this fields to the opensearch-dashboards.yml

It worked but with this exception:

[pablo]

@nap608 Your change has enabled basic authentication in OpenSearch Dashboards with OpenSearch cluster.
Maybe there is an issue with the certificate (opensearch.ssl.certificate) or certificate authentication wasn’t enabled in OpenSearch’s security plugin.

Could you share the output of the following command?

curl --insecure -u admin:<password> https://<OpenSearch_node>:19200/_plugins/_security/api/securityconfig?pretty

[nap608]

Hi @pablo ,

Do you get any response when running the following API GET call?

{
  "config" : {
    "dynamic" : {
      "filtered_alias_mode" : "warn",
      "disable_rest_auth" : false,
      "disable_intertransport_auth" : false,
      "respect_request_indices_options" : false,
      "kibana" : {
        "multitenancy_enabled" : true,
        "private_tenant_enabled" : true,
        "default_tenant" : "",
        "server_username" : "kibanaserver",
        "index" : ".kibana",
        "sign_in_options" : [
          "BASIC"
        ]
      },
      "http" : {
        "anonymous_auth_enabled" : false,
        "xff" : {
          "enabled" : false,
          "internalProxies" : "192\\.168\\.0\\.10|192\\.168\\.0\\.11",
          "remoteIpHeader" : "X-Forwarded-For"
        }
      },
      "authc" : {
        "jwt_auth_domain" : {
          "http_enabled" : false,
          "order" : 0,
          "http_authenticator" : {
            "challenge" : false,
            "type" : "jwt",
            "config" : {
              "signing_key" : "base64 encoded HMAC key or public RSA/ECDSA pem key",
              "jwt_header" : "Authorization",
              "jwt_clock_skew_tolerance_seconds" : 30
            }
          },
          "authentication_backend" : {
            "type" : "noop",
            "config" : { }
          },
          "description" : "Authenticate via Json Web Token"
        },
        "ldap" : {
          "http_enabled" : false,
          "order" : 5,
          "http_authenticator" : {
            "challenge" : false,
            "type" : "basic",
            "config" : { }
          },
          "authentication_backend" : {
            "type" : "ldap",
            "config" : {
              "enable_ssl" : false,
              "enable_start_tls" : false,
              "enable_ssl_client_auth" : false,
              "verify_hostnames" : true,
              "hosts" : [
                "localhost:8389"
              ],
              "userbase" : "ou=people,dc=example,dc=com",
              "usersearch" : "(sAMAccountName={0})"
            }
          },
          "description" : "Authenticate via LDAP or Active Directory"
        },
        "basic_internal_auth_domain" : {
          "http_enabled" : true,
          "order" : 4,
          "http_authenticator" : {
            "challenge" : true,
            "type" : "basic",
            "config" : { }
          },
          "authentication_backend" : {
            "type" : "intern",
            "config" : { }
          },
          "description" : "Authenticate via HTTP Basic against internal users database"
        },
        "proxy_auth_domain" : {
          "http_enabled" : false,
          "order" : 3,
          "http_authenticator" : {
            "challenge" : false,
            "type" : "proxy",
            "config" : {
              "user_header" : "x-proxy-user",
              "roles_header" : "x-proxy-roles"
            }
          },
          "authentication_backend" : {
            "type" : "noop",
            "config" : { }
          },
          "description" : "Authenticate via proxy"
        },
        "clientcert_auth_domain" : {
          "http_enabled" : false,
          "order" : 2,
          "http_authenticator" : {
            "challenge" : false,
            "type" : "clientcert",
            "config" : {
              "username_attribute" : "cn"
            }
          },
          "authentication_backend" : {
            "type" : "noop",
            "config" : { }
          },
          "description" : "Authenticate via SSL client certificates"
        },
        "kerberos_auth_domain" : {
          "http_enabled" : false,
          "order" : 6,
          "http_authenticator" : {
            "challenge" : true,
            "type" : "kerberos",
            "config" : {
              "krb_debug" : false,
              "strip_realm_from_principal" : true
            }
          },
          "authentication_backend" : {
            "type" : "noop",
            "config" : { }
          }
        }
      },
      "authz" : {
        "roles_from_another_ldap" : {
          "http_enabled" : false,
          "authorization_backend" : {
            "type" : "ldap",
            "config" : { }
          },
          "description" : "Authorize via another Active Directory"
        },
        "roles_from_myldap" : {
          "http_enabled" : false,
          "authorization_backend" : {
            "type" : "ldap",
            "config" : {
              "enable_ssl" : false,
              "enable_start_tls" : false,
              "enable_ssl_client_auth" : false,
              "verify_hostnames" : true,
              "hosts" : [
                "localhost:8389"
              ],
              "rolebase" : "ou=groups,dc=example,dc=com",
              "rolesearch" : "(member={0})",
              "userrolename" : "disabled",
              "rolename" : "cn",
              "resolve_nested_roles" : true,
              "userbase" : "ou=people,dc=example,dc=com",
              "usersearch" : "(uid={0})"
            }
          },
          "description" : "Authorize via LDAP or Active Directory"
        }
      },
      "auth_failure_listeners" : { },
      "do_not_fail_on_forbidden" : false,
      "multi_rolespan_enabled" : true,
      "hosts_resolver_mode" : "ip-only",
      "do_not_fail_on_forbidden_empty" : false,
      "on_behalf_of" : {
        "enabled" : false
      }
    }
  }
}

opensearch-dashboard yml

[pablo]

@nap608 Thanks for sharing the config. As per config.yml, only basic authentication is enabled in OpenSearch security plugin.
If you want to use certificate authentication between OpenSearch Dashboards and OpenSearch (opensearch.ssl.certificate, opensearch.ssl.key) you must enable certificate authentication (clientcert_auth_domain).

Client certificate authentication

[nap608]

Its the default config.yml, not changed.
How exactly am I enabling it?

---

This is the main OpenSearch Security configuration file where authentication

and authorization is defined.

You need to configure at least one authentication domain in the authc of this file.

An authentication domain is responsible for extracting the user credentials from

the request and for validating them against an authentication backend like Active Directory for example.

If more than one authentication domain is configured the first one which succeeds wins.

If all authentication domains fail then the request is unauthenticated.

In this case an exception is thrown and/or the HTTP status is set to 401.

After authentication authorization (authz) will be applied. There can be zero or more authorizers which collect

the roles from a given backend for the authenticated user.

Both, authc and auth can be enabled/disabled separately for REST and TRANSPORT layer. Default is true for both.

http_enabled: true

transport_enabled: true

For HTTP it is possible to allow anonymous authentication. If that is the case then the HTTP authenticators try to

find user credentials in the HTTP request. If credentials are found then the user gets regularly authenticated.

If none can be found the user will be authenticated as an “anonymous” user. This user has always the username “anonymous”

and one role named “anonymous_backendrole”.

If you enable anonymous authentication all HTTP authenticators will not challenge.

Note: If you define more than one HTTP authenticators make sure to put non-challenging authenticators like “proxy” or “clientcert”

first and the challenging one last.

Because it’s not possible to challenge a client with two different authentication methods (for example

Kerberos and Basic) only one can have the challenge flag set to true. You can cope with this situation

by using pre-authentication, e.g. sending a HTTP Basic authentication header in the request.

Default value of the challenge flag is true.

HTTP

basic (challenging)

proxy (not challenging, needs xff)

kerberos (challenging)

clientcert (not challenging, needs https)

jwt (not challenging)

host (not challenging) #DEPRECATED, will be removed in a future version.

host based authentication is configurable in roles_mapping

Authc

internal

noop

ldap

Authz

ldap

noop

_meta:
type: “config”
config_version: 2

config:
dynamic:
# Set filtered_alias_mode to ‘disallow’ to forbid more than 2 filtered aliases per index
# Set filtered_alias_mode to ‘warn’ to allow more than 2 filtered aliases per index but warns about it (default)
# Set filtered_alias_mode to ‘nowarn’ to allow more than 2 filtered aliases per index silently
#filtered_alias_mode: warn
#do_not_fail_on_forbidden: false
#kibana:
# Kibana multitenancy
#multitenancy_enabled: true
#private_tenant_enabled: true
#default_tenant: “”
#server_username: kibanaserver
#index: ‘.kibana’
http:
anonymous_auth_enabled: false
xff:
enabled: false
internalProxies: ‘192.168.0.10|192.168.0.11’ # regex pattern
#internalProxies: ‘.’ # trust all internal proxies, regex pattern
#remoteIpHeader: ‘x-forwarded-for’
###### see Pattern (Java Platform SE 7 ) for regex help
###### more information about XFF X-Forwarded-For - Wikipedia
###### and here RFC 7239 - Forwarded HTTP Extension
###### and Apache Tomcat 8 Configuration Reference (8.0.53) - The Valve Component
authc:
kerberos_auth_domain:
http_enabled: false
transport_enabled: false
order: 6
http_authenticator:
type: kerberos
challenge: true
config:
# If true a lot of kerberos/security related debugging output will be logged to standard out
krb_debug: false
# If true then the realm will be stripped from the user name
strip_realm_from_principal: true
authentication_backend:
type: noop
basic_internal_auth_domain:
description: “Authenticate via HTTP Basic against internal users database”
http_enabled: true
transport_enabled: true
order: 4
http_authenticator:
type: basic
challenge: true
authentication_backend:
type: intern
proxy_auth_domain:
description: “Authenticate via proxy”
http_enabled: false
transport_enabled: false
order: 3
http_authenticator:
type: proxy
challenge: false
config:
user_header: “x-proxy-user”
roles_header: “x-proxy-roles”
authentication_backend:
type: noop
jwt_auth_domain:
description: “Authenticate via Json Web Token”
http_enabled: false
transport_enabled: false
order: 0
http_authenticator:
type: jwt
challenge: false
config:
signing_key: “base64 encoded HMAC key or public RSA/ECDSA pem key”
jwt_header: “Authorization”
jwt_url_parameter: null
jwt_clock_skew_tolerance_seconds: 30
roles_key: null
subject_key: null
authentication_backend:
type: noop
clientcert_auth_domain:
description: “Authenticate via SSL client certificates”
http_enabled: false
transport_enabled: false
order: 2
http_authenticator:
type: clientcert
config:
username_attribute: cn #optional, if omitted DN becomes username
challenge: false
authentication_backend:
type: noop
ldap:
description: “Authenticate via LDAP or Active Directory”
http_enabled: false
transport_enabled: false
order: 5
http_authenticator:
type: basic
challenge: false
authentication_backend:
# LDAP authentication backend (authenticate users against a LDAP or Active Directory)
type: ldap
config:
# enable ldaps
enable_ssl: false
# enable start tls, enable_ssl should be false
enable_start_tls: false
# send client certificate
enable_ssl_client_auth: false
# verify ldap hostname
verify_hostnames: true
hosts:
- localhost:8389
bind_dn: null
password: null
userbase: ‘ou=people,dc=example,dc=com’
# Filter to search for users (currently in the whole subtree beneath userbase)
# {0} is substituted with the username
usersearch: ‘(sAMAccountName={0})’
# Use this attribute from the user as username (if not set then DN is used)
username_attribute: null
authz:
roles_from_myldap:
description: “Authorize via LDAP or Active Directory”
http_enabled: false
transport_enabled: false
authorization_backend:
# LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)
type: ldap
config:
# enable ldaps
enable_ssl: false
# enable start tls, enable_ssl should be false
enable_start_tls: false
# send client certificate
enable_ssl_client_auth: false
# verify ldap hostname
verify_hostnames: true
hosts:
- localhost:8389
bind_dn: null
password: null
rolebase: ‘ou=groups,dc=example,dc=com’
# Filter to search for roles (currently in the whole subtree beneath rolebase)
# {0} is substituted with the DN of the user
# {1} is substituted with the username
# {2} is substituted with an attribute value from user’s directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute
rolesearch: ‘(member={0})’
# Specify the name of the attribute which value should be substituted with {2} above
userroleattribute: null
# Roles as an attribute of the user entry
userrolename: disabled
#userrolename: memberOf
# The attribute in a role entry containing the name of that role, Default is “name”.
# Can also be “dn” to use the full DN as rolename.
rolename: cn
# Resolve nested roles transitive (roles which are members of other roles and so on …)
resolve_nested_roles: true
userbase: ‘ou=people,dc=example,dc=com’
# Filter to search for users (currently in the whole subtree beneath userbase)
# {0} is substituted with the username
usersearch: ‘(uid={0})’
# Skip users matching a user name, a wildcard or a regex pattern
#skip_users:
# - 'cn=Michael Jackson,oupeople,o=TEST’
# - ‘/\S*/’
roles_from_another_ldap:
description: “Authorize via another Active Directory”
http_enabled: false
transport_enabled: false
authorization_backend:
type: ldap
#config goes here …

auth_failure_listeners:

ip_rate_limiting:

type: ip

allowed_tries: 10

time_window_seconds: 3600

block_expiry_seconds: 600

max_blocked_clients: 100000

max_tracked_clients: 100000

internal_authentication_backend_limiting:

type: username

authentication_backend: intern

allowed_tries: 10

time_window_seconds: 3600

block_expiry_seconds: 600

max_blocked_clients: 100000

max_tracked_clients: 100000

[pablo]

@nap608 Your config is default. If that was working for you in the other cluster, then just copy client certificate authentication section from security configuration from that cluster and apply to this one.
Alternatively follow the shared documentation.

[nap608]

I did as you requested and finally I get different error in the logs:

on the opensearch log i get:

[2024-07-24T14:57:43,379][INFO ][o.o.s.p.PrivilegesEvaluator] [dba-tlv-wqfst2] No cluster-level perm match for User [name=dba-tlv-wqfst2, backend_roles=, requestedTenant=null] Resolved [aliases=[], allIndices=[], types=[], originalRequested=[], remoteIndices=] [Action [cluster:monitor/nodes/info]] [RolesChecked [own_index]]. No permissions for [cluster:monitor/nodes/info]

on opensearch dashboards log i get:

{“type”:“log”,“@timestamp”:“2024-07-24T11:58:43Z”,“tags”:[“error”,“opensearch”,“data”],“pid”:6660,“message”:“[security_exception]: no permissions for [cluster:monitor/nodes/info] and User [name=dba-tlv-wqfst2, backend_roles=, requestedTenant=null]”}

I feel i’m missing something in the users and roles definition, IDK which user and which role im missing.

also dont understand how I assign a certificate a role.

[pablo]

@nap608 Based on that error message, the CN in the certificate is dba-tlv-wqfst2.
The default user is kibanaserver and has already proper role assigned.

Check if you have kibana_server role in your roles_mapping.yml. If so, edit it and add user dba-tlv-wqfst2

kibana_server:
  reserved: true
  hidden: false
  backend_roles: []
  hosts: []
  users:
  - "kibanaserver"
  - "dba-tlv-wqfst2"
  and_backend_roles: []