TLS/SSL multinode

Security
1

Versions
OpenSearch 2.15
Ubutnu 22.04

Describe the issue:
On Manager node i have this

# /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -cacert /etc/opensearch/ca.pem -cert /etc/opensearch/cert.pem -key /etc/opensearch/key_pks.pem -cd /etc/opensearch/opensearch-security --accept-red-cluster -h rnd-os-mgr.devops.nova
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to rnd-os-mgr.devops.nova:9200 ... done
Connected as "CN=rnd-os-mgr.devops.nova,OU=IT,O=OP,C=EN"
OpenSearch Version: 2.15.0
Contacting opensearch cluster 'opensearch' ...
Clustername: opensearch
Clusterstate: RED
Number of nodes: 1
Number of data nodes: 0
.opendistro_security index already exists, so we do not need to create one.
ERR: .opendistro_security index state is RED.
Populate config from /etc/opensearch/opensearch-security/
Will update '/config' with /etc/opensearch/opensearch-security/config.yml 
   FAIL: Configuration for 'config' failed because of java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-6 [ACTIVE]
Will update '/roles' with /etc/opensearch/opensearch-security/roles.yml

and in this in log

[2024-07-02T16:40:52,673][ERROR][o.o.s.t.SecurityRequestHandler] [rnd-os-mgr] OpenSearchException[Transport client authentication no longer supported.]

On rnd-os-node01 i have this in log

[2024-07-02T16:42:40,581][WARN ][o.o.d.HandshakingTransportAddressConnector] [rnd-os-node01] handshake failed for [connectToRemoteMasterNode[172.24.49.127:9300]]
======
root@rnd-os-node01:/etc/opensearch# /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -cacert /etc/opensearch/ca.pem -cert /etc/opensearch/cert.pem -key /etc/opensearch/key_pks.pem -cd /etc/opensearch/opensearch-security --accept-red-cluster -h rnd-os-node01.devops.nova
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to rnd-os-node01.devops.nova:9200 ... done
Connected as "CN=rnd-os-mgr.devops.nova,OU=IT,O=OP,C=EN"
OpenSearch Version: 2.15.0
Contacting opensearch cluster 'opensearch' ...
Cannot retrieve cluster state due to: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE]. This is not an error, will keep on trying ...
  Root cause: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE] (java.net.SocketTimeoutException/java.net.SocketTimeoutException)
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.

Configuration:
Node01

cluster.name: opensearch
node.roles: data, ingest
node.name: rnd-os-node01
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
network.host: 0.0.0.0
discovery.seed_hosts: ["rnd-os-mgr", "rnd-os-node01", "rnd-os-node02"]
cluster.initial_cluster_manager_nodes: ["rnd-os-mgr.devops.nova"]
plugins.security.ssl.transport.pemcert_filepath: cert.pem
plugins.security.ssl.transport.pemkey_filepath: key_pkcs.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: cert.pem
plugins.security.ssl.http.pemkey_filepath: key_pkcs.pem
plugins.security.ssl.http.pemtrustedcas_filepath: ca.pem
plugins.security.allow_unsafe_democertificates: false
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=rnd-os-mgr.devops.nova, OU=IT, O=OP, C=EN
  - C=EN, O=OP, OU=IT, CN=rnd-os-mgr.devops.nova
plugins.security.nodes_dn:
  - CN=rnd-os-mgr.devops.nova, OU=IT, O=OP, C=EN
  - C=EN, O=OP, OU=IT, CN=rnd-os-mgr.devops.nova
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [all_access, security_rest_api_access]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [.plugins-ml-agent, .plugins-ml-config, .plugins-ml-connector,
  .plugins-ml-controller, .plugins-ml-model-group, .plugins-ml-model, .plugins-ml-task,
  .plugins-ml-conversation-meta, .plugins-ml-conversation-interactions, .plugins-ml-memory-meta,
  .plugins-ml-memory-message, .plugins-ml-stop-words, .opendistro-alerting-config,
  .opendistro-alerting-alert*, .opendistro-anomaly-results*, .opendistro-anomaly-detector*,
  .opendistro-anomaly-checkpoints, .opendistro-anomaly-detection-state, .opendistro-reports-*,
  .opensearch-notifications-*, .opensearch-notebooks, .opensearch-observability, .ql-datasources,
  .opendistro-asynchronous-search-response*, .replication-metadata-store, .opensearch-knn-models,
  .geospatial-ip2geo-data*, .plugins-flow-framework-config, .plugins-flow-framework-templates,
  .plugins-flow-framework-state]
node.max_local_storage_nodes: 3

Manager

node.roles: cluster_manager
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
network.host: 0.0.0.0
discovery.seed_hosts: ["rnd-os-mgr", "rnd-os-node01", "rnd-os-node01"]
cluster.initial_cluster_manager_nodes: ["rnd-os-mgr"]
plugins.security.ssl.transport.pemcert_filepath: cert.pem
plugins.security.ssl.transport.pemkey_filepath: key_pkcs.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: cert.pem
plugins.security.ssl.http.pemkey_filepath: key_pkcs.pem
plugins.security.ssl.http.pemtrustedcas_filepath: ca.pem
plugins.security.allow_unsafe_democertificates: false
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=rnd-os-mgr.devops.nova, OU=IT, O=OP, C=EN
  - C=EN, O=OP, OU=IT, CN=rnd-os-mgr.devops.nova
plugins.security.nodes_dn:
  - CN=rnd-os-mgr.devops.nova, OU=IT, O=OP, C=EN
  - C=EN, O=OP, OU=IT, CN=rnd-os-mgr.devops.nova
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [all_access, security_rest_api_access]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [.plugins-ml-agent, .plugins-ml-config, .plugins-ml-connector,
  .plugins-ml-controller, .plugins-ml-model-group, .plugins-ml-model, .plugins-ml-task,
  .plugins-ml-conversation-meta, .plugins-ml-conversation-interactions, .plugins-ml-memory-meta,
  .plugins-ml-memory-message, .plugins-ml-stop-words, .opendistro-alerting-config,
  .opendistro-alerting-alert*, .opendistro-anomaly-results*, .opendistro-anomaly-detector*,
  .opendistro-anomaly-checkpoints, .opendistro-anomaly-detection-state, .opendistro-reports-*,
  .opensearch-notifications-*, .opensearch-notebooks, .opensearch-observability, .ql-datasources,
  .opendistro-asynchronous-search-response*, .replication-metadata-store, .opensearch-knn-models,
  .geospatial-ip2geo-data*, .plugins-flow-framework-config, .plugins-flow-framework-templates,
  .plugins-flow-framework-state]
node.max_local_storage_nodes: 3
2

Hi @DmitriiKuvshinov ,

Could you please share your config.yml file?

3

It’s a default configuration from package (install with apt-manager)
Main problem is - i can’t create cluster
On manager

root@rnd-os-mgr:~# curl -XGET https://rnd-os-mgr.devops.nova:9200/_cat/nodes?v -u 'admin:0G1hcd7klg[JESuJDzmC' --cert /etc/opensearch/cert.pem --key /etc/opensearch/key_pks.pem
ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role node.roles      cluster_manager name
172.24.49.121           58          92   3    0.09    0.07     0.03 m         cluster_manager *               rnd-os-mgr

On node01

root@rnd-os-node01:/etc/opensearch# curl -XGET https://rnd-os-node01.devops.nova:9200/_cat/nodes?v -u 'admin:0G1hcd7klg[JESuJDzmC' --cert /etc/opensearch/cert.pem --key /etc/opensearch/key_pks.pem
{"error":{"root_cause":[{"type":"cluster_manager_not_discovered_exception","reason":null}],"type":"cluster_manager_not_discovered_exception","reason":null},"status":503}

config.yml

---
_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
    authc:
      kerberos_auth_domain:
        http_enabled: false
        transport_enabled: false
        order: 6
        http_authenticator:
          type: kerberos
          challenge: true
          config:
            krb_debug: false
            strip_realm_from_principal: true
        authentication_backend:
          type: noop
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 4
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern
      proxy_auth_domain:
        description: "Authenticate via proxy"
        http_enabled: false
        transport_enabled: false
        order: 3
        http_authenticator:
          type: proxy
          challenge: false
          config:
            user_header: "x-proxy-user"
            roles_header: "x-proxy-roles"
        authentication_backend:
          type: noop
      jwt_auth_domain:
        description: "Authenticate via Json Web Token"
        http_enabled: false
        transport_enabled: false
        order: 0
        http_authenticator:
          type: jwt
          challenge: false
          config:
            signing_key: "base64 encoded HMAC key or public RSA/ECDSA pem key"
            jwt_header: "Authorization"
            jwt_url_parameter: null
            jwt_clock_skew_tolerance_seconds: 30
            roles_key: null
            subject_key: null
        authentication_backend:
          type: noop
      clientcert_auth_domain:
        description: "Authenticate via SSL client certificates"
        http_enabled: false
        transport_enabled: false
        order: 2
        http_authenticator:
          type: clientcert
          config:
          challenge: false
        authentication_backend:
          type: noop
      ldap:
        description: "Authenticate via LDAP or Active Directory"
        http_enabled: false
        transport_enabled: false
        order: 5
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: ldap
          config:
            enable_ssl: false
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: true
            hosts:
            - localhost:8389
            bind_dn: null
            password: null
            userbase: 'ou=people,dc=example,dc=com'
            usersearch: '(sAMAccountName={0})'
            username_attribute: null
    authz:
      roles_from_myldap:
        description: "Authorize via LDAP or Active Directory"
        http_enabled: false
        transport_enabled: false
        authorization_backend:
          type: ldap
          config:
            enable_ssl: false
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: true
            hosts:
            - localhost:8389
            bind_dn: null
            password: null
            rolebase: 'ou=groups,dc=example,dc=com'
            rolesearch: '(member={0})'
            userroleattribute: null
            userrolename: disabled
            rolename: cn
            resolve_nested_roles: true
            userbase: 'ou=people,dc=example,dc=com'
            usersearch: '(uid={0})'
      roles_from_another_ldap:
        description: "Authorize via another Active Directory"
        http_enabled: false
        transport_enabled: false
        authorization_backend:
          type: ldap
4

And every yml file in /etc/opensearch/opensearch-security fail with error

   FAIL: Configuration for 'config' failed because of java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-6 [ACTIVE]
5

So
Magic happen
Fix node config (set NOT fqdn of master node) and node connect to master

1 Like