TLS error on multi-node cluster

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OS: 2.10.0
OD: 2.10.0

Describe the issue:
I am trying to configure multi-node cluster using helm chart, i created the tls certs using istio cert-manager. everything works fine when i go with the normal setup(node as both master and data).
but when i try to deploy the multi-node cluster, i am facing the tls error.

[2023-12-13T11:44:43,622][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [opensearch-cluster-data-0] Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:

Configuration:

security

 network.host: 0.0.0.0
 plugins:
  security:
    nodes_dn:
      - 'CN=opensearch.dev.abcd.net'
      - 'CN=opensearch-cluster-master.logquery.svc.cluster.local'
      - 'CN=opensearch-cluster-data.logquery.svc.cluster.local'
      - 'CN=*.cluster.local'
    ssl:
      transport:
        pemcert_filepath: certs/tls.crt
        pemkey_filepath: certs/tls.key
        pemtrustedcas_filepath: certs/tls.crt
        enforce_hostname_verification: false
        resolve_hostname: false
      http:
        enabled: true
        pemcert_filepath: certs/tls.crt
        pemkey_filepath: certs/tls.key
        pemtrustedcas_filepath: certs/tls.crt
    allow_unsafe_democertificates: false
    allow_default_init_securityindex: true
    authcz:
      admin_dn:
        - CN=opensearch.dev.abcd.net
        - CN=opensearch-cluster-master.logquery.svc.cluster.local
        - CN=opensearch-cluster-data.logquery.svc.cluster.local
    audit.type: internal_opensearch
    enable_snapshot_restore_privilege: true
    check_snapshot_restore_write_privileges: true
    restapi:
      roles_enabled: ["all_access", "security_rest_api_access"]

secret mount

secretMounts:

  • name: opensearch-certs
    secretName: tls-for-opensearch
    path: /usr/share/opensearch/config/certs
    defaultMode: 0400

Can someone help me on this issue.
Thanks in advance.

2

hi @pablo, @jasonrojas any advice here.

3

This could mean the client connects to HTTP endpoint with HTTPS or HTTPS endpoint with HTTP.

Did you use official OpenSearch helm charts or custom ones?

4

Hi @pablo , thanks for the response.
the issue resolved after disabled the labels which used to scrape the jvm metrics.

podAnnotations:
sidecar.istio.io/inject: “false”
#prometheus.io/scrape: “true”
#prometheus.io/path: “/_prometheus/metrics”
#prometheus.io/port: “9200”

but now i am not able to scrape the jvm metrics :pensive:

5

@arun_udaiyar I’m unfamiliar with Prometheus but does that config tries to connect with OpenSearch HTTP endpoint 9200?

If so, maybe you should consider Nginx reverse proxy that would route HTTP to HTTPS traffic.
If you run it in the Kubernetes environment, ingress could do the same trick.

6

Hi @pablo ,
Got it working now,
i used ServiceMonitor to give the tls certs to prometheus and removed this labels now its working.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: opensearch-metrics-tls
namespace: opensearch-ns
labels:
release: kube-prometheus-stack
spec:
endpoints:
- path: /_prometheus/metrics
port: http
scheme: https
tlsConfig:
insecureSkipVerify: true
ca:
secret:
name: tls-for-opensearch
key: tls.crt
cert:
secret:
name: tls-for-opensearch
key: tls.crt
keySecret:
name: tls-for-opensearch
key: tls.key
selector:
matchLabels:
app.kubernetes.io/name: opensearch