Versions: Opensearch 3.1.0, Helm Opensearch Operator 2.8.0
Describe the issue:
I’m trying to set up Cross Cluster Replication (CCR) between two OpenSearch clusters running on separate Kubernetes clusters. Both are exposed via LoadBalancer services. The OpenSearch Operator was installed via Helm, but the OpenSearch clusters themselves were deployed using kubectl apply with custom YAML manifests. TLS is enabled and auto-generated (generate: true) on both clusters.
I followed the official documentation from here and adjusted the CN/OU settings in the nodes_dn configuration so that each cluster recognizes the other’s nodes. However, when I try to configure the remote cluster on the follower using this command:
curl -XPUT -k -H 'Content-Type: application/json' -u 'admin:MyPassword' 'https://FOLLOWER-LOADBALANCER-IP:9200/_cluster/settings?pretty' -d '
{
"persistent": {
"cluster": {
"remote": {
"my-connection-alias": {
"mode": "proxy",
"proxy_address": "LEADER-LOADBALANCER-IP:9300"
}
}
}
}
}'
I get this error:
{
"my-connection-alias" : {
"connected" : false,
"mode" : "proxy",
"proxy_address" : "LEADER-LOADBALANCER-IP:9300",
"server_name" : "",
"num_proxy_sockets_connected" : 0,
"max_proxy_socket_connections" : 18,
"initial_connect_timeout" : "30s",
"skip_unavailable" : false
}
}
and pod logs give this errors
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
I’m stuck at this point and would appreciate any guidance or suggestions.
Configuration:
I installed OpenSearch and OpenSearch Operator on both clusters using the steps below. Because the CN and OU sections weren’t the same, I made the necessary changes to the other cluster’s YAML file. I changed the CN and OU values because I didn’t think it would be appropriate to share them on the forum.
kubectl apply -f my-secrets.yaml -n opensearch
kubectl create namespace opensearch-operator-system
helm install opensearch-operator opensearch-operator/opensearch-operator -n opensearch-operator-system
leader-values.yaml
apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
name: opensearch-leader-cluster
namespace: opensearch
spec:
general:
serviceName: leader-cluster
version: 3.1.0
setVMMaxMapCount: true
additionalConfig:
plugins.security.nodes_dn: |
- "CN=follower-cluster-masters-1,OU=follower-cluster"
- "CN=follower-cluster-nodes-0,OU=follower-cluster
dashboards:
enable: true
opensearchCredentialsSecret:
name: dashboards-credentials
tls:
enable: true
generate: true # Have the operator generate and sign a certificate
version: 3.1.0
replicas: 1
resources:
requests:
memory: "515Mi"
cpu: "200m"
limits:
memory: "515Mi"
cpu: "200m"
additionalConfig:
opensearch_security.multitenancy.enabled: "true"
nodePools:
- component: masters
replicas: 2
diskSize: "5Gi"
resources:
requests:
memory: "1Gi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "1000m"
roles:
- "cluster_manager"
- "remote_cluster_client"
- component: nodes
replicas: 2
diskSize: "10Gi"
resources:
requests:
memory: "2Gi"
cpu: "1005m"
limits:
memory: "3Gi"
cpu: "2000m"
roles:
- "data"
- "remote_cluster_client"
- component: coordinators
replicas: 2
diskSize: "3Gi"
resources:
requests:
memory: "1Gi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "1000m"
roles:
- "ingest"
- "remote_cluster_client"
security:
config:
adminCredentialsSecret:
name: admin-credentials-secret
securityConfigSecret:
name: securityconfig-secret
tls: # Everything related to TLS configuration
transport: # Configuration of the transport endpoint
generate: true # Have the operator generate and sign certificates
perNode: true # Separate certificate per node
http: # Configuration of the http endpoint
generate: true
After applying this YAML file, and after all the pods were seen running I used the following command for the LoadBalancer.
kubectl -n opensearch patch svc leader-cluster \
--type='merge' \
-p '{
"spec": {
"type": "LoadBalancer",
"ports": [
{"port": 9200, "targetPort": 9200, "protocol": "TCP", "name": "http"},
{"port": 9300, "targetPort": 9300, "protocol": "TCP", "name": "transport"},
{"port": 9600, "targetPort": 9600, "protocol": "TCP", "name": "metrics"},
{"port": 9650, "targetPort": 9650, "protocol": "TCP", "name": "rca"}
]
}
}'
follower-values.yaml
apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
name: opensearch-follower-cluster
namespace: opensearch
spec:
general:
serviceName: follower-cluster
version: 3.1.0
setVMMaxMapCount: true
additionalConfig:
plugins.security.nodes_dn: |
- "CN=opensearch-leader-cluster-masters-0,OU=leader-cluster"
- "CN=opensearch-leader-cluster-masters-1,OU=leader-cluster"
- "CN=opensearch-leader-cluster-nodes-0,OU=leader-cluster"
- "CN=opensearch-leader-cluster-nodes-1,OU=leader-cluster"
dashboards:
enable: true
opensearchCredentialsSecret:
name: dashboards-credentials
tls:
enable: true # Configure TLS
generate: true # Have the operator generate and sign a certificate
version: 3.1.0
replicas: 1
resources:
requests:
memory: "512Mi"
cpu: "200m"
limits:
memory: "512Mi"
cpu: "200m"
additionalConfig:
opensearch_security.multitenancy.enabled: "true"
nodePools:
- component: masters
replicas: 2
diskSize: "5Gi"
resources:
requests:
memory: "1Gi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "1000m"
roles:
- "cluster_manager"
- "remote_cluster_client"
- component: nodes
replicas: 2
diskSize: "10Gi"
resources:
requests:
memory: "2Gi"
cpu: "1000m"
limits:
memory: "3Gi"
cpu: "2000m"
roles:
- "data"
- "remote_cluster_client"
- component: coordinators
replicas: 2
diskSize: "3Gi"
resources:
requests:
memory: "1Gi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "1000m"
roles:
- "ingest"
- "remote_cluster_client"
security:
config:
adminCredentialsSecret:
name: admin-credentials-secret # The secret with the admin credentials for the operator to use
securityConfigSecret:
name: securityconfig-secret
tls: # Everything related to TLS configuration
transport: # Configuration of the transport endpoint
generate: true # Have the operator generate and sign certificates
perNode: true # Separate certificate per node
http: # Configuration of the http endpoint
generate: true
the remaining parts have the same steps as leader-cluster.