Hi, I want to configure opensearch cross cluster replication across two different k8s clusters. How can i achieve this and i am using the 2.6.0 version of Opensearch

OpenSearch
,
1

Opensearch cross cluster replication

2

Hi @zainakram,

Have you checked Docs?

Best,
mj

3

yes @Mantas I have gone through this documentation but I am unable to achieve the cross cluster replication on two k8s clusters.

4

Would you mind elaborate a bit more? Any errors? what exactly is not working, and what steps did you already take?

thanks,
mj

5

hi @Mantas following are my config files of opensearch for DC-1 and DC-2

DC-1
cat opensearch-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: opensearch-config
  namespace: elk
data:
  opensearch.yml: |
    cluster.name: "dc1-cluster"
    node.name: "opensearch-leader"
    node.attr.zone: "dc1"
    network.host: "0.0.0.0"
    discovery.seed_hosts: ["10.142.0.3"]
    cluster.initial_cluster_manager_nodes: ["opensearch-leader"]
    cluster.remote.dc2-cluster.seeds: ["10.142.0.7:32088"]
    http.port: 9200
    transport.port: 9300

cat opensearch-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: opensearch-leader
  labels:
    app: opensearch-leader
spec:
  replicas: 1  # Adjust for more nodes in the cluster
  selector:
    matchLabels:
      app: opensearch-leader
  template:
    metadata:
      labels:
        app: opensearch-leader
    spec:
      containers:
      - name: opensearch
        image: opensearchproject/opensearch:2.6.0  # Docker image
        env:
        - name: plugins.security.disabled
          value: "true"  # Disable security plugin if not needed
        ports:
        - containerPort: 9200  # HTTP port
        - containerPort: 9600  # Performance Analyzer port
        - containerPort: 9300  # Transport port for CCR
        volumeMounts:
        - name: config-volume
          mountPath: /usr/share/opensearch/config/opensearch.yml
          subPath: opensearch.yml
        resources:
          limits:
            memory: "2Gi"
            cpu: "1000m"
          requests:
            memory: "1Gi"
            cpu: "500m"
      volumes:
      - name: config-volume
        configMap:
          name: opensearch-config  # Reference to the ConfigMap you created

cat opensearch-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: opensearch-leader-service
spec:
  selector:
    app: opensearch-leader
  ports:
    - name: http
      protocol: TCP
      port: 9200
      targetPort: 9200
    - name: transport
      protocol: TCP
      port: 9300
      targetPort: 9300  # Expose the transport port for cross-cluster communication
  type: NodePort  # Changed from NodePort to LoadBalancer

DC-2

cat opensearch-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: opensearch-config
  namespace: elk
data:
  opensearch.yml: |
    cluster.name: "dc2-cluster"
    node.name: "opensearch-follower"
    node.attr.zone: "dc2"
    network.host: "0.0.0.0"
    discovery.seed_hosts: ["10.142.0.3", "10.142.0.7"]
    cluster.initial_cluster_manager_nodes: ["opensearch-follower"]
    cluster.remote.dc1-cluster.seeds: ["104.196.41.2:32462"]
    http.port: 9200
    transport.port: 9300

cat opensearch-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: opensearch-follower
  labels:
    app: opensearch-follower
spec:
  replicas: 1  # Adjust for more nodes in the cluster
  selector:
    matchLabels:
      app: opensearch-follower
  template:
    metadata:
      labels:
        app: opensearch-follower
    spec:
      containers:
      - name: opensearch
        image: opensearchproject/opensearch:2.6.0  # Docker image
        env:
        - name: plugins.security.disabled
          value: "true"  # Disable security plugin if not needed
        ports:
        - containerPort: 9200  # HTTP port
        - containerPort: 9600  # Performance Analyzer port
        - containerPort: 9300  # Transport port for CCR
        volumeMounts:
        - name: config-volume
          mountPath: /usr/share/opensearch/config/opensearch.yml
          subPath: opensearch.yml
        resources:
          limits:
            memory: "2Gi"
            cpu: "1000m"
          requests:
            memory: "1Gi"
            cpu: "500m"
      volumes:
      - name: config-volume
        configMap:
          name: opensearch-config  # Reference to the ConfigMap you created

cat opensearch-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: opensearch-follower-service
spec:
  selector:
    app: opensearch-follower
  ports:
    - name: http
      protocol: TCP
      port: 9200
      targetPort: 9200
    - name: transport
      protocol: TCP
      port: 9300
      targetPort: 9300  # Expose the transport port for cross-cluster communication
  type: NodePort  # Or LoadBalancer depending on your setup

after applying these files i am getting the following error in the logs of Opensearch for both pods of DC-1 and DC-2

[2024-11-19T05:25:43,402][INFO ][o.o.t.TransportService   ] [opensearch-follower] publish_address {192.168.217.109:9300}, bound_addresses {[::]:9300}
[2024-11-19T05:26:13,443][WARN ][o.o.t.RemoteClusterService] [opensearch-follower] failed to connect to remote clusters within 30s
[2024-11-19T05:26:13,503][WARN ][o.o.t.SniffConnectionStrategy] [opensearch-follower] fetching nodes from external cluster [dc1-cluster] failed
org.opensearch.transport.ConnectTransportException: [][104.196.41.2:32462] connect_timeout[30s]
        at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onTimeout(TcpTransport.java:1082) ~[opensearch-2.6.0.jar:2.6.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-2.6.0.jar:2.6.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
6

@zainakram, thanks for sharing. Ill run some tests in my lab and get back to you if I find something.

best,
mj