Cross cluster replication not started

curl -XPUT -k -H ‘Content-Type: application/json’ -u ‘admin:v.kuMH05SDqsDb.’ ‘https://0.0.0.0:9200/_plugins/_replication/follower-01/_start?pretty’ -d ’
{
“leader_alias”: “all-in-one-test”,
“leader_index”: “leader-01”,
“use_roles”:{
“leader_cluster_role”: “admin”,
“follower_cluster_role”: “admin”
}
}’
{
“error” : {
“root_cause” : [
{
“type” : “exception”,
“reason” : “Transport client authentication no longer supported.”
}
],
“type” : “exception”,
“reason” : “Transport client authentication no longer supported.”
},
“status” : 500
}

I used all-in-one deployment and I face this error to start the replication from leader to follower node. please help me out
Kindley give quick response.

Thankyou

Hi @sahunikita1609,

Have you tried using your admin certs?

something like:


curl -XPUT --cacert /path/to/opensearch/root-ca-cert/<root-ca>.pem --key /path/to/opensearch/admin-key/<admin-key>.pem --cert /path/to/opensearch/admin-cert/<admin>.pem --insecure -H ‘Content-Type: application/json’ ‘https://0.0.0.0:9200/_plugins/_replication/follower-01/_start?pretty’ -d ’
{
“leader_alias”: “all-in-one-test”,
“leader_index”: “leader-01”,
“use_roles”:{
“leader_cluster_role”: “admin”,
“follower_cluster_role”: “admin”
}
}’

best,
mj

Thankyou for the quick response.

I solved the issue mentioned above. However, I am now facing a new problem. When I run the following command, an error occurs:

curl -XPUT -k -H ‘Content-Type: application/json’ -u ‘admin:v.kuMH05SDqsDb.’ ‘https://0.0.0.0:9200/_plugins/_replication/follower-01/_start?pretty’ -d ’
{
“leader_alias”: “all-in-one-test”,
“leader_index”: “leader-01”,
“use_roles”:{
“leader_cluster_role”: “all_access”,
“follower_cluster_role”: “all_access”
}
}’

ERROR:

{
“error” : {
“root_cause” : [
{
“type” : “index_not_found_exception”,
“reason” : “no such index [leader-01]”,
“index” : “leader-01”,
“resource.id” : “leader-01”,
“resource.type” : “index_expression”,
“index_uuid” : “na
}
],
“type” : “index_not_found_exception”,
“reason” : “no such index [leader-01]”,
“index” : “leader-01”,
“resource.id” : “leader-01”,
“resource.type” : “index_expression”,
“index_uuid” : “na
},
“status” : 404
}

Note: This index is only created in the leader node, and I want to replicate it in the follower cluster using the ‘CCR’ (Cross-Cluster Replication) method. I checked in the path /var/lib/wazuh-indexer/nodes/0/indices, and the leader-01 index is present on the leader node. However, when I run the command mentioned above, I am facing this error.

Guide me if I am following wrong steps:

Thankyou

Hi Mantas,

kindly give the response on this ticket @Mantas

Could you run the below on both of your clusters and share the output:

curl -XGET -k  -u ‘admin:v.kuMH05SDqsDb.’  https://0.0.0.0:9200/_cat/indices?v

best,
mj

Hi Mantas,

I have ruined this command on leader with leader’s password and same with follower credentials on follower’s vm. so i have these are the right method to run command. I’m attaching the result of both the vm.

Result of Leader node:

green  open   wazuh-archives-4.x-2024.05.30 w0f_P7sAQ3qujzobz2Qr8Q   3   0      10779            0      6.7mb          6.7mb
yellow open   leader-02                     Po0PkR8US-qQ74yE25iAoQ   1   1          0            0       208b           208b
yellow open   leader-03                     AxcHAmRNQ6mjBGX-IqEXJw   1   1          0            0       208b           208b
green  open   wazuh-statistics-2024.24w     Mzidy9dRSs6pUrp4fSRNig   1   0       4022            0      1.4mb          1.4mb
yellow open   leader-01                     Oviudis3TPKSPkGWQBfrDQ   1   1          0            0       208b           208b
green  open   wazuh-archives-4.x-2024.05.31 EfDhwJTLRyaq0u_r_w7UYA   3   0       7272            0      4.9mb          4.9mb

Result of follower node:

health status index                         uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   wazuh-monitoring-2024.26w     EzWm7IyrT3m5-9lvVFNxfg   1   0          0            0       208b           208b
green  open   .opensearch-observability     Ff-WnFsjSJeLPcJPNK16YA   1   0          0            0       208b           208b
green  open   wazuh-alerts-4.x-2024.06.26   XNWkg2QwRWWadpaA44TkrQ   3   0        921            0      1.3mb          1.3mb
green  open   wazuh-alerts-4.x-2024.06.25   sIywYyGbRH-FYSx81h-fGg   3   0        996            0      1.6mb          1.6mb
green  open   wazuh-archives-4.x-2024.06.25 r43AswB2SbmYys6WNSQYAw   3   0       2801            0      2.8mb          2.8mb
green  open   wazuh-archives-4.x-2024.06.26 GpzQcIDCQF2_JulfmYS_Ug   3   0       3792            0      3.6mb          3.6mb
green  open   wazuh-statistics-2024.26w     Gv9WRs2vTvOYKz_FlqRBEg   1   0        591            0    458.3kb        458.3kb
green  open   .opendistro_security          E4I84EFgTa6bZpUuJ2S0IA   1   0         10            2     54.9kb         54.9kb
green  open   .kibana_1                     EfjQldwASzmBbhb2Vtk8VA   1   0          4            1       26kb           26kb

Please help here.
Regards,
Nikita Sahu

Hi Mantas,

Please look into this ticket. @Mantas

Sorry to interrupt you.

Thanks
Nikita Sahu

What is your leader node address and port and what is your follower node address and port?

Thank you for the reply @Mantas .

port is 9200
is opened in both nodes.

both nodes are present in same region and same compartment.

sorry I cann’t reveal the Ip address for company policy.

Please help here.

Thankyou,
Nikita

I’ll make some IPs up to illustrate:

leader ip 1.1.1.1
follower ip 2.2.2.2

could you please run the below accordingly and share the output (note: both calls are addressed to the follower node):

curl -XPUT -k -H 'Content-Type: application/json' -u 'admin:<custom-admin-password>' 'https://2.2.2.2:9200/_cluster/settings?pretty' -d '
{
  "persistent": {
    "cluster": {
      "remote": {
        "my-connection-alias": {
          "seeds": ["1.1.1.1:9300"]
        }
      }
    }
  }
}'
curl -XPUT -k -H 'Content-Type: application/json' -u 'admin:<custom-admin-password>' 'https://2.2.2.2:9200/_plugins/_replication/follower-01/_start?pretty' -d '
{
   "leader_alias": "my-connection-alias",
   "leader_index": "leader-01",
   "use_roles":{
      "leader_cluster_role": "all_access",
      "follower_cluster_role": "all_access"
   }
}'

Thankyou @Mantas

I run the query that is provided by you on follower node.

Result of first query:

curl -XPUT -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://2.2.2.2:9200/_cluster/settings?pretty' -d '
{
  "persistent": {
    "cluster": {
      "remote": {
        "ccr-leader01": {
          "seeds": ["1.1.1.1:9300"]
        }
      }
    }
  }
}'
{
  "acknowledged" : true,
  "persistent" : {
    "cluster" : {
      "remote" : {
        "ccr-leader01" : {
          "seeds" : [
            "1.1.1.1:9300"
          ]
        }
      }
    }
  },
  "transient" : { }
}

Result of 2nd query:

 curl -XPUT -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://2.2.2.2:9200/_plugins/_replication/follower-01/_start?pretty' -d '
{
   "leader_alias": "ccr-leader01",
   "leader_index": "leader-01",
   "use_roles":{
      "leader_cluster_role": "all_access",
      "follower_cluster_role": "all_access"
   }
}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "exception",
        "reason" : "Transport client authentication no longer supported."
      }
    ],
    "type" : "exception",
    "reason" : "Transport client authentication no longer supported."
  },
  "status" : 500
}

I generated the tls certificate manually and the root and admin certificates are common on both nodes (leader and follower) & leader node certificates are separately generated on leader node and follower node’s certificates generated on follower node.
Note: root and admin certificate generated on leader node and same certificate used for follower node.

Regards,
Nikita Sahu

would you mind sharing your opensearch.yml ?

best,
mj

Please look in to below attach yaml files @Mantas

opensearch.yml of follower node:

node.host: "0.0.0.0"
node.name: "ccr-follower01"
cluster.initial_master_nodes:
- "ccr-follower01"
cluster.name: "esdl-cluster"

node.max_local_storage_nodes: "3"
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch

plugins.security.unsupported.inject_user.enabled: true
plugins.security.nodes_dn_dynamic_config_enabled: true
node.remote_cluster_client: true

plugins.security.ssl.http.pemcert_filepath: /etc/opensearch/ccr-follower01.pem
plugins.security.ssl.http.pemkey_filepath: /etc/opensearch/ccr-follower01-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/ccr-follower01.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/ccr-follower01-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.transport.enforce_hostname_verification: false
  #plugins.security.allow_default_init_securityindex: true
plugins.security.ssl.transport.resolve_hostname: false
plugins.security.ssl.http.enabled_ciphers:
  - "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
  - "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
  - "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
  - "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
plugins.security.ssl.http.enabled_protocols:
  - "TLSv1.2"
plugins.security.authcz.admin_dn:
- "CN=admin,OU=EVENTUS,O=EVENTUS,L=MUMBAI,ST=MAHARASHTRA,C=IN"
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.enable_snapshot_restore_privilege: true
  #plugins.security.cache.ttl_minutes: 60
plugins.security.nodes_dn:
- "CN=ccr-follower01,OU=EVENTUS,O=EVENTUS,L=MUMBAI,ST=MAHARASHTRA,C=IN"
plugins.security.restapi.roles_enabled:
- "all_access"
- "security_rest_api_access"

  #plugins.security.system_indices.permission.enabled: true
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]

### Option to allow Filebeat-oss 7.10.2 to work ###
compatibility.override_main_response_version: true

opensearch.yml of leader:

network.host: "0.0.0.0"
node.name: "ccr-leader01"
cluster.initial_master_nodes:
- "ccr-leader01"
cluster.name: "esdl-cluster"

node.max_local_storage_nodes: "3"
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch

plugins.security.unsupported.inject_user.enabled: true
plugins.security.nodes_dn_dynamic_config_enabled: true
node.remote_cluster_client: true


plugins.security.ssl.http.pemcert_filepath: /etc/opensearch/ccr-leader01.pem
plugins.security.ssl.http.pemkey_filepath: /etc/opensearch/ccr-leader01-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/ccr-leader01.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/ccr-leader01-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.resolve_hostname: false
plugins.security.ssl.http.enabled_ciphers:
  - "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
  - "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
  - "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
  - "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
plugins.security.ssl.http.enabled_protocols:
  - "TLSv1.2"
plugins.security.authcz.admin_dn:
- "CN=admin,OU=EVENTUS,O=EVENTUS,L=MUMBAI,ST=MAHARASHTRA,C=IN"
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.nodes_dn:
- "CN=ccr-leader01,OU=EVENTUS,O=EVENTUS,L=MUMBAI,ST=MAHARASHTRA,C=IN"
plugins.security.restapi.roles_enabled:
- "all_access"
- "security_rest_api_access"

plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]

### Option to allow Filebeat-oss 7.10.2 to work ###
compatibility.override_main_response_version: true

Thanks
Nikita

Hi @Mantas

I hope you review the yaml files .
Please give me suggestion here.

Thankyou

Hi @sahunikita1609,

Can you add your follower cluster DN on the leader cluster ( plugins.security.nodes_dn:) to allow connections from the followers to the leader and test it again?

plugins.security.nodes_dn:
- "CN=ccr-leader01,OU=EVENTUS,O=EVENTUS,L=MUMBAI,ST=MAHARASHTRA,C=IN"
- "CN=ccr-follower01,OU=EVENTUS,O=EVENTUS,L=MUMBAI,ST=MAHARASHTRA,C=IN"

Please see more here:

Best,
mj

Hi @Mantas,

Thank you for assisting me.

I tried the steps, but the same error is occurring.

I have a question: Should the root and admin certificates be the same on both clusters, or should they be created separately for each cluster?

Thank you.
Nikita Sahu

Hi @sahunikita1609,

I don’t believe there is a difference as long as one of the below is true and you have the Basic permissions for cross-cluster replication set up:

*     Security plugin fully enabled on both clusters
*     Security plugin enabled only for TLS on both clusters (plugins.security.ssl_only)
*     Security plugin absent or disabled on both clusters (not recommended)

best,
mj

Hi @Mantas ,

Thank you for giving me your valuable time.

With your guidance, I have successfully completed my task. Could you please confirm if my cross-cluster replication (CCR) has started successfully?

I am attaching a screenshot of the CCR status for your review.

Thank you.

1 Like

Hi @sahunikita1609,

Glad to read you got there.

It looks good, the status is “SYNCING” and the checkpoints are matching meaning that the indexes are fully synced.

I would still advise keeping an eye on and doing some spot-checks (run some search on the follower).

Best,
mj

1 Like