Cross cluster replication not started

Security
,
1

curl -XPUT -k -H ‘Content-Type: application/json’ -u ‘admin:v.kuMH05SDqsDb.’ ‘https://0.0.0.0:9200/_plugins/_replication/follower-01/_start?pretty’ -d ’
{
“leader_alias”: “all-in-one-test”,
“leader_index”: “leader-01”,
“use_roles”:{
“leader_cluster_role”: “admin”,
“follower_cluster_role”: “admin”
}
}’
{
“error” : {
“root_cause” : [
{
“type” : “exception”,
“reason” : “Transport client authentication no longer supported.”
}
],
“type” : “exception”,
“reason” : “Transport client authentication no longer supported.”
},
“status” : 500
}

I used all-in-one deployment and I face this error to start the replication from leader to follower node. please help me out
Kindley give quick response.

Thankyou

2

Hi @sahunikita1609,

Have you tried using your admin certs?

something like:


curl -XPUT --cacert /path/to/opensearch/root-ca-cert/<root-ca>.pem --key /path/to/opensearch/admin-key/<admin-key>.pem --cert /path/to/opensearch/admin-cert/<admin>.pem --insecure -H ‘Content-Type: application/json’ ‘https://0.0.0.0:9200/_plugins/_replication/follower-01/_start?pretty’ -d ’
{
“leader_alias”: “all-in-one-test”,
“leader_index”: “leader-01”,
“use_roles”:{
“leader_cluster_role”: “admin”,
“follower_cluster_role”: “admin”
}
}’

best,
mj

3

Thankyou for the quick response.

I solved the issue mentioned above. However, I am now facing a new problem. When I run the following command, an error occurs:

curl -XPUT -k -H ‘Content-Type: application/json’ -u ‘admin:v.kuMH05SDqsDb.’ ‘https://0.0.0.0:9200/_plugins/_replication/follower-01/_start?pretty’ -d ’
{
“leader_alias”: “all-in-one-test”,
“leader_index”: “leader-01”,
“use_roles”:{
“leader_cluster_role”: “all_access”,
“follower_cluster_role”: “all_access”
}
}’

ERROR:

{
“error” : {
“root_cause” : [
{
“type” : “index_not_found_exception”,
“reason” : “no such index [leader-01]”,
“index” : “leader-01”,
“resource.id” : “leader-01”,
“resource.type” : “index_expression”,
“index_uuid” : “na”
}
],
“type” : “index_not_found_exception”,
“reason” : “no such index [leader-01]”,
“index” : “leader-01”,
“resource.id” : “leader-01”,
“resource.type” : “index_expression”,
“index_uuid” : “na”
},
“status” : 404
}

Note: This index is only created in the leader node, and I want to replicate it in the follower cluster using the ‘CCR’ (Cross-Cluster Replication) method. I checked in the path /var/lib/wazuh-indexer/nodes/0/indices, and the leader-01 index is present on the leader node. However, when I run the command mentioned above, I am facing this error.

Guide me if I am following wrong steps:

Thankyou

4

Hi Mantas,

kindly give the response on this ticket @Mantas

5

Could you run the below on both of your clusters and share the output:

curl -XGET -k  -u ‘admin:v.kuMH05SDqsDb.’  https://0.0.0.0:9200/_cat/indices?v

best,
mj

6

Hi Mantas,

I have ruined this command on leader with leader’s password and same with follower credentials on follower’s vm. so i have these are the right method to run command. I’m attaching the result of both the vm.

7

Result of Leader node:

green  open   wazuh-archives-4.x-2024.05.30 w0f_P7sAQ3qujzobz2Qr8Q   3   0      10779            0      6.7mb          6.7mb
yellow open   leader-02                     Po0PkR8US-qQ74yE25iAoQ   1   1          0            0       208b           208b
yellow open   leader-03                     AxcHAmRNQ6mjBGX-IqEXJw   1   1          0            0       208b           208b
green  open   wazuh-statistics-2024.24w     Mzidy9dRSs6pUrp4fSRNig   1   0       4022            0      1.4mb          1.4mb
yellow open   leader-01                     Oviudis3TPKSPkGWQBfrDQ   1   1          0            0       208b           208b
green  open   wazuh-archives-4.x-2024.05.31 EfDhwJTLRyaq0u_r_w7UYA   3   0       7272            0      4.9mb          4.9mb

Result of follower node:

health status index                         uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   wazuh-monitoring-2024.26w     EzWm7IyrT3m5-9lvVFNxfg   1   0          0            0       208b           208b
green  open   .opensearch-observability     Ff-WnFsjSJeLPcJPNK16YA   1   0          0            0       208b           208b
green  open   wazuh-alerts-4.x-2024.06.26   XNWkg2QwRWWadpaA44TkrQ   3   0        921            0      1.3mb          1.3mb
green  open   wazuh-alerts-4.x-2024.06.25   sIywYyGbRH-FYSx81h-fGg   3   0        996            0      1.6mb          1.6mb
green  open   wazuh-archives-4.x-2024.06.25 r43AswB2SbmYys6WNSQYAw   3   0       2801            0      2.8mb          2.8mb
green  open   wazuh-archives-4.x-2024.06.26 GpzQcIDCQF2_JulfmYS_Ug   3   0       3792            0      3.6mb          3.6mb
green  open   wazuh-statistics-2024.26w     Gv9WRs2vTvOYKz_FlqRBEg   1   0        591            0    458.3kb        458.3kb
green  open   .opendistro_security          E4I84EFgTa6bZpUuJ2S0IA   1   0         10            2     54.9kb         54.9kb
green  open   .kibana_1                     EfjQldwASzmBbhb2Vtk8VA   1   0          4            1       26kb           26kb

Please help here.
Regards,
Nikita Sahu

8

Hi Mantas,

Please look into this ticket. @Mantas

Sorry to interrupt you.

Thanks
Nikita Sahu

9

What is your leader node address and port and what is your follower node address and port?

10

Thank you for the reply @Mantas .

port is 9200
is opened in both nodes.

both nodes are present in same region and same compartment.

sorry I cann’t reveal the Ip address for company policy.

Please help here.

Thankyou,
Nikita

11

I’ll make some IPs up to illustrate:

leader ip 1.1.1.1
follower ip 2.2.2.2

could you please run the below accordingly and share the output (note: both calls are addressed to the follower node):

curl -XPUT -k -H 'Content-Type: application/json' -u 'admin:<custom-admin-password>' 'https://2.2.2.2:9200/_cluster/settings?pretty' -d '
{
  "persistent": {
    "cluster": {
      "remote": {
        "my-connection-alias": {
          "seeds": ["1.1.1.1:9300"]
        }
      }
    }
  }
}'

curl -XPUT -k -H 'Content-Type: application/json' -u 'admin:<custom-admin-password>' 'https://2.2.2.2:9200/_plugins/_replication/follower-01/_start?pretty' -d '
{
   "leader_alias": "my-connection-alias",
   "leader_index": "leader-01",
   "use_roles":{
      "leader_cluster_role": "all_access",
      "follower_cluster_role": "all_access"
   }
}'
12

Thankyou @Mantas

I run the query that is provided by you on follower node.

Result of first query:

curl -XPUT -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://2.2.2.2:9200/_cluster/settings?pretty' -d '
{
  "persistent": {
    "cluster": {
      "remote": {
        "ccr-leader01": {
          "seeds": ["1.1.1.1:9300"]
        }
      }
    }
  }
}'
{
  "acknowledged" : true,
  "persistent" : {
    "cluster" : {
      "remote" : {
        "ccr-leader01" : {
          "seeds" : [
            "1.1.1.1:9300"
          ]
        }
      }
    }
  },
  "transient" : { }
}

Result of 2nd query:

 curl -XPUT -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://2.2.2.2:9200/_plugins/_replication/follower-01/_start?pretty' -d '
{
   "leader_alias": "ccr-leader01",
   "leader_index": "leader-01",
   "use_roles":{
      "leader_cluster_role": "all_access",
      "follower_cluster_role": "all_access"
   }
}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "exception",
        "reason" : "Transport client authentication no longer supported."
      }
    ],
    "type" : "exception",
    "reason" : "Transport client authentication no longer supported."
  },
  "status" : 500
}

I generated the tls certificate manually and the root and admin certificates are common on both nodes (leader and follower) & leader node certificates are separately generated on leader node and follower node’s certificates generated on follower node.
Note: root and admin certificate generated on leader node and same certificate used for follower node.

Regards,
Nikita Sahu

13

would you mind sharing your opensearch.yml ?

best,
mj

14

(post deleted by author)

15

Hi @Mantas

I hope you review the yaml files .
Please give me suggestion here.

Thankyou

16

Hi @sahunikita1609,

Can you add your follower cluster DN on the leader cluster ( plugins.security.nodes_dn:) to allow connections from the followers to the leader and test it again?

plugins.security.nodes_dn:
- "CN=ccr-leader01,OU=EVENTUS,O=EVENTUS,L=MUMBAI,ST=MAHARASHTRA,C=IN"
- "CN=ccr-follower01,OU=EVENTUS,O=EVENTUS,L=MUMBAI,ST=MAHARASHTRA,C=IN"

Please see more here:

Best,
mj

17

Hi @Mantas,

Thank you for assisting me.

I tried the steps, but the same error is occurring.

I have a question: Should the root and admin certificates be the same on both clusters, or should they be created separately for each cluster?

Thank you.
Nikita Sahu

18

Hi @sahunikita1609,

I don’t believe there is a difference as long as one of the below is true and you have the Basic permissions for cross-cluster replication set up:

*     Security plugin fully enabled on both clusters
*     Security plugin enabled only for TLS on both clusters (plugins.security.ssl_only)
*     Security plugin absent or disabled on both clusters (not recommended)

best,
mj

19

Hi @Mantas ,

Screenshot 2024-07-13 035419
Screenshot 2024-07-13 0354191207×213 6.49 KB

Thank you for giving me your valuable time.

With your guidance, I have successfully completed my task. Could you please confirm if my cross-cluster replication (CCR) has started successfully?

I am attaching a screenshot of the CCR status for your review.

Thank you.

1 Like
20

Hi @sahunikita1609,

Glad to read you got there.

It looks good, the status is “SYNCING” and the checkpoints are matching meaning that the indexes are fully synced.

I would still advise keeping an eye on and doing some spot-checks (run some search on the follower).

Best,
mj

1 Like