Opensearch 3 node cluster data folder handling for RDR case

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): 2.19.2

Describe the issue:

During our recent RDR exercise, we shut down the two PROD OpenSearch nodes and kept only the DR node active within the same cluster. In this setup, only the existing search functionality worked, but newly created indexes were not visible during the RDR window.

To ensure that both search and index creation work during the 7‑day RDR period, we are planning to run the DR node as a Single‑Node OpenSearch instance during the RDR drill. After the drill, we intend to switch the DR node back to the original cluster configuration so it can rejoin the same 3‑node cluster with the PROD nodes.

Our current cluster data directory is: /app/data

We would like to clarify the following:

Question:

For RDR, can we purge the entire data folder on the DR node, run the DR node in single‑node mode using the same /app/data directory, and after RDR is completed, switch back to the original cluster configuration and rejoin the DR node to the existing cluster? since prod node data folder still same existing data and hope it will sync to DR data folder after we started the DR Service.

We need to confirm whether this approach is supported or will cause any issues, especially related to:

  • Cluster UUID mismatch

  • Metadata incompatibility

  • Rejoining the DR node to the original cluster

  • Index/state corruption after switching back

Please advise if this approach is technically feasible or if there is a recommended method for enabling full search + indexing during RDR without breaking future cluster reintegration.

Configuration:

cluster.name: opensearch-cluster

node.name: host1.sg.net.com
node.roles: [cluster_manager, data, ingest]

cluster.routing.allocation.enable: all
cluster.routing.allocation.allow_rebalance: always
path.data: /app/opensearch-bitbucket/application-data

network.host: 0.0.0.0
network.bind_host: x.x.x.x
network.publish_host: x.x.x.x

http.port: 9200
transport.port: 9300

discovery.seed_hosts:

  • host1:9300
  • host2:9300
  • host3:9300

cluster.initial_cluster_manager_nodes:

  • host1
  • host2
  • host3

######## Start OpenSearch Security Demo Configuration ########

WARNING: revise all the lines below before you go into production

plugins.security.ssl.transport.enabled: true
plugins.security.ssl.transport.pemcert_filepath: “/app/ssl/server.crt”
plugins.security.ssl.transport.pemkey_filepath: “/app/ssl/server-decrypted.key”
plugins.security.ssl.transport.pemtrustedcas_filepath: “/app/ssl/rootCA.pem”
plugins.security.ssl.transport.enabled_protocols:

  • TLSv1.3
  • TLSv1.2

plugins.security.ssl.http.enabled_ciphers:

  • “TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256”
  • “TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256”
  • “TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256”
  • “TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256”
  • “TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384”
  • “TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384”

plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: “/app/ssl/server.crt”
plugins.security.ssl.http.pemkey_filepath: “/app/ssl/server-decrypted.key”
plugins.security.ssl.http.pemtrustedcas_filepath: “/app/ssl/rootCA.pem”
plugins.security.ssl.http.enabled_protocols:

  • TLSv1.3
  • TLSv1.2

plugins.security.ssl.transport.enabled_ciphers:

  • “TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256”
  • “TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256”
  • “TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256”
  • “TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256”
  • “TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384”
  • “TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384”

plugins.security.nodes_dn:

  • “CN=bbopensearchp.sg.net.com,OU=Technology,O=Limited,L=Country,ST=Country,C=SA”
    plugins.security.allow_unsafe_democertificates: true
    Plugins.security.allow_default_init_securityindex: true
    plugins.security.authcz.admin_dn: [‘CN=admin-opensearch.sg.uobnet.com,OU=Technology,O=Limited,L=Country,ST=Country,C=SA’]
    plugins.security.audit.type: internal_opensearch
    plugins.security.enable_snapshot_restore_privilege: true
    plugins.security.check_snapshot_restore_write_privileges: true
    plugins.security.restapi.roles_enabled: [“all_access”]
    plugins.security.allow_default_init_securityindex: true
    plugins.security.system_indices.enabled: true
    plugins.security.system_indices.indices: [.opensearch-security, .plugins-ml-agent, .plugins-ml-config, .plugins-ml-connector,
    .plugins-ml-controller, .plugins-ml-model-group, .plugins-ml-model, .plugins-ml-task,
    .plugins-ml-conversation-meta, .plugins-ml-conversation-interactions, .plugins-ml-memory-meta,
    .plugins-ml-memory-message, .plugins-ml-stop-words, .opensearch-alerting-config,
    .opensearch-alerting-alert*, .opensearch-anomaly-results*, .opensearch-anomaly-detector*,
    .opensearch-anomaly-checkpoints, .opensearch-anomaly-detection-state, .opensearch-reports-,
    .opensearch-notifications-    , .opensearch-notebooks, .opensearch-observability, .ql-datasources,
    .opensearch-asynchronous-search-response*, .replication-metadata-store, .opensearch-knn-models,
    .geospatial-ip2geo-data*, .plugins-flow-framework-config, .plugins-flow-framework-templates,
    .plugins-flow-framework-state]
    node.max_local_storage_nodes: 3
    ######## End OpenSearch Security Demo Configuration ########

Relevant Logs or Screenshots: NA

2

@vinay this approach would not work as there would be new cluster UUID created and divergent cluster state, which will fail to merge into prod after. Any indices created during the single‑node period would be stranded and won’t replicate back to PROD.

I would recommend to look at Cross‑Cluster Replication, as a means to DR. Failing over to the follower cluster during an outage and later setting up the original cluster as the follower.

Hope this helps