Anyone successful in using autofollow rules for data streams?

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

OpenSearch Version: 2.8.0
Server OS: 5.15.49-linuxkit-pr, Amazon Linux 2 (Opensearch docker container)

Describe the issue:
Not sure if supported but I’m having issues using Auto-follow for CCR, specifically for data streams

Configuration:

2 clusters:
localhost:9200 = leader
localhost:9202 = follower

$ curl -XGET -u 'admin:admin' -k 'http://localhost:9200'                                                             
{
  "name" : "opensearch-node1",
  "cluster_name" : "leader-cluster",
  "cluster_uuid" : "QB6yIBJ5TMWfovr_AayxTg",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.8.0",
    "build_type" : "tar",
    "build_hash" : "db90a415ff2fd428b4f7b3f800a51dc229287cb4",
    "build_date" : "2023-06-03T06:24:11.331474107Z",
    "build_snapshot" : false,
    "lucene_version" : "9.6.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}
$ curl -XGET -u 'admin:admin' -k 'http://localhost:9202'
{
  "name" : "25f9a2a2c444",
  "cluster_name" : "follower-cluster",
  "cluster_uuid" : "TmfI38TTQb2IdtYrWUopSw",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.8.0",
    "build_type" : "tar",
    "build_hash" : "db90a415ff2fd428b4f7b3f800a51dc229287cb4",
    "build_date" : "2023-06-03T06:24:11.331474107Z",
    "build_snapshot" : false,
    "lucene_version" : "9.6.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

$ curl -XGET -k -H 'Content-Type: application/json' -u 'admin:admin' 'http://localhost:9202/_cluster/settings?pretty'
{
  "persistent" : {
    "cluster" : {
      "remote" : {
        "replication-cluster1" : {
          "seeds" : [
            "172.20.0.2:9300",
            "172.20.0.5:9300",
            "172.20.0.4:9300"
          ]
        }
      }
    },
    "plugins" : {
      "index_state_management" : {
        "template_migration" : {
          "control" : "-1"
        }
      }
    }
  },
  "transient" : { }
}

$ curl -XGET -u 'admin:admin' -k 'http://localhost:9202/_plugins/_replication/autofollow_stats'
{
  "num_success_start_replication" : 1,
  "num_failed_start_replication" : 0,
  "num_failed_leader_calls" : 0,
  "failed_indices" : [
    ".ds-shows-asia-000001",
    ".ds-shows-namerica-000001"
  ],
  "autofollow_stats" : [
    {
      "name" : "auto-follow-shows",
      "pattern" : "shows*",
      "num_success_start_replication" : 1,
      "num_failed_start_replication" : 0,
      "num_failed_leader_calls" : 0,
      "failed_indices" : [
        ".ds-shows-asia-000001",
        ".ds-shows-namerica-000001"
      ],
      "last_execution_time" : 1690314894780
    }
  ]
}

$ curl -XGET -u 'admin:admin' -k 'http://localhost:9200/_index_template/shows-template/?pretty'
{
  "index_templates" : [
    {
      "name" : "shows-template",
      "index_template" : {
        "index_patterns" : [
          "shows*"
        ],
        "data_stream" : {
          "timestamp_field" : {
            "name" : "@timestamp"
          }
        }
      }
    }
  ]
}

Relevant Logs or Screenshots:

Failed to start replication for replication-cluster1:.ds-shows-namerica-000001 -> .ds-shows-namerica-000001.
replication-node1  | org.opensearch.action.ActionRequestValidationException: Validation Failed: 1: Value .ds-shows-namerica-000001 must not start with '.';2: Value .ds-shows-namerica-000001 must not start with '.';
2

@ajmirabal OpenSearch is a fork of ElasticSearch 7.10.2 and the data stream support for CCR has been introduced in Elasticsearch 7.11.

There is an open feature request in the OpenSearch GitHub.

Based on the above, OpenSearch doesn’t support data streams for CCR.

3

Thanks for the quick response.