Remotestore restore doesn't do anything

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): OpenSearch 2.15.0

Describe the issue:
I’ve configured remotestate for my OpenSearch cluster and can see that data has been replicated into s3 buckets.
After that to test possibility to recover from red state I’ve deleted all the data nodes from my cluster by terminating underlying instances.
I waited till new data nodes come to the running state and tried to run the restore command as decsribed in the doc - Remote-backed storage - OpenSearch Documentation, but for some reason OpenSearch doesn’t do anything:

POST /_remotestore/_restore?restore_all_shards=true
{
    "indices": ["my_index"]
}

and the response is:

{
  "remote_store": {
    "snapshot": "remote_store",
    "indices": [],
    "shards": {
      "total": 0,
      "failed": 0,
      "successful": 0
    }
  }
}

Configuration:

"cluster.remote_store.state.enabled": "true",
# Remote cluster state repository settings
"node.attr.remote_store.state.repository": "cluster-state-repo",
"node.attr.remote_store.repository.cluster-state-repo.type": "s3",
"node.attr.remote_store.repository.cluster-state-repo.settings.bucket": "REDUCTED",
"node.attr.remote_store.repository.cluster-state-repo.settings.base_path": "cluster_state",
"node.attr.remote_store.repository.cluster-state-repo.settings.region": "us-east-1",
# Segment repository settings
"node.attr.remote_store.segment.repository": "segment-repo",
"node.attr.remote_store.repository.segment-repo.type": "s3",
"node.attr.remote_store.repository.segment-repo.settings.bucket": "REDUCTED",
"node.attr.remote_store.repository.segment-repo.settings.base_path": "segments",
"node.attr.remote_store.repository.segment-repo.settings.region": "us-east-1",

# Translog repository settings
"node.attr.remote_store.translog.repository": "translog-repo",
"node.attr.remote_store.repository.translog-repo.type": "s3",
"node.attr.remote_store.repository.translog-repo.settings.bucket": "REDUCTED",
"node.attr.remote_store.repository.translog-repo.settings.base_path": "translog",
"node.attr.remote_store.repository.translog-repo.settings.region": "us-east-1",

Relevant Logs or Screenshots:

2

I guess i figured that out.
The index has been created before I’ve updated settings to use remote-backed storage on all nodes and this index (even if all the segments and translog for it have been saved into s3) doesn’t have remote_store settings defined.

It would be nice if OpenSearch raise a proper error on the restore operation though.

Best,
Evgeny

1 Like