Failing to read from opensearch-keystore

OpenSearch
1

When trying to create a snapshot repository using an S3 compatible API the creation fails with the following error:

{
  "error": {
    "root_cause": [
      {
        "type": "repository_verification_exception",
        "reason": "[search] path [my/snapshot/directory] is not accessible on cluster-manager node"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[search] path [my/snapshot/directory] is not accessible on cluster-manager node",
    "caused_by": {
      "type": "i_o_exception",
      "reason": "Unable to upload object [my/snapshot/directory/tests-BjniINtaTreGkZPpQu4zRw/master.dat] using a single upload",
      "caused_by": {
        "type": "sdk_client_exception",
        "reason": "sdk_client_exception: Failed to connect to service endpoint: ",
        "caused_by": {
          "type": "i_o_exception",
          "reason": "Read timed out"
        }
      }
    }
  },
  "status": 500
}

The creation request is being done via the Dev Tools in OpenSearch. Here is the request:

PUT /_snapshot/search
{
  "type": "s3",
  "settings": {
    "bucket": "search-backup",
    "base_path": "my/snapshot/directory",
    "endpoint": "https://my-endpoint"
  }
}

If I manually specify the Access and Secret key, then the request is successful.

PUT /_snapshot/search
{
  "type": "s3",
  "settings": {
    "bucket": "search-backup",
    "base_path": "my/snapshot/directory",
    "endpoint": "https://my-endpoint",
    "access_key": "xxx",
    "secret_key": "xxx"
  }
}

The opensearch-keystore should be providing these values to opensearch. They are being set with the following commands:

echo {{ backup_access_key }} | podman exec opensearch /usr/share/opensearch/bin/opensearch-keystore add --force --stdin s3.client.default.access_key

echo {{ backup_secret_key }} | podman exec opensearch /usr/share/opensearch/bin/opensearch-keystore add --force --stdin s3.client.default.secret_key

If the opensearch-keystore is listed, then I can see the keys are present:

podman exec opensearch /usr/share/opensearch/bin/opensearch-keystore list
keystore.seed
s3.client.default.access_key
s3.client.default.secret_key

Am I missing something here?

Opensearch Version is 2.8.0

2

@baazzaar Hello! Did you solve your problem? I have the same problem and i don’t know how to solve it

3

@baazzaar i found the solution.
Command /usr/share/opensearch/bin/opensearch-keystore add creates keystore at /usr/share/opensearch/config/ or /etc/opensearch/. And there are two possible problems:

  1. You run command opensearch-keystore add from sudo user, but opensearch works under another custom user. And opensearch can’t open keystore due to filesystem permissions
  2. You use custom config folder. For example your systemd service /lib/systemd/system/opensearch.service has parameter Environment=OPENSEARCH_PATH_CONF=/opt/opensearch/etc. And opensearch looks for keystore at /opt/opensearch/etc folder. But ./opensearch-keystore add command creates it in another folder - /usr/share/opensearch/config/ or /etc/opensearch/
    And opensearch doesn’t see your keystore.