Failing to read from opensearch-keystore

OpenSearch
1

When trying to create a snapshot repository using an S3 compatible API the creation fails with the following error:

{
  "error": {
    "root_cause": [
      {
        "type": "repository_verification_exception",
        "reason": "[search] path [my/snapshot/directory] is not accessible on cluster-manager node"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[search] path [my/snapshot/directory] is not accessible on cluster-manager node",
    "caused_by": {
      "type": "i_o_exception",
      "reason": "Unable to upload object [my/snapshot/directory/tests-BjniINtaTreGkZPpQu4zRw/master.dat] using a single upload",
      "caused_by": {
        "type": "sdk_client_exception",
        "reason": "sdk_client_exception: Failed to connect to service endpoint: ",
        "caused_by": {
          "type": "i_o_exception",
          "reason": "Read timed out"
        }
      }
    }
  },
  "status": 500
}

The creation request is being done via the Dev Tools in OpenSearch. Here is the request:

PUT /_snapshot/search
{
  "type": "s3",
  "settings": {
    "bucket": "search-backup",
    "base_path": "my/snapshot/directory",
    "endpoint": "https://my-endpoint"
  }
}

If I manually specify the Access and Secret key, then the request is successful.

PUT /_snapshot/search
{
  "type": "s3",
  "settings": {
    "bucket": "search-backup",
    "base_path": "my/snapshot/directory",
    "endpoint": "https://my-endpoint",
    "access_key": "xxx",
    "secret_key": "xxx"
  }
}

The opensearch-keystore should be providing these values to opensearch. They are being set with the following commands:

echo {{ backup_access_key }} | podman exec opensearch /usr/share/opensearch/bin/opensearch-keystore add --force --stdin s3.client.default.access_key

echo {{ backup_secret_key }} | podman exec opensearch /usr/share/opensearch/bin/opensearch-keystore add --force --stdin s3.client.default.secret_key

If the opensearch-keystore is listed, then I can see the keys are present:

podman exec opensearch /usr/share/opensearch/bin/opensearch-keystore list
keystore.seed
s3.client.default.access_key
s3.client.default.secret_key

Am I missing something here?

Opensearch Version is 2.8.0