Aws s3 verfication exception

OpenSearch
, ,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch and Dashboards version - 2.9.0

Describe the issue:
I added the credentials (aws_access_key_id, aws_secret_access_key, aws_session_token) to the Opensearch keystore.
Following were the commands used
adding access_key to all nodes

ansible -i <inventory_file>  elasticsearch_servers  -f20 -m shell -a "docker exec -it -u elasticsearch elasticsearch bash -c \"echo  <access_key> | /usr/share/elasticsearch/bin/elasticsearch-keystore add -f --stdin s3.client.default.access_key\""

adding secret_key to all nodes

ansible -i <inventory_file>  elasticsearch_servers  -f20 -m shell -a "docker exec -it -u elasticsearch elasticsearch bash -c \"echo <secret_key_here> | /usr/share/elasticsearch/bin/elasticsearch-keystore add -f --stdin s3.client.default.secret_key\""

adding session_token to all nodes

ansible -i <inventory_file>  elasticsearch_servers  -f20 -m shell -a "docker exec -it -u elasticsearch elasticsearch bash -c \"echo <session_token_here> | /usr/share/elasticsearch/bin/elasticsearch-keystore add -f --stdin s3.client.default.session_token\"" usr/share/elasticsearch/bin/elasticsearch-keystore add -f --stdin s3.client.default.access_key

I reloaded the cluster secure settings using below API:
POST _nodes/reload_secure_settings

Then I tried to register the repository, using the below command:

PUT _snapshot/repository
{
  "type": "s3",
  "settings": {
    "bucket": "aws-bucket",
    "endpoint": "s3.eu-central-1.amazonaws.com",
    "storage_class": "standard_ia"
  }
}

Relevant Logs or Screenshots:
When trying to register the repo using the above command, I am getting the below error:

{
  "error": {
    "root_cause": [
      {
        "type": "repository_verification_exception",
        "reason": "[repository] path  is not accessible on cluster-manager node"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[repository] path  is not accessible on cluster-manager node",
    "caused_by": {
      "type": "i_o_exception",
      "reason": "Unable to upload object [tests-yr9yJbIAS763NYM69Qni4A/master.dat] using a single upload",
      "caused_by": {
        "type": "sdk_client_exception",
        "reason": "sdk_client_exception: Unable to load region from any of the providers in the chain software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@7358f1fb: [software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@5c6470db: Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or  system property (aws.region)., software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@330298e5: No region provided in profile: default, software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@6dc70c2: Unable to retrieve region information from EC2 Metadata service. Please make sure the application is running on EC2.]"
      }
    }
  },
  "status": 500
}

This happened after I upgraded the cluster from 2.4.0 to 2.9.0

Any idea why this is happening?

Thanks in advance

2

Something maybe changed, need to add region parameter to the settings field when registering repository, like this:

PUT /_snapshot/my-s3-repository
{
  "type": "s3",
  "settings": {
    "bucket": "x",
    "base_path": "y",
    "region": "us-west-2"
  }
}