Issue with Configuring repository-s3 Plugin in OpenSearch Helm Deployment

DevOps Kubernetes
, ,
1

I am deploying OpenSearch using the official Helm chart and want to add the repository-s3 plugin for snapshot storage in AWS S3. To achieve this, I have enabled the plugin in the Helm values file under the plugins section as follows:

plugins:
  enabled: true
  installList:
    - "repository-s3"

Additionally, I have configured my AWS credentials by setting the access key, secret key, and region as environment variables:

extraEnvs:
  - name: AWS_ACCESS_KEY_ID
    value: "<my_access_key>"
  - name: AWS_SECRET_ACCESS_KEY
    value: "<my_secret_key>"
  - name: AWS_REGION
    value: "<my_region>"

The credentials have the required permissions to perform snapshot operations in S3. However, after deploying OpenSearch, I encountered the following error when attempting to create an S3 snapshot repository:

{"error":{"root_cause":[{"type":"repository_exception","reason":"[s3-repo] Could not determine repository generation from root blobs"}],"type":"repository_exception","reason":"[s3-repo] Could not determine repository generation from root blobs","caused_by":{"type":"i_o_exception","reason":"Exception when listing blobs by prefix [index-]","caused_by":{"type":"sdk_client_exception","reason":"Failed to load credentials from IMDS.","caused_by":{"type":"sdk_client_exception","reason":"The requested metadata is not found at http://<ip>/latest/meta-data/iam/security-credentials/"}}}},"status":500}

Despite setting the AWS credentials explicitly, OpenSearch appears to be attempting to retrieve them from the Instance Metadata Service (IMDS) instead of using the provided environment variables. I have also verified that the repository-s3 plugin is installed inside the pod by running:

kubectl exec -it <opensearch-pod> -- bin/opensearch-plugin list

Even after this, the error remains. It seems OpenSearch is still failing to authenticate properly with S3. I would like to know if there is a specific configuration required to ensure OpenSearch correctly reads the AWS credentials from environment variables. Additionally, if any extra configurations need to be added to the values.yaml file for repository-s3 to work correctly, please provide guidance on that.

2

@swetha AWS access and secret keys must be placed in opensearch.keystore as per OpenSearch documentation.

To achieve that in OpenSearch helm you must create a secret with access and secret keys and pass it as keystore element in values.yml

Please bear in mind that both key and secret must be encoded with base64 before adding to Secret manifest.

apiVersion: v1
kind: Secret
metadata:
  name: aws-credentials
type: Opaque
data:
  s3.client.default.access_key: <encoded key with base64>
  s3.client.default.secret_key: <encoded with base64>

values.yml

config:
  # Values must be YAML literal style scalar / YAML multiline string.
  # <filename>: |
  #   <formatted-value(s)>
  # log4j2.properties: |
  #   status = error
  #
  #   appender.console.type = Console
  #   appender.console.name = console
  #   appender.console.layout.type = PatternLayout
  #   appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
  #
  #   rootLogger.level = info
  #   rootLogger.appenderRef.console.ref = console
  opensearch.yml: |
    cluster.name: opensearch-cluster
    # Bind to all interfaces because we don't know what IP address Docker will assign to us.
    network.host: 0.0.0.0
    s3.client.default.region: eu-west-1
keystore:
# To add secrets to the keystore:
#  - secretName: opensearch-encryption-key
  - secretName: aws-credentials
## Enable to add 3rd Party / Custom plugins not offered in the default OpenSearch image.
plugins:
  enabled: true
  installList:
  - repository-s3
1 Like
3

Thanks, @pablo

Now, I can able to create the s3 repository in opensearch.

1 Like