OpenSearch vs DynamoDB table: Request failed: [security_exception] no permissions for [cluster:monitor/state] and User

OpenSearch
1

Hi,
I’m trying to enable full-text search on a DynamoDB table. I’ve created an OpenSearch domain for this, specific role and policy, and tried to create an ingestion pipeline. The pipeline is active, but I cannot see the index with the data from the table in the OpenSearch Dashboard on my domain, and I’m getting a warning in the logs:
2024-01-16T15:57:49.613 [dynamodb-pipeline-sink-worker-2-thread-1] WARN org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Failed to initialize OpenSearch sink with a retryable exception. org.opensearch.client.opensearch._types.OpenSearchException: Request failed: [security_exception] no permissions for [cluster:monitor/state] and User [name=arn:aws:iam::{account_number}:role/OSPipelineRole, backend_roles=[arn:aws:iam::{account_number}:role/OSPipelineRole], requestedTenant=null] at org.opensearch.client.transport.aws.AwsSdk2Transport.parseResponse(AwsSdk2Transport.java:479) ~[opensearch-java-2.5.0.jar:?] at org.opensearch.client.transport.aws.AwsSdk2Transport.executeSync(AwsSdk2Transport.java:393) ~[opensearch-java-2.5.0.jar:?] at org.opensearch.client.transport.aws.AwsSdk2Transport.performRequest(AwsSdk2Transport.java:191) ~[opensearch-java-2.5.0.jar:?] at org.opensearch.client.opensearch.cluster.OpenSearchClusterClient.getSettings(OpenSearchClusterClient.java:282) ~[opensearch-java-2.5.0.jar:?] at org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager.checkISMEnabled(AbstractIndexManager.java:212) ~[opensearch-2.6.0.jar:?] at org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager.checkAndCreateIndexTemplate(AbstractIndexManager.java:239) ~[opensearch-2.6.0.jar:?] at org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager.setupIndex(AbstractIndexManager.java:234) ~[opensearch-2.6.0.jar:?] at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.doInitializeInternal(OpenSearchSink.java:229) ~[opensearch-2.6.0.jar:?] at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.doInitialize(OpenSearchSink.java:191) ~[opensearch-2.6.0.jar:?] at org.opensearch.dataprepper.model.sink.AbstractSink.initialize(AbstractSink.java:52) ~[data-prepper-api-2.6.0.jar:?] at org.opensearch.dataprepper.pipeline.Pipeline.isReady(Pipeline.java:200) ~[data-prepper-core-2.6.0.jar:?] at org.opensearch.dataprepper.pipeline.Pipeline.lambda$execute$2(Pipeline.java:252) ~[data-prepper-core-2.6.0.jar:?] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.base/java.lang.Thread.run(Thread.java:829) [?:?]

Here’s my policy json:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "es:*",
            "Resource": "arn:aws:es:us-east-1:{account_number}:domain/applicants-domain/*"
        },
        {
            "Effect": "Allow",
            "Action": "es:*",
            "Resource": "arn:aws:es:us-east-1:{account_number}:domain/applicants-domain"
        },
        {
            "Effect": "Allow",
            "Action": "es:DescribeDomain",
            "Resource": "arn:aws:es:us-east-1:{account_number}:domain/applicants-domain"
        },
        {
            "Effect": "Allow",
            "Action": "es:ESHttp*",
            "Resource": "arn:aws:es:us-east-1:{account_number}:domain/applicants-domain/*"
        }
    ]
}

the pipeline role’s trust permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "cloudformation.amazonaws.com",
                    "cloudsearch.amazonaws.com",
                    "s3.amazonaws.com",
                    "lambda.amazonaws.com",
                    "osis-pipelines.amazonaws.com",
                    "dynamodb.amazonaws.com"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

and the pipeline yaml file:

version: "2"
dynamodb-pipeline:
  source:
    dynamodb:
      acknowledgments: true
      tables:
        - table_arn: "arn:aws:dynamodb:us-east-1:{account_number}:table/applicants"
          stream:
            start_position: "LATEST"
          export:
            s3_bucket: "pioneers-education-opensearch-dev"
            s3_region: "us-east-1"
            s3_prefix: "ddb-to-opensearch/"
      aws:
        sts_role_arn: "arn:aws:iam::{account_number}:role/OSPipelineRole"
        region: "us-east-1"

  sink:
    - opensearch:
        hosts: [ "https://search-applicants-domain-lpki4i4m3pz6jweojyqzru.aos.us-east-1.on.aws" ]
        index: "users"
        aws:
          sts_role_arn: "arn:aws:iam::{account_number}:role/OSPipelineRole"
          region: "us-east-1"

Actually, I have the real number instead of {account_number} everywhere, I just removed it for the forum.

Seems like I have the wrong permissions here, please guide me on what I might be doing wrong.

2

Have you managed to find the solution to this problem, I have the same experience

3

Yes, thank you for asking!

In my OpenSearch dashboard (you can find the link in the domain description):

a) Choose the Security option from left hand side menu.

b) Choose Roles.

c) Select all_access.

d) Inside all_access, select Mapped Users and select Manage Mapping.

e) In the Backend roles, provide the IAM role ARN which you have created and attached the same in the access policy of the cluster and choose Map.

That worked!

1 Like
4

Thank you, it worked. Couldnt find any other documentation that indicated this.
Appreciate the help