Unable set up OpenSearch manual snapshots with the master user from an "internal user database"

OpenSearch
,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch 2.13

Describe the issue:
We have an OpenSearch domain with manual snapshots, it was working fine until we enable fine grained access control with master user from “Internal user database”.

This is the related CDK code:

this.domainPass = new secretsmanager.Secret(this, 'search-master-pass', {
  description: `${domainId} OpenSearch master user password`,
})

this.domain = new opensearch.Domain(scope, domainId, {
  fineGrainedAccessControl: {
    masterUserName: this.domainPass.secretValueFromJson('username').toString(),
    masterUserPassword: this.domainPass.secretValueFromJson('password'),
  },
  encryptionAtRest: {
    enabled: true,
  },
  nodeToNodeEncryption: true,
  enforceHttps: true,
  version: opensearch.EngineVersion.openSearch('2.13'),
  ...
})
this.domain.addAccessPolicies(
  new iam.PolicyStatement({
    actions: ['es:*'],
    principals: [new iam.AnyPrincipal()],
    resources: [`${this.domain.domainArn}`, `${this.domain.domainArn}/*`],
  })
)

// Lambda that connects OpenSearch
this.domainLambda = new nodejs.NodejsFunction(this, `domain-lambda`, {
  architecture: lambda.Architecture.ARM_64,
  handler: 'index.handler',
  runtime: lambda.Runtime.NODEJS_20_X,
  environment: {
      OPENSEARCH_USERNAME: this.domainPass.secretValueFromJson('username').toString(),
      OPENSEARCH_PASSWORD: this.domainPass.secretValueFromJson('password').toString(),
  },
})

this.domain.grantReadWrite(this.domainLambda)
this.domainLambda.node.addDependency(this.domain)

// Snapshot setup
const snapshotBucket = new s3.Bucket(this, 'os-snapshots', {
  blockPublicAccess: s3.BlockPublicAccess.BLOCK_ALL,
})
const snapshotRole = new iam.Role(this, 'snapshot-role', {
  assumedBy: new iam.ServicePrincipal('es.amazonaws.com'),
})
snapshotRole.addToPolicy(
  new iam.PolicyStatement({
    actions: ['s3:ListBucket'],
    resources: [`${snapshotBucket.bucketArn}`],
  })
)
snapshotRole.addToPolicy(
  new iam.PolicyStatement({
    actions: ['s3:GetObject', 's3:PutObject', 's3:DeleteObject'],
    resources: [snapshotBucket.arnForObjects('*')],
  })
)

this.domainLambda.node.addDependency(snapshotBucket)
this.domainLambda.addToRolePolicy(
  new iam.PolicyStatement({
    actions: ['iam:PassRole'],
    resources: [snapshotRole.roleArn],
  })
)

This is how we connect from domainLambda function:

const client = new Client({
  node: `https://${domainEndpoint}`,
  auth: {
    username: process.env.OPENSEARCH_USERNAME,
    password: process.env.OPENSEARCH_PASSWORD,
  },
})

This works fine for searching, indexing and updating cluster settings etc.

But it fails with the following error when setting up snapshots:

{
  "body": {
    "Message": "User: anonymous is not authorized to perform: iam:PassRole on resource: arn:aws:iam::1234567890:role/dev-search-snapshot-role because no resource-based policy allows the iam:PassRole action"
  },
  "statusCode": 403,
}

When we use AwsSigv4Signer domainLambda function:

const client = new Client({
  node: `https://${domainEndpoint}`,
  ...AwsSigv4Signer({
      region: process.env.AWS_REGION,
      getCredentials: () => defaultProvider()(),
    }),
})

We get the following error when setting up snapshots:

{
  "body": {
    "error": {
      "reason": "no permissions for [cluster:admin/repository/put] and User [name=arn:aws:iam::1234567890:role/dev-resources-searchdomainlambda-88MJ3WaAhb57, backend_roles=[arn:aws:iam::1234567890:role/dev-resources-searchdomainlambda-88MJ3WaAhb57], requestedTenant=null]"
    }
  },
  "statusCode": 403,
}

I think we need to replace the following code with master user instead of lambda but not sure how to do that as master user is just a plain username and password:

this.domainLambda.addToRolePolicy(
  new iam.PolicyStatement({
    actions: ['iam:PassRole'],
    resources: [snapshotRole.roleArn],
  })
)

We can’t use “IAM Principal” instead of “Internal user database” because we have this issue mentioned here.

Originally posted here