OpenSearch Ingestion sink fails with security_exception on existsIndexTemplate despite es:* + cluster_all permissions

Data Prepper
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

OpenSearch: 3.3
Deployment: Amazon OpenSearch Ingestion (managed Data Prepper), domain with fine-grained access control enabled, VPC-based domain

Describe the issue:

Our OpenSearch Ingestion pipeline’s OpenSearch sink fails to initialize with a retryable security_exception: authentication/authorization failure, specifically on the composable index template existence check (existsIndexTemplate). This persists even after granting maximum permissions on both the OpenSearch internal security role and the IAM execution role.

Stack trace:

org.opensearch.client.opensearch.\_types.OpenSearchException: Request failed: \[security_exception\] authentication/authorization failure
at org.opensearch.client.transport.aws.AwsSdk2Transport.parseResponse(AwsSdk2Transport.java:541)
at org.opensearch.client.transport.aws.AwsSdk2Transport.executeSync(AwsSdk2Transport.java:440)
at org.opensearch.client.transport.aws.AwsSdk2Transport.performRequest(AwsSdk2Transport.java:217)
at org.opensearch.client.opensearch.indices.OpenSearchIndicesClient.existsIndexTemplate(OpenSearchIndicesClient.java:571)
at org.opensearch.dataprepper.plugins.sink.opensearch.index.ComposableTemplateAPIWrapper.getTemplate(ComposableTemplateAPIWrapper.java:45)
at org.opensearch.dataprepper.plugins.sink.opensearch.index.ComposableIndexTemplateStrategy.getExistingTemplateVersion(ComposableIndexTemplateStrategy.java:28)
at org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager.shouldCreateTemplate(AbstractIndexManager.java:292)
at org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager.checkAndCreateIndexTemplate(AbstractIndexManager.java:246)
at org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager.setupIndex(AbstractIndexManager.java:224)
at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.doInitializeInternal(OpenSearchSink.java:173)

The exact same underlying operation succeeds when run manually via OpenSearch Dashboards Dev Tools as the master user:
GET _index_template/main-fashchat-vectorindex
→ 404 resource_not_found_exception (expected, template doesn’t exist yet)

Troubleshooting already performed (all unsuccessful):

  • Confirmed OpenSearch role mapping (backend role → security role) is correct, under “Backend roles” not “Users”
  • Escalated OpenSearch internal role permissions to cluster_all + indices_all on index pattern * — no change
  • Escalated IAM policy on the pipeline’s execution role to es:* wildcard on the domain resource — no change
  • Purged OpenSearch security plugin cache — no change
  • Fully stopped/restarted the pipeline (multiple times) — no change
  • Confirmed domain access policy is fully open (Principal: , Allow, es:)
  • Confirmed cluster health is green, no relocating/unassigned shards

Given that maximum permissions on both IAM and OpenSearch security layers don’t resolve this, and the identical operation succeeds via a different auth path, this looks like it could be related to how the sink signs/sends this specific request, or an incompatibility between the Data Prepper client bundled in OpenSearch Ingestion and OpenSearch 3.3’s composable template API.

Configuration:

Sink config:
{
“opensearch”: {
“hosts”: [“https://vpc-vectordb-cjeqgna22xddlonz3uar3drrxe.ap-south-1.es.amazonaws.com”],
“aws”: {
“serverless”: false,
“region”: “ap-south-1”
},
“index_type”: “custom”,
“index”: “main-fashchat-vectorindex”,
“template_type”: “index-template”
}
}

Relevant Logs or Screenshots:

image
image1687×568 155 KB

image
image1768×452 101 KB

Thanks

Aditya Aggarwal

2

@Aadis_11 I understand that both OpenSearch and Data Prepper are AWS-managed solutions. I think you should contact AWS support and report this issue.

The sink configuration looks fine. The error doesn’t state whether this is a role or user/password issue.

Maybe try this policy temporarily and see if it helps. It is full access, but at least it will tell you if the Data Prepper sink can access the OS at all.

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "*",
            "Resource": "*"
        }
    ]
}