Issues with DynamoDB to OpenSearch Pipeline: No New Shards Acquired

Data Prepper
1

Versions:

  • OpenSearch: 2.13

Describe the issue:
I have configured an OpenSearch ingestion pipeline to stream data from a DynamoDB table into OpenSearch. The export appears to have completed, but I’m encountering a persistent issue where no new events are being streamed from DynamoDB to OpenSearch. Specifically, the log message indicates:

2024-07-24T12:26:22.493 [pool-14-thread-4] INFO org.opensearch.dataprepper.plugins.source.dynamodb.stream.StreamScheduler - No new shards acquired after 14100 attempts. This means that all shards are currently being consumed, or that the export is still in progress. New shards will not be consumed until the export is fully processed.

Configuration:

version: "2"
dynamodb-pipeline:
  source:
    dynamodb:
      acknowledgments: true
      tables:
        - table_arn: "arn:aws:dynamodb:xxxxxxxxxxxxxxx"
          stream:
            start_position: "LATEST"
          export:
            s3_bucket: "xxxxxx
            s3_region: "xxxxxx"
            s3_prefix: "dev/"
      aws:
         sts_role_arn: "arn:aws:iam::xxxxxxxxxxx"
         region: "xxxxx"
  sink:
    - opensearch:
        hosts: [ https://vpc-xxxxxxxxx ]
        index: "table-index"
        index_type: custom
        document_id: "${getMetadata(\"primary_key\")}"
        action: "${getMetadata(\"opensearch_action\")}"
        document_version: "${getMetadata(\"document_version\")}"
        document_version_type: "external"
        aws:
          sts_role_arn: "xxxxxxxxx"
          region: "xxxxxxx2"

Additional Information:

  • DynamoDB Streams are enabled for the table. And the stream should have no other consumers.
  • Permissions are configured as per AWS recommendations
  • The pipeline seems to have initially processed some data but is now stuck.

Questions:

  1. What might be causing the ingestion pipeline to stop acquiring new shards?
  2. How can I debug why new events are not being streamed to OpenSearch?
  3. Are there specific configurations or settings I should review to resolve this issue?

Thank you for your assistance!