[Data Prepper] - S3 Source Error

Data Prepper
1

Hi,

I am using data prepper with S3 data source, where I have configure SNS queue to read logs from aws cloudtrail but it’s throwing an error related to parsing. It seems the event which is consumed from the SNS queue is not parsable.

I tried with all possible combination of Codec and Compression listed in the documentation, but the error didn’t solve.

image
image1887×977 121 KB

2

Hi @mann

Based on the error, it looks like the JSON string is invalid. Can you please share your pipeline configuration?

3

Hi @mann

Looks like you have configured SNS topic, we only support SQS queue in S3 source. Please try to modify your S3 bucket to send events to SQS queue and try again.

4

Thanks @asif I see, so it seems we can configure data prepper to support any kind of log collection like AWS Cloudtrail, Cloudwatch, VPC Flow Logs etc. Which can be collected via S3 bucket through SQS queue.

5

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.