Ingesting json files within aws pipeline(dataprepper)

Data Prepper
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
opensearch (AWS) 2.13

Describe the issue:
I am using a pipeline to ingest json logs in opensearch. In the pipeline . My pipeline is very simple. I just use codec : newline and in the processor section, I delete the s3 entries
here is an exerp of my pipeline
source when the record is inserted in the sink(s)
compression: “none”
codec:
newline:
sqs:
queue_url: https://sqs.eu-west-1.amazonaws.com/blablabla/myappr
visibility_timeout: 60s
aws:
sts_role_arn: arn:aws:iam::104276265064:role/mypipeline
region: eu-west-1
delete_s3_objects_on_read: false
processor:

  • delete_entries:
    with_keys: [“s3”]

sink:

Configuration:

Here is an an example of a json record I try to ingest
{“timestamp”:“2024-09-04T07:23:39.144Z”,“sequence”:24744,“loggerClassName”:“org.slf4j.impl.Slf4jLogger”,“loggerName”:“patrimoine.edixit.app.config.jaxrs.JaxRsLoggingInterceptor”,“level”:“DEBUG”,“message”:“Calling : class patrimoine.edixit.app.api.rest.auth.AppApiResourceAuth#currentUser”,“threadName”:“default task-129”,“threadId”:332,“mdc”:{“userlogin”:“SM”},“ndc”:“”,“hostName”:“8af329c7723c”,“processName”:“jboss-modules.jar”,“processId”:91}

When the pipeline process the logs It doesn’t recognize the fields .

when I do a simple put /index/_doc/1 it works like a charm. I don’t know why I have this issue because the json is well structured

Describe the issue:

Configuration:

Relevant Logs or Screenshots:

2

I you want parse json file with aws pipeline, you must set codec to json no newline

And the document must be in events root node

3

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.