Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
opensearch (AWS) 2.13
Describe the issue:
I am using a pipeline to ingest json logs in opensearch. In the pipeline . My pipeline is very simple. I just use codec : newline and in the processor section, I delete the s3 entries
here is an exerp of my pipeline
source when the record is inserted in the sink(s)
compression: “none”
codec:
newline:
sqs:
queue_url: https://sqs.eu-west-1.amazonaws.com/blablabla/myappr
visibility_timeout: 60s
aws:
sts_role_arn: arn:aws:iam::104276265064:role/mypipeline
region: eu-west-1
delete_s3_objects_on_read: false
processor:
- delete_entries:
with_keys: [“s3”]
sink:
- opensearch:
hosts: [ https://myopensearchdomain ]
index: mydomain
aws:
Configuration:
Here is an an example of a json record I try to ingest
{“timestamp”:“2024-09-04T07:23:39.144Z”,“sequence”:24744,“loggerClassName”:“org.slf4j.impl.Slf4jLogger”,“loggerName”:“patrimoine.edixit.app.config.jaxrs.JaxRsLoggingInterceptor”,“level”:“DEBUG”,“message”:“Calling : class patrimoine.edixit.app.api.rest.auth.AppApiResourceAuth#currentUser”,“threadName”:“default task-129”,“threadId”:332,“mdc”:{“userlogin”:“SM”},“ndc”:“”,“hostName”:“8af329c7723c”,“processName”:“jboss-modules.jar”,“processId”:91}
When the pipeline process the logs It doesn’t recognize the fields .
when I do a simple put /index/_doc/1 it works like a charm. I don’t know why I have this issue because the json is well structured
Relevant Logs or Screenshots: