How to retry the failed documents in the index

OpenSearch
,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): logstash version 8.3.2. Server OS RHEL 8.10. OpenSearch v1.2.0

Describe the issue: We are getting below error.


Could not index event to OpenSearch

"error"=>("type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [loggerData.content] of type [text] in document

"caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START OBJECT at 1:336")

To resolve this error temporarily we have to delete the existing index but the error occurs again after few days as we are using dynamic mapping in opensearch.

Is there a way where we can retrigger the failed logs from logstash to opensearch.

As this error is occurring for few applications and the index contains many application. If we delete the index it will delete large amount of logs for that particular day. Hence, is there any suggestion where we can resend or retry the logs to opensearch from logstash server.

We know we can configure the dead letter queue but is there any other way apart from dead letter queue configuration.

Also, can we setup a predfined mapping schema for one application that can help us resolve this issue.

Thanks!

Configuration:

Relevant Logs or Screenshots:

2

@rohit_dhiman Logstash doesn’t register processed documents. Also, it is not aware of incorrect mapping in the OpenSearch. Logstash only filters and modifies ingested data.

You could try to create a new field for that specific type of document or drop it in Logstash.