Error 400 - Rejected by OpenSearch [error type]: mapper_parsing_exception

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

OpenSearch 2.6

Describe the issue:

While trying to send documents from Fluentd to OpenSearch, I am seeing multiple instances of below error message, I can’t seem to identify which character is causing the parsing exception.

I am not using any parsers on Fluentd side, any help on how to overcome this error is appreciated

Relevant Logs or Screenshots:

2023-05-29 16:17:47 +0530 [warn]: #0 fluent/log.rb:351:warn: dump an error event: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error="400 - Rejected by OpenSearch [error type]: mapper_parsing_exception [reason]: 'failed to parse field [log] of type [text] in document with id 'zeEeZ4gBmGsd-CaUkJdZ'. Preview of field's value: '''" location=nil tag="Connectship.BEL.PDC00CSBA304W" time=2023-05-29 16:16:09.356339400 +0530 record={"filename"=>"C:\\ConnectShip\\Logs\\BelAudit.log", "log"=>"03/02/2022 21:31:55 [1] INFO  AUDITLOGGER - Computer :TR7-PF2BM35S - User :MNGIMDEV - Carton Id : - Company Name :Miriam Hernandez Contreras - Contact Name :Miriam Hernandez Contreras - Address: Av.Rio Nilo 3220, Luis Alfaro y Pi\xBFa y J, , Guadalajara, Ja, 44860 ", "hostname"=>"PDC00CSBA304W", "Type"=>"BEL", "host"=>"10.202.115.51"}
2023-05-29 16:17:47 +0530 [warn]: #0 fluent/log.rb:351:warn: dump an error event: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error="400 - Rejected by OpenSearch [error type]: mapper_parsing_exception [reason]: 'failed to parse field [log] of type [text] in document with id '2OEeZ4gBmGsd-CaUkJdZ'. Preview of field's value: '''" location=nil tag="Connectship.BEL.PDC00CSBA304W" time=2023-05-29 16:16:09.356339400 +0530 record={"filename"=>"C:\\ConnectShip\\Logs\\BelAudit.log", "log"=>"03/03/2022 15:59:55 [17] INFO  AUDITLOGGER - Computer :TR3-PF2BM3N2 - User :MNGIMDEV - Carton Id : - Company Name :Ana Lorena Zendejas velazquez - Contact Name :Ana Lorena Zendejas velazquez - Address: L\xBFpez Mateos 722 col petrolera , Mexico, , Coatzacoalcos, Ve, 96500 ", "hostname"=>"PDC00CSBA304W", "Type"=>"BEL", "host"=>"10.202.115.51"}
2023-05-29 16:17:47 +0530 [warn]: #0 fluent/log.rb:351:warn: dump an error event: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error="400 - Rejected by OpenSearch [error type]: mapper_parsing_exception [reason]: 'failed to parse field [log] of type [text] in document with id 'OOEeZ4gBmGsd-CaUkJpc'. Preview of field's value: '''" location=nil tag="Connectship.BEL.PDC00CSBA304W" time=2023-05-29 16:16:09.403140000 +0530 record={"filename"=>"C:\\ConnectShip\\Logs\\BelAudit.log", "log"=>"04/04/2022 14:54:32 [1] INFO  AUDITLOGGER - Computer :5CG9285X9T-LA - User :MNGDEV - Carton Id : - Company Name :Ana Lorena Zendejas velazquez - Contact Name :Ana Lorena Zendejas velazquez - Address: L\xBFpez Mateos 722 col petrolera , Mexico, , Coatzacoalcos, Ve, 96500 ", "hostname"=>"PDC00CSBA304W", "Type"=>"BEL", "host"=>"10.202.115.51"}
2023-05-29 16:17:47 +0530 [warn]: #0 fluent/log.rb:351:warn: dump an error event: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error="400 - Rejected by OpenSearch [error type]: mapper_parsing_exception [reason]: 'failed to parse field [log] of type [text] in document with id '2uEeZ4gBmGsd-CaUkLp2'. Preview of field's value: '''" location=nil tag="Connectship.BEL.PDC00CSBA304W" time=2023-05-29 16:16:10.168549900 +0530 record={"filename"=>"C:\\ConnectShip\\Logs\\BelAudit.log", "log"=>"10/28/2022 11:53:07 [54] INFO  AUDITLOGGER - Computer :TR7-PF2BM35S - User :MNGIMDEV - Carton Id : - Company Name :Dennise \xBFlvarez - Contact Name :Dennise \xBFlvarez - Address: Vista Horizonte s/n torre bosque 1 depto, , Huixquilucan, M\xBF, 52779 ", "hostname"=>"PDC00CSBA304W", "Type"=>"BEL", "host"=>"10.202.115.51"}
2023-05-29 16:17:47 +0530 [warn]: #0 fluent/log.rb:351:warn: dump an error event: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error="400 - Rejected by OpenSearch [error type]: mapper_parsing_exception [reason]: 'failed to parse field [log] of type [text] in document with id '5-EeZ4gBmGsd-CaUkLp2'. Preview of field's value: '''" location=nil tag="Connectship.BEL.PDC00CSBA304W" time=2023-05-29 16:16:10.168549900 +0530 record={"filename"=>"C:\\ConnectShip\\Logs\\BelAudit.log", "log"=>"10/28/2022 12:50:00 [26] INFO  AUDITLOGGER - Computer :5CG9285X9T-LA - User :mngDev - Carton Id : - Company Name :Dennise \xBFlvarez - Contact Name :Dennise \xBFlvarez - Address: Vista Horizonte s/n torre bosque 1 depto, , Huixquilucan, M\xBF, 52779 ", "hostname"=>"PDC00CSBA304W", "Type"=>"BEL", "host"=>"10.202.115.51"}
2023-05-29 16:17:47 +0530 [warn]: #0 fluent/log.rb:351:warn: dump an error event: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error="400 - Rejected by OpenSearch [error type]: mapper_parsing_exception [reason]: 'failed to parse field [log] of type [text] in document with id '6OEeZ4gBmGsd-CaUkLp2'. Preview of field's value: '''" location=nil tag="Connectship.BEL.PDC00CSBA304W" time=2023-05-29 16:16:10.168549900 +0530 record={"filename"=>"C:\\ConnectShip\\Logs\\BelAudit.log", "log"=>"10/28/2022 12:50:25 [26] INFO  AUDITLOGGER - Computer :5CG9285X9T-LA - User :mngDev - Carton Id : - Company Name :Dennise \xBFlvarez - Contact Name :Dennise \xBFlvarez - Address: Vista Horizonte s/n torre bosque 1 depto, , Huixquilucan, M\xBF, 52779 ", "hostname"=>"PDC00CSBA304W", "Type"=>"BEL", "host"=>"10.202.115.51"}
2023-05-29 16:17:47 +0530 [warn]: #0 fluent/log.rb:351:warn: dump an error event: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error="400 - Rejected by OpenSearch [error type]: mapper_parsing_exception [reason]: 'failed to parse field [log] of type [text] in document with id '7eEeZ4gBmGsd-CaUkLp2'. Preview of field's value: '''" location=nil tag="Connectship.BEL.PDC00CSBA304W" time=2023-05-29 16:16:10.168549900 +0530 record={"filename"=>"C:\\ConnectShip\\Logs\\BelAudit.log", "log"=>"10/31/2022 13:25:15 [48] INFO  AUDITLOGGER - Computer :5CG9285X9T-LA - User :mngDev - Carton Id : - Company Name :Dennise \xBFlvarez - Contact Name :Dennise \xBFlvarez - Address: Vista Horizonte s/n torre bosque 1 depto, , Huixquilucan, M\xBF, 52779 ", "hostname"=>"PDC00CSBA304W", "Type"=>"BEL", "host"=>"10.202.115.51"}
2023-05-29 16:17:47 +0530 [warn]: #0 fluent/log.rb:351:warn: dump an error event: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error="400 - Rejected by OpenSearch [error type]: mapper_parsing_exception [reason]: 'failed to parse field [log] of type [text] in document with id '8-EeZ4gBmGsd-CaUkLp2'. Preview of field's value: '''" location=nil tag="Connectship.BEL.PDC00CSBA304W" time=2023-05-29 16:16:10.168549900 +0530 record={"filename"=>"C:\\ConnectShip\\Logs\\BelAudit.log", "log"=>"10/31/2022 14:01:14 [45] INFO  AUDITLOGGER - Computer :TR7-PF2BM35S - User :MNGIMDEV - Carton Id : - Company Name :Dennise \xBFlvarez - Contact Name :Dennise \xBFlvarez - Address: Vista Horizonte s/n torre bosque 1 depto, , Huixquilucan, M\xBF, 52779 ", "hostname"=>"PDC00CSBA304W", "Type"=>"BEL", "host"=>"10.202.115.51"}
2023-05-29 16:17:47 +0530 [warn]: #0 fluent/log.rb:351:warn: dump an error event: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error="400 - Rejected by OpenSearch [error type]: mapper_parsing_exception [reason]: 'failed to parse field [log] of type [text] in document with id 'leEeZ4gBmGsd-CaUkMB5'. Preview of field's value: '''" location=nil tag="Connectship.BEL.PDC00CSBA304W" time=2023-05-29 16:16:11.963573000 +0530 record={"filename"=>"C:\\ConnectShip\\Logs\\BelError.log", "log"=>"04/28/2023 14:28:28 [62] ERROR ERRORLOGGER - Computer : 5CG7330M0B - User : JECDEV - Carton Id : 235986723958629 - Error Message : 30005, Package #1: Endicia Web Service error: The Certified Intermediary\x92s pass phrase is incorrect. Error encountered (Log ID: 40657) (3001) .", "hostname"=>"PDC00CSBA304W", "Type"=>"BEL", "host"=>"10.202.115.51"}
2023-05-29 16:17:47 +0530 [warn]: #0 fluent/log.rb:351:warn: dump an error event: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error="400 - Rejected by OpenSearch [error type]: mapper_parsing_exception [reason]: 'failed to parse field [log] of type [text] in document with id 'FOEeZ4gBmGsd-CaUkMx9'. Preview of field's value: '''" location=nil tag="Connectship.BEL.PDC00CSBA304W" time=2023-05-29 16:16:11.510167100 +0530 record={"filename"=>"C:\\ConnectShip\\Logs\\BelError.log", "log"=>"11/09/2022 11:26:32 [62] ERROR ERRORLOGGER - Computer : KMX-PF2G31AN - User : NMXIMDEV - Carton Id : 540500000200090 - Error Message : 30008, Error Code: 513  Error message : default:[{\"code\":400,\"description\":\"The Parameter [apiRequest.labelDefinition.location.destination.homeAddress.contact.contactName] fails with Se\xF1ora Ana Gabriela Martinez Acosta - value must be match: .{1,30}\"},{\"code\":400,\"description\":\"The Parameter [apiRequest.labelDefinition.location.destination.homeAddress.contact.contactName] fails with Se\xF1ora Ana Gabriela Martinez Acosta - size must be max: 30 chars\"}].", "hostname"=>"PDC00CSBA304W", "Type"=>"BEL", "host"=>"10.202.115.51"}
2023-05-29 16:17:47 +0530 [warn]: #0 fluent/log.rb:351:warn: dump an error event: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error="400 - Rejected by OpenSearch [error type]: mapper_parsing_exception [reason]: 'failed to parse field [log] of type [text] in document with id 'IuEeZ4gBmGsd-CaUkNuS'. Preview of field's value: '''" location=nil tag="Connectship.BEL.PDC00CSBA304W" time=2023-05-29 16:16:11.853371500 +0530 record={"filename"=>"C:\\ConnectShip\\Logs\\BelError.log", "log"=>"03/13/2023 17:13:48 [44] ERROR ERRORLOGGER - Computer : 5CG9398QYP - User : tcdDEV - Carton Id : 943756348634845 - Error Message : 30005, Package #1: Endicia Web Service error: The Certified Intermediary\x92s pass phrase is incorrect. Error encountered (Log ID: 45929) (3001) .", "hostname"=>"PDC00CSBA304W", "Type"=>"BEL", "host"=>"10.202.115.51"}
2023-05-29 16:17:47 +0530 [warn]: #0 fluent/log.rb:351:warn: dump an error event: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error="400 - Rejected by OpenSearch [error type]: mapper_parsing_exception [reason]: 'failed to parse field [log] of type [text] in document with id 'I-EeZ4gBmGsd-CaUkNuS'. Preview of field's value: '''" location=nil tag="Connectship.BEL.PDC00CSBA304W" time=2023-05-29 16:16:11.853371500 +0530 record={"filename"=>"C:\\ConnectShip\\Logs\\BelError.log", "log"=>"03/13/2023 17:14:14 [44] ERROR ERRORLOGGER - Computer : 5CG9398QYP - User : tcdDEV - Carton Id : 983475934875983 - Error Message : 30005, Package #1: Endicia Web Service error: The Certified Intermediary\x92s pass phrase is incorrect. Error encountered (Log ID: 45961) (3001) .", "hostname"=>"PDC00CSBA304W", "Type"=>"BEL", "host"=>"10.202.115.51"}
2023-05-29 16:17:47 +0530 [warn]: #0 fluent/log.rb:351:warn: dump an error event: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error="400 - Rejected by OpenSearch [error type]: mapper_parsing_exception [reason]: 'failed to parse field [log] of type [text] in document with id 'JOEeZ4gBmGsd-CaUkNuS'. Preview of field's value: '''" location=nil tag="Connectship.BEL.PDC00CSBA304W" time=2023-05-29 16:16:11.853371500 +0530 record={"filename"=>"C:\\ConnectShip\\Logs\\BelError.log", "log"=>"03/13/2023 17:14:37 [44] ERROR ERRORLOGGER - Computer : 5CG9398QYP - User : tcdDEV - Carton Id : 983475934875983 - Error Message : 30005, Package #1: Endicia Web Service error: The Certified Intermediary\x92s pass phrase is incorrect. Error encountered (Log ID: 45983) (3001) .", "hostname"=>"PDC00CSBA304W", "Type"=>"BEL", "host"=>"10.202.115.51"}
2

Het @Amith

These two

failed to parse field [log] of type [text] in document with id ‘zeEeZ4gBmGsd-CaUkJdZ’

Endicia Web Service error: The Certified Intermediary\x92s pass phrase is incorrect. Error encountered (Log ID: 40657) (3001) .", “hostname”=>“PDC00CSBA304W”, “Type”=>“BEL”, “host”=>“10.202.115.51”

Not sure how you setup your connection/configurations

3

The errors generated in the “log” field are actual application logs, which seems to fail parsing for some reason. I do not have any field mapping specified, its using opensearch dynamic mapping.
Below is my fluentd configuration file :

<system>
  log_level debug
</system>
## Source descriptions:
<source>
  @type forward
  bind 0.0.0.0
  port 24225
  source_hostname_key host
</source>
<match Connectship.**>
  @type opensearch
  hosts 10.223.16.103:9200,10.223.16.100:9200,10.223.16.101:9200
  retry_on_failure true
  user admin
  password admin
  scheme https
  index_name connectship
  ssl_verify false
  @log_level debug
  with_transporter_log true
</match>
4

Usually, I use Divide et Impera in these cases. If I can find an offending document and an offending field (like you seem to do), I would try to index all the text to reproduce the problem, then just half of it until I identify the half that reproduces it and so on.