Versions 2.9
Describe the issue: Some of the kubernetes logs are being rejected by the OpenSearch
Configuration: I am using fluentd so here is my source config
<source>
@type tail
@id in_tail_container_logs
@label @KUBERNETES_POD_LOGS
path /var/log/containers/*.log
pos_file /var/log/fluentd-containers.log.pos
tag kubernetes.*
read_from_head true
<parse>
@type multi_format
<pattern>
format json
time_key time
time_type string
time_format "%Y-%m-%dT%H:%M:%S.%NZ"
keep_time_key false
</pattern>
<pattern>
format regexp
expression /^(?<time>.+) (?<stream>stdout|stderr)( (.))? (?<log>.*)$/
time_format '%Y-%m-%dT%H:%M:%S.%NZ'
keep_time_key false
</pattern>
</parse>
emit_unmatched_lines true
</source>
Here is a snippet from my filter section
`<match kubernetes.var.log.containers.fluentd-*_opensearch_*.log>
@type relabel
@label @FLUENT_LOG
</match>
<filter kubernetes.**>
@type kubernetes_metadata
@id filter_kube_metadata
skip_labels false
skip_pod_labels true
skip_namespace_labels false
skip_namespace_metadata false
skip_container_metadata true
skip_master_url true
stats_interval 0
</filter>
<filter kubernetes.**>
@type record_transformer
remove_keys $.docker, $.kubernetes.pod_id, $.kubernetes.labels, $.kubernetes.namespace_id, $['kubernetes']['namespace_labels']['``argocd.argoproj.io/instance``'], $['kubernetes']['namespace_labels']['``kubernetes.io/metadata.name``'], $.kubernetes.namespace_labels.namespace-type
</filter>
<match kubernetes.**>
@type detect_exceptions
remove_tag_prefix kubernetes
message log
languages java, python, js
force_line_breaks true
multiline_flush_interval 0.1
</match>`
Relevant Logs or Screenshots:
These are some of the errors I am getting in fluentd
2025-10-15 18:59:58 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error=“400 - Rejected by OpenSearch” location=nil tag=“var.log.containers.learch-api-798dfb9d78-2xsj8_learch-api-prod_learch-api-43c621204daf6176c53307fa366832874605d8312e86d6b4de7f082459943cda.log” time=2025-10-15 18:59:53.299882872 +0000 record={“stream”=>“stdout”, “log”=>“15 Oct 2025 18:59:53 com.yp.search.evaluators.DefaultTrueEvaluator:25 INFO com.yp.search.evaluators.DefaultTrueEvaluator - Evaluating…”, “kubernetes”=>{“container_name”=>“learch-api”, “namespace_name”=>“learch-api-prod”, “pod_name”=>“learch-api-798dfb9d78-2xsj8”, “pod_ip”=>“172.19.137.1”, “host”=>“phx1-q17-host20”, “namespace_labels”=>{“app”=>“learch-api”, “app.kubernetes.io/managed-by"=>"Helm”, “field.cattle.io/projectId"=>"consumer-search”, “istio-injection”=>“enabled”, “objectset.rio.cattle.io/hash"=>"12dc83ae61371d4aa04ae98ecb1fa7c8c09a7440”}}}
2025-10-15 18:59:58 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error=“400 - Rejected by OpenSearch” location=nil tag=“var.log.containers.spellchecker-api-b56f54b56-pz88b_spellchecker-api-prod_spellchecker-api-7ecacabe114cd1d813a1680f89b75acac9de3a078be56d3ec245dc171d960fe8.log” time=2025-10-15 18:59:52.860257146 +0000 record={“stream”=>“stdout”, “log”=>“15 Oct 2025 18:59:52:860 com.yp.qis.spellchecker.modules.LuceneSpellcheckerModule:590 INFO com.yp.qis.spellchecker.modules.LuceneSpellcheckerModule - Suggestion search will use a max of 100 candidates.”, “kubernetes”=>{“container_name”=>“spellchecker-api”, “namespace_name”=>“spellchecker-api-prod”, “pod_name”=>“spellchecker-api-b56f54b56-pz88b”, “pod_ip”=>“172.19.137.229”, “host”=>“phx1-q17-host20”, “namespace_labels”=>{“app”=>“spellchecker-api”, “app.kubernetes.io/managed-by"=>"Helm”, “field.cattle.io/projectId"=>"consumer-search”, “istio-injection”=>“enabled”, “objectset.rio.cattle.io/hash"=>"12dc83ae61371d4aa04ae98ecb1fa7c8c09a7440”}}}