Getting logs from kubernetes pods rejected by OpenSearch

Versions 2.9

Describe the issue: Some of the kubernetes logs are being rejected by the OpenSearch

Configuration: I am using fluentd so here is my source config

<source>
  @type tail
  @id in_tail_container_logs
  @label @KUBERNETES_POD_LOGS
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag kubernetes.*
  read_from_head true
  <parse>
    @type multi_format
    <pattern>
      format json
      time_key time
      time_type string
      time_format "%Y-%m-%dT%H:%M:%S.%NZ"
      keep_time_key false
    </pattern>
    <pattern>
      format regexp
      expression /^(?<time>.+) (?<stream>stdout|stderr)( (.))? (?<log>.*)$/
      time_format '%Y-%m-%dT%H:%M:%S.%NZ'
      keep_time_key false
    </pattern>
  </parse>
  emit_unmatched_lines true
</source>

Here is a snippet from my filter section

    `<match kubernetes.var.log.containers.fluentd-*_opensearch_*.log>
      @type relabel
      @label @FLUENT_LOG
    </match>

    <filter kubernetes.**>
      @type kubernetes_metadata
      @id filter_kube_metadata
      skip_labels false
      skip_pod_labels true
      skip_namespace_labels false
      skip_namespace_metadata false
      skip_container_metadata true
      skip_master_url true
      stats_interval 0
    </filter>

    <filter kubernetes.**>
      @type record_transformer
      remove_keys $.docker, $.kubernetes.pod_id, $.kubernetes.labels, $.kubernetes.namespace_id, $['kubernetes']['namespace_labels']['``argocd.argoproj.io/instance``'], $['kubernetes']['namespace_labels']['``kubernetes.io/metadata.name``'], $.kubernetes.namespace_labels.namespace-type
    </filter>

    <match kubernetes.**>
      @type detect_exceptions
      remove_tag_prefix kubernetes
      message log
      languages java, python, js
      force_line_breaks true
      multiline_flush_interval 0.1
    </match>`

Relevant Logs or Screenshots:

These are some of the errors I am getting in fluentd

2025-10-15 18:59:58 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error=“400 - Rejected by OpenSearch” location=nil tag=“var.log.containers.learch-api-798dfb9d78-2xsj8_learch-api-prod_learch-api-43c621204daf6176c53307fa366832874605d8312e86d6b4de7f082459943cda.log” time=2025-10-15 18:59:53.299882872 +0000 record={“stream”=>“stdout”, “log”=>“15 Oct 2025 18:59:53 com.yp.search.evaluators.DefaultTrueEvaluator:25 INFO com.yp.search.evaluators.DefaultTrueEvaluator - Evaluating…”, “kubernetes”=>{“container_name”=>“learch-api”, “namespace_name”=>“learch-api-prod”, “pod_name”=>“learch-api-798dfb9d78-2xsj8”, “pod_ip”=>“172.19.137.1”, “host”=>“phx1-q17-host20”, “namespace_labels”=>{“app”=>“learch-api”, “app.kubernetes.io/managed-by"=>"Helm”, “field.cattle.io/projectId"=>"consumer-search”, “istio-injection”=>“enabled”, “objectset.rio.cattle.io/hash"=>"12dc83ae61371d4aa04ae98ecb1fa7c8c09a7440”}}}

2025-10-15 18:59:58 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error=“400 - Rejected by OpenSearch” location=nil tag=“var.log.containers.spellchecker-api-b56f54b56-pz88b_spellchecker-api-prod_spellchecker-api-7ecacabe114cd1d813a1680f89b75acac9de3a078be56d3ec245dc171d960fe8.log” time=2025-10-15 18:59:52.860257146 +0000 record={“stream”=>“stdout”, “log”=>“15 Oct 2025 18:59:52:860 com.yp.qis.spellchecker.modules.LuceneSpellcheckerModule:590 INFO com.yp.qis.spellchecker.modules.LuceneSpellcheckerModule - Suggestion search will use a max of 100 candidates.”, “kubernetes”=>{“container_name”=>“spellchecker-api”, “namespace_name”=>“spellchecker-api-prod”, “pod_name”=>“spellchecker-api-b56f54b56-pz88b”, “pod_ip”=>“172.19.137.229”, “host”=>“phx1-q17-host20”, “namespace_labels”=>{“app”=>“spellchecker-api”, “app.kubernetes.io/managed-by"=>"Helm”, “field.cattle.io/projectId"=>"consumer-search”, “istio-injection”=>“enabled”, “objectset.rio.cattle.io/hash"=>"12dc83ae61371d4aa04ae98ecb1fa7c8c09a7440”}}}

Is it possible that it’s caused by these nested fields in the logs?

“app.kubernetes.io/managed-by"=>"Helm”, “field.cattle.io/projectId"=>"consumer-search” and

“objectset.rio.cattle.io/hash"=>"12dc83ae61371d4aa04ae98ecb1fa7c8c09a7440”

@stecino I was able to reproduce the 400 rejection.

I was able to use the followoing configuration to get this indexed in OS:

fluetd/Dockerfile

FROM fluent/fluentd:v1.16-debian
USER root

# (Optional) build deps if any gem needs native extensions
RUN apt-get update && apt-get install -y --no-install-recommends make gcc g++ && rm -rf /var/lib/apt/lists/*

# Install plugins
RUN fluent-gem install fluent-plugin-opensearch -v "~> 1.1" --no-document \
 && fluent-gem install fluent-plugin-kubernetes_metadata_filter -v "~> 3.8" --no-document \
 && fluent-gem install fluent-plugin-rename-key -v "~> 0.4" --no-document

# (Optional) remove build deps
RUN apt-get purge -y make gcc g++ && apt-get autoremove -y && apt-get clean

USER fluent

fluent.conf

# ========= SOURCE =========
<source>
  @type tail
  @id in_tail_app
  path /var/log/app/app.log
  pos_file /fluentd/log/app.pos
  tag app.raw
  read_from_head true
  <parse>
    @type json
    time_key time
    time_type string
    time_format %iso8601
  </parse>
</source>

# ========= NORMALIZE K8S LABEL/ANNOTATION KEYS =========
# de_dot + de_slash rewrites keys that contain '.' or '/' so OpenSearch accepts them
<filter app.raw>
  @type rename_key
  @id fix_field_names
  deep_rename true

  # replace dots in any key with underscore
  replace_rule1 \. _

  # replace slashes in any key with double underscore
  replace_rule2 / __
</filter>

# ========= OUTPUT TO OPENSEARCH =========
<match app.**>
  @type opensearch
  host opensearch
  port 9200
  scheme http

  # Rolling daily index with your prefix:
  logstash_format true
  logstash_prefix logs-test

  include_tag_key true
  reconnect_on_error true
  request_timeout 10s
  suppress_type_name true
</match>

If I test with the following command, this is not rejected and index correctly:

echo '{"time":"2025-10-16T10:12:30Z","log":"hello test","stream":"stdout","kubernetes":{"namespace_labels":{"app":"learch-api","app.kubernetes.io/managed-by":"Helm","field.cattle.io/projectId":"consumer-search","objectset.rio.cattle.io/hash":"333333ae6137..."}}}' >> logs/app.log

See basic dc.yml

version: "3.8"

services:
  opensearch:
    image: opensearchproject/opensearch:2.9.0
    container_name: os-29
    environment:
      - discovery.type=single-node
      - plugins.security.disabled=true
      - OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g
    ulimits:
      memlock: { soft: -1, hard: -1 }
    ports:
      - "9200:9200"
      - "9600:9600"

  dashboards:
    image: opensearchproject/opensearch-dashboards:2.9.0
    container_name: osd-29
    depends_on: [opensearch]
    environment:
      - OPENSEARCH_HOSTS=["http://opensearch:9200"]
    ports:
      - "5601:5601"

  fluentd:
    build: ./fluentd
    container_name: fluentd
    depends_on: [opensearch]
    volumes:
      - ./logs:/var/log/app:ro
      - ./fluentd/fluent.conf:/fluentd/etc/fluent.conf:ro
    environment:
      - FLUENT_UID=0

@Anthony thank you for your reply. I actually ended up removing these fileds in fluentd to make the data little smaller, and not to deal with . and \ conversions. In reality I don’t need them in my records.

“app.kubernetes.io/managed-by"=>"Helm”, “field.cattle.io/projectId"=>"consumer-search” and

“objectset.rio.cattle.io/hash"=>"12dc83ae61371d4aa04ae98ecb1fa7c8c09a7440”

But I am still getting 400 rejections from OpenSearch.

2025-10-16 18:44:31 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error=“400 - Rejected by OpenSearch” location=nil tag=“var.log.containers.learch-api-798dfb9d78-xv9k6_learch-api-prod_learch-api-56139593bb43178192b9f8ac3f9a7f42ab9a3e339128504fec2b7a86278082fd.log” time=2025-10-16 18:44:27.296827316 +0000 record={“stream”=>“stdout”, “log”=>“127.0.0.6 - - [16/Oct/2025:18:44:25 +0000] “GET /learch?q=pizza&g=glendale,ca&debugQuery.performCategorization=true&debugQuery.performGeo=true&debugQuery.performSpellcheck=true&app_id=web&wt=json&rows=1&fl=id HTTP/1.1” 200 8369 “-” “kube-probe/1.31” 84”, “kubernetes”=>{“container_name”=>“learch-api”, “namespace_name”=>“learch-api-prod”, “pod_name”=>“learch-api-798dfb9d78-xv9k6”, “pod_ip”=>“172.19.133.56”, “host”=>“phx1-q17-host18”, “namespace_labels”=>{“app”=>“learch-api”, “istio-injection”=>“enabled”}}}

Is it maybe due to the log field? Should I try to convert it into string?

This is what I implemented in my fluentd config to remove the fields in addition to what I have

  <filter kubernetes.**>
    @type record_transformer
    remove_keys  $['kubernetes']['namespace_labels']['kubernetes.io/metadata.name'], $['kubernetes']['namespace_labels']['app.kubernetes.io/managed-by'], $['kubernetes']['namespace_labels']['field.cattle.io/projectId'], $['kubernetes']['namespace_labels']['objectset.rio.cattle.io/hash'], $.kubernetes.namespace_labels.namespace-type
  </filter>
1 Like

I was able to identify the problem (highlighted)

POST /kubernetes-na1-2025.10.16/_doc
{
“stream”: “stdout”,
“log”: “127.0.0.6 - - [16/Oct/2025:18:44:25 +0000] "GET /learch?q=pizza&g=glendale,ca&debugQuery.performCategorization=true&debugQuery.performGeo=true&debugQuery.performSpellcheck=true&app_id=web&wt=json&rows=1&fl=id HTTP/1.1" 200 8369 "-" "kube-probe/1.31" 84”,
“kubernetes”: {
“container_name”: “learch-api”,
“namespace_name”: “learch-api-prod”,
“pod_name”: “learch-api-798dfb9d78-xv9k6”,
“pod_ip”: “172.19.133.56”,
“host”: “phx1-q17-host18”,
“namespace_labels”: {
“app”: “learch-api”,
“istio-injection”: “enabled”
}

}
}

it was generating this error

“error”: {
“root_cause”: [
{
“type”: “mapper_parsing_exception”,
“reason”: “object mapping for [kubernetes.namespace_labels.app] tried to parse field [app] as object, but found a concrete value”
}
],
“type”: “mapper_parsing_exception”,
“reason”: “object mapping for [kubernetes.namespace_labels.app] tried to parse field [app] as object, but found a concrete value”
},
“status”: 400
}

By removing namespace_labels logs are going through now. Is there a way to fix this without a removal?

@stecino Are you able to provide the exact line being parsed by fluentd, as I have tried to test with the following:

echo '{"time":"2025-10-16T12:00:00Z","log":"flat dotted only","stream":"stdout","kubernetes.namespace_labels.app.kubernetes.io/managed-by":"Helm"}' >> logs/app.log

Using the following fluent.conf file:

<source>
  @type tail
  @id in_tail_app
  path /var/log/app/app.log
  pos_file /fluentd/log/app.pos
  tag app.raw
  read_from_head true
  <parse>
    @type json
    time_key time
    time_type string
    time_format %iso8601
  </parse>
</source>

# ========= NORMALIZE PROBLEM KEYS =========
<filter app.raw>
  @type rename_key
  @id fix_field_names
  deep_rename true

  # rule order matters; run dot replacement first, then slashes
  replace_rule1 \. _
  replace_rule2 / __
</filter>

# ========= OUTPUT TO OPENSEARCH =========
<match app.**>
  @type opensearch
  host opensearch
  port 9200
  scheme http
  logstash_format true
  logstash_prefix repro-flat-bad
  include_tag_key true
  reconnect_on_error true
  request_timeout 10s
  suppress_type_name true
  log_es_400_reason true
</match>

And the error is not triggered. The resulting doc in OS looks like this:

"_index": "repro-flat-bad-2025.10.16",
        "_id": "AGGMAZoBKcKMD74Uu2Gu",
        "_score": 1,
        "_source": {
          "log": "flat dotted only",
          "stream": "stdout",
          "kubernetes_namespace_labels_app_kubernetes_io/managed-by": "Helm",
          "@timestamp": "2025-10-16T12:00:00.000000000+00:00",
          "tag": "app.raw"
        }

@Anthony yes this is the log entry in the fluentd, before removing kubernetes.namespace_labels

2025-10-16 18:44:31 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error=“400 - Rejected by OpenSearch” location=nil tag=“var.log.containers.learch-api-798dfb9d78-xv9k6_learch-api-prod_learch-api-56139593bb43178192b9f8ac3f9a7f42ab9a3e339128504fec2b7a86278082fd.log” time=2025-10-16 18:44:27.296827316 +0000 record={“stream”=>“stdout”, “log”=>“127.0.0.6 - - [16/Oct/2025:18:44:25 +0000] “GET /learch?q=pizza&g=glendale,ca&debugQuery.performCategorization=true&debugQuery.performGeo=true&debugQuery.performSpellcheck=true&app_id=web&wt=json&rows=1&fl=id HTTP/1.1” 200 8369 “-” “kube-probe/1.31” 84”, “kubernetes”=>{“container_name”=>“learch-api”, “namespace_name”=>“learch-api-prod”, “pod_name”=>“learch-api-798dfb9d78-xv9k6”, “pod_ip”=>“172.19.133.56”, “host”=>“phx1-q17-host18”, “namespace_labels”=>{“app”=>“learch-api”, “istio-injection”=>“enabled”}}}

This is the Ruby hash conversion into JSON, that’s what’s being ingested into OpenSearch

{
“stream”: “stdout”,
“log”: “127.0.0.6 - - [16/Oct/2025:18:44:25 +0000] \“GET /learch?q=pizza&g=glendale,ca&debugQuery.performCategorization=true&debugQuery.performGeo=true&debugQuery.performSpellcheck=true&app_id=web&wt=json&rows=1&fl=id HTTP/1.1\” 200 8369 \”-\" \“kube-probe/1.31\” 84",
“kubernetes”: {
“container_name”: “learch-api”,
“namespace_name”: “learch-api-prod”,
“pod_name”: “learch-api-798dfb9d78-xv9k6”,
“pod_ip”: “172.19.133.56”,
“host”: “phx1-q17-host18”,
“namespace_labels”: {
“app”: “learch-api”,
“istio-injection”: “enabled”
}

}
}

@stecino can you provide the mappings used for this index please, its possible previous docs already created a mapping as an object, therefore this fails.

This is dynamically generated, I don’t have any strict mapping in place

{
“kubernetes-na1-2025.10.21” : {
“mappings” : {
“properties” : {
“@timestamp” : {
“type” : “date”
},
“kubernetes” : {
“properties” : {
“container_name” : {
“type” : “text”,
“fields” : {
“keyword” : {
“type” : “keyword”,
“ignore_above” : 256
}
}
},
“host” : {
“type” : “text”,
“fields” : {
“keyword” : {
“type” : “keyword”,
“ignore_above” : 256
}
}
},
“namespace_labels” : {
“properties” : {
“app” : {
“type” : “text”,
“fields” : {
“keyword” : {
“type” : “keyword”,
“ignore_above” : 256
}
}
},
“istio-injection” : {
“type” : “text”,
“fields” : {
“keyword” : {
“type” : “keyword”,
“ignore_above” : 256
}
}
},
“managed-by” : {
“type” : “text”,
“fields” : {
“keyword” : {
“type” : “keyword”,
“ignore_above” : 256
}
}
},
“name” : {
“type” : “text”,
“fields” : {
“keyword” : {
“type” : “keyword”,
“ignore_above” : 256
}
}
}
}
},
“namespace_name” : {
“type” : “text”,
“fields” : {
“keyword” : {
“type” : “keyword”,
“ignore_above” : 256
}
}
},
“pod_ip” : {
“type” : “text”,
“fields” : {
“keyword” : {
“type” : “keyword”,
“ignore_above” : 256
}
}
},
“pod_name” : {
“type” : “text”,
“fields” : {
“keyword” : {
“type” : “keyword”,
“ignore_above” : 256
}
}
}
}
},
“log” : {
“type” : “text”,
“fields” : {
“keyword” : {
“type” : “keyword”,
“ignore_above” : 256
}
}
},
“stream” : {
“type” : “text”,
“fields” : {
“keyword” : {
“type” : “keyword”,
“ignore_above” : 256
}
}
}
}
}
}
}

I can see it here

    "namespace_labels" : {
      "properties" : {
        "app" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },

@stecino do you need to search/aggregate on those label keys (kubernetes.namespace_labels.app)?

The issue seems to be coming from OpenSearch trying to interpret the lines as nested object, but the mapping doesn’t match, see following example.

“kubernetes.namespace_labels.app.kubernetes.io/managed-by”:“Helm”

“kubernetes.namespace_labels.app”:“learch-api”.

The simple way to work around this is to create an index template with enabled: false, OpenSearch will store the whole sub-object in _source but will not parse or index any of its inner keys.

You can use a template similar to the following:

PUT _index_template/kube-logs
{
  "index_patterns": ["logs-test-disabled-*"],
  "template": {
    "mappings": {
      "properties": {
        "kubernetes": {
          "properties": {
            "namespace_labels": {
              "type": "object",
              "enabled": false
            }
          }
        }
      }
    }
  }
}

OpenSearch will then index the 2 lines listed above as follows:

"hits" : [
      {
        "_index" : "logs-test-disabled-2025.10.16",
        "_id" : "uBVhC5oBDa2gRCgZfrzG",
        "_score" : 1.0,
        "_source" : {
          "kubernetes.namespace_labels.app.kubernetes.io/managed-by" : "Helm",
          "@timestamp" : "2025-10-16T13:00:00.000000000+00:00",
          "log" : "flat dotted only",
          "tag" : "app.raw"
        }
      },
      {
        "_index" : "logs-test-disabled-2025.10.16",
        "_id" : "uRVjC5oBDa2gRCgZlLx5",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2025-10-16T13:01:00.000000000+00:00",
          "log" : "flat scalar app",
          "kubernetes.namespace_labels.app" : "learch-api",
          "tag" : "app.raw"
        }
      }