Opensearch Rejecting Logs With App Label

OpenDistro
1

Hi,
I am currently facing an issue with OpenSearch as a log storage. I am using fluentbit as a DaemonSet, in combination with Fluentd and OpenSearch on Kubernetes. Fluentbit collects the logs with the help of the Kubernetes plugin and forwards them to fluentd which then attempts stores them in OpenSearch.
While it woks fine for most logs, if the logs come from a pod which use the app label (so for example app: example-app), I am getting an error:

[warn]: #0 dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch [error type]: mapper_parsing_exception [reason]: 'object mapping for [kubernetes.labels.app] tried to parse field [app] as object, but found a concrete value'"

I looked into the IndexMapping of OpenSearch and found that labels section is parsed like this :

"labels": {"properties": {
      "app": {"properties": {"kubernetes": {"properties": {
        "io/version": {
          "type": "text",
          "fields": {"keyword": {
            "type": "keyword",
            "ignore_above": 256
          }}
        },
        "io/instance": {
          "type": "text",
          "fields": {"keyword": {
            "type": "keyword",
            "ignore_above": 256
          }}
        },
        "io/component": {
          "type": "text",
          "fields": {"keyword": {
            "type": "keyword",
            "ignore_above": 256
          }}
        },
        "io/name": {
          "type": "text",
          "fields": {"keyword": {
            "type": "keyword",
            "ignore_above": 256
          }}
        },
        "io/part-of": {
          "type": "text",
          "fields": {"keyword": {
            "type": "keyword",
            "ignore_above": 256
          }}
        }
      }}}},

Which seems to be derived from the new recommended labels for Kubernetes, which are also used in my cluster. I am unsure how to configure OpenSearch in a way that it does not reject these logs, maybe someone can help.

Thanks
Philipp

1 Like