Flatten nested maps

Data Prepper
1

Hi @all,

I was trying to process events coming from DynamoDB using an ingestion pipeline. The main operation that I’m trying to achieve is to flatten a map and simply getting rid of the outer level (level1)

For example -

{
   "level1": {
         "level2": {
                    "key": "value"
               
           }
    }
}

I want to parse this object, in such a way that level1.level2.key should rename to level2.key.

{
         "level2": {
                    "key": "value"
           }
    }
}

Thank you for your help

2

You mean the ingest pipeline in OpenSearch? You can use both rename processor and remove processor to achieve that:

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "rename": {
          "field": "level1.level2.key",
          "target_field": "level2.key"
        }
      },
      {
        "remove": {
          "field": "level1"
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "level1": {
          "level2": {
            "key": "value"
          }
        }
      }
    }
  ]
}
1 Like
3

I’m definitively gonna give that a try :), and come with feedback. My question was refering to the ingestion pipelines from Data prepper rather than “classical ones”.

Can the same logic be applied there?

Thank for the help :slight_smile:

4

I think they are similar, Data prepper has rename_keys and delete_entries processor: Mutate event - OpenSearch documentation, you can have a try.

1 Like
5

Hello there,

Found a neat way to do it by using a rename_keys processor

processor:
- rename_keys:
entries:
- from_key: “level1/level2”
to_key: “level2”

is transforming
{
“level1”: {
“level2”: “value”
}
}
to
{
“level2”: “value”

}

the way to access nested object is by using ‘/’ delimiter rather than ‘.’. pretty neat and simple, but sadly not documented :frowning:

6

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.