How to use pipeline field vars to construct opensearch sink index param

Data Prepper
1

With logstash i would use some filters to construct the index name based on certain other params. Then I would use that newly constructed field in the output config for the index param.

For example, in Logstash:

filter {
    mutate {
        add_field => { "index_lowercase" => "index-%{team}-%{role}" }

    }

    mutate {
        lowercase => [ "index_lowercase" ]
    }
}

output {
    opensearch {
            index => "%{index_lowercase}"
    }
}

When I try to do the same in Dataprepper I get:

Caused by: java.lang.IllegalArgumentException: Index time pattern contains time patterns that are less than one hour: [m, s, S, A, n, N]

The config I’m trying in Dataprepper is:

processor:
    - add_entries:
        entries:
        - key: "index_lowercase"
          value: "index-%{team}-%{role}"

    - lowercase_string:
        with_keys:
          - "index_lowercase"
  
  sink:
      - opensearch:
          hosts: [https://my_opensource_host:9200]
          username: "someusername"
          password: "some_password_here"
          insecure: true
          index_type: management_disabled
          index: "%{index_lowercase}"
          dlq_file: "/usr/share/data-prepper/index_failures.dlq"

I see in the documentation for this sink type that it can support date/time macro formatting but I would like to take it a step further and use field values to make the index name dynamic. Is this supported?

2

Hi @searchspark2310,

Currently, we do not have a feature to customize index names by key-value. Is that what you are asking?

Thanks,
–David

3

@searchspark2310 , There is an open feature request to add this type of support. At the moment, we don’t have it scheduled for any given release.

4

Thanks for this information, i have tried to use this pattern unsuccessfully.
Im using SQS to get events from S3 and ingest to domain.
I tried to use dynamic index by add suffix such this in the example - also tried ${data}, ${time}, and more option (some of them from the SQS event JSON) and nothing works.
The Error is - “The key ___ could not be found in the Event when formatting”.
any idea what is the problem and can i get the Event fields in order to use them?

5

I would be interested in the solution as well.