How to use pipeline field vars to construct opensearch sink index param

With logstash i would use some filters to construct the index name based on certain other params. Then I would use that newly constructed field in the output config for the index param.

For example, in Logstash:

filter {
    mutate {
        add_field => { "index_lowercase" => "index-%{team}-%{role}" }


    mutate {
        lowercase => [ "index_lowercase" ]

output {
    opensearch {
            index => "%{index_lowercase}"

When I try to do the same in Dataprepper I get:

Caused by: java.lang.IllegalArgumentException: Index time pattern contains time patterns that are less than one hour: [m, s, S, A, n, N]

The config I’m trying in Dataprepper is:

    - add_entries:
        - key: "index_lowercase"
          value: "index-%{team}-%{role}"

    - lowercase_string:
          - "index_lowercase"
      - opensearch:
          hosts: [https://my_opensource_host:9200]
          username: "someusername"
          password: "some_password_here"
          insecure: true
          index_type: management_disabled
          index: "%{index_lowercase}"
          dlq_file: "/usr/share/data-prepper/index_failures.dlq"

I see in the documentation for this sink type that it can support date/time macro formatting but I would like to take it a step further and use field values to make the index name dynamic. Is this supported?

Hi @searchspark2310,

Currently, we do not have a feature to customize index names by key-value. Is that what you are asking?


@searchspark2310 , There is an open feature request to add this type of support. At the moment, we don’t have it scheduled for any given release.