Data Prepper Dynamic Index using keys from logs

[MatanBaruch]

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Opensearch - 2.3 (AWS)
Data Prepper 2.0.1 (opensearchproject/data-prepper:2.0.1)

Describe the issue:
Unable to use key from the log as Index name suffix

Configuration:

grok-pipeline:
  source:
    http:
  processor:
  - grok:
      match:
        log: [ "%{COMMONAPACHELOG}" ]
  - date:
      from_time_received: true
      destination: "@timestamp"

  sink:
  - stdout:
  - opensearch:
      hosts: [ "https://logs.examples.com:443" ]
      username: "data-prepper"
      password: "XXXX" 
      index: "data-prepper_${Cluster}"

Relevant Logs or Screenshots:

Log Line

{"date":1.677753494596472E9,"log":"2023-03-02T10:38:14.59644441Z stdout F response_data = self.request(","kubernetes":{"pod_name":"nginx-698bd9dc77-87j24","namespace_name":"default","pod_id":"153c9fb1-06b4-4bb9-81f1-078ca95b6dc6","labels":{"app":"nginx","pod-template-hash":"698bd9dc77"},"annotations":{"kubernetes.io/psp":"eks.privileged"},"host":"ip-10-0-0-10.ec2.internal","container_name":"nginx","docker_id":"67d10791bb12481f4af2afbab82e0026e1632137cc39bf40bcc43ba4045f80ca","container_hash":"docker.io/nginx/nginx@sha256:2b59a3735cc6eec3548584eed63f415e7d824f9941590354c836fe56c7d0f352","container_image":"docker.io/nginx/nginx:latest"},"Cluster":"infra","Region":"us-east-1","@timestamp":"2023-03-02T10:38:35.432Z"}

Error from Data Prepper

Caused by: java.io.IOException: [data-prepper_${Cluster}] OpenSearchStatusException[OpenSearch exception [type=invalid_index_name_exception, reason=Invalid index name [data-prepper_${Cluster}], must be lowercase]]

[oeyh]

Hi @MatanBaruch , thank you for posting!

I reproduced the error by using Data Prepper 2.0.1 and was able to resolve it with v2.1.0, the latest version. Data Prepper 2.1.0 should support Dynamic Index name. Please try it out and see if that helps.

[MatanBaruch]

Thanks I will check.