# Data Prepper Dynamic Index using keys from logs

**URL:** <https://forum.opensearch.org/t/data-prepper-dynamic-index-using-keys-from-logs/12868>\
**Category:** Data Prepper\
**Tags:** configure\
**Created:** [March 2, 2023, 10:34am UTC](https://forum.opensearch.org/t/data-prepper-dynamic-index-using-keys-from-logs/12868 "2023-03-02T10:34:55Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![MatanBaruch](https://avatars.discourse-cdn.com/v4/letter/m/5f8ce5/32.png) [@MatanBaruch](https://forum.opensearch.org/u/MatanBaruch)\
**Post date:** [March 2, 2023, 10:34am UTC](https://forum.opensearch.org/t/data-prepper-dynamic-index-using-keys-from-logs/12868/1 "2023-03-02T10:34:55Z")

</div>

**Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser):

> Opensearch - 2.3 (AWS)  
> Data Prepper 2.0.1 (opensearchproject/data-prepper:2.0.1)

**Describe the issue** :  
Unable to use key from the log as Index name suffix

**Configuration** :

```auto
grok-pipeline:
  source:
    http:
  processor:
  - grok:
      match:
        log: ["%{COMMONAPACHELOG}"]
  - date:
      from_time_received: true
      destination: "@timestamp"

  sink:
  - stdout:
  - opensearch:
      hosts: ["https://logs.examples.com:443"]
      username: "data-prepper"
      password: "XXXX" 
      index: "data-prepper_${Cluster}"

```

**Relevant Logs or Screenshots** :

_ **Log Line** _

`{"date":1.677753494596472E9,"log":"2023-03-02T10:38:14.59644441Z stdout F response_data = self.request(","kubernetes":{"pod_name":"nginx-698bd9dc77-87j24","namespace_name":"default","pod_id":"153c9fb1-06b4-4bb9-81f1-078ca95b6dc6","labels":{"app":"nginx","pod-template-hash":"698bd9dc77"},"annotations":{"kubernetes.io/psp":"eks.privileged"},"host":"ip-10-0-0-10.ec2.internal","container_name":"nginx","docker_id":"67d10791bb12481f4af2afbab82e0026e1632137cc39bf40bcc43ba4045f80ca","container_hash":"docker.io/nginx/nginx@sha256:2b59a3735cc6eec3548584eed63f415e7d824f9941590354c836fe56c7d0f352","container_image":"docker.io/nginx/nginx:latest"},"Cluster":"infra","Region":"us-east-1","@timestamp":"2023-03-02T10:38:35.432Z"}`

_ **Error from Data Prepper** _

`Caused by: java.io.IOException: [data-prepper_${Cluster}] OpenSearchStatusException[OpenSearch exception [type=invalid_index_name_exception, reason=Invalid index name [data-prepper_${Cluster}], must be lowercase]]`

---

<div class="post-metadata">

**Author:** ![oeyh](https://avatars.discourse-cdn.com/v4/letter/o/8edcca/32.png) [@oeyh](https://forum.opensearch.org/u/oeyh)\
**Post date:** [March 12, 2023, 12:34am UTC](https://forum.opensearch.org/t/data-prepper-dynamic-index-using-keys-from-logs/12868/2 "2023-03-12T00:34:32Z")

</div>

Hi @MatanBaruch , thank you for posting!

I reproduced the error by using Data Prepper 2.0.1 and was able to resolve it with v2.1.0, the latest version. Data Prepper 2.1.0 should support Dynamic Index name. Please try it out and see if that helps.

---

<div class="post-metadata">

**Author:** ![MatanBaruch](https://avatars.discourse-cdn.com/v4/letter/m/5f8ce5/32.png) [@MatanBaruch](https://forum.opensearch.org/u/MatanBaruch)\
**Post date:** [March 12, 2023, 8:42am UTC](https://forum.opensearch.org/t/data-prepper-dynamic-index-using-keys-from-logs/12868/3 "2023-03-12T08:42:04Z")

</div>

Thanks I will check.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/33547ea01a5b12dcca2958411d3edd97ae2ea8c1.png) [@system](https://forum.opensearch.org/u/system)\
**Post date:** [May 11, 2023, 8:42am UTC](https://forum.opensearch.org/t/data-prepper-dynamic-index-using-keys-from-logs/12868/4 "2023-05-11T08:42:30Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.