Dynamic Index Creation based on document field

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Describe the issue:

Hi,

I have below document:
{“transactionId”:“1234”,“transactionType”:“card”, “timestamp”: “2023-11-30T14:41:12”}

I want to insert this document in the index based on transactionType, for example above document will goto index payment_{transactionType}_{timestamp in date format} i.e., payment_card_2024-11-30

Please help here.

Configuration:

Relevant Logs or Screenshots:

2

You can use ingest pipeline to achieve that:

  1. Create a ingest pipeline named date-index-name
PUT _ingest/pipeline/date_index_name
{
  "processors": [
      {
        "date_index_name": {
          "field": "timestamp",
          "index_name_prefix":"payment_{{transactionType}}_",
          "date_rounding" : "d",
          "date_formats": ["yyyy-MM-dd'T'HH:mm:ss"]
        }
      }
    ]
}
  1. Index documents with the ingest pipeline
POST test1/_doc?pipeline=date_index_name
{
  "transactionId": "1234",
  "transactionType": "card",
  "timestamp": "2023-11-30T14:41:12"
}
  1. Check the index and document:

GET payment_card_2023-11-30/_search
3

Has anyone confirmed this works? I can’t get it to work in my open search cluster… here’s my pipeline configuration:

“processors”: [
{
“script”: {
“lang”: “painless”,
“source”: “if (ctx.retention == null) { ctx.retention = ‘R2Y’; }”
}
},
{
“date_index_name”: {
“field”: “date”,
“date_formats”: [
“ISO8601”
],
“index_name_prefix”: “logs_{{retention}}_”,
“date_rounding”: “M”,
“index_name_format”: “yyyy-MM”,
“ignore_failure”: false
}
}
]

I tested successfully that the “retention” field is added the the document before i added the template snippet to the index_name_prefix… after adding the template snippet, no new indexes are created and also I don’t see any failures in the processor stats:

{
“date_index_name” : {
“type” : “date_index_name”,
“stats” : {
“count” : 245,
“time_in_millis” : 0,
“current” : 0,
“failed” : 0
}
}

changing index_name_prefix to a static string again fixes it… seems like the template snippets dont really work there?

4

I’ve got it working! The index wasn’t being created because the value had uppercase letters.