Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Docker based v2.8
Describe the issue:
I use OpenSearch along with Logstash and Filebeat to ingest and analyze logs and extract metrics.
The index names are created in the Logstash, here some examples:
- component1_r_m-2023.39, where r_m states for monthly retention and 39 is the week number which means that the index is created weekly; or
- component2_r_w-2023.10.04, where r_w states for weekly retention and the index is created daily
I have three ISM policies which delete the indices:
- monthly for the index pattern r_m
- weekly for the index pattern r_w
- yearly for the index pattern r_y
What I want to achieve is to add via API a rollup job which would take the weekly index, aggregate data into a new index which would be deleted after a year. Ideally I would like to use the current retention policies.
For instance all indices component1_r_m-2023.30 … - … component1_r_m-2023.30 would be aggregated to rollup-component1_r_y-2023.3 and component1_r_m-2023.40 … - … component1_r_m-2023.49 would be aggregated to rollup-component1_r_y-2023.4, and so on.
In a nutshell, I want to achieve what index rollup is intended to do, aggregated data and keep if for longer in less number of indices or shards.
Configuration:
First I tried the Rollup Jobs. Here though there are 2 issues.
- It’s not possible to create the dynamic target_index from an index pattern
- But even though I prepared several rollup jobs which should be installed now the ones which are intended to be used in the future are failing and not getting re-eanbled when the index matching the source_index pattern has been created.
For the reference there were 2 rollup jobs created:
Job1
“source_index”: “component1_r_m-2023.3*”,
“target_index”: “rollup-component1_r_y-2023.3”
Job2
“source_index”: “component1_r_m-2023.4*”,
“target_index”: “rollup-component1_r_y-2023.4”
The Job1 was enabled but Job 2 failed due to unavailability of the source_index and disabled.
So, this method will not work.
So I’m trying now with the ISM rollover with rollup action.
I’ve added the alias and ism rollover alias to the index template.
PUT _index_template/component1
{
“index_patterns”: [
“component1*”
],
“priority”: 5,
“template”: {
“aliases”: {
“xxyyzz”: {
“is_write_index”: true
}
},
“settings”: {
“number_of_shards”: “1”,
“refresh_interval”: “30s”,
“plugins.index_state_management.rollover_alias”: “xxyyzz”
},
“mappings”: {
“dynamic”: false,
When checking the details of the index I see that the alias, write index and rollover_alias are there
{
“component1_r_w-2023.39-0001”: {
“aliases”: {
“xxyyzz”: {
“is_write_index”: true
}
},
…
“settings”: {
“index”: {
“refresh_interval”: “30s”,
“number_of_shards”: “1”,
“plugins”: {
“index_state_management”: {
“rollover_alias”: “xxyyzz”
}
},
The ISM policy is a per the prescription in the OpenSearch docu:
PUT _plugins/_ism/policies/rollover_policy_component1
{
“policy”: {
“description”: “Example rollover policy component1.”,
“default_state”: “rollover”,
“states”: [
{
“name”: “rollover”,
“actions”: [
{
“rollover”: {
“min_doc_count”: 1
}
}
],
“transitions”: [
{
“state_name”: “rp”
}
]
},
{
“name”: “rp”,
“actions”: [
{
“rollup”: {
“ism_rollup”: {
“target_index”: “rollup_ndx-{{ctx.source_index}}”,
“description”: “Example rollup job”,
“page_size”: 5,
“dimensions”: [
{
“date_histogram”: {
“source_field”: “ts”,
“fixed_interval”: “5m”,
“timezone”: “UTC”
…
],
“metrics”:
}
}
}
],
“transitions”:
}
],
“ism_template”: {
“index_patterns”: [“component1*”],
“priority”: 80
}
}
}
This is the latest error I’m facing. The rollover alias should be pointing to multiple indices therefore I’m puzzled how to move on and if I will be able to fulfill my requirements.
{
“cause”: “Rollover alias [xxyyzz] can point to multiple indices, found duplicated alias [[xxyyzz]] in index template [component1]”,
“message”: “Failed to rollover index [index=component1_r_w-2023.39-0001]”
}
It would be much easier and intuitive if the target_index could be dynamically created but without dependency to the source_index. This could have been done using solution similar to the one in Logstash.
If I could create a rollup job
“source_index”: “component1_r_m-*”,
“target_index”: “rollup-component1_r_y-%{+YYYY.MM}”
then data in source indices would be aggregated to the monthly indices which would be dropped after a year.
Relevant Logs or Screenshots: