Index alias for multiple indices

Plugins Index Management
1

Hi,

I am currently working with Opendistro elasticsearch and we have been using ISM. The process is when a new Logstash configuration is created with a new index, we manually create a template for the index and create a new index with an alias and place the alias in Logstash.

We have a new requirement, where we will need to setup the same way we use currently but for new application. For example the new index will look like logstash-%%{app} where the app will be a tag, or a field from filebeat.

I am stuck now, because in this situation we may not know the application name in advance and hence not sure how to proceed. Can someone help here?

2

Any response for this issue?

3

Same issue here. We are looking for some ISM options in in the Logstash plugin.
Something like this → Elasticsearch output plugin | Logstash Reference [8.4] | Elastic

4

Also same issue here. I have logs being sent via logstash to indices named syslog-YYYY.MM.DD.

logstash.conf output looks like this:

output {
   elasticsearch { 
       hosts => ["https://localhost:9200"]
       index => "%{type}-%{+YYYY.MM.dd}"
       ssl => true

These are associated to an alias named syslog using the index regex syslog-20*

{
    "actions": [
      {
        "add": {
          "index": "syslog-20*",
          "alias": "syslog"
        }
      }
    ]
  }

Since the indices are named with a date suffix, and not the -00001 format that the rollover function looks for, I need to create a separate index named syslog-00001 and set is_write_index: true for the syslog alias. Then my logs get rolled, but I’m constantly incrementing a separate (albeit smaller) index…

It would be nice to be able to configure logstash output to specify alias/pattern as @ larwied mentioned or have the option to rollover indices matching a custom regex and not just “^.*-\d$”

5

@dbbaughe Don’t know if you saw this since it wasn’t under the Index Management category…can you provide any guidance here?

6

I did not see this @GSmith, thanks :slight_smile:
Moving to ISM section and will look into this

7

Bumping for any updates?
We have similar indices named with variables such as beat version and date, that we don’t know in advance. Our setup is FB>LS>ES and we want to roll based on size.
filebeat_testing-7.2.4
filebeat_testing-7.5.0
…
filebeat_testing-7.5.0-2020-07-29

Any help is appreciated,
Currently we are using the following template with no luck. Bootstrapping works only if LS outputs to a constant “filebeat_testing”.

PUT _template/ism-test-template
{
“index_patterns”: [
“filebeat_testing*”
],
“order”: 10,
“settings”: {
“opendistro.index_state_management.rollover_alias”: “filebeat_testing”,
“opendistro.index_state_management.policy_id”: “simple_rotate_100GB”,
“number_of_shards”: 4,
“number_of_replicas”: 1,
“refresh_interval”: “30s”
}
}