[Google Translate]
Hello. Can you help me explain how rollover works? I have the index received by the logstash. I expect to get a new index every 40GB.
OK. I pointed to the police index_patterns logstash- * after which the policy began to be applied to logstash-0000002. The data is still sent to logstash-000001. Most likely the matter is in logstash and in the output you need to specify not the index but the alias of the index
I’ll let here my experience with fluentd configuration and ISM
fluentd.conf
<store>
@type elasticsearch
hosts https://opensearch-node1:9200,https://opensearch-node2:9200,https://opensearch-node3:9200
ssl_verify false
user "admin"
password "passpasspass"
###
### Needed options to configure ISM
include_timestamp true
index_name fluentd-logs-alias # In this case (ISM), this should be the ALIAS name
### Rollover index configuration
rollover_index true # Specify this as true when an index with rollover capability needs to be created.
application_name fluentd
index_date_pattern "" # Specify this to override the index date pattern for creating a rollover index.
deflector_alias fluentd-logs-alias # Specify the deflector alias which would be assigned to the rollover index created.
# This is useful in case of using the Elasticsearch rollover API
# If rollover_index is set, then this parameter will be in effect otherwise ignored.
template_name fluentd-template # The name of the template to define.
Run docker-compose to an opensearch cluster (in my case) and opendashboards
version: '3'
services:
opensearch-node1:
image: opensearchproject/opensearch:${OSVERSION}
container_name: opensearch-node1
environment:
- cluster.name=opensearch-cluster
- node.name=opensearch-node1
- discovery.seed_hosts=opensearch-node2,opensearch-node3
- cluster.initial_master_nodes=opensearch-node1,opensearch-node2,opensearch-node3
- bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
hard: 65536
volumes:
- opensearch-data1:/usr/share/opensearch/data
ports:
- 9200:9200
- 9600:9600 # required for Performance Analyzer
networks:
- opensearch-net
# healthcheck:
#test: curl -k https://admin:admin@localhost:9200/_cluster/health | grep -vq '"status":"red"'
#interval: 30s
#timeout: 10s
#retries: 5
opensearch-node2:
image: opensearchproject/opensearch:${OSVERSION}
container_name: opensearch-node2
environment:
- cluster.name=opensearch-cluster
- node.name=opensearch-node2
- discovery.seed_hosts=opensearch-node1,opensearch-node3
- cluster.initial_master_nodes=opensearch-node1,opensearch-node2,opensearch-node3
- bootstrap.memory_lock=true
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
volumes:
- opensearch-data2:/usr/share/opensearch/data
networks:
- opensearch-net
opensearch-node3:
image: opensearchproject/opensearch:${OSVERSION}
container_name: opensearch-node3
environment:
- cluster.name=opensearch-cluster
- node.name=opensearch-node3
- discovery.seed_hosts=opensearch-node1,opensearch-node2
- cluster.initial_master_nodes=opensearch-node1,opensearch-node2,opensearch-node3
- bootstrap.memory_lock=true
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
volumes:
- opensearch-data3:/usr/share/opensearch/data
networks:
- opensearch-net
opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:${ODVERSION}
container_name: opensearch-dashboards
#depends_on: {"opensearch-node1": {"condition": "service_healthy"}}
ports:
- 5601:5601
expose:
- "5601"
environment:
OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200","https://opensearch-node3:9200"]'
networks:
- opensearch-net
volumes:
opensearch-data1:
opensearch-data2:
opensearch-data3:
networks:
opensearch-net:
Run Dev-tools to create policy, template and start a new index writable
Hi @aamarques ,
Is there any difference between elasticsearch and opensearch plugin for Fluentd?
However, I’ve done some researches and didn’t find the rollover_index configuration on opensearch plugin docs.
Hi @saeed.kazemi
I think they differ after e-search 7.10 version, but ISM depends both opensearch and fluend config.
“OpenSearch is a community-driven, open-source search and analytics suite derived from Apache 2.0 licensed Elasticsearch 7.10.2 & Kibana 7.10.2." https://opensearch.org/
Rollover index configuration
rollover_index true # Specify this as true when an index with rollover capability needs to be created.
application_name fluentd
index_date_pattern "" # Specify this to override the index date pattern for creating a rollover index.
deflector_alias fluentd-logs-alias # Specify the deflector alias which would be assigned to the rollover index created.
# This is useful in case of using the Elasticsearch rollover API
# If rollover_index is set, then this parameter will be in effect otherwise ignored.
rollover_index is configured in fluentd.conf as rollover_alias belongs to filebeat.yml and etc. deflector_alias is deprecated in ES but was necessary in this config.
Try to follow this setup above.
I want to remove index alias after the rollover occur. Is there any way to setup policy or template that do this magic?
Tank you for you time and consideration though.
Hi @saeed.kazemi
I can’t imagine any way to do it “auto-magically” as this feature was concept to create rollover indexes that can be searched buy alias
Thank you for your reply.
Is there any problem with this approach? (have alias on rollover indexes or closed indexes.)
Cause we search on alias name as well. @aamarques