Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
I used opensearch 2.11.1 version
Describe the issue:
I used an OpenSearch cluster to store my data and implemented an ILM (Index Lifecycle Management) policy to manage it. For visualization, I utilized the Wazuh Dashboard, version 4.7.3.
The OpenSearch cluster setup includes one master node, one hot data node, and one cold data node.
To store snapshots, I connected an S3 bucket, while for other data, I attached a block volume to a dedicated node.
Configuration:
ILM Policy:
{
"id": "INDEX-MANAGEMENT-POLICY",
"seqNo": 6416514,
"primaryTerm": 87,
"policy": {
"policy_id": "INDEX-MANAGEMENT-POLICY",
"description": "hot -> cold ->snapshot->delete",
"last_updated_time": 1735018704417,
"schema_version": 19,
"error_notification": null,
"default_state": "hot",
"states": [
{
"name": "hot",
"actions": [
{
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"allocation": {
"require": {
"temp": "hot"
},
"include": {},
"exclude": {},
"wait_for": false
}
},
{
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"replica_count": {
"number_of_replicas": 1
}
}
],
"transitions": [
{
"state_name": "cold",
"conditions": {
"min_index_age": "2d"
}
}
]
},
{
"name": "cold",
"actions": [
{
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"allocation": {
"require": {
"temp": "cold"
},
"include": {},
"exclude": {},
"wait_for": false
}
},
{
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"replica_count": {
"number_of_replicas": 1
}
}
],
"transitions": [
{
"state_name": "snapshot",
"conditions": {
"min_index_age": "3d"
}
}
]
},
{
"name": "snapshot",
"actions": [
{
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"snapshot": {
"repository": "new_snapshots",
"snapshot": "test-snapshot"
}
}
],
"transitions": [
{
"state_name": "delete",
"conditions": {
"min_index_age": "6d"
}
}
]
},
{
"name": "delete",
"actions": [
{
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"delete": {}
}
],
"transitions": []
}
],
"ism_template": [
{
"index_patterns": [
"wazuh-alerts-*",
"wazuh-archives-*"
],
"priority": 1,
"last_updated_time": 1712053708756
}
]
}
}
Opensearch.yml:
# ======================== OpenSearch Configuration =========================
#
# NOTE: OpenSearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.opensearch.org
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: esdl-cluster
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: ${HOSTNAME}
node.roles: [ data, ingest ]
node.attr.zone: zoneA
node.attr.temp: hot
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/opensearch
#
# Path to log files:
#
path.logs: /var/log/opensearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# OpenSearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts:
- "master-ip"
- "hot-ip"
- "cold-ip"
#
# Bootstrap the cluster using an initial set of cluster-manager-eligible nodes:
#
cluster.initial_cluster_manager_nodes:
- "master"
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#
# ---------------------------------- Remote Store -----------------------------------
# Controls whether cluster imposes index creation only with remote store enabled
# cluster.remote_store.enabled: true
#
# Repository to use for segment upload while enforcing remote store for an index
# node.attr.remote_store.segment.repository: my-repo-1
#
# Repository to use for translog upload while enforcing remote store for an index
# node.attr.remote_store.translog.repository: my-repo-1
#
# ---------------------------------- Experimental Features -----------------------------------
# Gates the visibility of the experimental segment replication features until they are production ready.
#
#opensearch.experimental.feature.segment_replication_experimental.enabled: false
#
# Gates the functionality of a new parameter to the snapshot restore API
# that allows for creation of a new index type that searches a snapshot
# directly in a remote repository without restoring all index data to disk
# ahead of time.
#
#opensearch.experimental.feature.searchable_snapshot.enabled: false
#
#
# Gates the functionality of enabling extensions to work with OpenSearch.
# This feature enables applications to extend features of OpenSearch outside of
# the core.
#
#opensearch.experimental.feature.extensions.enabled: false
#
#
# Gates the concurrent segment search feature. This feature enables concurrent segment search in a separate
# index searcher threadpool.
#
#opensearch.experimental.feature.concurrent_segment_search.enabled: false
######## Start OpenSearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production
plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/hot-02.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/hot-02-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: /etc/opensearch/hot-02.pem
plugins.security.ssl.http.pemkey_filepath: /etc/opensearch/hot-02-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
- "CN=ADMIN,OU=EVENTUS,O=EVENTUS,L=MUMBAI,ST=MAHARASHTRA,C=IN"
plugins.security.nodes_dn:
- "CN=uat-elmaster-01,OU=EVENTUS,O=EVENTUS,L=MUMBAI,ST=MAHARASHTRA,C=IN"
- "CN=uat-eldatanode-01,OU=EVENTUS,O=EVENTUS,L=MUMBAI,ST=MAHARASHTRA,C=IN"
- "CN=uat-elhot-02,OU=EVENTUS,O=EVENTUS,L=MUMBAI,ST=MAHARASHTRA,C=IN"
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-config", ".plugins-ml-connector", ".plugins-ml-model-group", ".plugins-ml-model", ".plugins-ml-task", ".plugins-ml-conversation-meta", ".plugins-ml-conversation-interactions", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".ql-datasources", ".opendistro-asynchronous-search-response*", ".replication-metadata-store", ".opensearch-knn-models", ".geospatial-ip2geo-data*"]
node.max_local_storage_nodes: 3
######## End OpenSearch Security Demo Configuration ########
compatibility.override_main_response_version: true
plugins.ml_commons.allow_registering_model_via_local_file: true
plugins.ml_commons.allow_registering_model_via_url: true
path.repo: ["/mnt/snapshots/snapshots"]
I attached the snapshot repository connected to an S3 bucket, but I’m facing two issues:
- Snapshots indices are still consuming storage on the local nodes.
- An error occurs when I attempt to restore a snapshot (details below).
Note: Snapshot is searchable without restoring the snapshot data.
Thankyou in advance.
Best Regards,
Nikita S.