Remote snapshots

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Describe the issue: Currently have Hot Cold Delete. after 7days the indecies goto cold and are flagged Read-only.

Script. detects read only indices and executes Snaphot, Closure of original index and remote restore.

However the filesystem I have allocated to the warm nodes filled up.
currently have 7TB of snapshots and the warm node logical volume was allocated 500GB. The data on the warmnode logical volume appears to be meta data about the snapshot

How much space whould I allocate to the warm nodes when i am expecting 150TB of remote searchable snapshots?

Configuration: Opensearch 3.0. locallay hosted on prem. NFS share with snapshot Warm nodes configure with 500GB allocated to opensearch_warmlv.

Remote Repository 150TB.

Relevant Logs or Screenshots:

/dev/mapper/almalinux-var_log 100G 7.9G 93G 8% /var/log

/dev/mapper/OpenSearch-managerlv 187G 1.4G 185G 1% /opensearch_man

/dev/mapper/OpenSearch-searchlv 931G 6.6G 925G 1% /opensearch_search

/dev/mapper/OpenSearch-warmlv 466G 466G 4.2M 100% /opensearch_warm

2

Hey @GuyS ,

Can you share any config or scripts used?

Leeroy.

3

This is a snippet of my script that creates the snapshot:
snapshot_name = f"{clean_index_name}-{timestamp}"

        try:

snapshot_body = {

‘indices’: index,

‘ignore_unavailable’: True,

‘include_global_state’: False

            }

client.snapshot.create(

repository = repo,

snapshot=snapshot_name,

body=snapshot_body,

wait_for_completion=False

            )

logger.info(f"Snapshot ‘{snapshot_name}’ creation initiated…")

while True:

try:

status_response = client.snapshot.status(repository=repo, snapshot=snapshot_name)

snapshot_state = status_response[‘snapshots’][0][‘state’]

logger.info(f"Snapshot ‘{snapshot_name}’ state: {snapshot_state}")

except:

logger.error(f"Error checking snapshot status: {e}", exc_info=True)

time.sleep(30)

continue

This is the restore snippt:
logger.info(f"Restoring snapshot ‘{snapshot_name}’ as searchable index…")

        try:

restore_body = {

‘indices’: index,

‘storage_type’: ‘remote_snapshot’,

‘rename_pattern’: ‘.ds-(.+)’,

‘rename_replacement’: f’{clean_index_name}',

‘index_settings’:{‘hidden’:‘false’}

            }

client.snapshot.restore(

repository=repo,

snapshot=snapshot_name,

body=restore_body,

wait_for_completion=True

            )

log_snapshot(snapshot_name, clean_index_name, restored=True)

time.sleep(15)

client = opensearch.build_client()

#check health

still_healthy = opensearch.get_health(client)

if not still_healthy:

logger.error(“Cluster health check failed after snapshot restore”)

sys.exit(1)

else:
My repo settings:
{
“search_snap”: {
“type”: “fs”,
“settings”: {
“chunk_size”: null,
“location”: “/mnt/opensearch_snapshots”,
“max_restore_bytes_per_sec”: “200m”,
“readonly”: “false”,
“compress”: “true”,
“max_snapshot_bytes_per_sec”: “200m”
}
}
}

4

Also before i implemeted the compression. I calculated that the searchable snapchot where taking up about 1GB for every 30GB of data snapshotted