Warm node space

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):3.7

Describe the issue:warm nodes loosing space

Configuration: locally hosted

Relevant Logs or Screenshots:

smallwog@M6GSTIN344STCL \~ % curl -k -"https://mccoy01:9200/_cat/nodes?h=name,disk.used_percent,disk.avail,node.roles&v"

name            disk.used_percent disk.avail node.roles

mccoy05-warm               100.00         0b warm

mccoy02-warm                 5.13     17.2tb warm

But if i restart the warm node with no space it regains the space and the other warm node loses its storage.

At the OS level. the storage is fine

/dev/mapper/OpenSearch-warmlv          4.6T  1.5T  3.2T  31% /opensearch_warm

it looks alot like this:

But not quite

node           disk.percent disk.indices disk.used disk.avail
mccoy03-data             42        5.7tb     5.8tb      7.7tb
mccoy04-search            0           0b     6.5gb    924.3gb
mccoy02-search            0           0b     6.5gb    924.3gb
mccoy05-data             43        5.8tb     5.9tb      7.7tb
mccoy04-data             36        4.8tb     4.9tb      8.6tb
mccoy01-data             36        4.8tb     4.9tb      8.6tb
mccoy06-data             27        3.5tb     3.7tb      9.9tb
mccoy06-search            0           0b     6.5gb    924.3gb
mccoy05-warm            100       20.3tb    18.2tb         0b
mccoy03-search            0           0b     6.5gb    924.3gb
mccoy01-search            0           0b     6.5gb    924.5gb
mccoy02-data             27        3.6tb     3.7tb      9.8tb
mccoy05-search            0           0b     6.5gb    924.3gb
mccoy02-warm              0       20.4tb    50.6gb     18.1tb

Filesystem                             Size  Used Avail Use% Mounted on

devtmpfs                               4.0M     0  4.0M   0% /dev

tmpfs                                  252G     0  252G   0% /dev/shm

tmpfs                                  101G   28M  101G   1% /run

efivarfs                               304K  188K  111K  63% /sys/firmware/efi/efivars

/dev/mapper/almalinux-root             100G  4.4G   96G   5% /

/dev/sda3                              961M  397M  564M  42% /boot

/dev/mapper/almalinux-home             100G  3.5G   97G   4% /home

/dev/mapper/almalinux-opt              100G  803M  100G   1% /opt

/dev/mapper/OpenSearch-opensearchlv     14T  5.9T  7.8T  44% /opensearch_data

/dev/mapper/almalinux-tmp               20G  218M   20G   2% /tmp

/dev/mapper/almalinux-var              100G  1.1G   99G   2% /var

/dev/mapper/OpenSearch-searchlv        931G  6.6G  925G   1% /opensearch_search

/dev/mapper/OpenSearch-warmlv          4.6T  1.4T  3.2T  31% /opensearch_warm

/dev/sda1                              256M  7.5M  249M   3% /boot/efi

/dev/mapper/almalinux-var_log          100G   11G   90G  11% /var/log

opensearchnfs1.bc.edu:/opensearchvol1  310T  170T  141T  55% /mnt/opensearch_snapshots

tmpfs                                   51G     0   51G   0% /run/user/1000

tmpfs                                   51G     0   51G   0% /run/user/1001
2

@GuyS Do _cat/shards correspond with that node disk usage?

3

Hi Pablo,

Thank you for getting back to me. the output of GET _cat/shards show that:

mccoy02-warm has 625 shards

mccoy05-warm has 625 shards

4

One note this was an artifact from an upgrade from 3.1 to 3.7. we didnt notice the fault until 24 hours after the upgrade.

5

in the ls ||wc or the warm node filesystem:
[smallwog@mccoy02 indices]$ ls|wc

354     354    8142

/opensearch_warm/nodes/0/indices

[smallwog@mccoy05 indices]$ ls|wc

354     354    8142

Not sure how that tracks with the GET _cat/shards

I am about ready to unmount the searchable snapshots and remount as searchable snapshots

6

I was able to fix this by setting:

PUT /_cluster/settings
{
“persistent”: {
“cluster.max_shards_per_node”: 2000
}
}