GuyS
1
Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):3.7
Describe the issue:warm nodes loosing space
Configuration: locally hosted
Relevant Logs or Screenshots:
smallwog@M6GSTIN344STCL \~ % curl -k -"https://mccoy01:9200/_cat/nodes?h=name,disk.used_percent,disk.avail,node.roles&v"
name disk.used_percent disk.avail node.roles
mccoy05-warm 100.00 0b warm
mccoy02-warm 5.13 17.2tb warm
But if i restart the warm node with no space it regains the space and the other warm node loses its storage.
At the OS level. the storage is fine
/dev/mapper/OpenSearch-warmlv 4.6T 1.5T 3.2T 31% /opensearch_warm
it looks alot like this:
But not quite
node disk.percent disk.indices disk.used disk.avail
mccoy03-data 42 5.7tb 5.8tb 7.7tb
mccoy04-search 0 0b 6.5gb 924.3gb
mccoy02-search 0 0b 6.5gb 924.3gb
mccoy05-data 43 5.8tb 5.9tb 7.7tb
mccoy04-data 36 4.8tb 4.9tb 8.6tb
mccoy01-data 36 4.8tb 4.9tb 8.6tb
mccoy06-data 27 3.5tb 3.7tb 9.9tb
mccoy06-search 0 0b 6.5gb 924.3gb
mccoy05-warm 100 20.3tb 18.2tb 0b
mccoy03-search 0 0b 6.5gb 924.3gb
mccoy01-search 0 0b 6.5gb 924.5gb
mccoy02-data 27 3.6tb 3.7tb 9.8tb
mccoy05-search 0 0b 6.5gb 924.3gb
mccoy02-warm 0 20.4tb 50.6gb 18.1tb
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs 252G 0 252G 0% /dev/shm
tmpfs 101G 28M 101G 1% /run
efivarfs 304K 188K 111K 63% /sys/firmware/efi/efivars
/dev/mapper/almalinux-root 100G 4.4G 96G 5% /
/dev/sda3 961M 397M 564M 42% /boot
/dev/mapper/almalinux-home 100G 3.5G 97G 4% /home
/dev/mapper/almalinux-opt 100G 803M 100G 1% /opt
/dev/mapper/OpenSearch-opensearchlv 14T 5.9T 7.8T 44% /opensearch_data
/dev/mapper/almalinux-tmp 20G 218M 20G 2% /tmp
/dev/mapper/almalinux-var 100G 1.1G 99G 2% /var
/dev/mapper/OpenSearch-searchlv 931G 6.6G 925G 1% /opensearch_search
/dev/mapper/OpenSearch-warmlv 4.6T 1.4T 3.2T 31% /opensearch_warm
/dev/sda1 256M 7.5M 249M 3% /boot/efi
/dev/mapper/almalinux-var_log 100G 11G 90G 11% /var/log
opensearchnfs1.bc.edu:/opensearchvol1 310T 170T 141T 55% /mnt/opensearch_snapshots
tmpfs 51G 0 51G 0% /run/user/1000
tmpfs 51G 0 51G 0% /run/user/1001
pablo
2
@GuyS Do _cat/shards correspond with that node disk usage?
GuyS
3
Hi Pablo,
Thank you for getting back to me. the output of GET _cat/shards show that:
mccoy02-warm has 625 shards
mccoy05-warm has 625 shards
GuyS
4
One note this was an artifact from an upgrade from 3.1 to 3.7. we didnt notice the fault until 24 hours after the upgrade.
GuyS
5
in the ls ||wc or the warm node filesystem:
[smallwog@mccoy02 indices]$ ls|wc
354 354 8142
/opensearch_warm/nodes/0/indices
[smallwog@mccoy05 indices]$ ls|wc
354 354 8142
Not sure how that tracks with the GET _cat/shards
I am about ready to unmount the searchable snapshots and remount as searchable snapshots
GuyS
6
I was able to fix this by setting:
PUT /_cluster/settings
{
“persistent”: {
“cluster.max_shards_per_node”: 2000
}
}
GuyS
7
wel that wasnt exactly a fix. but is was a work around
IS there a way i can submit a bug report?
pablo
8
@GuyS This GitHub issue looks related to your reported scenario.
If you think it doesn’t cover it in full, you can open another issue in the same repo.
If you do so, please share the GitHub link here.
GuyS
9
We think this does apply to us as well.
We are not on kubernetes my cluster is 6 soon to be 7 physical nodes with 21 opensearch processes on bare metal. Same indication different platform.
If there is any way I can assist with logs and or test cases let me know. As this is going to be very limiting for my environment.