Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.14.0
Describe the issue:
WE are facing issue with CPU/Memory being 100% on hotnodes most of the times, Increasing CPU from 2-4 still doesnt seems to be effective.
Currently we have a 3 hot node cluster with 6 primary shards and 1 replica.
If we increase to 5 or 6 nodes cluster, can tell me whats the impact.
Please note we cant take a chance in losing any data in the cluster
Configuration:
HOT
#-------------------------------------------------------------------------------------------
# Idea is keep only initial 7 days in SSD type of disk (premium-rwo)
#-------------------------------------------------------------------------------------------
- component: hot-data
labels:
lv_status: “BUILD”
lv_application: “OPENSEARCH”
lv_sla: “GOLD”
lv_contact: “”
replicas: 3
pdb:
enable: true
minAvailable: 2
diskSize: “750Gi”
jvm: -Xmx5200M -Xms5200M
nodeSelector:
resources:
requests:
memory: “8Gi”
cpu: 4
limits:
memory: “8Gi”
cpu: 4
roles:
- “data”
persistence:
pvc:
storageClass: premium-rwo
accessModes:
- ReadWriteOnce
additionalConfig:
allowVolumeExpansion: “true”
node.attr.temp: “hot”
#-------------------------------------------------------------------------------------------
#-------------------------------------------------------------------------------------------
plugins.security.ssl.http.enabled_protocols: '[TLSv1, TLSv1.1, TLSv1.2, TLSv1.3]'
plugins.security.ssl.transport.enabled_protocols: '[TLSv1, TLSv1.1, TLSv1.2, TLSv1.3]'
plugins.security.system_indices.enabled: 'true'
plugins.security.system_indices.indices: '[".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opendistro-asynchronous-search-response*"]'
Relevant Logs or Screenshots: