Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
- Opensearch : 3.2.0
- Opensearch Dashboard : 3.2.0
- Operator : v2.8.0
Describe the issue:
Default Shard: 1, Replication Shard: 0,
When Primary Shard: 1, Replica Shard: 0,
Operator kills the data node without properly draining the Primary Shard.
The reason for testing the Replication Shard to zero is to temporarily zero the Replica Shard to increase performance when performing hundreds of GB indexing, and to increase the Replica Shard when indexing is complete.
I think there is only the Primary Shard and no Replica Shard,
so I think operator need to safely completely relocate to another node and kill the data node, but operator didn`t
Is this normal situation?
If not, How do i configure in order to safely drain data node without red status ?
Configuration:
- Master : 3EA
- Data : 3EA
- Coord : 2EA
- test-index
- Primary shard : 1EA
- Replica shard : 0EA (for testing, for indexing)
apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
annotations:
argocd.argoproj.io/sync-options: ServerSideApply=true
labels:
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/version: 2.8.0
helm.sh/chart: opensearch-cluster-3.1.0
name: opensearch
namespace: opensearch-stag
spec:
bootstrap:
additionalConfig: {}
affinity: {}
jvm: ""
nodeSelector: {}
resources: {}
tolerations: []
confMgmt:
smartScaler: true
dashboards:
additionalConfig:
opensearch_security.auth.anonymous_auth_enabled: "true"
affinity: {}
annotations: {}
basePath: ""
enable: true
env:
- name: DISABLE_INSTALL_DEMO_CONFIG
value: "true"
image: docker.io/opensearchproject/opensearch-dashboards:3.2.0
imagePullPolicy: IfNotPresent
imagePullSecrets: []
labels: {}
nodeSelector: {}
opensearchCredentialsSecret:
name: admin-credentials-secret
pluginsList: []
podSecurityContext: {}
replicas: 2
resources:
limits:
cpu: 500m
memory: 2Gi
requests:
cpu: 500m
memory: 2Gi
securityContext: {}
service:
loadBalancerSourceRanges: []
type: ClusterIP
tls:
caSecret: {}
enable: false
generate: false
secret:
name: ""
tolerations: []
version: 3.2.0
general:
additionalConfig:
plugins.security.ssl_cert_reload_enabled: "true"
additionalVolumes:
- configMap:
name: log4j2-properties
name: log4j2-properties
path: /usr/share/opensearch/config/log4j2.properties
restartPods: true
subPath: log4j2.properties
drainDataNodes: true
httpPort: 9200
image: docker.io/opensearchproject/opensearch:3.2.0
imagePullPolicy: IfNotPresent
keystore: []
monitoring:
enable: false
monitoringUserSecret: ""
pluginUrl: ""
scrapeInterval: 30s
tlsConfig: {}
pluginsList:
- analysis-nori
podSecurityContext: {}
securityContext: {}
serviceAccount: ""
serviceName: opensearch
setVMMaxMapCount: true
snapshotRepositories: []
vendor: Opensearch
version: 3.2.0
initHelper:
imagePullPolicy: IfNotPresent
imagePullSecrets: []
resources: {}
version: "1.36"
nodePools:
- additionalConfig:
plugins.security.enable_snapshot_restore_privilege: "false"
plugins.security.ssl_cert_reload_enabled: "true"
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
opster.io/opensearch-nodepool: master
topologyKey: kubernetes.io/hostname
weight: 100
annotations:
ad.datadoghq.com/opensearch.checks: |
{
"elastic": {
"init_config": {},
"instances": [
{
"tls_verify": false,
"url": "https://%%host%%:9200",
"username": "ENC[k8s_secret@opensearch-stag/admin-credentials-secret/username]",
"password": "ENC[k8s_secret@opensearch-stag/admin-credentials-secret/password]",
"index_stats": "true",
"pshard_stats": "true",
"cat_allocation_stats": "true"
}
]
}
}
component: master
diskSize: 10Gi
env:
- name: DISABLE_INSTALL_DEMO_CONFIG
value: "true"
nodeSelector:
karpenter.sh/nodepool: opensearch
pdb:
enable: true
maxUnavailable: 1
persistence:
pvc:
accessModes:
- ReadWriteOnce
storageClass: ebs-gp3
replicas: 3
resources:
limits:
memory: 6Gi
requests:
cpu: 1000m
memory: 4Gi
roles:
- cluster_manager
tolerations:
- effect: NoSchedule
key: karpenter.sh/nodepool
operator: Equal
value: opensearch
- additionalConfig:
plugins.security.audit.config.enable_rest: "false"
plugins.security.audit.config.enable_transport: "false"
plugins.security.enable_snapshot_restore_privilege: "false"
plugins.security.ssl_cert_reload_enabled: "true"
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
opster.io/opensearch-nodepool: data
topologyKey: kubernetes.io/hostname
weight: 100
annotations:
ad.datadoghq.com/opensearch.checks: |
{
"elastic": {
"init_config": {},
"instances": [
{
"tls_verify": false,
"url": "https://%%host%%:9200",
"username": "ENC[k8s_secret@opensearch-stag/admin-credentials-secret/username]",
"password": "ENC[k8s_secret@opensearch-stag/admin-credentials-secret/password]",
"index_stats": "true",
"pshard_stats": "true",
"cat_allocation_stats": "true"
}
]
}
}
component: data
diskSize: 1000Gi
env:
- name: DISABLE_INSTALL_DEMO_CONFIG
value: "true"
nodeSelector:
karpenter.sh/nodepool: opensearch
pdb:
enable: true
maxUnavailable: 1
persistence:
pvc:
accessModes:
- ReadWriteOnce
storageClass: ebs-gp3
replicas: 3
resources:
limits:
memory: 20Gi
requests:
cpu: 3000m
memory: 20Gi
roles:
- data
- ingest
tolerations:
- effect: NoSchedule
key: karpenter.sh/nodepool
operator: Equal
value: opensearch
- additionalConfig:
node.data: "false"
node.ingest: "false"
node.master: "false"
node.remote_cluster_client: "false"
plugins.security.audit.config.enable_rest: "false"
plugins.security.audit.config.enable_transport: "false"
plugins.security.enable_snapshot_restore_privilege: "false"
plugins.security.ssl_cert_reload_enabled: "true"
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
opster.io/opensearch-nodepool: coord
topologyKey: kubernetes.io/hostname
weight: 100
annotations:
ad.datadoghq.com/opensearch.checks: |
{
"elastic": {
"init_config": {},
"instances": [
{
"tls_verify": false,
"url": "https://%%host%%:9200",
"username": "ENC[k8s_secret@opensearch-stag/admin-credentials-secret/username]",
"password": "ENC[k8s_secret@opensearch-stag/admin-credentials-secret/password]",
"index_stats": "true",
"pshard_stats": "true",
"cat_allocation_stats": "true"
}
]
}
}
component: coord
diskSize: 30Gi
env:
- name: DISABLE_INSTALL_DEMO_CONFIG
value: "true"
nodeSelector:
karpenter.sh/nodepool: opensearch
pdb:
enable: true
maxUnavailable: 1
persistence:
pvc:
accessModes:
- ReadWriteOnce
storageClass: ebs-gp3
replicas: 2
resources:
limits:
memory: 6Gi
requests:
cpu: 2000m
memory: 4Gi
roles:
- ""
tolerations:
- effect: NoSchedule
key: karpenter.sh/nodepool
operator: Equal
value: opensearch
security:
config:
adminCredentialsSecret:
name: admin-credentials-secret
adminSecret:
name: opensearch-admin-cert
securityConfigSecret:
name: securityconfig-secret
tls:
http:
secret:
name: opensearch-http-cert
transport:
adminDn:
- CN=admin,OU=opensearch
caSecret:
name: opensearch-ca
nodesDn:
- CN=opensearch,OU=opensearch
secret:
name: opensearch-transport-cert
Relevant Logs or Screenshots: