# Kubernetes Opensearch operator scaling issue (what is banzaicloud.com/last-applied ?) **URL:** \ **Category:** OpenSearch\ **Tags:** troubleshoot\ **Created:** [September 9, 2025, 12:40am UTC](https://forum.opensearch.org/t/kubernetes-opensearch-operator-scaling-issue-what-is-banzaicloud-com-last-applied/26949 "2025-09-09T00:40:16Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![jameskim](https://avatars.discourse-cdn.com/v4/letter/j/eb9ed0/32.png) [@jameskim](https://forum.opensearch.org/u/jameskim)\ **Post date:** [September 9, 2025, 12:40am UTC](https://forum.opensearch.org/t/kubernetes-opensearch-operator-scaling-issue-what-is-banzaicloud-com-last-applied/26949/1 "2025-09-09T00:40:16Z")
**Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser) - Operator : 2.8.0 - Opensearch : 3.2.0 - Kubernetes Environment - EKS 1.30 - AWS **Describe the issue** Hi. I`m running opensearch cluster with opensearch operator. There are two issues when changing the Nodepool replicas. _**[Issue**_ #1\*\*\*] When increasing the node pool replicas, the existing pods also restart.\*\*\* **(Test : data nodepool replicas 3 → 6)** When the number of OpenSearch node pool replicas changes, a new ControllerRevision is also created. ```auto ❯ k get pods | grep data opensearch-data-0 1/1 Running 0 16m opensearch-data-1 1/1 Running 0 14m opensearch-data-2 1/1 Running 0 5m25s opensearch-data-3 0/1 Init:0/2 0 65s opensearch-data-4 0/1 Init:0/2 0 65s opensearch-data-5 0/1 Init:0/2 0 65s kubectl get controllerrevision | egrep -e 'statefulset.apps/opensearch-data|NAME' NAME CONTROLLER REVISION AGE opensearch-data-58bc78989d statefulset.apps/opensearch-data 28 88s opensearch-data-84788b946b statefulset.apps/opensearch-data 27 20m ``` After that, the existing pods are also restarted. ```auto kubectl get pods -L controller-revision-hash | egrep -e 'data|NAME' NAME READY STATUS RESTARTS AGE CONTROLLER-REVISION-HASH NAME READY STATUS RESTARTS AGE CONTROLLER-REVISION-HASH # Existing Pods (Why this restarting ?) opensearch-data-0 1/1 Running 0 5m31s opensearch-data-5dd76687b6 opensearch-data-1 1/1 Running 0 3m43s opensearch-data-5dd76687b6 opensearch-data-2 1/1 Running 0 118s opensearch-data-5dd76687b6 # New Pods (Re - Starting because controller hash changed) opensearch-data-3 0/1 Init:0/2 0 4s opensearch-data-5dd76687b6 opensearch-data-4 1/1 Running 0 8m39s opensearch-data-58bc78989d opensearch-data-5 1/1 Running 0 8m39s opensearch-data-58bc78989d ``` **If anyone knows, please help me. Is this normal behavior?** **If not, is there a way to prevent it?** **Or, How to troubleshoot this ?** ```auto kubectl get controllerrevision opensearch-data-84788b946b -n opensearch-prod -o yaml > old.yamlkubectl get controllerrevision opensearch-data-5dd76687b6 -n opensearch-prod -o yaml > new.yaml diff -u old.yaml new.yaml ``` ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/8/8e046023639b79c062bb693f63a800cf699d4421.jpeg) [banzaicloud.com/last-applied](http://banzaicloud.com/last-applied) * * * **[Issue** #2\*\*] When reducing the OpenSearch node pool replicas, the SmartScaler does not work properly.\*\* **(Test : data nodepool replicas 6** → **3)** ```auto # kubectl get opensearchclusters opensearch -o jsonpath="{.status}" { "availableNodes": 10, "componentsStatus": [ { "component": "Restarter", "status": "InProgress" }, { "component": "Scaler", "description": "data", "status": "Excluded" } ], "health": "yellow", "initialized": true, "phase": "RUNNING", "version": "3.2.0" } # kubectl get pods opensearch-data-0 1/1 Running 0 2m8s opensearch-data-1 1/1 Running 0 13m opensearch-data-2 1/1 Running 0 11m # kubectl get opensearchclusters opensearch -o jsonpath="{.status}" { "availableNodes": 9, "componentsStatus": [ { "component": "Restarter", "status": "InProgress" } ], "health": "red", "initialized": true, "phase": "RUNNING", "version": "3.2.0" } ``` Since SmartScaler is enabled, I expected it to properly migrate the shards from the data nodes being scaled down before reducing them, but it seems this is not happening correctly. **[DEV Tools]** ```auto # GET _cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state index shard prirep state node unassigned.reason security-auditlog-2025.09.08 0 p UNASSIGNED NODE_LEFT security-auditlog-2025.09.08 0 r UNASSIGNED NODE_LEFT .kibana_1 0 p UNASSIGNED NODE_LEFT .kibana_1 0 r UNASSIGNED NODE_LEFT security-auditlog-2025.08.30 0 p UNASSIGNED NODE_LEFT security-auditlog-2025.08.30 0 r UNASSIGNED NODE_LEFT top_queries-2025.09.02-00378 0 p STARTED opensearch-data-1 top_queries-2025.09.02-00378 0 r STARTED opensearch-data-2 # GET _cluster/settings "transient": { "cluster": { "routing": { "allocation": { "enable": "all" } } } } ``` Is this the intended behavior, or could there be something misconfigured on my side? **Configuration** - The reason I set `drainDataNodes: false` is that the PVC volumes already exist, and I wanted to prevent shard relocation during a simple restart. ```auto spec: .. .. confMgmt: smartScaler: true general: .. drainDataNodes: false .. .. nodePools: - additionalConfig: plugins.security.audit.config.enable_rest: "false" plugins.security.audit.config.enable_transport: "false" plugins.security.enable_snapshot_restore_privilege: "false" plugins.security.ssl_cert_reload_enabled: "true" affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: opster.io/opensearch-nodepool: data topologyKey: kubernetes.io/hostname annotations: ad.datadoghq.com/opensearch.checks: | { "elastic": { "init_config": {}, "instances": [ { "tls_verify": false, "url": "https://%%host%%:9200", "username": "ENC[k8s_secret@opensearch-prod/admin-credentials-secret/username]", "password": "ENC[k8s_secret@opensearch-prod/admin-credentials-secret/password]", "index_stats": "true", "pshard_stats": "true", "cat_allocation_stats": "true", "pending_task_stats": "true" } ] } } component: data diskSize: 1000Gi env: - name: DISABLE_INSTALL_DEMO_CONFIG value: "true" nodeSelector: karpenter.sh/nodepool: opensearch-nodepool pdb: enable: true maxUnavailable: 1 persistence: pvc: accessModes: - ReadWriteOnce storageClass: ebs-gp3 replicas: 3 resources: limits: memory: 20Gi requests: cpu: 3000m memory: 20Gi roles: - data - ingest tolerations: - effect: NoSchedule key: karpenter.sh/nodepool operator: Equal value: opensearch-nodepool ``` **Relevant Logs or Screenshots** : Thanks ! ---
**Author:** ![Anthony](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/anthony/32/9939_2.png) [@Anthony](https://forum.opensearch.org/u/Anthony)\ **Post date:** [September 10, 2025, 9:24am UTC](https://forum.opensearch.org/t/kubernetes-opensearch-operator-scaling-issue-what-is-banzaicloud-com-last-applied/26949/2 "2025-09-10T09:24:27Z")
@jameskim The restart is a known issue, I would recommend to raise a bug [here](https://github.com/opensearch-project/opensearch-k8s-operator/issues), this happens because the configurations gets reshuffled and is therefore seen as a change that needs to be reapplied. Regarding the graceful scale down, both of the settings (`smartScaler` and `drainDataNodes`) need to be set to `true`, otherwise the Scaler component is skipped. ---
**Author:** ![jameskim](https://avatars.discourse-cdn.com/v4/letter/j/eb9ed0/32.png) [@jameskim](https://forum.opensearch.org/u/jameskim)\ **Post date:** [September 11, 2025, 1:14am UTC](https://forum.opensearch.org/t/kubernetes-opensearch-operator-scaling-issue-what-is-banzaicloud-com-last-applied/26949/3 "2025-09-11T01:14:15Z")
Hi! thanks for replying. 1. Pod was restarting because of RecoveryMode, so I’m going to disable this setting If not necessary. (I’m going to see the source code more) 2. The reason SmartScaler doesn’t work properly is because there is a bug in the code (When using CatShards API) So i will modify source code myself and use this. 3. DrainDataNode disabled. Because PVC is preserving data anyway, so I turned it off because the shards should not have to relocate when pod restarts.