Opensearch ALL Data node restart after increasing the number of replicas

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

  • OS : 3.2.0
  • Dashboard : 3.2.0
  • Operator : 2.8.0
  • Server OS : AWS Bottlerocket

Describe the issue

I am doing a test to increase the number of the Data node replicas.

With node 0 and node 1 already in place, we expected only new node 2 to be provisioned.

However, after the new node provisioned, node0 and node 1, respectively restarted.

Configuration

  general:
..
    drainDataNodes: false
..
    - component: data
      annotations: *commonAnnotations
      persistence: *commonPersistence
      pdb: *commonPDB
      diskSize: "1000Gi"
      additionalConfig:
        plugins.security.enable_snapshot_restore_privilege: "false"
        plugins.security.ssl_cert_reload_enabled: "true"
        # https://docs.opensearch.org/latest/security/audit-logs/index/
        plugins.security.audit.config.enable_rest: 'false'
        plugins.security.audit.config.enable_transport: 'false'
      env:
        - name: DISABLE_INSTALL_DEMO_CONFIG
          value: "true"
      replicas: 2 -> 3

..

  confMgmt:
    smartScaler: true

Relevant Logs or Screenshots

After data-2 provisioned,

data-0, data-1 restarted.

opensearch-data-0                        1/1     Running     0             9m3s
opensearch-data-1                        1/1     Running     0             7m19s
opensearch-data-2                        1/1     Running     0             5m25s

Is this normal situation ?

If not, How can I prevent this from happening?

Any helps ?