Opensearch nodePool Data nodes Not getting Initialized and change to Running state

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch v2.14.0
Server OS: Ubuntu 22.04.2 LTS
Cluster: Onprem 3 node

Describe the issue:
Setting up Opensearch on k8s environment with Opensearch-Operator approach.

  1. Setup of Operator via helm
  2. Setup of Opensearch-cluster by applying yaml file

Issue i am facing is that, When we apply the opensearch-cluster.yaml with nodePools master/data/coordinates, master pods comes up. However data pods and coordinating pods doesnt come up. Added the yaml file in configuration section for reference.

Pod logs are attached for reference.

Can someone help with this issue?

Post setup, i am able to curl to the service

k -n demo exec -it opensearch-masters-0 -- curl -k -u 'admin:admin' https://opens
earch.demo.svc.cluster.local:9200
{
  "name" : "opensearch-masters-0",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "ztclGe3uTKua0e9iySinbg",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.14.0",
    "build_type" : "tar",
    "build_hash" : "aaa555453f4713d652b52436874e11ba258d8f03",
    "build_date" : "2024-05-09T18:51:00.973564994Z",
    "build_snapshot" : false,
    "lucene_version" : "9.10.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

Additional Observations:

  1. To initialize the pods or to be able to view kubectl get pods after applying the yaml file, would take ~3-5 mins
  2. After pods starts visible when we get pods, it take ~5-10 mins to bring up the pods into running state.

Configuration:
Opensearch-Cluster.yaml code

cat opensearch-cluster-extension.yaml
#Minimal configuration of a cluster with version 2.X of the operator.
#Note the replacement of 'master' role with 'cluster_manager' on line 49
apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: opensearch
  namespace: demo
spec:
  security:
    config:
    tls:
      http:
        generate: true
      transport:
        generate: true
        perNode: true
  general:
    setVMMaxMapCount: false
    httpPort: 9200
    serviceName: opensearch
    version: 2.14.0
    pluginsList: ["repository-s3"]
    monitoring:
      enable: true
  dashboards:
    annotations:
      traffic.sidecar.istio.io/includeInboundPorts: "*"
      traffic.sidecar.istio.io/excludeOutboundPorts: "9200,9300"
      traffic.sidecar.istio.io/excludeInboundPorts: "9200,9300"
    tls:
      enable: false
      generate: true
    version: 2.14.0
    enable: true
    replicas: 2
    resources:
      requests:
         memory: "512Mi"
         cpu: "200m"
      limits:
         memory: "512Mi"
         cpu: "200m"
  confMgmt:
    smartScaler: true
  nodePools:
    - component: masters
      replicas: 3
      diskSize: "30Gi"
      labels:
        nodeType: masters
      resources:
         requests:
            memory: "4Gi"
            cpu: "1000m"
         limits:
            memory: "4Gi"
            cpu: "1000m"
      roles:
        - "data"
        - "cluster_manager"
      annotations:
        traffic.sidecar.istio.io/includeInboundPorts: "*"
        traffic.sidecar.istio.io/excludeOutboundPorts: "9200,9300"
        traffic.sidecar.istio.io/excludeInboundPorts: "9200,9300"
    - component: nodes
      replicas: 3
      diskSize: "30Gi"
      nodeSelector:
      resources:
         requests:
            memory: "2Gi"
            cpu: "500m"
         limits:
            memory: "2Gi"
            cpu: "500m"
      roles:
        - "data"
    - component: coordinators
      replicas: 3
      diskSize: "30Gi"
      nodeSelector:
      resources:
         requests:
            memory: "2Gi"
            cpu: "500m"
         limits:
            memory: "2Gi"
            cpu: "500m"
      roles:
        - "ingest"

Relevant Logs or Screenshots:

kns get po
NAME                                     READY   STATUS      RESTARTS        AGE
opensearch-coordinators-0                0/1     Running     5 (3m37s ago)   23m
opensearch-masters-0                     1/1     Running     0               23m
opensearch-masters-1                     1/1     Running     0               21m
opensearch-masters-2                     1/1     Running     0               18m
opensearch-nodes-0                       0/1     Running     5 (3m40s ago)   23m
opensearch-securityconfig-update-jlspx   0/1     Completed   0               23m
kns logs opensearch-nodes-0 -p
-> Installing https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/download/2.14.0.0/prometheus-exporter-2.14.0.0.zip
-> Downloading https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/download/2.14.0.0/prometheus-exporter-2.14.0.0.zip
-> Installed prometheus-exporter with folder name prometheus-exporter
-> Installing repository-s3
-> Downloading repository-s3 from opensearch
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.io.FilePermission config#plus read
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission getClassLoader
* java.lang.RuntimePermission setContextClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.NetPermission setDefaultAuthenticator
* java.net.SocketPermission * connect,resolve
* java.util.PropertyPermission aws.configFile read,write
* java.util.PropertyPermission aws.sharedCredentialsFile read,write
* java.util.PropertyPermission opensearch.allow_insecure_settings read,write
* java.util.PropertyPermission opensearch.path.conf read,write
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.
-> Installed repository-s3 with folder name repository-s3
Enabling OpenSearch Security Plugin
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user.
Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string.
If a password is not provided, the setup will quit.
 For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/
### OpenSearch Security Demo Installer
### ** Warning: Do not use on production or public reachable systems **
OpenSearch install type: rpm/deb on Linux 5.15.0-52-generic amd64
OpenSearch config dir: /usr/share/opensearch/config/
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 2.14.0
Detected OpenSearch Security Version: 2.14.0.0
/usr/share/opensearch/config/opensearch.yml seems to be already configured for Security. Quit.
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.14.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
Oct 17, 2024 3:33:46 AM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.14.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2024-10-17T03:33:52,763][INFO ][o.o.n.Node               ] [opensearch-nodes-0] version[2.14.0], pid[88], build[tar/aaa555453f4713d652b52436874e11ba258d8f03/2024-05-09T18:51:00.973564994Z], OS[Linux/5.15.0-52-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.3/21.0.3+9-LTS]
[2024-10-17T03:33:52,853][INFO ][o.o.n.Node               ] [opensearch-nodes-0] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true]
[2024-10-17T03:33:52,854][INFO ][o.o.n.Node               ] [opensearch-nodes-0] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-1661847594692135011, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, --add-modules=jdk.incubator.vector, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xmx1024M, -Xms1024M, -Dopensearch.transport.cname_in_publish_address=true, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2024-10-17T03:34:05,160][INFO ][o.o.p.p.PrometheusExporterPlugin] [opensearch-nodes-0] starting Prometheus exporter plugin
[2024-10-17T03:34:10,860][INFO ][o.o.s.s.t.SSLConfig      ] [opensearch-nodes-0] SSL dual mode is disabled
[2024-10-17T03:34:10,861][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] OpenSearch Config path is /usr/share/opensearch/config
[2024-10-17T03:34:14,562][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-nodes-0] JVM supports TLSv1.3
[2024-10-17T03:34:14,567][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-nodes-0] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively
[2024-10-17T03:34:25,556][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-nodes-0] TLS Transport Client Provider : JDK
[2024-10-17T03:34:25,557][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-nodes-0] TLS Transport Server Provider : JDK
[2024-10-17T03:34:25,557][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-nodes-0] TLS HTTP Provider             : JDK
[2024-10-17T03:34:25,558][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-nodes-0] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2024-10-17T03:34:25,558][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-nodes-0] Enabled TLS protocols for HTTP layer      : [TLSv1.3, TLSv1.2]
[2024-10-17T03:34:26,149][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] Clustername: opensearch
[2024-10-17T03:34:26,171][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] Directory /usr/share/opensearch/config has insecure file permissions (should be 0700)
[2024-10-17T03:34:26,172][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/opensearch.yml has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,247][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] Directory /usr/share/opensearch/config/tls-transport has insecure file permissions (should be 0700)
[2024-10-17T03:34:26,248][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-nodes-2.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,249][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-masters-0.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,249][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-masters-1.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,250][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-masters-1.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,250][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-masters-0.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,251][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-nodes-1.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,251][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-bootstrap-0.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,251][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-masters-2.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,252][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-nodes-1.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,252][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-bootstrap-0.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,253][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-nodes-0.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,253][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-nodes-0.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,254][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/ca.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,254][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-masters-2.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,254][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-nodes-2.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,255][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..data has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,255][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] Directory /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564 has insecure file permissions (should be 0700)
[2024-10-17T03:34:26,256][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-nodes-2.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,256][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-masters-0.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,257][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-masters-1.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,257][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-masters-0.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,258][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-nodes-1.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,258][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-bootstrap-0.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,259][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-masters-2.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,259][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-masters-1.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,259][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-nodes-1.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,260][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-bootstrap-0.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,260][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-nodes-0.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,261][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-nodes-0.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,262][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/ca.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,262][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-masters-2.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,262][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-nodes-2.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,263][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] Directory /usr/share/opensearch/config/tls-http has insecure file permissions (should be 0700)
[2024-10-17T03:34:26,263][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-http/tls.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,263][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-http/tls.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,263][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-http/ca.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,264][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-http/..data has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,264][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] Directory /usr/share/opensearch/config/tls-http/..2024_10_17_03_31_46.402380427 has insecure file permissions (should be 0700)
[2024-10-17T03:34:26,264][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-http/..2024_10_17_03_31_46.402380427/ca.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,348][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-http/..2024_10_17_03_31_46.402380427/tls.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,349][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-http/..2024_10_17_03_31_46.402380427/tls.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:36,355][INFO ][o.o.p.c.c.PluginSettings ] [opensearch-nodes-0] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2024-10-17T03:34:43,963][INFO ][o.o.i.r.ReindexPlugin    ] [opensearch-nodes-0] ReindexPlugin reloadSPI called
[2024-10-17T03:34:43,966][INFO ][o.o.i.r.ReindexPlugin    ] [opensearch-nodes-0] Unable to find any implementation for RemoteReindexExtension
[2024-10-17T03:34:44,767][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-nodes-0] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2024-10-17T03:34:44,862][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-nodes-0] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2024-10-17T03:34:44,866][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-nodes-0] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2024-10-17T03:34:44,870][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-nodes-0] Loaded scheduler extension: scheduler_geospatial_ip2geo_datasource, index: .scheduler-geospatial-ip2geo-datasource
[2024-10-17T03:34:44,948][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-nodes-0] Loaded scheduler extension: opensearch_sap_job, index: .opensearch-sap--job
[2024-10-17T03:34:45,352][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded module [aggs-matrix-stats]
[2024-10-17T03:34:45,353][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded module [analysis-common]
[2024-10-17T03:34:45,353][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded module [cache-common]
[2024-10-17T03:34:45,353][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded module [geo]
[2024-10-17T03:34:45,353][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded module [ingest-common]
[2024-10-17T03:34:45,354][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded module [ingest-geoip]
[2024-10-17T03:34:45,354][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded module [ingest-user-agent]
[2024-10-17T03:34:45,354][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded module [lang-expression]
[2024-10-17T03:34:45,354][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded module [lang-mustache]
[2024-10-17T03:34:45,354][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded module [lang-painless]
[2024-10-17T03:34:45,354][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded module [mapper-extras]
[2024-10-17T03:34:45,355][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded module [opensearch-dashboards]
[2024-10-17T03:34:45,355][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded module [parent-join]
[2024-10-17T03:34:45,355][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded module [percolator]
[2024-10-17T03:34:45,355][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded module [rank-eval]
[2024-10-17T03:34:45,356][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded module [reindex]
[2024-10-17T03:34:45,356][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded module [repository-url]
[2024-10-17T03:34:45,356][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded module [search-pipeline-common]
[2024-10-17T03:34:45,356][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded module [systemd]
[2024-10-17T03:34:45,357][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded module [transport-netty4]
[2024-10-17T03:34:45,358][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded plugin [opensearch-alerting]
[2024-10-17T03:34:45,359][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded plugin [opensearch-anomaly-detection]
[2024-10-17T03:34:45,359][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded plugin [opensearch-asynchronous-search]
[2024-10-17T03:34:45,359][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded plugin [opensearch-cross-cluster-replication]
[2024-10-17T03:34:45,359][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded plugin [opensearch-custom-codecs]
[2024-10-17T03:34:45,360][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded plugin [opensearch-flow-framework]
[2024-10-17T03:34:45,360][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded plugin [opensearch-geospatial]
[2024-10-17T03:34:45,360][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded plugin [opensearch-index-management]
[2024-10-17T03:34:45,360][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded plugin [opensearch-job-scheduler]
[2024-10-17T03:34:45,447][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded plugin [opensearch-knn]
[2024-10-17T03:34:45,447][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded plugin [opensearch-ml]
[2024-10-17T03:34:45,448][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded plugin [opensearch-neural-search]
[2024-10-17T03:34:45,448][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded plugin [opensearch-notifications]
[2024-10-17T03:34:45,448][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded plugin [opensearch-notifications-core]
[2024-10-17T03:34:45,448][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded plugin [opensearch-observability]
[2024-10-17T03:34:45,449][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded plugin [opensearch-performance-analyzer]
[2024-10-17T03:34:45,449][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded plugin [opensearch-reports-scheduler]
[2024-10-17T03:34:45,449][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded plugin [opensearch-security]
[2024-10-17T03:34:45,449][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded plugin [opensearch-security-analytics]
[2024-10-17T03:34:45,449][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded plugin [opensearch-skills]
[2024-10-17T03:34:45,449][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded plugin [opensearch-sql]
[2024-10-17T03:34:45,450][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded plugin [prometheus-exporter]
[2024-10-17T03:34:45,450][INFO ][o.o.p.PluginsService     ] [opensearch-nodes-0] loaded plugin [repository-s3]
[2024-10-17T03:34:46,466][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2024-10-17T03:34:46,555][INFO ][o.o.e.ExtensionsManager  ] [opensearch-nodes-0] ExtensionsManager initialized
[2024-10-17T03:34:46,753][INFO ][o.a.l.s.MemorySegmentIndexInputProvider] [opensearch-nodes-0] Using MemorySegmentIndexInput with Java 21 or later; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
[2024-10-17T03:34:46,860][INFO ][o.o.e.NodeEnvironment    ] [opensearch-nodes-0] using [1] data paths, mounts [[/usr/share/opensearch/data (/dev/vdb1)]], net usable_space [1.8tb], net total_space [1.9tb], types [ext4]
[2024-10-17T03:34:46,861][INFO ][o.o.e.NodeEnvironment    ] [opensearch-nodes-0] heap size [1gb], compressed ordinary object pointers [true]
[2024-10-17T03:34:47,270][INFO ][o.o.n.Node               ] [opensearch-nodes-0] node name [opensearch-nodes-0], node ID [ienk-pUrT-aWnqiMT8UEoQ], cluster name [opensearch], roles [data]
[2024-10-17T03:35:15,051][DEPRECATION][o.o.d.c.s.Settings       ] [opensearch-nodes-0] [index.store.hybrid.mmap.extensions] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2024-10-17T03:35:15,056][DEPRECATION][o.o.d.c.s.Settings       ] [opensearch-nodes-0] [cluster.initial_master_nodes] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2024-10-17T03:35:15,057][DEPRECATION][o.o.d.c.s.Settings       ] [opensearch-nodes-0] [plugins.security.ssl.transport.enforce_hostname_verification] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2024-10-17T03:35:24,268][INFO ][o.o.n.p.NeuralSearch     ] [opensearch-nodes-0] Registering hybrid query phase searcher with feature flag [plugins.neural_search.hybrid_search_disabled]
[2024-10-17T03:35:32,548][WARN ][o.o.s.c.Salt             ] [opensearch-nodes-0] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2024-10-17T03:35:33,267][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-nodes-0] Message routing enabled: true
[2024-10-17T03:35:33,854][INFO ][o.o.s.f.SecurityFilter   ] [opensearch-nodes-0] <NONE> indices are made immutable.

3

@ar.shashikumar Your coordinator and data nodes have half of resources that were assigned to your master nodes.
Try increase to the values of the master nodes.

@pablo Thanks for the response, when we assigned similar configuration as of Master it worked. However, there is no mention of such occurrence in the master/data pod logs which is very difficult to troubleshoot the issue.

Is there a recommended sizing of resources to be allocated to master/data/coordinator nodes. To take one of the example while setting up JVM, it is clearly mentioned that the 50% of the allocated memory to be assigned.

Similarly if we have such instructions while setting up resources for Opensearch NodeSets it would be of great help to the Opensearch community.

@ar.shashikumar The amount of resource always depends on the use case scenario. Maybe that’s why documentation doesn’t have recommended sizing.
Your configuration is good for a startup but you need to consider higher values for a production environment.

@pablo Thank you!! I will explore more on resources of master/data/coordinators in my next release