Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Describe the issue: i am configure opeansearch but cluster is carsh and dashboard pod is run but not connecting for opensearch cluster
Configuration:how to configure opensearch using helm chart plz sent me valus.yaml dashboard and clustr
Relevant Logs or Screenshots:
@abdul have you followed the opensearch docs here
The sample values.yml is provided there also, which I just tested and it works as expected.
You would need to investigate what is causing the pods to crash, most likely you do not have a default storageClass configured perhaps.
@Anthony i am using gp2 storageclass and Can you send the values.yaml file that you used? That might help me a bit
@abdul The most basic values.yaml file is included in the docs:
config:
opensearch.yml: |-
cluster.name: opensearch-cluster
network.host: 0.0.0.0
extraEnvs:
- name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
value: <strong_password>
thanks @anthony plz cheak my values.yml and opensearch.yml file
---
clusterName: “opensearch-cluster”
nodeGroup: “master”
# If discovery.type in the opensearch configuration is set to “single-node”,
# this should be set to “true”
# If “true”, replicas will be forced to 1
singleNode: false
# The service that non master groups will try to connect to when joining the cluster
# This should be set to clusterName + “-” + nodeGroup for your master group
masterService: “opensearch-cluster-master”
# OpenSearch roles that will be applied to this nodeGroup
# These will be set as environment variable “node.roles”. E.g. node.roles=master,ingest,data,remote_cluster_client
roles:
* master
* ingest
* data
* remote_cluster_client
replicas: 3
# if not set, falls back to parsing .Values.imageTag, then .Chart.appVersion.
majorVersion: “”
global:
# Set if you want to change the default docker registry, e.g. a private one.
dockerRegistry: “”
# Allows you to add any config files in {{ .Values.opensearchHome }}/config
opensearchHome: /usr/share/opensearch
# such as opensearch.yml and log4j2.properties
config:
# Values must be YAML literal style scalar / YAML multiline string.
# : |
# <formatted-value(s)>
# log4j2.properties: |
# status = error
#
# appender.console.type = Console
# appender.console.name = console
# appender.console.layout.type = PatternLayout
# appender.console.layout.pattern = \[%d{ISO8601}\]\[%-5p\]\[%-25c{1.}\] \[%node_name\]%marker %m%n
#
# rootLogger.level = info
# rootLogger.appenderRef.console.ref = console
opensearch.yml: |
cluster.name: opensearch-cluster
node.name: ${HOSTNAME}
discovery.seed_hosts: \[“opensearch-cluster-master-headless”\]
cluster.initial_master_nodes: \[“opensearch-cluster-master-0”, “opensearch-cluster-master-1”, “opensearch-cluster-master-2”\]
node.roles: \[ “master”, “data”, “ingest”, “remote_cluster_client” \]
# Bind to all interfaces because we don't know what IP address Docker will assign to us.
network.host: 0.0.0.0
# Setting network.host to a non-loopback address enables the annoying bootstrap checks. "Single-node" mode disables them again.
# Implicitly done if ".singleNode" is set to "true".
# discovery.type: single-node
# Start OpenSearch Security Demo Configuration
# WARNING: revise all the lines below before you go into production
# plugins:
# security:
# ssl:
# transport:
# pemcert_filepath: esnode.pem
# pemkey_filepath: esnode-key.pem
# pemtrustedcas_filepath: root-ca.pem
# enforce_hostname_verification: false
# http:
# enabled: true
# pemcert_filepath: esnode.pem
# pemkey_filepath: esnode-key.pem
# pemtrustedcas_filepath: root-ca.pem
# allow_unsafe_democertificates: true
# allow_default_init_securityindex: true
# authcz:
# admin_dn:
# - CN=kirk,OU=client,O=client,L=test,C=de
# audit.type: internal_opensearch
# enable_snapshot_restore_privilege: true
# check_snapshot_restore_write_privileges: true
# restapi:
# roles_enabled: ["all_access", "security_rest_api_access"]
# system_indices:
# enabled: true
# indices:
# [
# ".opendistro-alerting-config",
# ".opendistro-alerting-alert*",
# ".opendistro-anomaly-results*",
# ".opendistro-anomaly-detector*",
# ".opendistro-anomaly-checkpoints",
# ".opendistro-anomaly-detection-state",
# ".opendistro-reports-*",
# ".opendistro-notifications-*",
# ".opendistro-notebooks",
# ".opendistro-asynchronous-search-response*",
# ]
######## End OpenSearch Security Demo Configuration ########
# log4j2.properties:
# Extra environment variables to append to this nodeGroup
# This will be appended to the current ‘env:’ key. You can use any of the kubernetes env
# syntax here
extraEnvs:
# - name: MY_ENVIRONMENT_VAR
# value: the_value_goes_here
# Chart version 2.18.0 and App Version OpenSearch 2.12.0 onwards a custom strong password needs to be provided in order to setup demo admin user.
# Cluster will not spin-up without this unless demo config install is disabled.
* name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
value: “Ht5@n!2admin”
* name: cluster.initial_master_nodes
value: “opensearch-cluster-master-0,opensearch-cluster-master-1,opensearch-cluster-master-2”
* name: node.roles
value: “master,ingest,data,remote_cluster_client”
# Allows you to load environment variables from kubernetes secret or config map
envFrom: [ ]
# - secretRef:
# name: env-secret
# - configMapRef:
# name: config-map
# A list of secrets and their paths to mount inside the pod
# This is useful for mounting certificates for security and for mounting
# the X-Pack license
secretMounts: [ ]
hostAliases: [ ]
# - ip: “127.0.0.1”
# hostnames:
# - “foo.local”
# - “bar.local”
image:
repository: “opensearchproject/opensearch”
# override image tag, which is .Chart.AppVersion by default
tag: “”
pullPolicy: “IfNotPresent”
podAnnotations: {}
# iam.amazonaws.com/role: es-cluster
# OpenSearch Statefulset annotations
openSearchAnnotations: {}
# additionals labels
labels: {}
opensearchJavaOpts: “-Xmx1G -Xms1G”
resources:
requests:
cpu: “1000m”
memory: “1Gi”
initResources: {}
# limits:
# cpu: “25m”
# memory: “128Mi”
# requests:
# cpu: “25m”
# memory: “128Mi”
sidecarResources: {}
# limits:
# cpu: “25m”
# memory: “128Mi”
# requests:
# cpu: “25m”
# memory: “128Mi”
networkHost: “0.0.0.0”
rbac:
create: false
serviceAccountAnnotations: {}
serviceAccountName: “”
# Controls whether or not the Service Account token is automatically mounted to /var/run/secrets/kubernetes.io/serviceaccount
automountServiceAccountToken: false
podSecurityPolicy:
create: false
name: “”
spec:
privileged: true
fsGroup:
rule: RunAsAny
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- secret
- configMap
- persistentVolumeClaim
- emptyDir
persistence:
enabled: true
# Set to false to disable the `fsgroup-volume` initContainer that will update permissions on the persistent disk.
enableInitChown: true
# override image, which is busybox by default
# image: busybox
# override image tag, which is latest by default
# imageTag:
labels:
# Add default labels for the volumeClaimTemplate of the StatefulSet
enabled: false
# Add custom labels for the volumeClaimTemplate of the StatefulSet
additionalLabels: {}
# OpenSearch Persistent Volume Storage Class
# If defined, storageClassName:
# If set to “-”, storageClassName: “”, which disables dynamic provisioning
# If undefined (the default) or set to null, no storageClassName spec is
# set, choosing the default provisioner. (gp2 on AWS, standard on
# GKE, AWS & OpenStack)
#
storageClass: “gp2”
accessModes:
- ReadWriteOnce
size: 10Gi
annotations: {}
extraVolumes: [ ]
# - name: extras
# emptyDir: {}
extraVolumeMounts: [ ]
# - name: extras
# mountPath: /usr/share/extras
# readOnly: true
extraContainers: [ ]
# - name: do-something
# image: busybox
# command: \[‘do’, ‘something’\]
extraInitContainers: [ ]
# - name: do-somethings
# image: busybox
# command: \[‘do’, ‘something’\]
# This is the PriorityClass settings as defined in
#
https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
priorityClassName: “”
# By default this will make sure two pods don’t end up on the same node
# Changing this to a region would allow you to spread pods across regions
antiAffinityTopologyKey: “kubernetes.io/hostname”
# Hard means that by default pods will only be scheduled if there are enough nodes for them
# and that they will never end up on the same node. Setting this to soft will do this “best effort”.
# Setting this to custom will use what is passed into customAntiAffinity.
antiAffinity: “soft”
# Allows passing in custom anti-affinity settings as defined in
#
https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#types-of-inter-pod-affinity-and-anti-affinity
# Using this parameter requires setting antiAffinity to custom.
customAntiAffinity: {}
# This is the node affinity settings as defined in
#
https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature
nodeAffinity: {}
# This is the pod affinity settings as defined in
#
https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#types-of-inter-pod-affinity-and-anti-affinity
podAffinity: {}
# This is the pod topology spread constraints
#
https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/
topologySpreadConstraints: [ ]
# The default is to deploy all pods serially. By setting this to parallel all pods are started at
# the same time when bootstrapping the cluster
podManagementPolicy: “Parallel”
# The environment variables injected by service links are not used, but can lead to slow OpenSearch boot times when
# there are many services in the current namespace.
# If you experience slow pod startups you probably want to set this to `false`.
enableServiceLinks: true
protocol: https
httpPort: 9200
transportPort: 9300
metricsPort: 9600
httpHostPort: “”
transportHostPort: “”
service:
labels: {}
labelsHeadless: {}
headless:
annotations: {}
type: ClusterIP
# The IP family and IP families options are to set the behaviour in a dual-stack environment
# Omitting these values will let the service fall back to whatever the CNI dictates the defaults
# should be
#
# ipFamilyPolicy: SingleStack
# ipFamilies:
# - IPv4
nodePort: “”
annotations: {}
httpPortName: http
transportPortName: transport
metricsPortName: metrics
loadBalancerIP: “”
loadBalancerSourceRanges: [ ]
externalTrafficPolicy: “”
updateStrategy: RollingUpdate
# This is the max unavailable setting for the pod disruption budget
# The default value of 1 will make sure that kubernetes won’t allow more than 1
# of your pods to be unavailable during maintenance
maxUnavailable: 1
podSecurityContext:
fsGroup: 1000
runAsUser: 1000
securityContext:
capabilities:
drop:
- ALL
# readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
securityConfig:
enabled: true
path: “/usr/share/opensearch/config/opensearch-security”
actionGroupsSecret:
configSecret:
internalUsersSecret:
rolesSecret:
rolesMappingSecret:
tenantsSecret:
# The following option simplifies securityConfig by using a single secret and
# specifying the config files as keys in the secret instead of creating
# different secrets for for each config file.
# Note that this is an alternative to the individual secret configuration
# above and shouldn’t be used if the above secrets are used.
config:
log4j2.properties: |
status = error
appender.console.type = Console
appender.console.name = console
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = \[%d{ISO8601}\]\[%-5p\]\[%-25c{1.}\] \[%node_name\]%marker %m%n
rootLogger.level = info
rootLogger.appenderRef.console.ref = console
# There are multiple ways to define the configuration here:
# \* If you define anything under data, the chart will automatically create
# a secret and mount it. This is best option to choose if you want to override all the
# existing yml files at once.
# \* If you define securityConfigSecret, the chart will assume this secret is
# created externally and mount it. This is best option to choose if your intention is to
# only update a single yml file.
# \* It is an error to define both data and securityConfigSecret.
securityConfigSecret: “”
dataComplete: true
data: {}
# config.yml: |-
# internal_users.yml: |-
# roles.yml: |-
# roles_mapping.yml: |-
# action_groups.yml: |-
# tenants.yml: |-
# How long to wait for opensearch to stop gracefully
terminationGracePeriod: 120
sysctlVmMaxMapCount: 262144
startupProbe:
tcpSocket:
port: 9200
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 30
livenessProbe: {}
# periodSeconds: 20
# timeoutSeconds: 5
# failureThreshold: 10
# successThreshold: 1
# initialDelaySeconds: 10
# tcpSocket:
# port: 9200
readinessProbe:
tcpSocket:
port: 9200
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 3
## Use an alternate scheduler.
## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
##
schedulerName: “”
imagePullSecrets: [ ]
nodeSelector: {}
tolerations: [ ]
# Enabling this will publically expose your OpenSearch instance.
# Only enable this if you have security enabled on your cluster
ingress:
enabled: false
# For Kubernetes >= 1.18 you should specify the ingress-controller via the field ingressClassName
# See https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress
# ingressClassName: nginx
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: “true”
ingressLabels: {}
path: /
hosts:
- chart-example.local
tls: [ ]
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
nameOverride: “”
fullnameOverride: “”
masterTerminationFix: false
opensearchLifecycle: {}
# preStop:
# exec:
# command: \[“/bin/sh”, “-c”, “echo Hello from the preStart handler > /usr/share/message”\]
# postStart:
# exec:
# command: \[“/bin/sh”, “-c”, “echo Hello from the postStart handler > /usr/share/message”\]
lifecycle: {}
# preStop:
# exec:
# command: \[“/bin/sh”, “-c”, “echo Hello from the postStart handler > /usr/share/message”\]
# postStart:
# exec:
# command:
# - bash
# - -c
# - |
# #!/bin/bash
# # Add a template to adjust number of shards/replicas1
# TEMPLATE_NAME=my_template
# INDEX_PATTERN=“logstash-\*”
# SHARD_COUNT=8
# REPLICA_COUNT=1
# ES_URL=http://localhost:9200
# while \[\[ “$(curl -s -o /dev/null -w ‘%{http_code}\\n’ $ES_URL)” != “200” \]\]; do sleep 1; done
# curl -XPUT “$ES_URL/\_template/$TEMPLATE_NAME” -H ‘Content-Type: application/json’ -d’{“index_patterns”:\[‘"“$INDEX_PATTERN”"’\],“settings”:{“number_of_shards”:‘$SHARD_COUNT’,“number_of_replicas”:‘$REPLICA_COUNT’}}’
keystore: [ ]
# To add secrets to the keystore:
# - secretName: opensearch-encryption-key
networkPolicy:
create: false
## Enable creation of NetworkPolicy resources. Only Ingress traffic is filtered for now.
## In order for a Pod to access OpenSearch, it needs to have the following label:
## {{ template “uname” . }}-client: “true”
## Example for default configuration to access HTTP port:
## opensearch-master-http-client: “true”
## Example for default configuration to access transport port:
## opensearch-master-transport-client: “true”
http:
enabled: false
# Deprecated
# please use the above podSecurityContext.fsGroup instead
fsGroup: “”
## Set optimal sysctl’s through securityContext. This requires privilege. Can be disabled if
## the system has already been preconfigured. (Ex: https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html)
## Also see: https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/
sysctl:
enabled: false
## Set optimal sysctl’s through privileged initContainer.
sysctlInit:
enabled: false
# override image, which is busybox by default
# image: busybox
# override image tag, which is latest by default
# imageTag:
## Enable to add 3rd Party / Custom plugins not offered in the default OpenSearch image.
plugins:
enabled: false
installList: [ ]
# - example-fake-plugin
removeList: [ ]
# - example-fake-plugin
# – Array of extra K8s manifests to deploy
extraObjects: [ ]
# - apiVersion: secrets-store.csi.x-k8s.io/v1
# kind: SecretProviderClass
# metadata:
# name: argocd-secrets-store
# spec:
# provider: aws
# parameters:
# objects: |
# - objectName: “argocd”
# objectType: “secretsmanager”
# jmesPath:
# - path: “client_id”
# objectAlias: “client_id”
# - path: “client_secret”
# objectAlias: “client_secret”
# secretObjects:
# - data:
# - key: client_id
# objectName: client_id
# - key: client_secret
# objectName: client_secret
# secretName: argocd-secrets-store
# type: Opaque
# labels:
# app.kubernetes.io/part-of: argocd
# - |
# apiVersion: policy/v1
# kind: PodDisruptionBudget
# metadata:
# name: {{ template “opensearch.uname” . }}
# labels:
# {{- include “opensearch.labels” . | nindent 4 }}
# spec:
# minAvailable: 1
# selector:
# matchLabels:
# {{- include “opensearch.selectorLabels” . | nindent 6 }}
# ServiceMonitor Configuration for Prometheus
# Enabling this option will create a ServiceMonitor resource that allows Prometheus to scrape metrics from the OpenSearch service.
# This only creates the serviceMonitor, to actually have metrics Make sure to install the prometheus-exporter plugin needed for
# serving metrics over the `.Values.plugins` value:
# plugins:
# enabled: true
# installList:
# - https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/download/x.x.x.x/prometheus-exporter-x.x.x.x.zip
serviceMonitor:
# Set to true to enable the ServiceMonitor resource
enabled: false
# HTTP path where metrics are exposed.
# Ensure this matches your OpenSearch service configuration.
path: /\_prometheus/metrics
# Scheme to use for scraping.
scheme: http
# Frequency at which Prometheus will scrape metrics.
# Adjust based on your needs.
interval: 10s
# additional labels to be added to the ServiceMonitor
# labels:
# k8s.example.com/prometheus: kube-prometheus
labels: {}
# additional tlsConfig to be added to the ServiceMonitor
tlsConfig: {}
# Basic Auth configuration for the service monitor
# You can either use existingSecret, which expects a secret to be already present with data.username and data.password
# or set the credentials over the helm values, making helm create a secret for you
# basicAuth:
# enaled: true
# existingSecret: my-secret
# username: my-username
# password: my-password
basicAuth:
enabled: false@Anthony How will the connectivity between OpenSearch Dashboard and OpenSearch master be established?
@abdul Please surround your configuration in code block
@abdul the indentation did not come out right on your values.yml
Can you please confirm if you can start opensearch using the minimal values.yml file that I provided?
The connection between the OpenSearch and OSD is established using services, if you run the below command you should see the following:
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 13m
opensearch-cluster-master ClusterIP 10.108.120.152 <none> 9200/TCP,9300/TCP,9600/TCP 11m
opensearch-cluster-master-headless ClusterIP None <none> 9200/TCP,9300/TCP,9600/TCP 11m
---
clusterName: "opensearch-cluster"
nodeGroup: "master"
# If discovery.type in the opensearch configuration is set to "single-node",
# this should be set to "true"
# If "true", replicas will be forced to 1
singleNode: false
# The service that non master groups will try to connect to when joining the cluster
# This should be set to clusterName + "-" + nodeGroup for your master group
masterService: "opensearch-cluster-master"
# OpenSearch roles that will be applied to this nodeGroup
# These will be set as environment variable "node.roles". E.g. node.roles=master,ingest,data,remote_cluster_client
roles:
- master
- ingest
- data
- remote_cluster_client
replicas: 3
# if not set, falls back to parsing .Values.imageTag, then .Chart.appVersion.
majorVersion: ""
global:
# Set if you want to change the default docker registry, e.g. a private one.
dockerRegistry: ""
# Allows you to add any config files in {{ .Values.opensearchHome }}/config
opensearchHome: /usr/share/opensearch
# such as opensearch.yml and log4j2.properties
config:
# Values must be YAML literal style scalar / YAML multiline string.
# <filename>: |
# <formatted-value(s)>
# log4j2.properties: |
# status = error
#
# appender.console.type = Console
# appender.console.name = console
# appender.console.layout.type = PatternLayout
# appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
#
# rootLogger.level = info
# rootLogger.appenderRef.console.ref = console
opensearch.yml: |
cluster.name: opensearch-cluster
# Bind to all interfaces because we don't know what IP address Docker will assign to us.
network.host: 0.0.0.0
# Setting network.host to a non-loopback address enables the annoying bootstrap checks. "Single-node" mode disables them again.
# Implicitly done if ".singleNode" is set to "true".
# discovery.type: single-node
# Start OpenSearch Security Demo Configuration
# WARNING: revise all the lines below before you go into production
# plugins:
# security:
# ssl:
# transport:
# pemcert_filepath: esnode.pem
# pemkey_filepath: esnode-key.pem
# pemtrustedcas_filepath: root-ca.pem
# enforce_hostname_verification: false
# http:
# enabled: true
# pemcert_filepath: esnode.pem
# pemkey_filepath: esnode-key.pem
# pemtrustedcas_filepath: root-ca.pem
# allow_unsafe_democertificates: true
# allow_default_init_securityindex: true
# authcz:
# admin_dn:
# - CN=kirk,OU=client,O=client,L=test,C=de
# audit.type: internal_opensearch
# enable_snapshot_restore_privilege: true
# check_snapshot_restore_write_privileges: true
# restapi:
# roles_enabled: ["all_access", "security_rest_api_access"]
# system_indices:
# enabled: true
# indices:
# [
# ".opendistro-alerting-config",
# ".opendistro-alerting-alert*",
# ".opendistro-anomaly-results*",
# ".opendistro-anomaly-detector*",
# ".opendistro-anomaly-checkpoints",
# ".opendistro-anomaly-detection-state",
# ".opendistro-reports-*",
# ".opendistro-notifications-*",
# ".opendistro-notebooks",
# ".opendistro-asynchronous-search-response*",
# ]
######## End OpenSearch Security Demo Configuration ########
# log4j2.properties:
# Extra environment variables to append to this nodeGroup
# This will be appended to the current 'env:' key. You can use any of the kubernetes env
# syntax here
extraEnvs:
# - name: DISABLE_INSTALL_DEMO_CONFIG
# value: "true"
# - name: MY_ENVIRONMENT_VAR
# value: the_value_goes_here
# Chart version 2.18.0 and App Version OpenSearch 2.12.0 onwards a custom strong password needs to be provided in order to setup demo admin user.
# Cluster will not spin-up without this unless demo config install is disabled.
- name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
value: "Op3n$earch!23"
# Allows you to load environment variables from kubernetes secret or config map
envFrom: []
# - secretRef:
# name: env-secret
# - configMapRef:
# name: config-map
# A list of secrets and their paths to mount inside the pod
# This is useful for mounting certificates for security and for mounting
# the X-Pack license
secretMounts: []
hostAliases: []
# - ip: "127.0.0.1"
# hostnames:
# - "foo.local"
# - "bar.local"
image:
repository: "opensearchproject/opensearch"
# override image tag, which is .Chart.AppVersion by default
tag: ""
pullPolicy: "IfNotPresent"
podAnnotations: {}
# iam.amazonaws.com/role: es-cluster
# OpenSearch Statefulset annotations
openSearchAnnotations: {}
# additionals labels
labels: {}
opensearchJavaOpts: "-Xmx1G -Xms1G"
resources:
requests:
cpu: "1000m"
memory: "1Gi"
initResources: {}
# limits:
# cpu: "25m"
# memory: "128Mi"
# requests:
# cpu: "25m"
# memory: "128Mi"
sidecarResources: {}
# limits:
# cpu: "25m"
# memory: "128Mi"
# requests:
# cpu: "25m"
# memory: "128Mi"
networkHost: "0.0.0.0"
rbac:
create: false
serviceAccountAnnotations: {}
serviceAccountName: ""
# Controls whether or not the Service Account token is automatically mounted to /var/run/secrets/kubernetes.io/serviceaccount
automountServiceAccountToken: false
podSecurityPolicy:
create: false
name: ""
spec:
privileged: true
fsGroup:
rule: RunAsAny
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- secret
- configMap
- persistentVolumeClaim
- emptyDir
persistence:
enabled: true
# Set to false to disable the `fsgroup-volume` initContainer that will update permissions on the persistent disk.
enableInitChown: true
# override image, which is busybox by default
# image: busybox
# override image tag, which is latest by default
# imageTag:
labels:
# Add default labels for the volumeClaimTemplate of the StatefulSet
enabled: false
# Add custom labels for the volumeClaimTemplate of the StatefulSet
additionalLabels: {}
# OpenSearch Persistent Volume Storage Class
# If defined, storageClassName: <storageClass>
# If set to "-", storageClassName: "", which disables dynamic provisioning
# If undefined (the default) or set to null, no storageClassName spec is
# set, choosing the default provisioner. (gp2 on AWS, standard on
# GKE, AWS & OpenStack)
#
storageClass: "gp2"
accessModes:
- ReadWriteOnce
size: 10Gi
annotations: {}
extraVolumes: []
# - name: extras
# emptyDir: {}
extraVolumeMounts: []
# - name: extras
# mountPath: /usr/share/extras
# readOnly: true
extraContainers: []
# - name: do-something
# image: busybox
# command: ['do', 'something']
extraInitContainers: []
# - name: do-somethings
# image: busybox
# command: ['do', 'something']
# This is the PriorityClass settings as defined in
# https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
priorityClassName: ""
# By default this will make sure two pods don't end up on the same node
# Changing this to a region would allow you to spread pods across regions
antiAffinityTopologyKey: "kubernetes.io/hostname"
# Hard means that by default pods will only be scheduled if there are enough nodes for them
# and that they will never end up on the same node. Setting this to soft will do this "best effort".
# Setting this to custom will use what is passed into customAntiAffinity.
antiAffinity: "soft"
# Allows passing in custom anti-affinity settings as defined in
# https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#types-of-inter-pod-affinity-and-anti-affinity
# Using this parameter requires setting antiAffinity to custom.
customAntiAffinity: {}
# This is the node affinity settings as defined in
# https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature
nodeAffinity: {}
# This is the pod affinity settings as defined in
# https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#types-of-inter-pod-affinity-and-anti-affinity
podAffinity: {}
# This is the pod topology spread constraints
# https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/
topologySpreadConstraints: []
# The default is to deploy all pods serially. By setting this to parallel all pods are started at
# the same time when bootstrapping the cluster
podManagementPolicy: "Parallel"
# The environment variables injected by service links are not used, but can lead to slow OpenSearch boot times when
# there are many services in the current namespace.
# If you experience slow pod startups you probably want to set this to `false`.
enableServiceLinks: true
protocol: https
httpPort: 9200
transportPort: 9300
metricsPort: 9600
httpHostPort: ""
transportHostPort: ""
service:
labels: {}
labelsHeadless: {}
headless:
annotations: {}
type: ClusterIP
# The IP family and IP families options are to set the behaviour in a dual-stack environment
# Omitting these values will let the service fall back to whatever the CNI dictates the defaults
# should be
#
# ipFamilyPolicy: SingleStack
# ipFamilies:
# - IPv4
nodePort: ""
annotations: {}
httpPortName: http
transportPortName: transport
metricsPortName: metrics
loadBalancerIP: ""
loadBalancerSourceRanges: []
externalTrafficPolicy: ""
updateStrategy: RollingUpdate
# This is the max unavailable setting for the pod disruption budget
# The default value of 1 will make sure that kubernetes won't allow more than 1
# of your pods to be unavailable during maintenance
maxUnavailable: 1
podSecurityContext:
fsGroup: 1000
runAsUser: 1000
securityContext:
capabilities:
drop:
- ALL
# readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
securityConfig:
enabled: true
path: "/usr/share/opensearch/config/opensearch-security"
actionGroupsSecret:
configSecret:
internalUsersSecret:
rolesSecret:
rolesMappingSecret:
tenantsSecret:
# The following option simplifies securityConfig by using a single secret and
# specifying the config files as keys in the secret instead of creating
# different secrets for for each config file.
# Note that this is an alternative to the individual secret configuration
# above and shouldn't be used if the above secrets are used.
config:
# There are multiple ways to define the configuration here:
# * If you define anything under data, the chart will automatically create
# a secret and mount it. This is best option to choose if you want to override all the
# existing yml files at once.
# * If you define securityConfigSecret, the chart will assume this secret is
# created externally and mount it. This is best option to choose if your intention is to
# only update a single yml file.
# * It is an error to define both data and securityConfigSecret.
securityConfigSecret: ""
dataComplete: true
data: {}
# config.yml: |-
# internal_users.yml: |-
# roles.yml: |-
# roles_mapping.yml: |-
# action_groups.yml: |-
# tenants.yml: |-
# How long to wait for opensearch to stop gracefully
terminationGracePeriod: 120
sysctlVmMaxMapCount: 262144
startupProbe:
tcpSocket:
port: 9200
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 30
livenessProbe: {}
# periodSeconds: 20
# timeoutSeconds: 5
# failureThreshold: 10
# successThreshold: 1
# initialDelaySeconds: 10
# tcpSocket:
# port: 9200
readinessProbe:
tcpSocket:
port: 9200
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 3
## Use an alternate scheduler.
## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
##
schedulerName: ""
imagePullSecrets: []
nodeSelector: {}
tolerations: []
# Enabling this will publically expose your OpenSearch instance.
# Only enable this if you have security enabled on your cluster
ingress:
enabled: false
# For Kubernetes >= 1.18 you should specify the ingress-controller via the field ingressClassName
# See https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress
# ingressClassName: nginx
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
ingressLabels: {}
path: /
hosts:
- chart-example.local
tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
nameOverride: ""
fullnameOverride: ""
masterTerminationFix: false
opensearchLifecycle: {}
# preStop:
# exec:
# command: ["/bin/sh", "-c", "echo Hello from the preStart handler > /usr/share/message"]
# postStart:
# exec:
# command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
lifecycle: {}
# preStop:
# exec:
# command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
# postStart:
# exec:
# command:
# - bash
# - -c
# - |
# #!/bin/bash
# # Add a template to adjust number of shards/replicas1
# TEMPLATE_NAME=my_template
# INDEX_PATTERN="logstash-*"
# SHARD_COUNT=8
# REPLICA_COUNT=1
# ES_URL=http://localhost:9200
# while [[ "$(curl -s -o /dev/null -w '%{http_code}\n' $ES_URL)" != "200" ]]; do sleep 1; done
# curl -XPUT "$ES_URL/_template/$TEMPLATE_NAME" -H 'Content-Type: application/json' -d'{"index_patterns":['\""$INDEX_PATTERN"\"'],"settings":{"number_of_shards":'$SHARD_COUNT',"number_of_replicas":'$REPLICA_COUNT'}}'
keystore: []
# To add secrets to the keystore:
# - secretName: opensearch-encryption-key
networkPolicy:
create: false
## Enable creation of NetworkPolicy resources. Only Ingress traffic is filtered for now.
## In order for a Pod to access OpenSearch, it needs to have the following label:
## {{ template "uname" . }}-client: "true"
## Example for default configuration to access HTTP port:
## opensearch-master-http-client: "true"
## Example for default configuration to access transport port:
## opensearch-master-transport-client: "true"
http:
enabled: false
# Deprecated
# please use the above podSecurityContext.fsGroup instead
fsGroup: ""
## Set optimal sysctl's through securityContext. This requires privilege. Can be disabled if
## the system has already been preconfigured. (Ex: https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html)
## Also see: https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/
sysctl:
enabled: false
## Set optimal sysctl's through privileged initContainer.
sysctlInit:
enabled: false
# override image, which is busybox by default
# image: busybox
# override image tag, which is latest by default
# imageTag:
## Enable to add 3rd Party / Custom plugins not offered in the default OpenSearch image.
plugins:
enabled: false
installList: []
# - example-fake-plugin
removeList: []
# - example-fake-plugin
# -- Array of extra K8s manifests to deploy
extraObjects: []
# - apiVersion: secrets-store.csi.x-k8s.io/v1
# kind: SecretProviderClass
# metadata:
# name: argocd-secrets-store
# spec:
# provider: aws
# parameters:
# objects: |
# - objectName: "argocd"
# objectType: "secretsmanager"
# jmesPath:
# - path: "client_id"
# objectAlias: "client_id"
# - path: "client_secret"
# objectAlias: "client_secret"
# secretObjects:
# - data:
# - key: client_id
# objectName: client_id
# - key: client_secret
# objectName: client_secret
# secretName: argocd-secrets-store
# type: Opaque
# labels:
# app.kubernetes.io/part-of: argocd
# - |
# apiVersion: policy/v1
# kind: PodDisruptionBudget
# metadata:
# name: {{ template "opensearch.uname" . }}
# labels:
# {{- include "opensearch.labels" . | nindent 4 }}
# spec:
# minAvailable: 1
# selector:
# matchLabels:
# {{- include "opensearch.selectorLabels" . | nindent 6 }}
# ServiceMonitor Configuration for Prometheus
# Enabling this option will create a ServiceMonitor resource that allows Prometheus to scrape metrics from the OpenSearch service.
# This only creates the serviceMonitor, to actually have metrics Make sure to install the prometheus-exporter plugin needed for
# serving metrics over the `.Values.plugins` value:
# plugins:
# enabled: true
# installList:
# - https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/download/x.x.x.x/prometheus-exporter-x.x.x.x.zip
serviceMonitor:
# Set to true to enable the ServiceMonitor resource
enabled: false
# HTTP path where metrics are exposed.
# Ensure this matches your OpenSearch service configuration.
path: /_prometheus/metrics
# Scheme to use for scraping.
scheme: http
# Frequency at which Prometheus will scrape metrics.
# Adjust based on your needs.
interval: 10s
# additional labels to be added to the ServiceMonitor
# labels:
# k8s.example.com/prometheus: kube-prometheus
labels: {}
# additional tlsConfig to be added to the ServiceMonitor
tlsConfig: {}
# Basic Auth configuration for the service monitor
# You can either use existingSecret, which expects a secret to be already present with data.username and data.password
# or set the credentials over the helm values, making helm create a secret for you
# basicAuth:
# enaled: true
# existingSecret: my-secret
# username: my-username
# password: my-password
basicAuth:
enabled: false
@Anthony opensearch dashboard file
# Copyright OpenSearch Contributors
# SPDX-License-Identifier: Apache-2.0
# Default values for opensearch-dashboards.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
opensearchHosts: "https://opensearch-cluster-master:9200"
replicaCount: 1
image:
repository: "opensearchproject/opensearch-dashboards"
# override image tag, which is .Chart.AppVersion by default
tag: ""
pullPolicy: "IfNotPresent"
startupProbe:
tcpSocket:
port: 5601
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 20
successThreshold: 1
initialDelaySeconds: 10
livenessProbe:
tcpSocket:
port: 5601
periodSeconds: 20
timeoutSeconds: 5
failureThreshold: 10
successThreshold: 1
initialDelaySeconds: 10
readinessProbe:
tcpSocket:
port: 5601
periodSeconds: 20
timeoutSeconds: 5
failureThreshold: 10
successThreshold: 1
initialDelaySeconds: 10
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
serviceAccount:
# Specifies whether a service account should be created
create: true
# Annotations to add to the service account
annotations: {}
# The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template
name: ""
# Controls whether the ServiceAccount API token is automatically mounted on pod
automountServiceAccountToken: true
rbac:
create: true
# A list of secrets and their paths to mount inside the pod
# This is useful for mounting certificates for security and for mounting
# the X-Pack license
secretMounts: []
# - name: certs
# secretName: dashboard-certs
# path: /usr/share/dashboards/certs
podAnnotations: {}
# Deployment annotations
dashboardAnnotations: {}
extraEnvs:
- name: OPENSEARCH_USERNAME
value: "admin"
- name: OPENSEARCH_PASSWORD
value: "Op3n$earch!23"
# - name: "NODE_OPTIONS"
# value: "--max-old-space-size=1800"
envFrom: []
extraVolumes: []
# - name: extras
# emptyDir: {}
extraVolumeMounts: []
# - name: extras
# mountPath: /usr/share/extras
# readOnly: true
extraInitContainers: ""
extraContainers: ""
podSecurityContext: {}
securityContext:
capabilities:
drop:
- ALL
# readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
config: {}
# Default OpenSearch Dashboards configuration from docker image of Dashboards
opensearch_dashboards.yml: |
server:
host: "0.0.0.0"
opensearch.hosts: ["https://opensearch-cluster-master:9200"]
opensearch.username: "admin"
opensearch.password: "Op3n$earch!23"
opensearch.ssl.verificationMode: "none"
logging.verbose: true
# name: dashboards
# host: "{{ .Values.serverHost }}"
# opensearch_dashboards.yml:
# server:
# name: dashboards
# host: "{{ .Values.serverHost }}"
# Dashboards TLS Config (Ensure the cert files are present before enabling SSL
# ssl:
# enabled: true
# key: /usr/share/opensearch-dashboards/certs/dashboards-key.pem
# certificate: /usr/share/opensearch-dashboards/certs/dashboards-crt.pem
# determines how dashboards will verify certificates (needs to be none for default opensearch certificates to work)
# opensearch:
# ssl:
# certificateAuthorities: /usr/share/opensearch-dashboards/certs/dashboards-root-ca.pem
# if utilizing custom CA certs for connection to opensearch, provide the CA here
opensearchDashboardsYml:
defaultMode:
# value should be 0-0777
priorityClassName: ""
opensearchAccount:
secret: ""
keyPassphrase:
enabled: false
labels: {}
hostAliases: []
# - ip: "127.0.0.1"
# hostnames:
# - "foo.local"
# - "bar.local"
serverHost: "0.0.0.0"
service:
type: ClusterIP
# The IP family and IP families options are to set the behaviour in a dual-stack environment
# Omitting these values will let the service fall back to whatever the CNI dictates the defaults
# should be
#
# ipFamilyPolicy: SingleStack
# ipFamilies:
# - IPv4
port: 5601
metricsPort: 9601
loadBalancerIP: ""
nodePort: ""
labels: {}
annotations: {}
loadBalancerSourceRanges: []
# 0.0.0.0/0
httpPortName: http
metricsPortName: metrics
ingress:
enabled: false
# For Kubernetes >= 1.18 you should specify the ingress-controller via the field ingressClassName
# See https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress
# ingressClassName: nginx
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
labels: {}
hosts:
- host: chart-example.local
paths:
- path: /
backend:
serviceName: ""
servicePort: ""
tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
resources:
requests:
cpu: "100m"
memory: "512M"
limits:
cpu: "1000m"
memory: "1Gi"
autoscaling:
# This requires metrics server to be installed, to install use kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
# See https://github.com/kubernetes-sigs/metrics-server
enabled: false
minReplicas: 1
maxReplicas: 10
# The legacy `targetCPUUtilizationPercentage` key has been deprecated in favor of `targetCPU`
targetCPU: "80"
targetMemory: "80"
updateStrategy:
type: "Recreate"
nodeSelector: {}
tolerations: []
affinity: {}
# This is the pod topology spread constraints
# https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/
topologySpreadConstraints: []
# -- Array of extra K8s manifests to deploy
extraObjects: []
# - apiVersion: secrets-store.csi.x-k8s.io/v1
# kind: SecretProviderClass
# metadata:
# name: argocd-secrets-store
# spec:
# provider: aws
# parameters:
# objects: |
# - objectName: "argocd"
# objectType: "secretsmanager"
# jmesPath:
# - path: "client_id"
# objectAlias: "client_id"
# - path: "client_secret"
# objectAlias: "client_secret"
# secretObjects:
# - data:
# - key: client_id
# objectName: client_id
# - key: client_secret
# objectName: client_secret
# secretName: argocd-secrets-store
# type: Opaque
# labels:
# app.kubernetes.io/part-of: argocd
# - |
# apiVersion: policy/v1
# kind: PodDisruptionBudget
# metadata:
# name: {{ template "opensearch-dashboards.fullname" . }}
# labels:
# {{- include "opensearch-dashboards.labels" . | nindent 4 }}
# spec:
# minAvailable: 1
# selector:
# matchLabels:
# {{- include "opensearch-dashboards.selectorLabels" . | nindent 6 }}
# pod lifecycle policies as outlined here:
# https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks
lifecycle: {}
# preStop:
# exec:
# command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
# postStart:
# exec:
# command:
# - bash
# - -c
# - |
# #!/bin/bash
# curl -I "http://admin:admin@127.0.0.1:5601/status -H "kbn-xsrf: true" -H 'kbn-xsrf: true' -H "Content-Type: application/json"
## Enable to add 3rd Party / Custom plugins not offered in the default OpenSearchDashboards image.
plugins:
enabled: false
installList: []
# - example-fake-plugin-downloadable-url
removeList: []
# - examplePluginName
# ServiceMonitor Configuration for Prometheus
# Enabling this option will create a ServiceMonitor resource that allows Prometheus to scrape metrics from the OpenSearch service.
serviceMonitor:
# Set to true to enable the ServiceMonitor resource for OpenSearch Dashboards
enabled: false
# HTTP path where metrics are exposed by OpenSearch Dashboards.
# Ensure this path is correctly set in your service.
path: /_prometheus/metrics
# Frequency at which Prometheus will scrape metrics.
# Modify as needed for your monitoring requirements.
interval: 10s
# additional labels to be added to the ServiceMonitor
# labels:
# k8s.example.com/prometheus: kube-prometheus
labels: {}
@abdul I am able to run execute your exact values.yml file and have the opensearch cluster running with 3 cluster-manager nodes, therefore the issue is not with the values.yml
Can you run commands k logs <pod_name> and k describe po <pod_name> to see if there are any errors listed.
@Anthony But then why are the pods crashing? Can I send you the error logs? Will you be able to help? And I have also sent you the screenshot of the pod crash
@Anthony this is pod logs
Defaulted container "opensearch" out of: opensearch, fsgroup-volume (init), configfile (init)
Enabling OpenSearch Security Plugin
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user.
Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string.
If a password is not provided, the setup will quit.
For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/
### OpenSearch Security Demo Installer
### ** Warning: Do not use on production or public reachable systems **
OpenSearch install type: rpm/deb on Linux 5.15.0-151-generic amd64
OpenSearch config dir: /usr/share/opensearch/config/
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 3.2.0
Detected OpenSearch Security Version: 3.2.0.0
/usr/share/opensearch/config/opensearch.yml seems to be already configured for Security. Quit.
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::objectFieldOffset has been called by net.bytebuddy.dynamic.loading.ClassInjector$UsingUnsafe$Dispatcher$CreationAction
WARNING: Please consider reporting this to the maintainers of class net.bytebuddy.dynamic.loading.ClassInjector$UsingUnsafe$Dispatcher$CreationAction
WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release
WARNING: A restricted method in java.lang.System has been called
WARNING: java.lang.System::load has been called by com.sun.jna.Native in an unnamed module (file:/usr/share/opensearch/lib/jna-5.16.0.jar)
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for callers in this module
WARNING: Restricted methods will be blocked in a future release unless native access is enabled
[2025-09-18T04:41:44,747][WARN ][stderr ] [opensearch-cluster-master-0] Sep 18, 2025 4:41:44 AM org.opensearch.javaagent.bootstrap.AgentPolicy setPolicy
[2025-09-18T04:41:44,749][WARN ][stderr ] [opensearch-cluster-master-0] INFO: Policy attached successfully: org.opensearch.bootstrap.OpenSearchPolicy@5ac6c4f2
[2025-09-18T04:41:44,758][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] version[3.2.0], pid[1], build[tar/6adc0bf476e1624190564d7fbe4aba00ccf49ad8/2025-08-12T03:55:01.226522683Z], OS[Linux/5.15.0-151-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/24.0.2/24.0.2+12]
[2025-09-18T04:41:44,758][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true]
[2025-09-18T04:41:44,758][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,CLDR, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-13047235266071804518, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, --add-modules=jdk.incubator.vector, -javaagent:agent/opensearch-agent.jar, --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xmx1G, -Xms1G, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2025-09-18T04:41:44,867][WARN ][stderr ] [opensearch-cluster-master-0] Sep 18, 2025 4:41:44 AM org.apache.lucene.internal.vectorization.PanamaVectorizationProvider <init>
[2025-09-18T04:41:44,867][WARN ][stderr ] [opensearch-cluster-master-0] INFO: Java vector incubator API enabled; uses preferredBitSize=256; FMA enabled
[2025-09-18T04:41:45,312][INFO ][o.o.s.s.t.SSLConfig ] [opensearch-cluster-master-0] SSL dual mode is disabled
[2025-09-18T04:41:45,312][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] OpenSearch Config path is /usr/share/opensearch/config
[2025-09-18T04:41:45,565][INFO ][o.o.s.s.SslSettingsManager] [opensearch-cluster-master-0] TLS HTTP Provider : JDK
[2025-09-18T04:41:45,565][INFO ][o.o.s.s.SslSettingsManager] [opensearch-cluster-master-0] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2]
[2025-09-18T04:41:45,565][INFO ][o.o.s.s.SslSettingsManager] [opensearch-cluster-master-0] TLS Transport Client Provider : JDK
[2025-09-18T04:41:45,565][INFO ][o.o.s.s.SslSettingsManager] [opensearch-cluster-master-0] TLS Transport Server Provider : JDK
[2025-09-18T04:41:45,565][INFO ][o.o.s.s.SslSettingsManager] [opensearch-cluster-master-0] Enabled TLS protocols for Transport layer : [TLSv1.3, TLSv1.2]
[2025-09-18T04:41:46,020][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] Clustername: opensearch-cluster
[2025-09-18T04:41:46,022][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] Directory /usr/share/opensearch/config has insecure file permissions (should be 0700)
[2025-09-18T04:41:46,022][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] File /usr/share/opensearch/config/opensearch.yml has insecure file permissions (should be 0600)
[2025-09-18T04:41:46,022][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] File /usr/share/opensearch/config/root-ca.pem has insecure file permissions (should be 0600)
[2025-09-18T04:41:46,022][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] File /usr/share/opensearch/config/esnode-key.pem has insecure file permissions (should be 0600)
[2025-09-18T04:41:46,022][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] File /usr/share/opensearch/config/esnode.pem has insecure file permissions (should be 0600)
[2025-09-18T04:41:46,022][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] File /usr/share/opensearch/config/kirk-key.pem has insecure file permissions (should be 0600)
[2025-09-18T04:41:46,022][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] File /usr/share/opensearch/config/kirk.pem has insecure file permissions (should be 0600)
[2025-09-18T04:41:46,539][ERROR][o.o.p.c.c.PluginSettings ] [opensearch-cluster-master-0] Loading config file /usr/share/opensearch/config/opensearch-performance-analyzer/performance-analyzer.properties failed with error: org.opensearch.performanceanalyzer.commons.config.ConfigFileException: java.io.FileNotFoundException: /usr/share/opensearch/config/opensearch-performance-analyzer/performance-analyzer.properties (No such file or directory). Disabling plugin.
[2025-09-18T04:41:46,539][INFO ][o.o.p.c.c.PluginSettings ] [opensearch-cluster-master-0] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: false, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2025-09-18T04:41:46,664][ERROR][o.o.p.c.c.StatsCollector ] [opensearch-cluster-master-0] Error in loading metadata for folderLocation: /usr/share/opensearch/config/opensearch-performance-analyzer/, fileLocation: plugin-stats-metadata
java.io.FileNotFoundException: /usr/share/opensearch/config/opensearch-performance-analyzer/plugin-stats-metadata (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method) ~[?:?]
at java.base/java.io.FileInputStream.open(FileInputStream.java:185) ~[?:?]
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:139) ~[?:?]
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:109) ~[?:?]
at org.opensearch.performanceanalyzer.commons.collectors.StatsCollector.loadMetadata(StatsCollector.java:158) [performance-analyzer-commons-2.0.0.jar:?]
at org.opensearch.performanceanalyzer.commons.collectors.StatsCollector.instance(StatsCollector.java:73) [performance-analyzer-commons-2.0.0.jar:?]
at org.opensearch.performanceanalyzer.PerformanceAnalyzerPlugin.scheduleRcaCollectors(PerformanceAnalyzerPlugin.java:264) [opensearch-performance-analyzer-3.2.0.0.jar:3.2.0.0]
at org.opensearch.performanceanalyzer.PerformanceAnalyzerPlugin.<init>(PerformanceAnalyzerPlugin.java:213) [opensearch-performance-analyzer-3.2.0.0.jar:3.2.0.0]
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:483) ~[?:?]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:872) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:820) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:615) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:229) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.node.Node.<init>(Node.java:554) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.node.Node.<init>(Node.java:482) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:249) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:249) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:411) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:168) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:159) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-3.2.0.jar:3.2.0]
at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-3.2.0.jar:3.2.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:125) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:91) [opensearch-3.2.0.jar:3.2.0]
[2025-09-18T04:41:46,667][INFO ][o.o.p.c.SearchBackPressureStatsCollector] [opensearch-cluster-master-0] SearchBackPressureStatsCollector started
[2025-09-18T04:41:46,816][INFO ][o.o.i.r.ReindexModulePlugin] [opensearch-cluster-master-0] ReindexPlugin reloadSPI called
[2025-09-18T04:41:46,816][INFO ][o.o.i.r.ReindexModulePlugin] [opensearch-cluster-master-0] Unable to find any implementation for RemoteReindexExtension
[2025-09-18T04:41:46,831][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-master-0] Loaded scheduler extension: opensearch_time_series_analytics, index: .opendistro-anomaly-detector-jobs
[2025-09-18T04:41:46,832][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-master-0] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2025-09-18T04:41:46,832][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-master-0] Loaded scheduler extension: opensearch_sap_job, index: .opensearch-sap--job
[2025-09-18T04:41:46,847][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-master-0] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2025-09-18T04:41:46,848][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-master-0] Loaded scheduler extension: opensearch_ml_commons_jobs, index: .plugins-ml-jobs
[2025-09-18T04:41:46,848][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-master-0] Loaded scheduler extension: scheduler_geospatial_ip2geo_datasource, index: .scheduler-geospatial-ip2geo-datasource
[2025-09-18T04:41:46,849][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-master-0] Loaded scheduler extension: async-query-scheduler, index: .async-query-scheduler
[2025-09-18T04:41:46,861][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [aggs-matrix-stats]
[2025-09-18T04:41:46,861][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [analysis-common]
[2025-09-18T04:41:46,861][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [cache-common]
[2025-09-18T04:41:46,861][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [geo]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [ingest-common]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [ingest-geoip]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [ingest-user-agent]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [lang-expression]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [lang-mustache]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [lang-painless]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [mapper-extras]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [opensearch-dashboards]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [parent-join]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [percolator]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [rank-eval]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [reindex]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [repository-url]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [rule-framework]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [search-pipeline-common]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [systemd]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [transport-grpc]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [transport-netty4]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-alerting]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-anomaly-detection]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-asynchronous-search]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-cross-cluster-replication]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-custom-codecs]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-flow-framework]
[2025-09-18T04:41:46,862][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-geospatial]
[2025-09-18T04:41:46,863][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-index-management]
[2025-09-18T04:41:46,863][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-job-scheduler]
[2025-09-18T04:41:46,863][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-knn]
[2025-09-18T04:41:46,863][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-ltr]
[2025-09-18T04:41:46,863][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-ml]
[2025-09-18T04:41:46,863][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-neural-search]
[2025-09-18T04:41:46,863][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-notifications]
[2025-09-18T04:41:46,863][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-notifications-core]
[2025-09-18T04:41:46,863][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-observability]
[2025-09-18T04:41:46,863][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-performance-analyzer]
[2025-09-18T04:41:46,863][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-reports-scheduler]
[2025-09-18T04:41:46,863][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-search-relevance]
[2025-09-18T04:41:46,863][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-security]
[2025-09-18T04:41:46,863][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-security-analytics]
[2025-09-18T04:41:46,863][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-skills]
[2025-09-18T04:41:46,863][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-sql]
[2025-09-18T04:41:46,863][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-system-templates]
[2025-09-18T04:41:46,863][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-ubi]
[2025-09-18T04:41:46,863][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [query-insights]
[2025-09-18T04:41:46,872][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2025-09-18T04:41:46,882][DEPRECATION][o.o.d.c.s.Settings ] [opensearch-cluster-master-0] [node.max_local_storage_nodes] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2025-09-18T04:41:46,918][INFO ][o.o.e.NodeEnvironment ] [opensearch-cluster-master-0] using [1] data paths, mounts [[/usr/share/opensearch/data (172.20.20.48:/opt/nfs-storage/default-opensearch-cluster-master-opensearch-cluster-master-0-pvc-752b955e-2343-458e-b2f8-b00f73f5cae5)]], net usable_space [847.2gb], net total_space [914.7gb], types [nfs4]
[2025-09-18T04:41:46,918][INFO ][o.o.e.NodeEnvironment ] [opensearch-cluster-master-0] heap size [1gb], compressed ordinary object pointers [true]
[2025-09-18T04:41:46,934][DEPRECATION][o.o.d.c.n.DiscoveryNodeRole] [opensearch-cluster-master-0] Assigning [master] role in setting [node.roles] is deprecated. To promote inclusive language, please use [cluster_manager] role instead.
[2025-09-18T04:41:46,991][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] node name [opensearch-cluster-master-0], node ID [3YkPgm9pQXKJMjYQCgsy5Q], cluster name [opensearch-cluster], roles [ingest, remote_cluster_client, master, data]
[2025-09-18T04:41:47,035][INFO ][o.o.e.ExtensionsManager ] [opensearch-cluster-master-0] ExtensionsManager initialized
[2025-09-18T04:41:47,906][WARN ][stderr ] [opensearch-cluster-master-0] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[2025-09-18T04:41:47,906][WARN ][stderr ] [opensearch-cluster-master-0] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2025-09-18T04:41:47,906][WARN ][stderr ] [opensearch-cluster-master-0] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2025-09-18T04:41:48,445][DEPRECATION][o.o.d.c.s.Settings ] [opensearch-cluster-master-0] [cluster.initial_master_nodes] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2025-09-18T04:41:48,445][DEPRECATION][o.o.d.c.s.Settings ] [opensearch-cluster-master-0] [plugins.security.ssl.transport.enforce_hostname_verification] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2025-09-18T04:41:49,116][INFO ][o.o.t.g.p.r.s.q.QueryBuilderProtoConverterRegistry] [opensearch-cluster-master-0] Registered 4 built-in query converters
[2025-09-18T04:41:49,116][INFO ][o.o.t.g.p.r.s.q.QueryBuilderProtoConverterRegistry] [opensearch-cluster-master-0] Loaded 0 external query converters (0 failed)
[2025-09-18T04:41:49,156][WARN ][stderr ] [opensearch-cluster-master-0] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[2025-09-18T04:41:49,156][WARN ][stderr ] [opensearch-cluster-master-0] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2025-09-18T04:41:49,156][WARN ][stderr ] [opensearch-cluster-master-0] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2025-09-18T04:41:49,189][INFO ][o.o.s.a.i.AuditLogImpl ] [opensearch-cluster-master-0] Message routing enabled: true
[2025-09-18T04:41:49,265][WARN ][o.o.s.c.Salt ] [opensearch-cluster-master-0] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2025-09-18T04:41:49,265][WARN ][o.o.s.c.Salt ] [opensearch-cluster-master-0] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2025-09-18T04:41:49,267][INFO ][o.o.s.f.SecurityFilter ] [opensearch-cluster-master-0] <NONE> indices are made immutable.
[2025-09-18T04:41:49,339][INFO ][o.o.l.b.LTRCircuitBreakerService] [opensearch-cluster-master-0] Registered memory breaker.
[2025-09-18T04:41:49,347][INFO ][o.o.r.m.c.i.SdkClientFactory] [opensearch-cluster-master-0] Using local opensearch cluster as metadata store.
[2025-09-18T04:41:49,364][INFO ][o.o.t.b.CircuitBreakerService] [opensearch-cluster-master-0] Registered memory breaker.
[2025-09-18T04:41:49,643][INFO ][o.o.s.e.ExperimentTaskManager] [opensearch-cluster-master-0] ExperimentTaskManager initialized with max 10 concurrent tasks (processors: 20)
[2025-09-18T04:41:49,653][INFO ][o.o.r.m.c.i.SdkClientFactory] [opensearch-cluster-master-0] Using local opensearch cluster as metadata store.
[2025-09-18T04:41:49,662][WARN ][o.o.s.p.SQLPlugin ] [opensearch-cluster-master-0] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
[2025-09-18T04:41:49,777][INFO ][o.o.s.e.f.PPLFuncImpTable] [opensearch-cluster-master-0] Cannot create type checker for function: ARRAY. Will skip its type checking
[2025-09-18T04:41:49,777][INFO ][o.o.s.e.f.PPLFuncImpTable] [opensearch-cluster-master-0] Cannot create type checker for function: FORALL. Will skip its type checking
[2025-09-18T04:41:49,777][INFO ][o.o.s.e.f.PPLFuncImpTable] [opensearch-cluster-master-0] Cannot create type checker for function: EXISTS. Will skip its type checking
[2025-09-18T04:41:49,777][INFO ][o.o.s.e.f.PPLFuncImpTable] [opensearch-cluster-master-0] Cannot create type checker for function: FILTER. Will skip its type checking
[2025-09-18T04:41:49,777][INFO ][o.o.s.e.f.PPLFuncImpTable] [opensearch-cluster-master-0] Cannot create type checker for function: TRANSFORM. Will skip its type checking
[2025-09-18T04:41:49,778][INFO ][o.o.s.e.f.PPLFuncImpTable] [opensearch-cluster-master-0] Cannot create type checker for function: REDUCE. Will skip its type checking
[2025-09-18T04:41:49,778][INFO ][o.o.s.e.f.PPLFuncImpTable] [opensearch-cluster-master-0] Cannot create type checker for function: JSON. Will skip its type checking
[2025-09-18T04:41:49,778][INFO ][o.o.s.e.f.PPLFuncImpTable] [opensearch-cluster-master-0] Cannot create type checker for function: JSON_EXTRACT. Will skip its type checking
[2025-09-18T04:41:49,778][INFO ][o.o.s.e.f.PPLFuncImpTable] [opensearch-cluster-master-0] Cannot create type checker for function: JSON_KEYS. Will skip its type checking
[2025-09-18T04:41:49,778][INFO ][o.o.s.e.f.PPLFuncImpTable] [opensearch-cluster-master-0] Cannot create type checker for function: JSON_SET. Will skip its type checking
[2025-09-18T04:41:49,778][INFO ][o.o.s.e.f.PPLFuncImpTable] [opensearch-cluster-master-0] Cannot create type checker for function: JSON_DELETE. Will skip its type checking
[2025-09-18T04:41:49,778][INFO ][o.o.s.e.f.PPLFuncImpTable] [opensearch-cluster-master-0] Cannot create type checker for function: JSON_APPEND. Will skip its type checking
[2025-09-18T04:41:49,778][INFO ][o.o.s.e.f.PPLFuncImpTable] [opensearch-cluster-master-0] Cannot create type checker for function: JSON_EXTEND. Will skip its type checking
[2025-09-18T04:41:49,885][INFO ][o.o.p.i.c.s.QueryInsightsService] [opensearch-cluster-master-0] Setting query insights index template priority to [1847]
[2025-09-18T04:41:50,063][INFO ][o.o.r.m.c.i.SdkClientFactory] [opensearch-cluster-master-0] Using local opensearch cluster as metadata store.
[2025-09-18T04:41:50,077][INFO ][o.o.m.b.MLCircuitBreakerService] [opensearch-cluster-master-0] Registered ML memory breaker.
[2025-09-18T04:41:50,077][INFO ][o.o.m.b.MLCircuitBreakerService] [opensearch-cluster-master-0] Registered ML disk breaker.
[2025-09-18T04:41:50,077][INFO ][o.o.m.b.MLCircuitBreakerService] [opensearch-cluster-master-0] Registered ML native memory breaker.
[2025-09-18T04:41:50,139][INFO ][o.r.Reflections ] [opensearch-cluster-master-0] Reflections took 30 ms to scan 1 urls, producing 31 keys and 82 values
[2025-09-18T04:41:50,154][INFO ][o.r.Reflections ] [opensearch-cluster-master-0] Reflections took 1 ms to scan 1 urls, producing 3 keys and 5 values
[2025-09-18T04:41:50,553][INFO ][o.o.t.NettyAllocator ] [opensearch-cluster-master-0] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
[2025-09-18T04:41:50,556][INFO ][o.o.s.s.t.SSLConfig ] [opensearch-cluster-master-0] SSL dual mode is disabled
[2025-09-18T04:41:50,608][INFO ][o.o.d.DiscoveryModule ] [opensearch-cluster-master-0] using discovery type [zen] and seed hosts providers [settings]
[2025-09-18T04:41:50,964][WARN ][o.o.g.DanglingIndicesState] [opensearch-cluster-master-0] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2025-09-18T04:41:51,442][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [opensearch-cluster-master-0] PerformanceAnalyzer Enabled: false
[2025-09-18T04:41:51,460][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] initialized
[2025-09-18T04:41:51,460][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] starting ...
[2025-09-18T04:41:51,498][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [windows_logtype.json] log type
[2025-09-18T04:41:51,498][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [waf_logtype.json] log type
[2025-09-18T04:41:51,498][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [vpcflow_logtype.json] log type
[2025-09-18T04:41:51,498][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [test_windows_logtype.json] log type
[2025-09-18T04:41:51,498][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [s3_logtype.json] log type
[2025-09-18T04:41:51,499][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [others_web_logtype.json] log type
[2025-09-18T04:41:51,499][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [others_proxy_logtype.json] log type
[2025-09-18T04:41:51,499][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [others_macos_logtype.json] log type
[2025-09-18T04:41:51,499][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [others_compliance_logtype.json] log type
[2025-09-18T04:41:51,499][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [others_cloud_logtype.json] log type
[2025-09-18T04:41:51,499][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [others_apt_logtype.json] log type
[2025-09-18T04:41:51,499][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [others_application_logtype.json] log type
[2025-09-18T04:41:51,499][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [okta_logtype.json] log type
[2025-09-18T04:41:51,500][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [network_logtype.json] log type
[2025-09-18T04:41:51,500][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [netflow_logtype.json] log type
[2025-09-18T04:41:51,500][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [m365_logtype.json] log type
[2025-09-18T04:41:51,500][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [linux_logtype.json] log type
[2025-09-18T04:41:51,500][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [gworkspace_logtype.json] log type
[2025-09-18T04:41:51,500][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [github_logtype.json] log type
[2025-09-18T04:41:51,500][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [dns_logtype.json] log type
[2025-09-18T04:41:51,500][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [cloudtrail_logtype.json] log type
[2025-09-18T04:41:51,501][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [azure_logtype.json] log type
[2025-09-18T04:41:51,501][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [apache_access_logtype.json] log type
[2025-09-18T04:41:51,501][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [ad_ldap_logtype.json] log type
[2025-09-18T04:41:51,571][INFO ][o.o.t.TransportService ] [opensearch-cluster-master-0] publish_address {10.44.0.6:9300}, bound_addresses {[::]:9300}
[2025-09-18T04:41:51,572][INFO ][o.o.t.TransportService ] [opensearch-cluster-master-0] Remote clusters initialized successfully.
[2025-09-18T04:41:51,862][INFO ][o.o.b.BootstrapChecks ] [opensearch-cluster-master-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
ERROR: OpenSearch did not exit normally - check the logs at /usr/share/opensearch/logs/opensearch-cluster.log
[2025-09-18T04:41:51,870][INFO ][o.o.s.a.r.AuditMessageRouter] [opensearch-cluster-master-0] Closing AuditMessageRouter
[2025-09-18T04:41:51,870][INFO ][o.o.s.a.s.SinkProvider ] [opensearch-cluster-master-0] Closing InternalOpenSearchSink
[2025-09-18T04:41:51,870][INFO ][o.o.s.a.s.SinkProvider ] [opensearch-cluster-master-0] Closing DebugSink
[2025-09-18T04:41:51,870][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] stopping ...
[2025-09-18T04:41:51,878][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] stopped
[2025-09-18T04:41:51,878][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] closing ...
[2025-09-18T04:41:51,880][INFO ][o.o.s.a.i.AuditLogImpl ] [opensearch-cluster-master-0] Closing AuditLogImpl
[2025-09-18T04:41:51,884][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] closed
@Anthony this describe pod
root@k8s-master-hotwax:/home/common/helm-charts1/opeansearch-uat-chart/helm-charts/charts/opensearch# k describe pod opensearch-cluster-master-0
Name: opensearch-cluster-master-0
Namespace: default
Priority: 0
Service Account: default
Node: k8s-node01-hotwax/172.20.20.50
Start Time: Wed, 17 Sep 2025 13:06:43 +0000
Labels: app.kubernetes.io/component=opensearch-cluster-master
app.kubernetes.io/instance=opensearch-3-1758114402
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=opensearch
app.kubernetes.io/version=3.2.0
apps.kubernetes.io/pod-index=0
controller-revision-hash=opensearch-cluster-master-6bf559fdfd
helm.sh/chart=opensearch-3.2.1
statefulset.kubernetes.io/pod-name=opensearch-cluster-master-0
Annotations: configchecksum: 528945a0dd709eb4156f55d3294288c951aba17b4d8851fde5294c372310b3d
Status: Running
IP: 10.44.0.6
IPs:
IP: 10.44.0.6
Controlled By: StatefulSet/opensearch-cluster-master
Init Containers:
fsgroup-volume:
Container ID: containerd://bde13809907f87b8f1087ea3c5b362e684acd53e95ed890501668afeb822e500
Image: busybox:latest
Image ID: docker.io/library/busybox@sha256:ab33eacc8251e3807b85bb6dba570e4698c3998eca6f0fc2ccb60575a563ea74
Port: <none>
Host Port: <none>
Command:
sh
-c
Args:
chown -R 1000:1000 /usr/share/opensearch/data
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 17 Sep 2025 13:06:44 +0000
Finished: Wed, 17 Sep 2025 13:06:44 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/usr/share/opensearch/data from opensearch-cluster-master (rw)
configfile:
Container ID: containerd://351fae8244caa0ef8616a6293ca4b02061a9f6d380f2bb441b38653dbede99c2
Image: opensearchproject/opensearch:3.2.0
Image ID: docker.io/opensearchproject/opensearch@sha256:23297b8d8545e129dd58c254ed08d786dc552410ba772983ad2af31048d2f04b
Port: <none>
Host Port: <none>
Command:
sh
-c
#!/usr/bin/env bash
cp -r /tmp/configfolder/* /tmp/config/
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 17 Sep 2025 13:06:44 +0000
Finished: Wed, 17 Sep 2025 13:06:44 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/tmp/config/ from config-emptydir (rw)
/tmp/configfolder/opensearch.yml from config (rw,path="opensearch.yml")
Containers:
opensearch:
Container ID: containerd://ed1e4e79399adabc943199a759cd1b13518c54bf7ebfcb5ddfe9623b290304b5
Image: opensearchproject/opensearch:3.2.0
Image ID: docker.io/opensearchproject/opensearch@sha256:23297b8d8545e129dd58c254ed08d786dc552410ba772983ad2af31048d2f04b
Ports: 9200/TCP, 9300/TCP, 9600/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 78
Started: Thu, 18 Sep 2025 04:41:42 +0000
Finished: Thu, 18 Sep 2025 04:41:51 +0000
Ready: False
Restart Count: 185
Requests:
cpu: 1
memory: 1Gi
Readiness: tcp-socket :9200 delay=0s timeout=3s period=5s #success=1 #failure=3
Startup: tcp-socket :9200 delay=5s timeout=3s period=10s #success=1 #failure=30
Environment:
node.name: opensearch-cluster-master-0 (v1:metadata.name)
cluster.initial_master_nodes: opensearch-cluster-master-0,opensearch-cluster-master-1,opensearch-cluster-master-2,
discovery.seed_hosts: opensearch-cluster-master-headless
cluster.name: opensearch-cluster
network.host: 0.0.0.0
OPENSEARCH_JAVA_OPTS: -Xmx1G -Xms1G
node.roles: master,ingest,data,remote_cluster_client,
OPENSEARCH_INITIAL_ADMIN_PASSWORD: Op3n$earch!23
Mounts:
/usr/share/opensearch/config/opensearch.yml from config-emptydir (rw,path="opensearch.yml")
/usr/share/opensearch/data from opensearch-cluster-master (rw)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
opensearch-cluster-master:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: opensearch-cluster-master-opensearch-cluster-master-0
ReadOnly: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: opensearch-cluster-master-config
Optional: false
config-emptydir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulled 4m13s (x186 over 15h) kubelet Container image "opensearchproject/opensearch:3.2.0" already present on machine
Warning BackOff 4m1s (x4293 over 15h) kubelet Back-off restarting failed container opensearch in pod opensearch-cluster-master-0_default(b4be3b02-35e8-4324-a098-c1675daea8a6)
This is the issue in your case. You need to increase the vm.max_pam_count, where are you running k8s, is it minikube, eks?
@Anthony thanx its working 