Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch v2.14.0
Server OS: Ubuntu 22.04.2 LTS
Cluster: Onprem 3 node
Describe the issue:
Setting up Opensearch on k8s environment with Opensearch-Operator approach.
- Setup of Operator via helm
- Setup of Opensearch-cluster by applying yaml file
Issue i am facing is that, When we apply the opensearch-cluster.yaml with nodePools master/data/coordinates, master pods comes up. However data pods and coordinating pods doesnt come up. Added the yaml file in configuration section for reference.
Pod logs are attached for reference.
Can someone help with this issue?
Post setup, i am able to curl to the service
k -n demo exec -it opensearch-masters-0 -- curl -k -u 'admin:admin' https://opens
earch.demo.svc.cluster.local:9200
{
"name" : "opensearch-masters-0",
"cluster_name" : "opensearch",
"cluster_uuid" : "ztclGe3uTKua0e9iySinbg",
"version" : {
"distribution" : "opensearch",
"number" : "2.14.0",
"build_type" : "tar",
"build_hash" : "aaa555453f4713d652b52436874e11ba258d8f03",
"build_date" : "2024-05-09T18:51:00.973564994Z",
"build_snapshot" : false,
"lucene_version" : "9.10.0",
"minimum_wire_compatibility_version" : "7.10.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "The OpenSearch Project: https://opensearch.org/"
}
Additional Observations:
- To initialize the pods or to be able to view kubectl get pods after applying the yaml file, would take ~3-5 mins
- After pods starts visible when we get pods, it take ~5-10 mins to bring up the pods into running state.
Configuration:
Opensearch-Cluster.yaml code
cat opensearch-cluster-extension.yaml
#Minimal configuration of a cluster with version 2.X of the operator.
#Note the replacement of 'master' role with 'cluster_manager' on line 49
apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
name: opensearch
namespace: demo
spec:
security:
config:
tls:
http:
generate: true
transport:
generate: true
perNode: true
general:
setVMMaxMapCount: false
httpPort: 9200
serviceName: opensearch
version: 2.14.0
pluginsList: ["repository-s3"]
monitoring:
enable: true
dashboards:
annotations:
traffic.sidecar.istio.io/includeInboundPorts: "*"
traffic.sidecar.istio.io/excludeOutboundPorts: "9200,9300"
traffic.sidecar.istio.io/excludeInboundPorts: "9200,9300"
tls:
enable: false
generate: true
version: 2.14.0
enable: true
replicas: 2
resources:
requests:
memory: "512Mi"
cpu: "200m"
limits:
memory: "512Mi"
cpu: "200m"
confMgmt:
smartScaler: true
nodePools:
- component: masters
replicas: 3
diskSize: "30Gi"
labels:
nodeType: masters
resources:
requests:
memory: "4Gi"
cpu: "1000m"
limits:
memory: "4Gi"
cpu: "1000m"
roles:
- "data"
- "cluster_manager"
annotations:
traffic.sidecar.istio.io/includeInboundPorts: "*"
traffic.sidecar.istio.io/excludeOutboundPorts: "9200,9300"
traffic.sidecar.istio.io/excludeInboundPorts: "9200,9300"
- component: nodes
replicas: 3
diskSize: "30Gi"
nodeSelector:
resources:
requests:
memory: "2Gi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "500m"
roles:
- "data"
- component: coordinators
replicas: 3
diskSize: "30Gi"
nodeSelector:
resources:
requests:
memory: "2Gi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "500m"
roles:
- "ingest"
Relevant Logs or Screenshots:
kns get po
NAME READY STATUS RESTARTS AGE
opensearch-coordinators-0 0/1 Running 5 (3m37s ago) 23m
opensearch-masters-0 1/1 Running 0 23m
opensearch-masters-1 1/1 Running 0 21m
opensearch-masters-2 1/1 Running 0 18m
opensearch-nodes-0 0/1 Running 5 (3m40s ago) 23m
opensearch-securityconfig-update-jlspx 0/1 Completed 0 23m
kns logs opensearch-nodes-0 -p
-> Installing https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/download/2.14.0.0/prometheus-exporter-2.14.0.0.zip
-> Downloading https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/download/2.14.0.0/prometheus-exporter-2.14.0.0.zip
-> Installed prometheus-exporter with folder name prometheus-exporter
-> Installing repository-s3
-> Downloading repository-s3 from opensearch
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: plugin requires additional permissions @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.io.FilePermission config#plus read
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission getClassLoader
* java.lang.RuntimePermission setContextClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.NetPermission setDefaultAuthenticator
* java.net.SocketPermission * connect,resolve
* java.util.PropertyPermission aws.configFile read,write
* java.util.PropertyPermission aws.sharedCredentialsFile read,write
* java.util.PropertyPermission opensearch.allow_insecure_settings read,write
* java.util.PropertyPermission opensearch.path.conf read,write
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.
-> Installed repository-s3 with folder name repository-s3
Enabling OpenSearch Security Plugin
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user.
Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string.
If a password is not provided, the setup will quit.
For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/
### OpenSearch Security Demo Installer
### ** Warning: Do not use on production or public reachable systems **
OpenSearch install type: rpm/deb on Linux 5.15.0-52-generic amd64
OpenSearch config dir: /usr/share/opensearch/config/
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 2.14.0
Detected OpenSearch Security Version: 2.14.0.0
/usr/share/opensearch/config/opensearch.yml seems to be already configured for Security. Quit.
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.14.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
Oct 17, 2024 3:33:46 AM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.14.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2024-10-17T03:33:52,763][INFO ][o.o.n.Node ] [opensearch-nodes-0] version[2.14.0], pid[88], build[tar/aaa555453f4713d652b52436874e11ba258d8f03/2024-05-09T18:51:00.973564994Z], OS[Linux/5.15.0-52-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.3/21.0.3+9-LTS]
[2024-10-17T03:33:52,853][INFO ][o.o.n.Node ] [opensearch-nodes-0] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true]
[2024-10-17T03:33:52,854][INFO ][o.o.n.Node ] [opensearch-nodes-0] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-1661847594692135011, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, --add-modules=jdk.incubator.vector, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xmx1024M, -Xms1024M, -Dopensearch.transport.cname_in_publish_address=true, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2024-10-17T03:34:05,160][INFO ][o.o.p.p.PrometheusExporterPlugin] [opensearch-nodes-0] starting Prometheus exporter plugin
[2024-10-17T03:34:10,860][INFO ][o.o.s.s.t.SSLConfig ] [opensearch-nodes-0] SSL dual mode is disabled
[2024-10-17T03:34:10,861][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] OpenSearch Config path is /usr/share/opensearch/config
[2024-10-17T03:34:14,562][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-nodes-0] JVM supports TLSv1.3
[2024-10-17T03:34:14,567][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-nodes-0] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively
[2024-10-17T03:34:25,556][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-nodes-0] TLS Transport Client Provider : JDK
[2024-10-17T03:34:25,557][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-nodes-0] TLS Transport Server Provider : JDK
[2024-10-17T03:34:25,557][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-nodes-0] TLS HTTP Provider : JDK
[2024-10-17T03:34:25,558][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-nodes-0] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2024-10-17T03:34:25,558][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-nodes-0] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2]
[2024-10-17T03:34:26,149][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] Clustername: opensearch
[2024-10-17T03:34:26,171][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] Directory /usr/share/opensearch/config has insecure file permissions (should be 0700)
[2024-10-17T03:34:26,172][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/opensearch.yml has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,247][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] Directory /usr/share/opensearch/config/tls-transport has insecure file permissions (should be 0700)
[2024-10-17T03:34:26,248][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-nodes-2.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,249][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-masters-0.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,249][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-masters-1.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,250][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-masters-1.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,250][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-masters-0.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,251][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-nodes-1.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,251][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-bootstrap-0.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,251][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-masters-2.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,252][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-nodes-1.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,252][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-bootstrap-0.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,253][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-nodes-0.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,253][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-nodes-0.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,254][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/ca.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,254][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-masters-2.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,254][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/opensearch-nodes-2.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,255][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..data has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,255][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] Directory /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564 has insecure file permissions (should be 0700)
[2024-10-17T03:34:26,256][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-nodes-2.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,256][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-masters-0.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,257][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-masters-1.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,257][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-masters-0.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,258][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-nodes-1.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,258][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-bootstrap-0.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,259][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-masters-2.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,259][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-masters-1.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,259][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-nodes-1.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,260][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-bootstrap-0.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,260][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-nodes-0.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,261][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-nodes-0.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,262][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/ca.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,262][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-masters-2.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,262][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-transport/..2024_10_17_03_31_46.2160498564/opensearch-nodes-2.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,263][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] Directory /usr/share/opensearch/config/tls-http has insecure file permissions (should be 0700)
[2024-10-17T03:34:26,263][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-http/tls.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,263][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-http/tls.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,263][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-http/ca.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,264][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-http/..data has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,264][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] Directory /usr/share/opensearch/config/tls-http/..2024_10_17_03_31_46.402380427 has insecure file permissions (should be 0700)
[2024-10-17T03:34:26,264][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-http/..2024_10_17_03_31_46.402380427/ca.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,348][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-http/..2024_10_17_03_31_46.402380427/tls.key has insecure file permissions (should be 0600)
[2024-10-17T03:34:26,349][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] File /usr/share/opensearch/config/tls-http/..2024_10_17_03_31_46.402380427/tls.crt has insecure file permissions (should be 0600)
[2024-10-17T03:34:36,355][INFO ][o.o.p.c.c.PluginSettings ] [opensearch-nodes-0] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2024-10-17T03:34:43,963][INFO ][o.o.i.r.ReindexPlugin ] [opensearch-nodes-0] ReindexPlugin reloadSPI called
[2024-10-17T03:34:43,966][INFO ][o.o.i.r.ReindexPlugin ] [opensearch-nodes-0] Unable to find any implementation for RemoteReindexExtension
[2024-10-17T03:34:44,767][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-nodes-0] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2024-10-17T03:34:44,862][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-nodes-0] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2024-10-17T03:34:44,866][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-nodes-0] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2024-10-17T03:34:44,870][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-nodes-0] Loaded scheduler extension: scheduler_geospatial_ip2geo_datasource, index: .scheduler-geospatial-ip2geo-datasource
[2024-10-17T03:34:44,948][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-nodes-0] Loaded scheduler extension: opensearch_sap_job, index: .opensearch-sap--job
[2024-10-17T03:34:45,352][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded module [aggs-matrix-stats]
[2024-10-17T03:34:45,353][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded module [analysis-common]
[2024-10-17T03:34:45,353][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded module [cache-common]
[2024-10-17T03:34:45,353][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded module [geo]
[2024-10-17T03:34:45,353][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded module [ingest-common]
[2024-10-17T03:34:45,354][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded module [ingest-geoip]
[2024-10-17T03:34:45,354][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded module [ingest-user-agent]
[2024-10-17T03:34:45,354][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded module [lang-expression]
[2024-10-17T03:34:45,354][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded module [lang-mustache]
[2024-10-17T03:34:45,354][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded module [lang-painless]
[2024-10-17T03:34:45,354][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded module [mapper-extras]
[2024-10-17T03:34:45,355][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded module [opensearch-dashboards]
[2024-10-17T03:34:45,355][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded module [parent-join]
[2024-10-17T03:34:45,355][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded module [percolator]
[2024-10-17T03:34:45,355][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded module [rank-eval]
[2024-10-17T03:34:45,356][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded module [reindex]
[2024-10-17T03:34:45,356][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded module [repository-url]
[2024-10-17T03:34:45,356][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded module [search-pipeline-common]
[2024-10-17T03:34:45,356][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded module [systemd]
[2024-10-17T03:34:45,357][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded module [transport-netty4]
[2024-10-17T03:34:45,358][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded plugin [opensearch-alerting]
[2024-10-17T03:34:45,359][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded plugin [opensearch-anomaly-detection]
[2024-10-17T03:34:45,359][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded plugin [opensearch-asynchronous-search]
[2024-10-17T03:34:45,359][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded plugin [opensearch-cross-cluster-replication]
[2024-10-17T03:34:45,359][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded plugin [opensearch-custom-codecs]
[2024-10-17T03:34:45,360][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded plugin [opensearch-flow-framework]
[2024-10-17T03:34:45,360][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded plugin [opensearch-geospatial]
[2024-10-17T03:34:45,360][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded plugin [opensearch-index-management]
[2024-10-17T03:34:45,360][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded plugin [opensearch-job-scheduler]
[2024-10-17T03:34:45,447][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded plugin [opensearch-knn]
[2024-10-17T03:34:45,447][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded plugin [opensearch-ml]
[2024-10-17T03:34:45,448][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded plugin [opensearch-neural-search]
[2024-10-17T03:34:45,448][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded plugin [opensearch-notifications]
[2024-10-17T03:34:45,448][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded plugin [opensearch-notifications-core]
[2024-10-17T03:34:45,448][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded plugin [opensearch-observability]
[2024-10-17T03:34:45,449][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded plugin [opensearch-performance-analyzer]
[2024-10-17T03:34:45,449][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded plugin [opensearch-reports-scheduler]
[2024-10-17T03:34:45,449][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded plugin [opensearch-security]
[2024-10-17T03:34:45,449][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded plugin [opensearch-security-analytics]
[2024-10-17T03:34:45,449][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded plugin [opensearch-skills]
[2024-10-17T03:34:45,449][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded plugin [opensearch-sql]
[2024-10-17T03:34:45,450][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded plugin [prometheus-exporter]
[2024-10-17T03:34:45,450][INFO ][o.o.p.PluginsService ] [opensearch-nodes-0] loaded plugin [repository-s3]
[2024-10-17T03:34:46,466][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-nodes-0] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2024-10-17T03:34:46,555][INFO ][o.o.e.ExtensionsManager ] [opensearch-nodes-0] ExtensionsManager initialized
[2024-10-17T03:34:46,753][INFO ][o.a.l.s.MemorySegmentIndexInputProvider] [opensearch-nodes-0] Using MemorySegmentIndexInput with Java 21 or later; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
[2024-10-17T03:34:46,860][INFO ][o.o.e.NodeEnvironment ] [opensearch-nodes-0] using [1] data paths, mounts [[/usr/share/opensearch/data (/dev/vdb1)]], net usable_space [1.8tb], net total_space [1.9tb], types [ext4]
[2024-10-17T03:34:46,861][INFO ][o.o.e.NodeEnvironment ] [opensearch-nodes-0] heap size [1gb], compressed ordinary object pointers [true]
[2024-10-17T03:34:47,270][INFO ][o.o.n.Node ] [opensearch-nodes-0] node name [opensearch-nodes-0], node ID [ienk-pUrT-aWnqiMT8UEoQ], cluster name [opensearch], roles [data]
[2024-10-17T03:35:15,051][DEPRECATION][o.o.d.c.s.Settings ] [opensearch-nodes-0] [index.store.hybrid.mmap.extensions] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2024-10-17T03:35:15,056][DEPRECATION][o.o.d.c.s.Settings ] [opensearch-nodes-0] [cluster.initial_master_nodes] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2024-10-17T03:35:15,057][DEPRECATION][o.o.d.c.s.Settings ] [opensearch-nodes-0] [plugins.security.ssl.transport.enforce_hostname_verification] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2024-10-17T03:35:24,268][INFO ][o.o.n.p.NeuralSearch ] [opensearch-nodes-0] Registering hybrid query phase searcher with feature flag [plugins.neural_search.hybrid_search_disabled]
[2024-10-17T03:35:32,548][WARN ][o.o.s.c.Salt ] [opensearch-nodes-0] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2024-10-17T03:35:33,267][INFO ][o.o.s.a.i.AuditLogImpl ] [opensearch-nodes-0] Message routing enabled: true
[2024-10-17T03:35:33,854][INFO ][o.o.s.f.SecurityFilter ] [opensearch-nodes-0] <NONE> indices are made immutable.
3