Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch 2.12.0
Opensearch Dashboards 2.12.0
Debian 12 x64
Describe the issue:
I just installed OpenSearch on a fresh Debian 12 server and am trying to get it to start.
I ran securityadmin.sh using some minimal yaml files and it succeeded.
Now when I try to connect to it (both cURL and OpenSearch Dashboards), it logs this:
Not yet initialized (you may need to run securityadmin)
I have let it sit for a while and nothing changed.
Configuration:
opensearch.yml
cluster.name: prism-logs-cluster
node.name: main-node
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
network.host: 0.0.0.0
http.port: 9200
discovery.type: single-node
plugins.security.disabled: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: /etc/opensearch/root-ca.pem
plugins.security.ssl.http.pemkey_filepath: /etc/opensearch/root-ca-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/root-ca.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/root-ca-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
action.auto_create_index: true
plugins.query.datasources.encryption.masterkey: 00000000000000000000000000000000000
plugins.security.authcz.admin_dn:
- "CN=admin,OU=admins"
Relevant Logs or Screenshots:
root@opensearch:~# sudo --user=opensearch /usr/share/opensearch/bin/opensearch
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.12.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
Apr 26, 2024 7:19:12 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.12.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2024-04-26T19:19:14,542][INFO ][o.o.n.Node ] [main-node] version[2.12.0], pid[31312], build[deb/2c355ce1a427e4a528778d4054436b5c4b756221/2024-02-20T02:18:31.541484890Z], OS[Linux/6.1.0-20-amd64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.2/21.0.2+13-LTS]
[2024-04-26T19:19:14,549][INFO ][o.o.n.Node ] [main-node] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true]
[2024-04-26T19:19:14,550][INFO ][o.o.n.Node ] [main-node] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-17957404929177023056, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/opensearch, -XX:ErrorFile=/var/log/opensearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/opensearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, --add-modules=jdk.incubator.vector, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadF>, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///etc/opensearch/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/etc/opensearch, -Dopensearch.distribution.type=deb, -Dopensearch.bundled_jdk=true]
[2024-04-26T19:19:18,290][INFO ][o.o.s.s.t.SSLConfig ] [main-node] SSL dual mode is disabled
[2024-04-26T19:19:18,291][INFO ][o.o.s.OpenSearchSecurityPlugin] [main-node] OpenSearch Config path is /etc/opensearch
[2024-04-26T19:19:19,130][INFO ][o.o.s.s.DefaultSecurityKeyStore] [main-node] JVM supports TLSv1.3
[2024-04-26T19:19:19,136][INFO ][o.o.s.s.DefaultSecurityKeyStore] [main-node] Config directory is /etc/opensearch/, from there the key- and truststore files are resolved relatively
[2024-04-26T19:19:20,521][INFO ][o.o.s.s.DefaultSecurityKeyStore] [main-node] TLS Transport Client Provider : JDK
[2024-04-26T19:19:20,528][INFO ][o.o.s.s.DefaultSecurityKeyStore] [main-node] TLS Transport Server Provider : JDK
[2024-04-26T19:19:20,530][INFO ][o.o.s.s.DefaultSecurityKeyStore] [main-node] TLS HTTP Provider : JDK
[2024-04-26T19:19:20,530][INFO ][o.o.s.s.DefaultSecurityKeyStore] [main-node] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2024-04-26T19:19:20,531][INFO ][o.o.s.s.DefaultSecurityKeyStore] [main-node] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2]
[2024-04-26T19:19:20,659][INFO ][o.o.s.OpenSearchSecurityPlugin] [main-node] Clustername: prism-logs-cluster
[2024-04-26T19:19:20,678][WARN ][o.o.s.OpenSearchSecurityPlugin] [main-node] File /etc/opensearch/tenants.yml has insecure file permissions (should be 0600)
[2024-04-26T19:19:20,684][WARN ][o.o.s.OpenSearchSecurityPlugin] [main-node] File /etc/opensearch/action_groups.yml has insecure file permissions (should be 0600)
[2024-04-26T19:19:20,686][WARN ][o.o.s.OpenSearchSecurityPlugin] [main-node] File /etc/opensearch/internal_users.yml has insecure file permissions (should be 0600)
[2024-04-26T19:19:20,694][WARN ][o.o.s.OpenSearchSecurityPlugin] [main-node] File /etc/opensearch/client.crt has insecure file permissions (should be 0600)
[2024-04-26T19:19:20,695][WARN ][o.o.s.OpenSearchSecurityPlugin] [main-node] File /etc/opensearch/roles.yml has insecure file permissions (should be 0600)
[2024-04-26T19:19:20,695][WARN ][o.o.s.OpenSearchSecurityPlugin] [main-node] File /etc/opensearch/roles_mapping.yml has insecure file permissions (should be 0600)
[2024-04-26T19:19:20,696][WARN ][o.o.s.OpenSearchSecurityPlugin] [main-node] File /etc/opensearch/admin.pem has insecure file permissions (should be 0600)
[2024-04-26T19:19:20,697][WARN ][o.o.s.OpenSearchSecurityPlugin] [main-node] File /etc/opensearch/root-ca.srl has insecure file permissions (should be 0600)
[2024-04-26T19:19:20,697][WARN ][o.o.s.OpenSearchSecurityPlugin] [main-node] File /etc/opensearch/config.yml has insecure file permissions (should be 0600)
[2024-04-26T19:19:20,698][WARN ][o.o.s.OpenSearchSecurityPlugin] [main-node] File /etc/opensearch/whitelist.yml has insecure file permissions (should be 0600)
[2024-04-26T19:19:20,700][WARN ][o.o.s.OpenSearchSecurityPlugin] [main-node] File /etc/opensearch/admin.csr has insecure file permissions (should be 0600)
[2024-04-26T19:19:20,702][WARN ][o.o.s.OpenSearchSecurityPlugin] [main-node] File /etc/opensearch/nodes_dn.yml has insecure file permissions (should be 0600)
[2024-04-26T19:19:20,707][WARN ][o.o.s.OpenSearchSecurityPlugin] [main-node] File /etc/opensearch/root-ca.pem has insecure file permissions (should be 0600)
[2024-04-26T19:19:22,682][INFO ][o.o.p.c.c.PluginSettings ] [main-node] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2024-04-26T19:19:24,113][INFO ][o.o.i.r.ReindexPlugin ] [main-node] ReindexPlugin reloadSPI called
[2024-04-26T19:19:24,115][INFO ][o.o.i.r.ReindexPlugin ] [main-node] Unable to find any implementation for RemoteReindexExtension
[2024-04-26T19:19:24,232][INFO ][o.o.j.JobSchedulerPlugin ] [main-node] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2024-04-26T19:19:24,312][INFO ][o.o.j.JobSchedulerPlugin ] [main-node] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2024-04-26T19:19:24,318][INFO ][o.o.j.JobSchedulerPlugin ] [main-node] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2024-04-26T19:19:24,339][INFO ][o.o.j.JobSchedulerPlugin ] [main-node] Loaded scheduler extension: scheduler_geospatial_ip2geo_datasource, index: .scheduler-geospatial-ip2geo-datasource
[2024-04-26T19:19:24,355][INFO ][o.o.j.JobSchedulerPlugin ] [main-node] Loaded scheduler extension: opensearch_sap_job, index: .opensearch-sap--job
[2024-04-26T19:19:24,492][INFO ][o.o.p.PluginsService ] [main-node] loaded module [aggs-matrix-stats]
[2024-04-26T19:19:24,494][INFO ][o.o.p.PluginsService ] [main-node] loaded module [analysis-common]
[2024-04-26T19:19:24,494][INFO ][o.o.p.PluginsService ] [main-node] loaded module [geo]
[2024-04-26T19:19:24,499][INFO ][o.o.p.PluginsService ] [main-node] loaded module [ingest-common]
[2024-04-26T19:19:24,499][INFO ][o.o.p.PluginsService ] [main-node] loaded module [ingest-geoip]
[2024-04-26T19:19:24,502][INFO ][o.o.p.PluginsService ] [main-node] loaded module [ingest-user-agent]
[2024-04-26T19:19:24,503][INFO ][o.o.p.PluginsService ] [main-node] loaded module [lang-expression]
[2024-04-26T19:19:24,507][INFO ][o.o.p.PluginsService ] [main-node] loaded module [lang-mustache]
[2024-04-26T19:19:24,511][INFO ][o.o.p.PluginsService ] [main-node] loaded module [lang-painless]
[2024-04-26T19:19:24,513][INFO ][o.o.p.PluginsService ] [main-node] loaded module [mapper-extras]
[2024-04-26T19:19:24,514][INFO ][o.o.p.PluginsService ] [main-node] loaded module [opensearch-dashboards]
[2024-04-26T19:19:24,514][INFO ][o.o.p.PluginsService ] [main-node] loaded module [parent-join]
[2024-04-26T19:19:24,515][INFO ][o.o.p.PluginsService ] [main-node] loaded module [percolator]
[2024-04-26T19:19:24,516][INFO ][o.o.p.PluginsService ] [main-node] loaded module [rank-eval]
[2024-04-26T19:19:24,523][INFO ][o.o.p.PluginsService ] [main-node] loaded module [reindex]
[2024-04-26T19:19:24,524][INFO ][o.o.p.PluginsService ] [main-node] loaded module [repository-url]
[2024-04-26T19:19:24,525][INFO ][o.o.p.PluginsService ] [main-node] loaded module [search-pipeline-common]
[2024-04-26T19:19:24,526][INFO ][o.o.p.PluginsService ] [main-node] loaded module [systemd]
[2024-04-26T19:19:24,527][INFO ][o.o.p.PluginsService ] [main-node] loaded module [transport-netty4]
[2024-04-26T19:19:24,528][INFO ][o.o.p.PluginsService ] [main-node] loaded plugin [opensearch-alerting]
[2024-04-26T19:19:24,528][INFO ][o.o.p.PluginsService ] [main-node] loaded plugin [opensearch-anomaly-detection]
[2024-04-26T19:19:24,528][INFO ][o.o.p.PluginsService ] [main-node] loaded plugin [opensearch-asynchronous-search]
[2024-04-26T19:19:24,530][INFO ][o.o.p.PluginsService ] [main-node] loaded plugin [opensearch-cross-cluster-replication]
[2024-04-26T19:19:24,531][INFO ][o.o.p.PluginsService ] [main-node] loaded plugin [opensearch-custom-codecs]
[2024-04-26T19:19:24,531][INFO ][o.o.p.PluginsService ] [main-node] loaded plugin [opensearch-flow-framework]
[2024-04-26T19:19:24,532][INFO ][o.o.p.PluginsService ] [main-node] loaded plugin [opensearch-geospatial]
[2024-04-26T19:19:24,533][INFO ][o.o.p.PluginsService ] [main-node] loaded plugin [opensearch-index-management]
[2024-04-26T19:19:24,534][INFO ][o.o.p.PluginsService ] [main-node] loaded plugin [opensearch-job-scheduler]
[2024-04-26T19:19:24,535][INFO ][o.o.p.PluginsService ] [main-node] loaded plugin [opensearch-knn]
[2024-04-26T19:19:24,535][INFO ][o.o.p.PluginsService ] [main-node] loaded plugin [opensearch-ml]
[2024-04-26T19:19:24,537][INFO ][o.o.p.PluginsService ] [main-node] loaded plugin [opensearch-neural-search]
[2024-04-26T19:19:24,537][INFO ][o.o.p.PluginsService ] [main-node] loaded plugin [opensearch-notifications]
[2024-04-26T19:19:24,539][INFO ][o.o.p.PluginsService ] [main-node] loaded plugin [opensearch-notifications-core]
[2024-04-26T19:19:24,540][INFO ][o.o.p.PluginsService ] [main-node] loaded plugin [opensearch-observability]
[2024-04-26T19:19:24,540][INFO ][o.o.p.PluginsService ] [main-node] loaded plugin [opensearch-performance-analyzer]
[2024-04-26T19:19:24,540][INFO ][o.o.p.PluginsService ] [main-node] loaded plugin [opensearch-reports-scheduler]
[2024-04-26T19:19:24,541][INFO ][o.o.p.PluginsService ] [main-node] loaded plugin [opensearch-security]
[2024-04-26T19:19:24,544][INFO ][o.o.p.PluginsService ] [main-node] loaded plugin [opensearch-security-analytics]
[2024-04-26T19:19:24,545][INFO ][o.o.p.PluginsService ] [main-node] loaded plugin [opensearch-skills]
[2024-04-26T19:19:24,545][INFO ][o.o.p.PluginsService ] [main-node] loaded plugin [opensearch-sql]
[2024-04-26T19:19:24,712][INFO ][o.o.s.OpenSearchSecurityPlugin] [main-node] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2024-04-26T19:19:24,724][INFO ][o.o.e.ExtensionsManager ] [main-node] ExtensionsManager initialized
[2024-04-26T19:19:24,787][INFO ][o.o.e.NodeEnvironment ] [main-node] using [1] data paths, mounts [[/ (/dev/vda2)]], net usable_space [32.5gb], net total_space [46.3gb], types [ext4]
[2024-04-26T19:19:24,788][INFO ][o.o.e.NodeEnvironment ] [main-node] heap size [1gb], compressed ordinary object pointers [true]
[2024-04-26T19:19:25,103][INFO ][o.o.n.Node ] [main-node] node name [main-node], node ID [aAs3rkCcQ72TDfBPZ3rfyw], cluster name [prism-logs-cluster], roles [ingest, remote_cluster_client, data, cluster_manager]
[2024-04-26T19:19:33,497][INFO ][o.o.n.p.NeuralSearch ] [main-node] Registering hybrid query phase searcher with feature flag [plugins.neural_search.hybrid_search_disabled]
[2024-04-26T19:19:35,139][WARN ][o.o.s.c.Salt ] [main-node] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2024-04-26T19:19:35,238][ERROR][o.o.s.a.s.SinkProvider ] [main-node] Default endpoint could not be created, auditlog will not work properly.
[2024-04-26T19:19:35,241][WARN ][o.o.s.a.r.AuditMessageRouter] [main-node] No default storage available, audit log may not work properly. Please check configuration.
[2024-04-26T19:19:35,243][INFO ][o.o.s.a.i.AuditLogImpl ] [main-node] Message routing enabled: false
[2024-04-26T19:19:35,309][INFO ][o.o.s.f.SecurityFilter ] [main-node] <NONE> indices are made immutable.
[2024-04-26T19:19:36,345][INFO ][o.o.a.b.ADCircuitBreakerService] [main-node] Registered memory breaker.
[2024-04-26T19:19:37,486][INFO ][o.o.m.b.MLCircuitBreakerService] [main-node] Registered ML memory breaker.
[2024-04-26T19:19:37,492][INFO ][o.o.m.b.MLCircuitBreakerService] [main-node] Registered ML disk breaker.
[2024-04-26T19:19:37,493][INFO ][o.o.m.b.MLCircuitBreakerService] [main-node] Registered ML native memory breaker.
[2024-04-26T19:19:37,840][INFO ][o.r.Reflections ] [main-node] Reflections took 184 ms to scan 1 urls, producing 21 keys and 59 values
[2024-04-26T19:19:40,127][INFO ][o.o.t.NettyAllocator ] [main-node] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
[2024-04-26T19:19:40,140][INFO ][o.o.s.s.t.SSLConfig ] [main-node] SSL dual mode is disabled
[2024-04-26T19:19:40,382][INFO ][o.o.d.DiscoveryModule ] [main-node] using discovery type [single-node] and seed hosts providers [settings]
[2024-04-26T19:19:42,205][WARN ][o.o.g.DanglingIndicesState] [main-node] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2024-04-26T19:19:44,343][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [main-node] PerformanceAnalyzer Enabled: false
[2024-04-26T19:19:44,455][INFO ][o.o.n.Node ] [main-node] initialized
[2024-04-26T19:19:44,455][INFO ][o.o.n.Node ] [main-node] starting ...
[2024-04-26T19:19:44,549][INFO ][o.o.s.l.BuiltinLogTypeLoader] [main-node] Loaded [windows_logtype.json] log type
[2024-04-26T19:19:44,556][INFO ][o.o.s.l.BuiltinLogTypeLoader] [main-node] Loaded [waf_logtype.json] log type
[2024-04-26T19:19:44,558][INFO ][o.o.s.l.BuiltinLogTypeLoader] [main-node] Loaded [vpcflow_logtype.json] log type
[2024-04-26T19:19:44,559][INFO ][o.o.s.l.BuiltinLogTypeLoader] [main-node] Loaded [test_windows_logtype.json] log type
[2024-04-26T19:19:44,564][INFO ][o.o.s.l.BuiltinLogTypeLoader] [main-node] Loaded [s3_logtype.json] log type
[2024-04-26T19:19:44,565][INFO ][o.o.s.l.BuiltinLogTypeLoader] [main-node] Loaded [others_web_logtype.json] log type
[2024-04-26T19:19:44,566][INFO ][o.o.s.l.BuiltinLogTypeLoader] [main-node] Loaded [others_proxy_logtype.json] log type
[2024-04-26T19:19:44,571][INFO ][o.o.s.l.BuiltinLogTypeLoader] [main-node] Loaded [others_macos_logtype.json] log type
[2024-04-26T19:19:44,572][INFO ][o.o.s.l.BuiltinLogTypeLoader] [main-node] Loaded [others_compliance_logtype.json] log type
[2024-04-26T19:19:44,573][INFO ][o.o.s.l.BuiltinLogTypeLoader] [main-node] Loaded [others_cloud_logtype.json] log type
[2024-04-26T19:19:44,574][INFO ][o.o.s.l.BuiltinLogTypeLoader] [main-node] Loaded [others_apt_logtype.json] log type
[2024-04-26T19:19:44,575][INFO ][o.o.s.l.BuiltinLogTypeLoader] [main-node] Loaded [others_application_logtype.json] log type
[2024-04-26T19:19:44,576][INFO ][o.o.s.l.BuiltinLogTypeLoader] [main-node] Loaded [okta_logtype.json] log type
[2024-04-26T19:19:44,584][INFO ][o.o.s.l.BuiltinLogTypeLoader] [main-node] Loaded [network_logtype.json] log type
[2024-04-26T19:19:44,584][INFO ][o.o.s.l.BuiltinLogTypeLoader] [main-node] Loaded [netflow_logtype.json] log type
[2024-04-26T19:19:44,585][INFO ][o.o.s.l.BuiltinLogTypeLoader] [main-node] Loaded [m365_logtype.json] log type
[2024-04-26T19:19:44,586][INFO ][o.o.s.l.BuiltinLogTypeLoader] [main-node] Loaded [linux_logtype.json] log type
[2024-04-26T19:19:44,589][INFO ][o.o.s.l.BuiltinLogTypeLoader] [main-node] Loaded [gworkspace_logtype.json] log type
[2024-04-26T19:19:44,591][INFO ][o.o.s.l.BuiltinLogTypeLoader] [main-node] Loaded [github_logtype.json] log type
[2024-04-26T19:19:44,592][INFO ][o.o.s.l.BuiltinLogTypeLoader] [main-node] Loaded [dns_logtype.json] log type
[2024-04-26T19:19:44,596][INFO ][o.o.s.l.BuiltinLogTypeLoader] [main-node] Loaded [cloudtrail_logtype.json] log type
[2024-04-26T19:19:44,599][INFO ][o.o.s.l.BuiltinLogTypeLoader] [main-node] Loaded [azure_logtype.json] log type
[2024-04-26T19:19:44,599][INFO ][o.o.s.l.BuiltinLogTypeLoader] [main-node] Loaded [apache_access_logtype.json] log type
[2024-04-26T19:19:44,605][INFO ][o.o.s.l.BuiltinLogTypeLoader] [main-node] Loaded [ad_ldap_logtype.json] log type
[2024-04-26T19:19:44,905][INFO ][o.o.t.TransportService ] [main-node] publish_address {140.82.2.192:9300}, bound_addresses {[::]:9300}
[2024-04-26T19:19:44,909][INFO ][o.o.t.TransportService ] [main-node] Remote clusters initialized successfully.
[2024-04-26T19:19:45,349][INFO ][o.o.c.c.Coordinator ] [main-node] cluster UUID [sqZxvnteR-mvVsONnVpoEg]
[2024-04-26T19:19:45,557][INFO ][o.o.c.s.MasterService ] [main-node] elected-as-cluster-manager ([1] nodes joined)[{main-node}{aAs3rkCcQ72TDfBPZ3rfyw}{k-6mgKrLRLiGcnX1zP5vKA}{140.82.2.192}{140.82.2.192:9300}{dimr}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 14, version: 86, delta: cluster-manager node changed {previous [], current [{main-node}{aAs3rkCcQ72TDfBPZ3rfyw}{k-6mgKrLRLiGcnX1zP5vKA}{140.82.2.192}{140.82.2.192:9300}{dimr}{shard_indexing_pressure_enabled=true}]}
[2024-04-26T19:19:45,655][INFO ][o.o.c.s.ClusterApplierService] [main-node] cluster-manager node changed {previous [], current [{main-node}{aAs3rkCcQ72TDfBPZ3rfyw}{k-6mgKrLRLiGcnX1zP5vKA}{140.82.2.192}{140.82.2.192:9300}{dimr}{shard_indexing_pressure_enabled=true}]}, term: 14, version: 86, reason: Publication{term=14, version=86}
[2024-04-26T19:19:45,681][INFO ][o.o.a.c.ADClusterEventListener] [main-node] Cluster is not recovered yet.
[2024-04-26T19:19:45,699][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [main-node] Detected cluster change event for destination migration
[2024-04-26T19:19:45,774][INFO ][o.o.i.i.ManagedIndexCoordinator] [main-node] Cache cluster manager node onClusterManager time: 1714159185774
[2024-04-26T19:19:45,802][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [main-node] Config override setting update called with empty string. Ignoring.
[2024-04-26T19:19:45,841][INFO ][o.o.d.PeerFinder ] [main-node] setting findPeersInterval to [1s] as node commission status = [true] for local node [{main-node}{aAs3rkCcQ72TDfBPZ3rfyw}{k-6mgKrLRLiGcnX1zP5vKA}{140.82.2.192}{140.82.2.192:9300}{dimr}{shard_indexing_pressure_enabled=true}]
[2024-04-26T19:19:45,852][INFO ][o.o.h.AbstractHttpServerTransport] [main-node] publish_address {140.82.2.192:9200}, bound_addresses {[::]:9200}
[2024-04-26T19:19:45,859][INFO ][o.o.n.Node ] [main-node] started
[2024-04-26T19:19:45,863][INFO ][o.o.s.OpenSearchSecurityPlugin] [main-node] Node started
[2024-04-26T19:19:45,865][INFO ][o.o.s.c.ConfigurationRepository] [main-node] Will not attempt to create index .opendistro_security and default configs if they are absent. Use securityadmin to initialize cluster
[2024-04-26T19:19:45,876][INFO ][o.o.s.OpenSearchSecurityPlugin] [main-node] 0 OpenSearch Security modules loaded so far: []
[2024-04-26T19:19:45,875][INFO ][o.o.s.c.ConfigurationRepository] [main-node] Background init thread started. Install default config?: false
[2024-04-26T19:19:45,879][INFO ][o.o.s.c.ConfigurationRepository] [main-node] Wait for cluster to be available ...
[2024-04-26T19:19:46,015][INFO ][o.o.c.s.ClusterSettings ] [main-node] updating [plugins.index_state_management.template_migration.control] from [0] to [-1]
[2024-04-26T19:19:46,020][INFO ][o.o.a.c.HashRing ] [main-node] Node added: [aAs3rkCcQ72TDfBPZ3rfyw]
[2024-04-26T19:19:46,029][INFO ][o.o.a.c.HashRing ] [main-node] Add data node to AD version hash ring: aAs3rkCcQ72TDfBPZ3rfyw
[2024-04-26T19:19:46,039][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [main-node] Detected cluster change event for destination migration
[2024-04-26T19:19:46,044][INFO ][o.o.a.c.HashRing ] [main-node] All nodes with known AD version: {aAs3rkCcQ72TDfBPZ3rfyw=ADNodeInfo{version=2.12.0, isEligibleDataNode=true}}
[2024-04-26T19:19:46,045][INFO ][o.o.a.c.HashRing ] [main-node] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 0
[2024-04-26T19:19:46,045][INFO ][o.o.a.c.HashRing ] [main-node] Build AD version hash ring successfully
[2024-04-26T19:19:46,047][INFO ][o.o.a.c.ADDataMigrator ] [main-node] Start migrating AD data
[2024-04-26T19:19:46,048][INFO ][o.o.a.c.ADDataMigrator ] [main-node] AD job index doesn't exist, no need to migrate
[2024-04-26T19:19:46,050][INFO ][o.o.a.c.ADClusterEventListener] [main-node] Init AD version hash ring successfully
[2024-04-26T19:19:46,104][INFO ][o.o.g.GatewayService ] [main-node] recovered [4] indices into cluster_state
[2024-04-26T19:19:46,108][INFO ][o.o.s.l.LogTypeService ] [main-node] Loading builtin types!
[2024-04-26T19:19:46,120][INFO ][o.o.s.l.LogTypeService ] [main-node] Indexing [429] fieldMappingDocs from logTypes: 24
[2024-04-26T19:19:46,196][WARN ][o.o.o.i.ObservabilityIndex] [main-node] message: index [.opensearch-observability/yUOca-2iSo6YL2KQ6WDrRQ] already exists
[2024-04-26T19:19:46,219][WARN ][o.o.s.SecurityAnalyticsPlugin] [main-node] Failed to initialize LogType config index and builtin log types
[2024-04-26T19:19:46,402][INFO ][o.o.p.PluginsService ] [main-node] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/_RT6rbyYRCq-2zv4oqTAaQ]
[2024-04-26T19:19:46,778][INFO ][o.o.p.PluginsService ] [main-node] PluginService:onIndexModule index:[.opendistro_security/5bA5LfqnQwu_dcozo7a2DA]
[2024-04-26T19:19:46,812][INFO ][o.o.s.l.LogTypeService ] [main-node] Loading builtin types!
[2024-04-26T19:19:46,813][INFO ][o.o.s.l.LogTypeService ] [main-node] Indexing [429] fieldMappingDocs from logTypes: 24
[2024-04-26T19:19:46,820][INFO ][o.o.p.PluginsService ] [main-node] PluginService:onIndexModule index:[.opensearch-observability/yUOca-2iSo6YL2KQ6WDrRQ]
[2024-04-26T19:19:46,824][INFO ][o.o.s.l.LogTypeService ] [main-node] Loading builtin types!
[2024-04-26T19:19:46,827][INFO ][o.o.s.l.LogTypeService ] [main-node] Indexing [429] fieldMappingDocs from logTypes: 24
[2024-04-26T19:19:46,836][INFO ][o.o.s.i.DetectorIndexManagementService] [main-node] info deleteOldIndices
[2024-04-26T19:19:46,860][INFO ][o.o.s.i.DetectorIndexManagementService] [main-node] No Old Correlation Indices to delete
[2024-04-26T19:19:46,890][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [main-node] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@66915b44] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-04-26T19:19:46,899][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [main-node] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@66915b44] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-04-26T19:19:46,900][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [main-node] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@66915b44] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-04-26T19:19:46,903][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [main-node] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@66915b44] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-04-26T19:19:46,897][INFO ][o.o.p.PluginsService ] [main-node] PluginService:onIndexModule index:[.plugins-ml-config/-YTeff8WSJaKF-A5Ina_QQ]
[2024-04-26T19:19:46,905][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [main-node] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@66915b44] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-04-26T19:19:46,907][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [main-node] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@66915b44] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-04-26T19:19:46,909][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [main-node] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@66915b44] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-04-26T19:19:46,909][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [main-node] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@66915b44] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-04-26T19:19:46,915][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [main-node] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@66915b44] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-04-26T19:19:46,917][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [main-node] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@66915b44] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-04-26T19:19:47,111][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [main-node] Detected cluster change event for destination migration
[2024-04-26T19:19:47,462][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [main-node] Detected cluster change event for destination migration
[2024-04-26T19:19:47,557][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [main-node] Detected cluster change event for destination migration
[2024-04-26T19:19:47,563][INFO ][o.o.c.r.a.AllocationService] [main-node] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.opendistro_security][0]]]).
[2024-04-26T19:19:47,594][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [main-node] Detected cluster change event for destination migration
[2024-04-26T19:19:55,813][INFO ][o.o.m.a.MLModelAutoReDeployer] [main-node] Index not found, not performing auto reloading!
[2024-04-26T19:19:55,816][INFO ][o.o.m.c.MLCommonsClusterManagerEventListener] [main-node] Starting ML sync up job...
[2024-04-26T19:20:00,644][INFO ][stdout ] [main-node] [FINE] No subscribers registered for event class org.opensearch.security.securityconf.DynamicConfigFactory$NodesDnModelImpl
[2024-04-26T19:20:00,646][INFO ][stdout ] [main-node] [FINE] No subscribers registered for event class org.greenrobot.eventbus.NoSubscriberEvent
[2024-04-26T19:20:00,647][INFO ][o.o.s.c.ConfigurationRepository] [main-node] Hot-reloading of audit configuration is disabled. Using configuration with defaults from opensearch settings. Populate the configuration in index using audit.yml or securityadmin to enable it.
[2024-04-26T19:20:00,648][INFO ][o.o.s.a.i.AuditLogImpl ] [main-node] Auditing on REST API is enabled.
[2024-04-26T19:20:00,648][INFO ][o.o.s.a.i.AuditLogImpl ] [main-node] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing.
[2024-04-26T19:20:00,653][INFO ][o.o.s.a.i.AuditLogImpl ] [main-node] Auditing on Transport API is enabled.
[2024-04-26T19:20:00,653][INFO ][o.o.s.a.i.AuditLogImpl ] [main-node] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing.
[2024-04-26T19:20:00,653][INFO ][o.o.s.a.i.AuditLogImpl ] [main-node] Auditing of request body is enabled.
[2024-04-26T19:20:00,654][INFO ][o.o.s.a.i.AuditLogImpl ] [main-node] Bulk requests resolution is disabled during request auditing.
[2024-04-26T19:20:00,654][INFO ][o.o.s.a.i.AuditLogImpl ] [main-node] Index resolution is enabled during request auditing.
[2024-04-26T19:20:00,654][INFO ][o.o.s.a.i.AuditLogImpl ] [main-node] Sensitive headers auditing is enabled.
[2024-04-26T19:20:00,657][INFO ][o.o.s.a.i.AuditLogImpl ] [main-node] Auditing requests from kibanaserver users is disabled.
[2024-04-26T19:20:00,659][INFO ][o.o.s.a.i.AuditLogImpl ] [main-node] Auditing request headers <NONE> is disabled.
[2024-04-26T19:20:00,662][INFO ][o.o.s.a.i.AuditLogImpl ] [main-node] Auditing of external configuration is disabled.
[2024-04-26T19:20:00,662][INFO ][o.o.s.a.i.AuditLogImpl ] [main-node] Auditing of internal configuration is disabled.
[2024-04-26T19:20:00,663][INFO ][o.o.s.a.i.AuditLogImpl ] [main-node] Auditing only metadata information for read request is disabled.
[2024-04-26T19:20:00,667][INFO ][o.o.s.a.i.AuditLogImpl ] [main-node] Auditing will watch {} for read requests.
[2024-04-26T19:20:00,668][INFO ][o.o.s.a.i.AuditLogImpl ] [main-node] Auditing read operation requests from kibanaserver users is disabled.
[2024-04-26T19:20:00,668][INFO ][o.o.s.a.i.AuditLogImpl ] [main-node] Auditing only metadata information for write request is disabled.
[2024-04-26T19:20:00,669][INFO ][o.o.s.a.i.AuditLogImpl ] [main-node] Auditing diffs for write requests is disabled.
[2024-04-26T19:20:00,670][INFO ][o.o.s.a.i.AuditLogImpl ] [main-node] Auditing write operation requests from kibanaserver users is disabled.
[2024-04-26T19:20:00,670][INFO ][o.o.s.a.i.AuditLogImpl ] [main-node] Auditing will watch <NONE> for write requests.
[2024-04-26T19:20:00,670][INFO ][o.o.s.a.i.AuditLogImpl ] [main-node] .opendistro_security is used as internal security index.
[2024-04-26T19:20:00,671][INFO ][o.o.s.a.i.AuditLogImpl ] [main-node] Internal index used for posting audit logs is null
[2024-04-26T19:20:00,671][INFO ][o.o.s.c.ConfigurationRepository] [main-node] Node 'main-node' initialized
[2024-04-26T19:20:05,866][INFO ][o.o.m.c.MLSyncUpCron ] [main-node] ML configuration already initialized, no action needed
[2024-04-26T19:20:30,425][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:20:30,535][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:20:30,546][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:20:30,557][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:20:32,492][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:20:32,512][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:20:32,525][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:20:32,536][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:20:34,985][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:20:34,999][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:20:35,009][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:20:35,019][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:20:37,486][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:20:37,491][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:20:37,503][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:20:37,516][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:20:39,988][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:20:39,995][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:20:40,006][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:20:40,011][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:20:45,777][INFO ][o.o.i.i.ManagedIndexCoordinator] [main-node] Performing move cluster state metadata.
[2024-04-26T19:20:45,779][INFO ][o.o.i.i.MetadataService ] [main-node] ISM config index not exist, so we cancel the metadata migration job.
[2024-04-26T19:21:45,780][INFO ][o.o.i.i.ManagedIndexCoordinator] [main-node] Cancel background move metadata process.
[2024-04-26T19:21:45,782][INFO ][o.o.i.i.ManagedIndexCoordinator] [main-node] Performing move cluster state metadata.
[2024-04-26T19:21:45,783][INFO ][o.o.i.i.MetadataService ] [main-node] Move metadata has finished.
[2024-04-26T19:22:05,509][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:22:05,552][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:22:05,558][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:22:05,570][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:22:07,944][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:22:07,951][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)
[2024-04-26T19:22:07,958][ERROR][o.o.s.a.BackendRegistry ] [main-node] Not yet initialized (you may need to run securityadmin)