Failed to Start Opensearch Dashboards - 3.2.0

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): chrome

Describe the issue: When trying to do status of opensearch dashboards - i get this error message “ failed to start opensearch Dashbaords

Configuration:

Relevant Logs or Screenshots:

[root@xxxxx ~]# sudo systemctl status opensearch-dashboards
× opensearch-dashboards.service - “OpenSearch Dashboards”
Loaded: loaded (/usr/lib/systemd/system/opensearch-dashboards.service; enabled; preset: disabled)
Active: failed (Result: exit-code) since Sun 2025-08-24 08:12:22 CEST; 3min 50s ago
Duration: 139ms
Process: 2340104 ExecStart=/usr/share/opensearch-dashboards/bin/opensearch-dashboards (code=exited, status=1/FAILURE)
Main PID: 2340104 (code=exited, status=1/FAILURE)
CPU: 172ms

Aug 24 08:12:22 xxxx systemd[1]: opensearch-dashboards.service: Scheduled restart job, restart counter is at 5.
Aug 24 08:12:22 Ixxx systemd[1]: Stopped “OpenSearch Dashboards”.
Aug 24 08:12:22 Ixxxx systemd[1]: opensearch-dashboards.service: Start request repeated too quickly.
Aug 24 08:12:22 xxxx systemd[1]: opensearch-dashboards.service: Failed with result ‘exit-code’.
Aug 24 08:12:22 xxx systemd[1]: Failed to start “OpenSearch Dashboards”.
[root@Ixxxxx ~]#

where can i find the logs, why it is unable to start the opensearch dashboards

does your opensearch cluster run without any issues? (where opensearch-dashboards is trying to connect to)

The Opensearch cluster runs, but when i do a curl -X GET https://localhost:9200/_cat/plugins?v -u ‘admin:xxxx’ --insecure
It gives a message - Unauthorized . so not sure what is the issue. i have set the admin password while installing opensearch( Do i need to change it and check it up)

In the Opensearch log messages - It says [2025-08-22T07:09:34,319][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure no such index [.opendistro_security] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS] (index=.opendistro_security)
org.opensearch.index.IndexNotFoundException: no such index [.opendistro_security]

I Checked it , it is been created in the etc/opensearch/openseach-security folder

Is there a need for HTTPS /SSL Certificates for the OSD. I have currently disabled the settings. i am testing the OSD with http. but i have my Open search server has https. Will this create an issue

I have solved the Opensearch cluster issue, it is running perfectly. but the OSD is not working it is still giving the same error - Failed to start Opensearch Dashboards

Currently , i dont get the failed to start opensearch dashboard error.. Currently in my remote machine - when i try to access the OSD- it shows up Opensearch dashboard server is not ready

@ak_amkum19 can you provide your opensearch.yml and opensearch_dashboards.yml files, redact any sensitive details if necessary.

@Anthony when i uncomment the network host, opensearch does’nt start. its very strange. once i comment it up, it works fine

@ak_amkum19 Can you provide the 2 files please, it will be easier to find the issue.

# ======================== OpenSearch Configuration =========================

# 

# NOTE: OpenSearch comes with reasonable defaults for most settings.

# Before you set out to tweak and tune the configuration, make sure you

# understand what are you trying to accomplish and the consequences.

# 

# The primary way of configuring a node is via this file. This template lists

# the most important settings you may want to configure for a production cluster.

# 

# Please consult the documentation for further information on configuration options:

# 

https://www.opensearch.org

# 

# ---------------------------------- Cluster -----------------------------------

# 

# Use a descriptive name for your cluster:

# 

cluster.name: Application-Opensearch

# 

# ------------------------------------ Node ------------------------------------

# 

# Use a descriptive name for the node:

# 

node.name: node-1

# 

# Add custom attributes to the node:

# 

#node.attr.rack: r1

# 

# ----------------------------------- Paths ------------------------------------

# 

# Path to directory where to store the data (separate multiple locations by comma):

# 

path.data: /var/lib/opensearch

# 

# Path to log files:

# 

path.logs: /var/log/opensearch

# 

# ----------------------------------- Memory -----------------------------------

# 

# Lock the memory on startup:

# 

bootstrap.memory_lock: true

# 

# Make sure that the heap size is set to about half the memory available

# on the system and that the owner of the process is allowed to use this

# limit.

# 

# OpenSearch performs poorly when the system is swapping the memory.

# 

# ---------------------------------- Network -----------------------------------

# 

# Set the bind address to a specific IP (IPv4 or IPv6):

# 

# network.host: 0.0.0.0

# 

# Set a custom port for HTTP:

# 

http.port: 9200

# 

# For more information, consult the network module documentation.

# 

# --------------------------------- Discovery ----------------------------------

# 

# Pass an initial list of hosts to perform discovery when this node is started:

# The default list of hosts is \[“127.0.0.1”, “\[::1\]”\]

# 

discovery.seed_hosts: \[“xxxxx”\]

# 

# Bootstrap the cluster using an initial set of cluster-manager-eligible nodes:

# 

cluster.initial_cluster_manager_nodes: \[“node-1”\]

# 

# For more information, consult the discovery and cluster formation module documentation.

# 

# ---------------------------------- Gateway -----------------------------------

# 

# Block initial recovery after a full cluster restart until N nodes are started:

# 

#gateway.recover_after_data_nodes: 3

# 

# For more information, consult the gateway module documentation.

# 

# ---------------------------------- Various -----------------------------------

# 

# Require explicit names when deleting indices:

# 

#action.destructive_requires_name: true

# 

# ---------------------------------- Remote Store -----------------------------------

# Controls whether cluster imposes index creation only with remote store enabled

# cluster.remote_store.enabled: true

# 

# Repository to use for segment upload while enforcing remote store for an index

# node.attr.remote_store.segment.repository: my-repo-1

# 

# Repository to use for translog upload while enforcing remote store for an index

# node.attr.remote_store.translog.repository: my-repo-1

# 

# ---------------------------------- Experimental Features -----------------------------------

# Gates the visibility of the experimental segment replication features until they are production ready.

# 

#opensearch.experimental.feature.segment_replication_experimental.enabled: false

# 

# Gates the functionality of a new parameter to the snapshot restore API

# that allows for creation of a new index type that searches a snapshot

# directly in a remote repository without restoring all index data to disk

# ahead of time.

# 

#opensearch.experimental.feature.searchable_snapshot.enabled: false

# 

# 

# Gates the functionality of enabling extensions to work with OpenSearch.

# This feature enables applications to extend features of OpenSearch outside of

# the core.

# 

#opensearch.experimental.feature.extensions.enabled: false

# 

# 

# Gates the optimization of datetime formatters caching along with change in default datetime formatter

# Once there is no observed impact on performance, this feature flag can be removed.

# 

#opensearch.experimental.optimization.datetime_formatter_caching.enabled: false

######## Start OpenSearch Security Demo Configuration ########

# WARNING: revise all the lines below before you go into production

plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/esnode.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: true
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: /etc/opensearch/esnode.pem
plugins.security.ssl.http.pemkey_filepath: /etc/opensearch/esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.authcz.admin_dn: \[‘CN=kirk,OU=client,O=client,L=test,C=de’\]
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: \[all_access, security_rest_api_access\]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: \[.plugins-ml-agent, .plugins-ml-config, .plugins-ml-connector,
.plugins-ml-controller, .plugins-ml-model-group, .plugins-ml-model, .plugins-ml-task,
.plugins-ml-conversation-meta, .plugins-ml-conversation-interactions, .plugins-ml-memory-meta,
.plugins-ml-memory-message, .plugins-ml-stop-words, .opendistro-alerting-config,
.opendistro-alerting-alert\*, .opendistro-anomaly-results\*, .opendistro-anomaly-detector\*,
.opendistro-anomaly-checkpoints, .opendistro-anomaly-detection-state, .opendistro-reports-*,
.opensearch-notifications-*, .opensearch-notebooks, .opensearch-observability, .ql-datasources,
.opendistro-asynchronous-search-response\*, .replication-metadata-store, .opensearch-knn-models,
.geospatial-ip2geo-data\*, .plugins-flow-framework-config, .plugins-flow-framework-templates,
.plugins-flow-framework-state, .plugins-search-relevance-experiment, .plugins-search-relevance-judgment-cache\]
node.max_local_storage_nodes: 3
######## End OpenSearch Security Demo Configuration ########***`strong text`***
---
# Copyright OpenSearch Contributors
# SPDX-License-Identifier: Apache-2.0

# Description:
# Default configuration for OpenSearch Dashboards

# OpenSearch Dashboards is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the OpenSearch Dashboards server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "xxxxxxx"

# Enables you to specify a path to mount OpenSearch Dashboards at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell OpenSearch Dashboards if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
# server.basePath: ""

# Specifies whether OpenSearch Dashboards should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# server.rewriteBasePath: false

# The maximum payload size in bytes for incoming server requests.
server.maxPayloadBytes: 1048576

# The OpenSearch Dashboards server's name.  This is used for display purposes.
# server.name: "your-hostname"

# The URLs of the OpenSearch instances to use for all your queries.
#opensearch.hosts: ["https://xxxxx:9200"]

# OpenSearch Dashboards uses an index in OpenSearch to store saved searches, visualizations and
# dashboards. OpenSearch Dashboards creates a new index if the index doesn't already exist.
opensearchDashboards.index: ".opensearch_dashboards"

# The default application to load.
opensearchDashboards.defaultAppId: "home"

# Setting for an optimized healthcheck that only uses the local OpenSearch node to do Dashboards healthcheck.
# This settings should be used for large clusters or for clusters with ingest heavy nodes.
# It allows Dashboards to only healthcheck using the local OpenSearch node rather than fan out requests across all nodes.
#
# It requires the user to create an OpenSearch node attribute with the same name as the value used in the setting
# This node attribute should assign all nodes of the same cluster an integer value that increments with each new cluster that is spun up
# e.g. in opensearch.yml file you would set the value to a setting using node.attr.cluster_id:
# Should only be enabled if there is a corresponding node attribute created in your OpenSearch config that matches the value here
# opensearch.optimizedHealthcheckId: "cluster_id"

# If your OpenSearch is protected with basic authentication, these settings provide
# the username and password that the OpenSearch Dashboards server uses to perform maintenance on the OpenSearch Dashboards
# index at startup. Your OpenSearch Dashboards users still need to authenticate with OpenSearch, which
# is proxied through the OpenSearch Dashboards server.
# opensearch.username: "opensearch_dashboards_system"
# opensearch.password: "pass"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the OpenSearch Dashboards server to the browser.
# server.ssl.enabled: false
# server.ssl.certificate: /path/to/your/server.crt
# server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of OpenSearch Dashboards to OpenSearch and are required when
# xpack.security.http.ssl.client_authentication in OpenSearch is set to required.
# opensearch.ssl.certificate: /path/to/your/client.crt
# opensearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your OpenSearch instance.
# opensearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#opensearch.ssl.verificationMode: none

# Time in milliseconds to wait for OpenSearch to respond to pings. Defaults to the value of
# the opensearch.requestTimeout setting.
#opensearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or OpenSearch. This value
# must be a positive integer.
#opensearch.requestTimeout: 30000

# List of OpenSearch Dashboards client-side headers to send to OpenSearch. To send *no* client-side
# headers, set this value to [] (an empty list).
# opensearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to OpenSearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the opensearch.requestHeadersWhitelist configuration.
# opensearch.customHeaders: {}

# Time in milliseconds for OpenSearch to wait for responses from shards. Set to 0 to disable.
# opensearch.shardTimeout: 30000

# Logs queries sent to OpenSearch. Requires logging.verbose set to true.
# opensearch.logQueries: false

# Specifies the path where OpenSearch Dashboards creates the process ID file.
# pid.file: /var/run/opensearchDashboards.pid

# Enables you to specify a file where OpenSearch Dashboards stores log output.
# logging.dest: stdout

# 2.15 Ignore 'ENOSPC' error for logging stream.
# When set to true, the 'ENOSPC' error message will not cause the OpenSearch Dashboards process to crash. Otherwise,
# the original behavior will be maintained. It is disabled by default.
# logging.ignoreEnospcError: false

# Set the value of this setting to true to suppress all logging output.
logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.
logging.quiet: false

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
logging.verbose: true

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
# ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
# i18n.locale: "en"

# Set the allowlist to check input graphite Url. Allowlist is the default check list.
# vis_type_timeline.graphiteAllowedUrls: ['https://www.hostedgraphite.com/UID/ACCESS_KEY/graphite']

# Set the blocklist to check input graphite Url. Blocklist is an IP list.
# Below is an example for reference
# vis_type_timeline.graphiteBlockedIPs: [
#  //Loopback
#  '127.0.0.0/8',
#  '::1/128',
#  //Link-local Address for IPv6
#  'fe80::/10',
#  //Private IP address for IPv4
#  '10.0.0.0/8',
#  '172.16.0.0/12',
#  '192.168.0.0/16',
#  //Unique local address (ULA)
#  'fc00::/7',
#  //Reserved IP address
#  '0.0.0.0/8',
#  '100.64.0.0/10',
#  '192.0.0.0/24',
#  '192.0.2.0/24',
#  '198.18.0.0/15',
#  '192.88.99.0/24',
#  '198.51.100.0/24',
#  '203.0.113.0/24',
#  '224.0.0.0/4',
#  '240.0.0.0/4',
#  '255.255.255.255/32',
#  '::/128',
#  '2001:db8::/32',
#  'ff00::/8',
# ]
# vis_type_timeline.graphiteBlockedIPs: []

# opensearchDashboards.branding:
#   logo:
#     defaultUrl: ""
#     darkModeUrl: ""
#   mark:
#     defaultUrl: ""
#     darkModeUrl: ""
#   loadingLogo:
#     defaultUrl: ""
#     darkModeUrl: ""
#   faviconUrl: ""
#   applicationTitle: ""

# Set the value of this setting to true to capture region blocked warnings and errors
# for your map rendering services.
# map.showRegionBlockedWarning: false%

# Set the value of this setting to false to suppress search usage telemetry
# for reducing the load of OpenSearch cluster.
# data.search.usageTelemetry.enabled: false

# 2.4 renames 'wizard.enabled: false' to 'vis_builder.enabled: false'
# Set the value of this setting to false to disable VisBuilder
# functionality in Visualization.
# vis_builder.enabled: false

# 2.4 New Experimental Feature
# Set the value of this setting to true to enable the experimental multiple data source
# support feature. Use with caution.
# data_source.enabled: false
# Set the value of these settings to customize crypto materials to encryption saved credentials
# in data sources.
# data_source.encryption.wrappingKeyName: 'changeme'
# data_source.encryption.wrappingKeyNamespace: 'changeme'
# data_source.encryption.wrappingKey: [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]

# 2.6 New ML Commons Dashboards Feature
# Set the value of this setting to true to enable the ml commons dashboards
# ml_commons_dashboards.enabled: false

# 2.12 New Experimental Assistant Dashboards Feature
# Set the value of this setting to true to enable the assistant dashboards
# assistant.chat.enabled: false

# 2.13 New Query Assistant Feature
# Set the value of this setting to false to disable the query assistant
# observability.query_assist.enabled: false

# 2.14 Enable Ui Metric Collectors in Usage Collector
# Set the value of this setting to true to enable UI Metric collections
# usageCollection.uiMetric.enabled: false

# 2.18 New Experimental Settings
# Set the value to true to enable
# assistant.alertInsight.enabled: false
# assistant.smartAnomalyDetector.enabled: false
# assistant.text2viz.enabled: false
# queryEnhancements.queryAssist.summary.enabled: false

opensearch.hosts: [https://xxxx:9200]
opensearch.ssl.verificationMode: none
opensearch.username: kibanaserver
opensearch.password: kibanaserver
opensearch.requestHeadersWhitelist: [authorization, securitytenant]

opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: [Private, Global]
opensearch_security.readonly_mode.roles: [kibana_read_only]
# Use this setting if you are running opensearch-dashboards without https
opensearch_security.cookie.secure: false

@Anthony i have added both the yaml files, let me know if there is some issues with the yaml file that is causing the issue

thanks
Amal

What errors are you seeing when you set network.host: 0.0.0.0 on opensearch.

If this is already set and the opensearch cluster is running, are you seeing any errors in opensearch dashboards logs?

Please find the error which i get from the Logs :

[2025-08-25T08:30:08,654][INFO ][o.o.b.BootstrapChecks ] [node-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2025-08-25T08:30:08,658][ERROR][o.o.b.Bootstrap ] [node-1] node validation exception
[1] bootstrap checks failed
[1]: memory locking requested for opensearch process but memory is not locked
[2025-08-25T08:30:08,660][INFO ][o.o.n.Node ] [node-1] stopping …
[2025-08-25T08:30:08,660][INFO ][o.o.s.a.r.AuditMessageRouter] [node-1] Closing AuditMessageRouter
[2025-08-25T08:30:08,661][INFO ][o.o.s.a.s.SinkProvider ] [node-1] Closing InternalOpenSearchSink
[2025-08-25T08:30:08,661][INFO ][o.o.s.a.s.SinkProvider ] [node-1] Closing DebugSink
[2025-08-25T08:30:08,671][INFO ][o.o.n.Node ] [node-1] stopped
[2025-08-25T08:30:08,672][INFO ][o.o.n.Node ] [node-1] closing …
[2025-08-25T08:30:08,677][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Closing AuditLogImpl
[2025-08-25T08:30:08,680][INFO ][o.o.n.Node ] [node-1] closed

********************************************************************************

[2025-08-25T08:30:05,419][ERROR][o.o.p.c.j.GCMetrics ] [node-1] MX bean missing: G1 Concurrent GC
[2025-08-25T08:30:05,422][ERROR][o.o.p.c.o.OSGlobals ] [node-1] Error in reading/parsing clk.tck value: java.lang.NumberFormatException: Cannot parse null string
java.lang.NumberFormatException: Cannot parse null string
at java.base/java.lang.Long.parseUnsignedLong(Long.java:881) ~[?:?]
at java.base/java.lang.Long.parseUnsignedLong(Long.java:1100) ~[?:?]
at org.opensearch.performanceanalyzer.commons.os.OSGlobals.getScClkTckFromConfig(OSGlobals.java:56) ~[performance-analyzer-commons-2.0.0.jar:?]
at org.opensearch.performanceanalyzer.commons.os.OSGlobals.(OSGlobals.java:32) ~[performance-analyzer-commons-2.0.0.jar:?]
at org.opensearch.performanceanalyzer.commons.os.ThreadCPU.(ThreadCPU.java:128) ~[performance-analyzer-commons-2.0.0.jar:?]
at org.opensearch.performanceanalyzer.commons.os.ThreadCPU.(ThreadCPU.java:19) ~[performance-analyzer-commons-2.0.0.jar:?]
at org.opensearch.performanceanalyzer.commons.metrics_generator.linux.LinuxOSMetricsGenerator.getPagingActivityGenerator(LinuxOSMetricsGenerator.java:41) ~[performance-analyzer-commons-2.0.0.jar:?]
at org.opensearch.performanceanalyzer.commons

****************************************************************************

[2025-08-25T08:30:00,230][ERROR][o.o.p.c.c.PluginSettings ] [node-1] Loading config file /etc/opensearch/opensearch-performance-analyzer/performance-analyzer.properties failed with error: org.opensearch.performanceanalyzer.commons.config.ConfigFileException: java.io.FileNotFoundException: /etc/opensearch/opensearch-performance-analyzer/performance-analyzer.properties (No such file or directory). Disabling plugin.
[2025-08-25T08:30:00,231][INFO ][o.o.p.c.c.PluginSettings ] [node-1] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: false, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2025-08-25T08:30:00,403][ERROR][o.o.p.c.c.StatsCollector ] [node-1] Error in loading metadata for folderLocation: /etc/opensearch/opensearch-performance-analyzer/, fileLocation: plugin-stats-metadata
java.io.FileNotFoundException: /etc/opensearch/opensearch-performance-analyzer/plugin-stats-metadata (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method) ~[?:?]

@ak_amkum19 The issue is the memory lock, your OS is not permitting it.

You can try to comment out this: bootstrap.memory_lock: true

If this doesn’t work, you will need allow memory lock for opensearch. Please let me know if the above worked.

i just tried commenting the bootstrap.memory_locl: true and i was able to start the opensearch. but the OSD is still showing up this error :Opensearch Dashboards server is not ready yet

is there a compatibility issue with 3.1.0 Opensearch Cluster and OSD 3.2 version . i have installed the 3..2 OSD Version

i was able to solve the issue by installing the 3.1 version of the OSD, Bcoz my Cluster was at 3.1 version. thanks for all the help

1 Like