Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSeach version: 1.3.15
OpenSearch Dashboard version: 1.3.15 [Customization: recreated docker image with repository s3 & prometheus-exporter i.e. =
RUN bin/opensearch-plugin install --batch repository-s3
RUN bin/opensearch-plugin install --batch https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/download/1.3.15.0/prometheus-exporter-1.3.15.0.zip
Platform: k8s
Describe the issue:
- After dpeloying OpenSearch dashboard in k8s, getting errors
- OpenSearch is UP in Single node env
- If I try to start OpenSearch dashboard with
env:
- name: DISABLE_INSTALL_DEMO_CONFIG
value: ‘true’
- name: DISABLE_SECURITY_DASHBOARDS_PLUGIN
value: ‘true’
then getting security related errors.
Configuration:
> apiVersion: apps/v1
> kind: Deployment
> metadata:
> name: gmsp-es-logging-dashboards
> namespace: gcs-logging-poc
> labels:
> app: gmsp-es-logging
> spec:
> replicas: 1
> selector:
> matchLabels:
> app: gmsp-es-logging-dashboards
> template:
> metadata:
> labels:
> app: gmsp-es-logging-dashboards
> spec:
> containers:
> - name: gmsp-es-logging-dashboards
> image: docker-repo.abc.net/opensearch-dashboards:1.3.15
> resources:
> limits:
> cpu: 2000m
> memory: 2Gi
> requests:
> cpu: 2000m
> memory: 2Gi
> ports:
> - containerPort: 5601
> env:
> - name: DISABLE_INSTALL_DEMO_CONFIG
> value: 'false'
> - name: DISABLE_SECURITY_DASHBOARDS_PLUGIN
> value: 'false'
> - name: OPENSEARCH_HOSTS
> value: '["http://gmsp-es-logging-headless:9200"]'
Relevant Logs or Screenshots:
k logs gmsp-es-logging-dashboards-957c7557c-dmz7r
> Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
> OpenSearch Security Demo Installer
> ** Warning: Do not use on production or public reachable systems **
> Basedir: /usr/share/opensearch
> OpenSearch install type: rpm/deb on NAME="Amazon Linux"
> OpenSearch config dir: /usr/share/opensearch/config
> OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
> OpenSearch bin dir: /usr/share/opensearch/bin
> OpenSearch plugins dir: /usr/share/opensearch/plugins
> OpenSearch lib dir: /usr/share/opensearch/lib
> Detected OpenSearch Version: x-content-1.3.15
> Detected OpenSearch Security Version: 1.3.15.0
>
> ### Success
> ### Execute this script now on all your nodes and then start all nodes
> ### OpenSearch Security will be automatically initialized.
> ### If you like to change the runtime configuration
> ### change the files in ../securityconfig and execute:
> "/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh" -cd "/usr/share/opensearch/plugins/opensearch-security/securityconfig" -icl -key "/usr/share/opensearch/config/kirk-key.pem" -cert "/usr/share/opensearch/config/kirk.pem" -cacert "/usr/share/opensearch/config/root-ca.pem" -nhnv
> ### or run ./securityadmin_demo.sh
> ### To use the Security Plugin ConfigurationGUI
> ### To access your secured cluster open https://<hostname>:<HTTP port> and log in with admin/admin.
> ### (Ignore the SSL certificate warning because we installed self-signed demo certificates)
> Enabling OpenSearch Security Plugin
> Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
> [2024-05-06T17:20:31,342][INFO ][o.o.n.Node ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] version[1.3.15], pid[108], build[tar/f1842d770471ab4a6e496e5b1e14c14478f059f1/2024-03-04T18:40:09.133023Z], OS[Linux/5.15.0-102-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/11.0.22/11.0.22+7]
> [2024-05-06T17:20:31,343][INFO ][o.o.n.Node ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] JVM home [/usr/share/opensearch/jdk], using bundled JDK [true]
> [2024-05-06T17:20:31,344][INFO ][o.o.n.Node ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-2355836678311344998, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
> [2024-05-06T17:20:32,415][INFO ][o.o.p.p.PrometheusExporterPlugin] [gmsp-es-logging-dashboards-957c7557c-dmz7r] starting Prometheus exporter plugin
> [2024-05-06T17:20:32,610][WARN ][stderr ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> [2024-05-06T17:20:32,610][WARN ][stderr ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] SLF4J: Defaulting to no-operation (NOP) logger implementation
> [2024-05-06T17:20:32,610][WARN ][stderr ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
> [2024-05-06T17:20:32,624][INFO ][o.o.s.s.t.SSLConfig ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] SSL dual mode is disabled
> [2024-05-06T17:20:32,624][INFO ][o.o.s.OpenSearchSecurityPlugin] [gmsp-es-logging-dashboards-957c7557c-dmz7r] OpenSearch Config path is /usr/share/opensearch/config
> [2024-05-06T17:20:32,871][INFO ][o.o.s.s.DefaultSecurityKeyStore] [gmsp-es-logging-dashboards-957c7557c-dmz7r] JVM supports TLSv1.3
> [2024-05-06T17:20:32,872][INFO ][o.o.s.s.DefaultSecurityKeyStore] [gmsp-es-logging-dashboards-957c7557c-dmz7r] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively
> [2024-05-06T17:20:33,319][INFO ][o.o.s.s.DefaultSecurityKeyStore] [gmsp-es-logging-dashboards-957c7557c-dmz7r] TLS Transport Client Provider : JDK
> [2024-05-06T17:20:33,319][INFO ][o.o.s.s.DefaultSecurityKeyStore] [gmsp-es-logging-dashboards-957c7557c-dmz7r] TLS Transport Server Provider : JDK
> [2024-05-06T17:20:33,319][INFO ][o.o.s.s.DefaultSecurityKeyStore] [gmsp-es-logging-dashboards-957c7557c-dmz7r] TLS HTTP Provider : JDK
> [2024-05-06T17:20:33,319][INFO ][o.o.s.s.DefaultSecurityKeyStore] [gmsp-es-logging-dashboards-957c7557c-dmz7r] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
> [2024-05-06T17:20:33,319][INFO ][o.o.s.s.DefaultSecurityKeyStore] [gmsp-es-logging-dashboards-957c7557c-dmz7r] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2]
> [2024-05-06T17:20:33,520][INFO ][o.o.s.OpenSearchSecurityPlugin] [gmsp-es-logging-dashboards-957c7557c-dmz7r] Clustername: docker-cluster
> [2024-05-06T17:20:33,525][WARN ][o.o.s.OpenSearchSecurityPlugin] [gmsp-es-logging-dashboards-957c7557c-dmz7r] Directory /usr/share/opensearch/config has insecure file permissions (should be 0700)
> [2024-05-06T17:20:34,325][INFO ][o.o.p.c.PluginSettings ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
> [2024-05-06T17:20:34,655][INFO ][o.o.i.r.ReindexPlugin ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] ReindexPlugin reloadSPI called
> [2024-05-06T17:20:34,656][INFO ][o.o.i.r.ReindexPlugin ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] Unable to find any implementation for RemoteReindexExtension
> [2024-05-06T17:20:34,701][INFO ][o.o.j.JobSchedulerPlugin ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
> [2024-05-06T17:20:34,705][INFO ][o.o.j.JobSchedulerPlugin ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
> [2024-05-06T17:20:34,706][INFO ][o.o.j.JobSchedulerPlugin ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
> [2024-05-06T17:20:34,709][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded module [aggs-matrix-stats]
> [2024-05-06T17:20:34,709][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded module [analysis-common]
> [2024-05-06T17:20:34,709][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded module [geo]
> [2024-05-06T17:20:34,709][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded module [ingest-common]
> [2024-05-06T17:20:34,709][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded module [ingest-geoip]
> [2024-05-06T17:20:34,709][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded module [ingest-user-agent]
> [2024-05-06T17:20:34,710][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded module [lang-expression]
> [2024-05-06T17:20:34,710][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded module [lang-mustache]
> [2024-05-06T17:20:34,710][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded module [lang-painless]
> [2024-05-06T17:20:34,710][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded module [mapper-extras]
> [2024-05-06T17:20:34,710][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded module [opensearch-dashboards]
> [2024-05-06T17:20:34,710][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded module [parent-join]
> [2024-05-06T17:20:34,710][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded module [percolator]
> [2024-05-06T17:20:34,710][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded module [rank-eval]
> [2024-05-06T17:20:34,710][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded module [reindex]
> [2024-05-06T17:20:34,710][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded module [repository-url]
> [2024-05-06T17:20:34,710][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded module [transport-netty4]
> [2024-05-06T17:20:34,711][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded plugin [opensearch-alerting]
> [2024-05-06T17:20:34,711][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded plugin [opensearch-anomaly-detection]
> [2024-05-06T17:20:34,711][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded plugin [opensearch-asynchronous-search]
> [2024-05-06T17:20:34,711][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded plugin [opensearch-cross-cluster-replication]
> [2024-05-06T17:20:34,711][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded plugin [opensearch-index-management]
> [2024-05-06T17:20:34,711][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded plugin [opensearch-job-scheduler]
> [2024-05-06T17:20:34,711][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded plugin [opensearch-knn]
> [2024-05-06T17:20:34,711][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded plugin [opensearch-ml]
> [2024-05-06T17:20:34,711][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded plugin [opensearch-observability]
> [2024-05-06T17:20:34,711][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded plugin [opensearch-performance-analyzer]
> [2024-05-06T17:20:34,711][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded plugin [opensearch-reports-scheduler]
> [2024-05-06T17:20:34,711][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded plugin [opensearch-security]
> [2024-05-06T17:20:34,711][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded plugin [opensearch-sql]
> [2024-05-06T17:20:34,712][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded plugin [prometheus-exporter]
> [2024-05-06T17:20:34,712][INFO ][o.o.p.PluginsService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] loaded plugin [repository-s3]
> [2024-05-06T17:20:34,727][INFO ][o.o.s.OpenSearchSecurityPlugin] [gmsp-es-logging-dashboards-957c7557c-dmz7r] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
> [2024-05-06T17:20:34,734][DEPRECATION][o.o.d.c.s.Settings ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] [node.max_local_storage_nodes] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
> [2024-05-06T17:20:34,741][INFO ][o.o.e.NodeEnvironment ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] using [1] data paths, mounts [[/ (overlay)]], net usable_space [610gb], net total_space [620.1gb], types [overlay]
> [2024-05-06T17:20:34,741][INFO ][o.o.e.NodeEnvironment ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] heap size [1gb], compressed ordinary object pointers [true]
> [2024-05-06T17:20:34,771][INFO ][o.o.n.Node ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] node name [gmsp-es-logging-dashboards-957c7557c-dmz7r], node ID [baFSf302Qsu5ZgtrYvZWPg], cluster name [docker-cluster], roles [master, remote_cluster_client, data, ingest]
> [2024-05-06T17:20:38,234][WARN ][o.o.s.c.Salt ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
> [2024-05-06T17:20:38,254][INFO ][o.o.s.a.i.AuditLogImpl ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] Message routing enabled: true
> [2024-05-06T17:20:38,308][INFO ][o.o.s.f.SecurityFilter ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] <NONE> indices are made immutable.
> [2024-05-06T17:20:38,645][INFO ][o.o.a.b.ADCircuitBreakerService] [gmsp-es-logging-dashboards-957c7557c-dmz7r] Registered memory breaker.
> [2024-05-06T17:20:39,007][INFO ][o.o.m.c.b.MLCircuitBreakerService] [gmsp-es-logging-dashboards-957c7557c-dmz7r] Registered ML memory breaker.
> [2024-05-06T17:20:39,419][INFO ][o.o.t.NettyAllocator ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
> [2024-05-06T17:20:39,422][INFO ][o.o.s.s.t.SSLConfig ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] SSL dual mode is disabled
> [2024-05-06T17:20:39,513][INFO ][o.o.d.DiscoveryModule ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] using discovery type [zen] and seed hosts providers [settings]
> [2024-05-06T17:20:39,973][WARN ][o.o.g.DanglingIndicesState] [gmsp-es-logging-dashboards-957c7557c-dmz7r] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
> [2024-05-06T17:20:40,363][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [gmsp-es-logging-dashboards-957c7557c-dmz7r] PerformanceAnalyzer Enabled: false
> [2024-05-06T17:20:40,388][INFO ][o.o.n.Node ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] initialized
> [2024-05-06T17:20:40,389][INFO ][o.o.n.Node ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] starting ...
> [2024-05-06T17:20:40,519][INFO ][o.o.t.TransportService ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] publish_address {10.219.143.38:9300}, bound_addresses {[::]:9300}
> [2024-05-06T17:20:40,671][INFO ][o.o.b.BootstrapChecks ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] bound or publishing to a non-loopback address, enforcing bootstrap checks
> ERROR: [1] bootstrap checks failed
> [1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
> ERROR: OpenSearch did not exit normally - check the logs at /usr/share/opensearch/logs/docker-cluster.log
> [2024-05-06T17:20:40,680][INFO ][o.o.n.Node ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] stopping ...
> [2024-05-06T17:20:40,680][INFO ][o.o.s.a.r.AuditMessageRouter] [gmsp-es-logging-dashboards-957c7557c-dmz7r] Closing AuditMessageRouter
> [2024-05-06T17:20:40,683][INFO ][o.o.s.a.s.SinkProvider ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] Closing InternalOpenSearchSink
> [2024-05-06T17:20:40,683][INFO ][o.o.s.a.s.SinkProvider ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] Closing DebugSink
> [2024-05-06T17:20:40,722][INFO ][o.o.n.Node ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] stopped
> [2024-05-06T17:20:40,722][INFO ][o.o.n.Node ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] closing ...
> [2024-05-06T17:20:40,729][INFO ][o.o.s.a.i.AuditLogImpl ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] Closing AuditLogImpl
> [2024-05-06T17:20:40,736][INFO ][o.o.n.Node ] [gmsp-es-logging-dashboards-957c7557c-dmz7r] closed