Opensearch will not start

tomusn83 · December 8, 2023, 9:46pm

Red Hat Enterprise Linux 7.9
OpenSearch 1.13-1
Default config with no changes (localhost, etc.)

Have installed OpenSearch and the service will not start with the out-of-the-box minimal config. Elasticsearch is installed on this machine but the service is turned off… trying to migrate to OpenSearch. Have set the logging to trace level and cannot find any clues. It seems like it’s timing out but I increased the time in the service file. Here’s the end of the log, which offers no clues just as the rest of the log doesn’t.

448][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [x.x.x.x] PerformanceAnalyzer Enabled: true
[2023-12-08T17:47:07,475][INFO ][o.o.n.Node               ] [x.x.x.x] initialized
[2023-12-08T17:47:07,476][INFO ][o.o.n.Node               ] [x.x.x.x] starting ...
[2023-12-08T17:47:17,595][INFO ][o.o.n.Node               ] [x.x.x.x] stopping ...
[2023-12-08T17:47:17,594][INFO ][o.o.s.a.r.AuditMessageRouter] [x.x.x.x] Closing AuditMessageRouter
[2023-12-08T17:47:17,602][INFO ][o.o.s.a.s.SinkProvider   ] [x.x.x.x] Closing InternalOpenSearchSink
[2023-12-08T17:47:17,604][INFO ][o.o.s.a.s.SinkProvider   ] [x.x.x.x] Closing DebugSink 
[2023-12-08T17:47:33,640][INFO ][o.o.t.TransportService   ] [x.x.x.x] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2023-12-08T17:47:33,721][INFO ][o.o.n.Node               ] [x.x.x.x] stopped
[2023-12-08T17:47:33,722][INFO ][o.o.n.Node               ] [x.x.x.x] closing ...
[2023-12-08T17:47:33,735][INFO ][o.o.s.a.i.AuditLogImpl   ] [x.x.x.x] Closing AuditLogImpl
[2023-12-08T17:47:33,746][INFO ][o.o.n.Node               ] [x.x.x.x] closed

Any guidance would be greatly appreciated.

Thanks,
Tom

pablo · December 11, 2023, 1:53am

@tomusn83 Could you share your opensearch.yml file?
Is there a chance to share full logs of the startup process?

tomusn83 · December 11, 2023, 3:00pm

Thanks Pablo! It appears our Graylog server was still trying to reach into our OpenSearch server, which we were upgrading from ElasticSearch. As soon as we turned off Graylog, OpenSearch started.

tomusn83 · December 11, 2023, 9:12pm

Pablo, I spoke too soon. OpenSearch WILL start if we turn off the NIC which talks to our Graylog server, so we assumed it was Graylog. Further troubleshooting revealed even with Graylog off, if the NIC is turned on OpenSearch times out. It is a public IP, but on a restricted network. Could OpenSearch see this public IP and try reaching out for updates or plugin updates and times out since it’s restricted and can’t get to the public Internet? YML & Log files follow.

Thank you… Tom

OpenSearch YML:

path.data: /var/lib/opensearch
path.logs: /var/log/opensearch

#bootstrap.memory_lock: true

#network.host: 192.168.0.1
#http.port: 9200

#discovery.seed_hosts: ["host1", "host2"]
#cluster.initial_master_nodes: ["node-1", "node-2"]

#gateway.recover_after_nodes: 3

plugins.security.disabled: true

plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=kirk,OU=client,O=client,L=test, C=de

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]
node.max_local_storage_nodes: 3

opensearch.log:

[2023-12-11T16:08:29,957][INFO ][o.o.n.Node               ] [FQDN removed] version[1.3.13], pid[4867], build[rpm/07ebac12b8942b51d9d4ada93a95aee76dc1e8e1/2023-09-19T22:09:07.067953Z], OS[Linux/3.10.0-1160.105.1.el7.x86_64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/11.0.20/11.0.20+8]
[2023-12-11T16:08:29,961][INFO ][o.o.n.Node               ] [FQDN removed] JVM home [/usr/share/opensearch/jdk], using bundled JDK [true]
[2023-12-11T16:08:29,962][INFO ][o.o.n.Node               ] [FQDN removed] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-7502121993970493572, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/opensearch, -XX:ErrorFile=/var/log/opensearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/opensearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///etc/opensearch/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/etc/opensearch, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]
[2023-12-11T16:08:31,523][WARN ][stderr                   ] [FQDN removed] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[2023-12-11T16:08:31,524][WARN ][stderr                   ] [FQDN removed] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2023-12-11T16:08:31,524][WARN ][stderr                   ] [FQDN removed] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2023-12-11T16:08:31,539][INFO ][o.o.s.s.t.SSLConfig      ] [FQDN removed] SSL dual mode is disabled
[2023-12-11T16:08:31,539][INFO ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] OpenSearch Config path is /etc/opensearch
[2023-12-11T16:08:31,819][INFO ][o.o.s.s.DefaultSecurityKeyStore] [FQDN removed] JVM supports TLSv1.3
[2023-12-11T16:08:31,821][INFO ][o.o.s.s.DefaultSecurityKeyStore] [FQDN removed] Config directory is /etc/opensearch/, from there the key- and truststore files are resolved relatively
[2023-12-11T16:08:32,199][INFO ][o.o.s.s.DefaultSecurityKeyStore] [FQDN removed] TLS Transport Client Provider : JDK
[2023-12-11T16:08:32,200][INFO ][o.o.s.s.DefaultSecurityKeyStore] [FQDN removed] TLS Transport Server Provider : JDK
[2023-12-11T16:08:32,200][INFO ][o.o.s.s.DefaultSecurityKeyStore] [FQDN removed] TLS HTTP Provider             : JDK
[2023-12-11T16:08:32,200][INFO ][o.o.s.s.DefaultSecurityKeyStore] [FQDN removed] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2023-12-11T16:08:32,201][INFO ][o.o.s.s.DefaultSecurityKeyStore] [FQDN removed] Enabled TLS protocols for HTTP layer      : [TLSv1.3, TLSv1.2]
[2023-12-11T16:08:32,393][INFO ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] Clustername: opensearch
[2023-12-11T16:08:32,403][WARN ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] Directory /etc/opensearch has insecure file permissions (should be 0700)
[2023-12-11T16:08:32,403][WARN ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] File /etc/opensearch/log4j2.properties.rpmsave has insecure file permissions (should be 0600)
[2023-12-11T16:08:32,405][WARN ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] File /etc/opensearch/jvm.options has insecure file permissions (should be 0600)
[2023-12-11T16:08:32,406][WARN ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] File /etc/opensearch/opensearch.yml has insecure file permissions (should be 0600)
[2023-12-11T16:08:32,406][WARN ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] Directory /etc/opensearch/opensearch-reports-scheduler has insecure file permissions (should be 0700)
[2023-12-11T16:08:32,406][WARN ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] File /etc/opensearch/opensearch-reports-scheduler/reports-scheduler.yml has insecure file permissions (should be 0600)
[2023-12-11T16:08:32,407][WARN ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] File /etc/opensearch/jvm.options.rpmsave has insecure file permissions (should be 0600)
[2023-12-11T16:08:32,407][WARN ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] Directory /etc/opensearch/opensearch-performance-analyzer has insecure file permissions (should be 0700)
[2023-12-11T16:08:32,407][WARN ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] File /etc/opensearch/opensearch-performance-analyzer/opensearch_security.policy has insecure file permissions (should be 0600)
[2023-12-11T16:08:32,408][WARN ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] File /etc/opensearch/opensearch-performance-analyzer/rca_idle_master.conf has insecure file permissions (should be 0600)
[2023-12-11T16:08:32,408][WARN ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] File /etc/opensearch/opensearch-performance-analyzer/supervisord.conf has insecure file permissions (should be 0600)
[2023-12-11T16:08:32,408][WARN ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] File /etc/opensearch/opensearch-performance-analyzer/rca_master.conf has insecure file permissions (should be 0600)
[2023-12-11T16:08:32,408][WARN ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] File /etc/opensearch/opensearch-performance-analyzer/log4j2.xml has insecure file permissions (should be 0600)
[2023-12-11T16:08:32,409][WARN ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] File /etc/opensearch/opensearch-performance-analyzer/agent-stats-metadata has insecure file permissions (should be 0600)
[2023-12-11T16:08:32,409][WARN ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] File /etc/opensearch/opensearch-performance-analyzer/plugin-stats-metadata has insecure file permissions (should be 0600)
[2023-12-11T16:08:32,409][WARN ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] File /etc/opensearch/opensearch-performance-analyzer/rca.conf has insecure file permissions (should be 0600)
[2023-12-11T16:08:32,409][WARN ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] File /etc/opensearch/opensearch-performance-analyzer/performance-analyzer.properties has insecure file permissions (should be 0600)
[2023-12-11T16:08:32,409][WARN ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] Directory /etc/opensearch/jvm.options.d has insecure file permissions (should be 0700)
[2023-12-11T16:08:32,410][WARN ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] File /etc/opensearch/opensearch.yml.rpmsave has insecure file permissions (should be 0600)
[2023-12-11T16:08:32,410][WARN ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] File /etc/opensearch/log4j2.properties has insecure file permissions (should be 0600)
[2023-12-11T16:08:32,410][WARN ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] Directory /etc/opensearch/opensearch-observability has insecure file permissions (should be 0700)
[2023-12-11T16:08:32,410][WARN ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] File /etc/opensearch/opensearch-observability/observability.yml has insecure file permissions (should be 0600)
[2023-12-11T16:08:33,299][INFO ][o.o.p.c.PluginSettings   ] [FQDN removed] Trying to create directory /dev/shm/performanceanalyzer/.
[2023-12-11T16:08:33,300][INFO ][o.o.p.c.PluginSettings   ] [FQDN removed] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2023-12-11T16:08:33,818][INFO ][o.o.i.r.ReindexPlugin    ] [FQDN removed] ReindexPlugin reloadSPI called
[2023-12-11T16:08:33,820][INFO ][o.o.i.r.ReindexPlugin    ] [FQDN removed] Unable to find any implementation for RemoteReindexExtension
[2023-12-11T16:08:33,884][INFO ][o.o.j.JobSchedulerPlugin ] [FQDN removed] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2023-12-11T16:08:33,891][INFO ][o.o.j.JobSchedulerPlugin ] [FQDN removed] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2023-12-11T16:08:33,892][INFO ][o.o.j.JobSchedulerPlugin ] [FQDN removed] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2023-12-11T16:08:33,896][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded module [aggs-matrix-stats]
[2023-12-11T16:08:33,897][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded module [analysis-common]
[2023-12-11T16:08:33,897][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded module [geo]
[2023-12-11T16:08:33,897][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded module [ingest-common]
[2023-12-11T16:08:33,897][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded module [ingest-geoip]
[2023-12-11T16:08:33,897][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded module [ingest-user-agent]
[2023-12-11T16:08:33,898][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded module [lang-expression]
[2023-12-11T16:08:33,898][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded module [lang-mustache]
[2023-12-11T16:08:33,898][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded module [lang-painless]
[2023-12-11T16:08:33,898][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded module [mapper-extras]
[2023-12-11T16:08:33,898][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded module [opensearch-dashboards]
[2023-12-11T16:08:33,899][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded module [parent-join]
[2023-12-11T16:08:33,899][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded module [percolator]
[2023-12-11T16:08:33,899][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded module [rank-eval]
[2023-12-11T16:08:33,899][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded module [reindex]
[2023-12-11T16:08:33,900][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded module [repository-url]
[2023-12-11T16:08:33,900][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded module [systemd]
[2023-12-11T16:08:33,901][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded module [transport-netty4]
[2023-12-11T16:08:33,901][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded plugin [opensearch-alerting]
[2023-12-11T16:08:33,902][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded plugin [opensearch-anomaly-detection]
[2023-12-11T16:08:33,902][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded plugin [opensearch-asynchronous-search]
[2023-12-11T16:08:33,902][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded plugin [opensearch-cross-cluster-replication]
[2023-12-11T16:08:33,902][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded plugin [opensearch-index-management]
[2023-12-11T16:08:33,903][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded plugin [opensearch-job-scheduler]
[2023-12-11T16:08:33,903][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded plugin [opensearch-knn]
[2023-12-11T16:08:33,903][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded plugin [opensearch-ml]
[2023-12-11T16:08:33,903][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded plugin [opensearch-observability]
[2023-12-11T16:08:33,905][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded plugin [opensearch-performance-analyzer]
[2023-12-11T16:08:33,905][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded plugin [opensearch-reports-scheduler]
[2023-12-11T16:08:33,905][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded plugin [opensearch-security]
[2023-12-11T16:08:33,905][INFO ][o.o.p.PluginsService     ] [FQDN removed] loaded plugin [opensearch-sql]
[2023-12-11T16:08:33,933][INFO ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2023-12-11T16:08:33,959][INFO ][o.o.e.NodeEnvironment    ] [FQDN removed] using [1] data paths, mounts [[/var (/dev/mapper/mscvg-lv_var)]], net usable_space [3.5gb], net total_space [7.7gb], types [ext4]
[2023-12-11T16:08:33,959][INFO ][o.o.e.NodeEnvironment    ] [FQDN removed] heap size [1gb], compressed ordinary object pointers [true]
[2023-12-11T16:08:34,089][INFO ][o.o.n.Node               ] [FQDN removed] node name [FQDN removed], node ID [X8uQ6E4_TkeoYAAYpIWALQ], cluster name [opensearch], roles [master, remote_cluster_client, data, ingest]
[2023-12-11T16:08:37,179][WARN ][o.o.s.c.Salt             ] [FQDN removed] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2023-12-11T16:08:37,210][INFO ][o.o.s.a.i.AuditLogImpl   ] [FQDN removed] Message routing enabled: true
[2023-12-11T16:08:37,269][INFO ][o.o.s.f.SecurityFilter   ] [FQDN removed] <NONE> indices are made immutable.
[2023-12-11T16:08:37,698][INFO ][o.o.a.b.ADCircuitBreakerService] [FQDN removed] Registered memory breaker.
[2023-12-11T16:08:38,085][INFO ][o.o.m.c.b.MLCircuitBreakerService] [FQDN removed] Registered ML memory breaker.
[2023-12-11T16:08:38,546][INFO ][o.o.t.NettyAllocator     ] [FQDN removed] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
[2023-12-11T16:08:38,598][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,698][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,700][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,700][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,700][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,701][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,701][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,701][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,702][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,702][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,702][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,703][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,703][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,707][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,707][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,708][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,710][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,710][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,710][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,711][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,711][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,711][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,712][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,712][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,712][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,713][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [FQDN removed] Fail to read queue capacity via reflection
[2023-12-11T16:08:38,735][INFO ][o.o.d.DiscoveryModule    ] [FQDN removed] using discovery type [zen] and seed hosts providers [settings]
[2023-12-11T16:08:39,355][WARN ][o.o.g.DanglingIndicesState] [FQDN removed] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2023-12-11T16:08:39,807][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [FQDN removed] PerformanceAnalyzer Enabled: true
[2023-12-11T16:08:39,836][INFO ][o.o.n.Node               ] [FQDN removed] initialized
[2023-12-11T16:08:39,837][INFO ][o.o.n.Node               ] [FQDN removed] starting ...
[2023-12-11T16:08:39,981][INFO ][o.o.t.TransportService   ] [FQDN removed] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2023-12-11T16:08:40,186][WARN ][o.o.b.BootstrapChecks    ] [FQDN removed] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2023-12-11T16:08:40,187][INFO ][o.o.c.c.Coordinator      ] [FQDN removed] cluster UUID [qSLwDjY9Tum69I5q7dCxCw]
[2023-12-11T16:08:40,200][INFO ][o.o.c.c.ClusterBootstrapService] [FQDN removed] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2023-12-11T16:08:40,337][INFO ][o.o.c.s.MasterService    ] [FQDN removed] elected-as-master ([1] nodes joined)[{FQDN removed}{X8uQ6E4_TkeoYAAYpIWALQ}{XtKjUcQaRESheEQPjt6bzw}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 4, version: 23, delta: master node changed {previous [], current [{FQDN removed}{X8uQ6E4_TkeoYAAYpIWALQ}{XtKjUcQaRESheEQPjt6bzw}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}]}
[2023-12-11T16:08:40,394][INFO ][o.o.c.s.ClusterApplierService] [FQDN removed] master node changed {previous [], current [{FQDN removed}{X8uQ6E4_TkeoYAAYpIWALQ}{XtKjUcQaRESheEQPjt6bzw}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}]}, term: 4, version: 23, reason: Publication{term=4, version=23}
[2023-12-11T16:08:40,408][INFO ][o.o.a.c.ADClusterEventListener] [FQDN removed] Cluster is not recovered yet.
[2023-12-11T16:08:40,421][INFO ][o.o.i.i.ManagedIndexCoordinator] [FQDN removed] Cache master node onMaster time: 1702310920421
[2023-12-11T16:08:40,429][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [FQDN removed] Config override setting update called with empty string. Ignoring.
[2023-12-11T16:08:40,458][INFO ][o.o.h.AbstractHttpServerTransport] [FQDN removed] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2023-12-11T16:08:40,458][INFO ][o.o.n.Node               ] [FQDN removed] started
[2023-12-11T16:08:40,503][INFO ][o.o.c.s.ClusterSettings  ] [FQDN removed] updating [plugins.index_state_management.template_migration.control] from [0] to [-1]
[2023-12-11T16:08:40,509][INFO ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] Node started
[2023-12-11T16:08:40,509][INFO ][o.o.s.c.ConfigurationRepository] [FQDN removed] Will attempt to create index .opendistro_security and default configs if they are absent
[2023-12-11T16:08:40,509][INFO ][o.o.a.c.HashRing         ] [FQDN removed] Node added: [X8uQ6E4_TkeoYAAYpIWALQ]
[2023-12-11T16:08:40,510][INFO ][o.o.s.OpenSearchSecurityPlugin] [FQDN removed] 0 OpenSearch Security modules loaded so far: []
[2023-12-11T16:08:40,510][INFO ][o.o.s.c.ConfigurationRepository] [FQDN removed] Background init thread started. Install default config?: true
[2023-12-11T16:08:40,521][INFO ][o.o.a.c.HashRing         ] [FQDN removed] Add data node to AD version hash ring: X8uQ6E4_TkeoYAAYpIWALQ
[2023-12-11T16:08:40,523][INFO ][o.o.a.c.HashRing         ] [FQDN removed] All nodes with known AD version: {X8uQ6E4_TkeoYAAYpIWALQ=ADNodeInfo{version=1.3.13, isEligibleDataNode=true}}
[2023-12-11T16:08:40,523][INFO ][o.o.a.c.HashRing         ] [FQDN removed] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 0
[2023-12-11T16:08:40,523][INFO ][o.o.a.c.HashRing         ] [FQDN removed] Build AD version hash ring successfully
[2023-12-11T16:08:40,529][INFO ][o.o.a.c.ADDataMigrator   ] [FQDN removed] Start migrating AD data
[2023-12-11T16:08:40,529][INFO ][o.o.a.c.ADDataMigrator   ] [FQDN removed] AD job index doesn't exist, no need to migrate
[2023-12-11T16:08:40,529][INFO ][o.o.a.c.ADClusterEventListener] [FQDN removed] Init AD version hash ring successfully
[2023-12-11T16:08:40,559][INFO ][o.o.g.GatewayService     ] [FQDN removed] recovered [1] indices into cluster_state
[2023-12-11T16:08:40,563][INFO ][o.o.s.c.ConfigurationRepository] [FQDN removed] Index .opendistro_security already exists
[2023-12-11T16:08:40,565][INFO ][o.o.s.c.ConfigurationRepository] [FQDN removed] Node started, try to initialize it. Wait for at least yellow cluster state....
[2023-12-11T16:08:41,097][INFO ][o.o.c.r.a.AllocationService] [FQDN removed] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.opendistro_security][0]]]).
[2023-12-11T16:08:41,152][INFO ][o.o.s.s.ConfigHelper     ] [FQDN removed] Will update 'config' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-12-11T16:08:41,250][INFO ][o.o.s.s.ConfigHelper     ] [FQDN removed] Index .opendistro_security already contains doc with id config, skipping update.
[2023-12-11T16:08:41,251][INFO ][o.o.s.s.ConfigHelper     ] [FQDN removed] Will update 'roles' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-12-11T16:08:41,261][INFO ][o.o.s.s.ConfigHelper     ] [FQDN removed] Index .opendistro_security already contains doc with id roles, skipping update.
[2023-12-11T16:08:41,262][INFO ][o.o.s.s.ConfigHelper     ] [FQDN removed] Will update 'rolesmapping' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-12-11T16:08:41,272][INFO ][o.o.s.s.ConfigHelper     ] [FQDN removed] Index .opendistro_security already contains doc with id rolesmapping, skipping update.
[2023-12-11T16:08:41,272][INFO ][o.o.s.s.ConfigHelper     ] [FQDN removed] Will update 'internalusers' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-12-11T16:08:41,283][INFO ][o.o.s.s.ConfigHelper     ] [FQDN removed] Index .opendistro_security already contains doc with id internalusers, skipping update.
[2023-12-11T16:08:41,283][INFO ][o.o.s.s.ConfigHelper     ] [FQDN removed] Will update 'actiongroups' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-12-11T16:08:41,290][INFO ][o.o.s.s.ConfigHelper     ] [FQDN removed] Index .opendistro_security already contains doc with id actiongroups, skipping update.
[2023-12-11T16:08:41,290][INFO ][o.o.s.s.ConfigHelper     ] [FQDN removed] Will update 'tenants' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/tenants.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-12-11T16:08:41,295][INFO ][o.o.s.s.ConfigHelper     ] [FQDN removed] Index .opendistro_security already contains doc with id tenants, skipping update.
[2023-12-11T16:08:41,295][INFO ][o.o.s.s.ConfigHelper     ] [FQDN removed] Will update 'nodesdn' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/nodes_dn.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
[2023-12-11T16:08:41,298][INFO ][o.o.s.s.ConfigHelper     ] [FQDN removed] Index .opendistro_security already contains doc with id nodesdn, skipping update.
[2023-12-11T16:08:41,298][INFO ][o.o.s.s.ConfigHelper     ] [FQDN removed] Will update 'whitelist' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/whitelist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
[2023-12-11T16:08:41,302][INFO ][o.o.s.s.ConfigHelper     ] [FQDN removed] Index .opendistro_security already contains doc with id whitelist, skipping update.
[2023-12-11T16:08:41,303][INFO ][o.o.s.s.ConfigHelper     ] [FQDN removed] Will update 'audit' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/audit.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-12-11T16:08:41,327][INFO ][o.o.s.s.ConfigHelper     ] [FQDN removed] Index .opendistro_security already contains doc with id audit, skipping update.
[2023-12-11T16:08:41,548][INFO ][o.o.s.a.i.AuditLogImpl   ] [FQDN removed] Auditing on REST API is enabled.
[2023-12-11T16:08:41,548][INFO ][o.o.s.a.i.AuditLogImpl   ] [FQDN removed] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing.
[2023-12-11T16:08:41,549][INFO ][o.o.s.a.i.AuditLogImpl   ] [FQDN removed] Auditing on Transport API is enabled.
[2023-12-11T16:08:41,549][INFO ][o.o.s.a.i.AuditLogImpl   ] [FQDN removed] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing.
[2023-12-11T16:08:41,549][INFO ][o.o.s.a.i.AuditLogImpl   ] [FQDN removed] Auditing of request body is enabled.
[2023-12-11T16:08:41,549][INFO ][o.o.s.a.i.AuditLogImpl   ] [FQDN removed] Bulk requests resolution is disabled during request auditing.
[2023-12-11T16:08:41,549][INFO ][o.o.s.a.i.AuditLogImpl   ] [FQDN removed] Index resolution is enabled during request auditing.
[2023-12-11T16:08:41,550][INFO ][o.o.s.a.i.AuditLogImpl   ] [FQDN removed] Sensitive headers auditing is enabled.
[2023-12-11T16:08:41,550][INFO ][o.o.s.a.i.AuditLogImpl   ] [FQDN removed] Auditing requests from kibanaserver users is disabled.
[2023-12-11T16:08:41,553][WARN ][o.o.s.a.r.AuditMessageRouter] [FQDN removed] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTERNAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint
[2023-12-11T16:08:41,553][INFO ][o.o.s.a.i.AuditLogImpl   ] [FQDN removed] Auditing of external configuration is disabled.
[2023-12-11T16:08:41,553][INFO ][o.o.s.a.i.AuditLogImpl   ] [FQDN removed] Auditing of internal configuration is enabled.
[2023-12-11T16:08:41,554][INFO ][o.o.s.a.i.AuditLogImpl   ] [FQDN removed] Auditing only metadata information for read request is enabled.
[2023-12-11T16:08:41,554][INFO ][o.o.s.a.i.AuditLogImpl   ] [FQDN removed] Auditing will watch {} for read requests.
[2023-12-11T16:08:41,554][INFO ][o.o.s.a.i.AuditLogImpl   ] [FQDN removed] Auditing read operation requests from kibanaserver users is disabled.
[2023-12-11T16:08:41,554][INFO ][o.o.s.a.i.AuditLogImpl   ] [FQDN removed] Auditing only metadata information for write request is enabled.
[2023-12-11T16:08:41,554][INFO ][o.o.s.a.i.AuditLogImpl   ] [FQDN removed] Auditing diffs for write requests is disabled.
[2023-12-11T16:08:41,555][INFO ][o.o.s.a.i.AuditLogImpl   ] [FQDN removed] Auditing write operation requests from kibanaserver users is disabled.
[2023-12-11T16:08:41,555][INFO ][o.o.s.a.i.AuditLogImpl   ] [FQDN removed] Auditing will watch <NONE> for write requests.
[2023-12-11T16:08:41,555][INFO ][o.o.s.a.i.AuditLogImpl   ] [FQDN removed] .opendistro_security is used as internal security index.
[2023-12-11T16:08:41,555][INFO ][o.o.s.a.i.AuditLogImpl   ] [FQDN removed] Internal index used for posting audit logs is null
[2023-12-11T16:08:41,556][INFO ][o.o.s.c.ConfigurationRepository] [FQDN removed] Hot-reloading of audit configuration is enabled
[2023-12-11T16:08:41,556][INFO ][o.o.s.c.ConfigurationRepository] [FQDN removed] Node 'FQDN removed' initialized
[2023-12-11T16:08:47,633][INFO ][o.o.n.Node               ] [FQDN removed] stopping ...
[2023-12-11T16:08:47,633][INFO ][o.o.s.a.r.AuditMessageRouter] [FQDN removed] Closing AuditMessageRouter
[2023-12-11T16:08:47,640][INFO ][o.o.s.a.s.SinkProvider   ] [FQDN removed] Closing InternalOpenSearchSink
[2023-12-11T16:08:47,641][INFO ][o.o.s.a.s.SinkProvider   ] [FQDN removed] Closing DebugSink
[2023-12-11T16:08:47,757][INFO ][o.o.n.Node               ] [FQDN removed] stopped
[2023-12-11T16:08:47,759][INFO ][o.o.n.Node               ] [FQDN removed] closing ...
[2023-12-11T16:08:47,776][INFO ][o.o.s.a.i.AuditLogImpl   ] [FQDN removed] Closing AuditLogImpl
[2023-12-11T16:08:47,789][INFO ][o.o.n.Node               ] [FQDN removed] closed

pablo · December 12, 2023, 10:25am

@tomusn83 According to the logs, OpenSearch exposes itself with localhost IP address.

[2023-12-11T16:08:39,981][INFO ][o.o.t.TransportService   ] [FQDN removed] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}

[2023-12-11T16:08:40,458][INFO ][o.o.h.AbstractHttpServerTransport] [FQDN removed] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}

Could you add the following line to opensearch.yml file?

network.host: 0.0.0.0

Do you deploy a single-node cluster?

tomusn83 · December 12, 2023, 1:28pm

I tried setting network.host: 0.0.0.0, but the result is the same.

The end of the log shows the machine’s public IP address, (but restricted network) as publish_address and bound_address as 0.0.0.0.

Thanks

[2023-12-12T13:16:55,546][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [u-bnoc-svr-elasticsearch-1.msc.navy.mil] PerformanceAnalyzer Enabled: true
[2023-12-12T13:16:55,571][INFO ][o.o.n.Node               ] [u-bnoc-svr-elasticsearch-1.msc.navy.mil] initialized
[2023-12-12T13:16:55,572][INFO ][o.o.n.Node               ] [u-bnoc-svr-elasticsearch-1.msc.navy.mil] starting ...
[2023-12-12T13:16:56,103][INFO ][o.o.n.Node               ] [u-bnoc-svr-elasticsearch-1.msc.navy.mil] stopping ...
[2023-12-12T13:17:21,790][INFO ][o.o.t.TransportService   ] [u-bnoc-svr-elasticsearch-1.msc.navy.mil] publish_address {xxx.xxx.72.90:9300}, bound_addresses {0.0.0.0:9300}
[2023-12-12T13:17:21,875][INFO ][o.o.n.Node               ] [u-bnoc-svr-elasticsearch-1.msc.navy.mil] stopped
[2023-12-12T13:17:21,875][INFO ][o.o.n.Node               ] [u-bnoc-svr-elasticsearch-1.msc.navy.mil] closing ...
[2023-12-12T13:17:22,040][INFO ][o.o.n.Node               ] [u-bnoc-svr-elasticsearch-1.msc.navy.mil] closed

pablo · December 12, 2023, 2:54pm

@tomusn83 I see now. Yeah, I think connecting public connection directly can cause that. Is it public DHCP or static?

0.0.0.0 means any available network interface as far as I remember.

pablo · December 12, 2023, 2:55pm

@tomusn83 Have you tried using exact internal IP address?

i.e.

network.host: 192.168.1.10

tomusn83 · December 15, 2023, 1:47pm

Finally got it working with a workaround.

Shut off the NIC with the public IP address (but on restricted network)
Start OpenSearch - successful
Turn the NIC back on - Opensearch still running

This appears to be something in the startup code since it stays running after the NIC is re-enabled. Seems to be reaching out to this public IP (even though nothing in the config) but times out because it is a restricted network.

Thanks again for the replies.

pablo · December 15, 2023, 5:18pm

@tomusn83 Have you tried to use network.host: _site_ or network.host: _[networkInterface]_ i.e. network.host: _en0_

_site_ will refer to any site-local address on the system i.e. 192.168.0.1

tomusn83 · December 26, 2023, 9:04pm

Network host is already set to 192.168.1.1 and is a direct connection to our Graylog server.

NIC #1 & #2 communicate with SAN over 10.x.x.x
NIC #3 is network host on 192.168.1.1 (connects to Graylog server)
NIC #4 has a public IP but gateway is to internal company network and used for SSH, etc.

It is only when NIC #4 is down that OpenSearch will start successfully. Only thing I can figure is OpenSearch sees this public IP and is trying to reach out but times out because it can’t reach the public Internet. Unfortunately, I don’t have the time to look through the source code to confirm my speculation.

Topic		Replies	Views
Webservice does not come up Performance Analyzer troubleshoot	1	485	February 7, 2023
Open Search cluster is running high cpu and response time is also high OpenSearch discuss	11	894	November 9, 2023
Opensearch restarts on Openshift On-Premises OpenSearch troubleshoot	2	529	June 2, 2022
Slow start after upgrading version General Feedback troubleshoot	0	305	June 2, 2023
Docker container intermittently failing to start in CI OpenDistro	8	2867	March 22, 2023

Opensearch will not start

Related Topics