# Opensearch with multiple nodes on different servers not working **URL:** \ **Category:** OpenSearch\ **Tags:** troubleshoot\ **Created:** [August 16, 2021, 12:19pm UTC](https://forum.opensearch.org/t/opensearch-with-multiple-nodes-on-different-servers-not-working/6801 "2021-08-16T12:19:20Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![thsul](https://avatars.discourse-cdn.com/v4/letter/t/eb8c5e/32.png) [@thsul](https://forum.opensearch.org/u/thsul)\ **Post date:** [August 16, 2021, 12:19pm UTC](https://forum.opensearch.org/t/opensearch-with-multiple-nodes-on-different-servers-not-working/6801/1 "2021-08-16T12:19:20Z")
Hello everyone, i have setup an development environment with one physical system which is running on proxmox. On the KVM system there should be running three virtual machines in the future. On each virtual machine (VM) a single node of an OpenSearch cluster should be running. Each VM is running on Debian 10 and installed the docker as well as the docker-compose components. There are multiple reasons for this structure and not building everything on a single host. I followed the instructions from [Docker - OpenSearch documentation](https://opensearch.org/docs/opensearch/install/docker/) The first VM was installed successfully with Opensearch v1.0.0 and is running fine. So the the first Opensearch (Elasticsearch) and the Opensearch-Dashboard (Kibana) is available. The next thing was, to setup the single VM as an second (elasticsearch) node. My problem is, that even if i have the same cluster.name configured, the secnd node is not joining the master. There are now visible errors in the logfiles or even tries. Node1: docker-compose.yml ```auto version: '3' services: opensearch-node1: image: opensearchproject/opensearch:1.0.0 container_name: opensearch-node1 environment: - cluster.name=opensearch-cluster-sul - node.name=opensearch-node1 - discovery.seed_hosts=opensearch-node1,XX.XX.XX.XX - cluster.initial_master_nodes=opensearch-node1,XX.XX.XX.XX - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM - "JAVA_HOME=/usr/share/opensearch/jdk" ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems hard: 65536 volumes: - opensearch-data1:/usr/share/opensearch/data ports: - 9200:9200 - 9600:9600 # required for Performance Analyzer networks: - opensearch-net opensearch-dashboards: image: opensearchproject/opensearch-dashboards:1.0.0 container_name: opensearch-dashboards ports: - 5601:5601 expose: - "5601" environment: OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://XX.XX.XX.XX"]' # must be a string with no spaces when specified as an environment variable networks: - opensearch-net volumes: opensearch-data1: networks: opensearch-net: ``` Node2: docker-compose.yml ```auto version: '3' services: opensearch-node2: image: opensearchproject/opensearch:1.0.0 container_name: opensearch-node2 environment: - cluster.name=opensearch-cluster-sul - node.name=opensearch-node2 - discovery.seed_hosts=XX.XX.XX.XX,YY.YY.YY.YY - cluster.initial_master_nodes=XX.XX.XX.XX,YY.YY.YY.YY - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM - "JAVA_HOME=/usr/share/opensearch/jdk" ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems hard: 65536 volumes: - opensearch-data2:/usr/share/opensearch/data ports: - 9200:9200 - 9600:9600 # required for Performance Analyzer networks: - opensearch-net volumes: opensearch-data2: networks: opensearch-net: ``` Output of “GET /\_nodes” within the Dev Tools from Node1: ```auto { "_nodes" : { "total" : 1, "successful" : 1, "failed" : 0 }, "cluster_name" : "opensearch-cluster-sul", ``` There should be two nodes running in one cluster. It is possible to access both elasticsearch instances from external via browser and seeing the status after authentication (basic auth). Node1 Output of elasticsearch (browser): ```auto // 20210816141518 // https://XX.XX.XX.XX:9200/ { "name": "opensearch-node1", "cluster_name": "opensearch-cluster-sul", "cluster_uuid": "vwpAdTh2QZWq_zASDcAQgQ", "version": { "distribution": "opensearch", "number": "1.0.0", "build_type": "tar", "build_hash": "34550c5b17124ddc59458ef774f6b43a086522e3", "build_date": "2021-07-02T23:22:21.383695Z", "build_snapshot": false, "lucene_version": "8.8.2", "minimum_wire_compatibility_version": "6.8.0", "minimum_index_compatibility_version": "6.0.0-beta1" }, "tagline": "The OpenSearch Project: https://opensearch.org/" } ``` Node2 elasticsearch output (browser): ```auto / 20210816141642 // https://YY.YY.YY.YY:9200/ { "name": "opensearch-node2", "cluster_name": "opensearch-cluster-sul", "cluster_uuid": "TjAai79fSieTTfXpNh9cqA", "version": { "distribution": "opensearch", "number": "1.0.0", "build_type": "tar", "build_hash": "34550c5b17124ddc59458ef774f6b43a086522e3", "build_date": "2021-07-02T23:22:21.383695Z", "build_snapshot": false, "lucene_version": "8.8.2", "minimum_wire_compatibility_version": "6.8.0", "minimum_index_compatibility_version": "6.0.0-beta1" }, "tagline": "The OpenSearch Project: https://opensearch.org/" } ``` Log from starting docker-compose on the second node: ```auto docker-compose up WARNING: Some networks were defined but are not used by any service: opensearch-net Starting opensearch-node2 ... done Attaching to opensearch-node2 opensearch-node2 | [2021-08-16T12:20:41,565][INFO][o.o.n.Node] [opensearch-node2] version[1.0.0], pid[11], build[tar/34550c5b17124ddc59458ef774f6b43a086522e3/2021-07-02T23:22:21.383695Z], OS[Linux/4.19.0-17-amd64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9] opensearch-node2 | [2021-08-16T12:20:41,568][INFO][o.o.n.Node] [opensearch-node2] JVM home [/usr/share/opensearch/jdk], using bundled JDK [true] opensearch-node2 | [2021-08-16T12:20:41,568][INFO][o.o.n.Node] [opensearch-node2] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-11897297000385487018, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/plugins/opensearch-performance-analyzer/pa_config/opensearch_security.policy, -Dopensearch.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -XX:MaxDirectMemorySize=268435456, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true] opensearch-node2 | [2021-08-16T12:20:42,674][INFO][o.o.s.s.t.SSLConfig] [opensearch-node2] SSL dual mode is disabled opensearch-node2 | [2021-08-16T12:20:42,675][INFO][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] OpenSearch Config path is /usr/share/opensearch/config opensearch-node2 | [2021-08-16T12:20:42,916][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-node2] JVM supports TLSv1.3 opensearch-node2 | [2021-08-16T12:20:42,918][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-node2] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively opensearch-node2 | [2021-08-16T12:20:43,481][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-node2] TLS Transport Client Provider : JDK opensearch-node2 | [2021-08-16T12:20:43,481][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-node2] TLS Transport Server Provider : JDK opensearch-node2 | [2021-08-16T12:20:43,481][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-node2] TLS HTTP Provider : JDK opensearch-node2 | [2021-08-16T12:20:43,481][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-node2] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2, TLSv1.1] opensearch-node2 | [2021-08-16T12:20:43,481][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-node2] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2, TLSv1.1] opensearch-node2 | [2021-08-16T12:20:43,708][INFO][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] Clustername: opensearch-cluster-sul opensearch-node2 | [2021-08-16T12:20:43,715][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] Directory /usr/share/opensearch/config has insecure file permissions (should be 0700) opensearch-node2 | [2021-08-16T12:20:43,716][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] File /usr/share/opensearch/config/opensearch.yml has insecure file permissions (should be 0600) opensearch-node2 | [2021-08-16T12:20:43,716][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] File /usr/share/opensearch/config/log4j2.properties has insecure file permissions (should be 0600) opensearch-node2 | [2021-08-16T12:20:43,717][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] File /usr/share/opensearch/config/esnode.pem has insecure file permissions (should be 0600) opensearch-node2 | [2021-08-16T12:20:43,717][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] File /usr/share/opensearch/config/esnode-key.pem has insecure file permissions (should be 0600) opensearch-node2 | [2021-08-16T12:20:43,717][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] File /usr/share/opensearch/config/kirk.pem has insecure file permissions (should be 0600) opensearch-node2 | [2021-08-16T12:20:43,717][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] File /usr/share/opensearch/config/kirk-key.pem has insecure file permissions (should be 0600) opensearch-node2 | [2021-08-16T12:20:43,717][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] File /usr/share/opensearch/config/root-ca.pem has insecure file permissions (should be 0600) opensearch-node2 | [2021-08-16T12:20:43,820][INFO][o.o.p.c.PluginSettings] [opensearch-node2] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600 opensearch-node2 | [2021-08-16T12:20:43,951][INFO][o.o.p.PerformanceAnalyzerPlugin] [opensearch-node2] Shard IndexingPressure not present in this OpenSearch version. Skipping ShardIndexingPressureMetricsCollector opensearch-node2 | [2021-08-16T12:20:44,337][INFO][o.o.i.r.ReindexPlugin] [opensearch-node2] ReindexPlugin reloadSPI called opensearch-node2 | [2021-08-16T12:20:44,338][INFO][o.o.i.r.ReindexPlugin] [opensearch-node2] Unable to find any implementation for RemoteReindexExtension opensearch-node2 | [2021-08-16T12:20:44,351][INFO][o.o.j.JobSchedulerPlugin] [opensearch-node2] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config opensearch-node2 | [2021-08-16T12:20:44,359][INFO][o.o.j.JobSchedulerPlugin] [opensearch-node2] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs opensearch-node2 | [2021-08-16T12:20:44,387][INFO][o.o.j.JobSchedulerPlugin] [opensearch-node2] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions opensearch-node2 | [2021-08-16T12:20:44,390][INFO][o.o.p.PluginsService] [opensearch-node2] loaded module [aggs-matrix-stats] opensearch-node2 | [2021-08-16T12:20:44,390][INFO][o.o.p.PluginsService] [opensearch-node2] loaded module [analysis-common] opensearch-node2 | [2021-08-16T12:20:44,390][INFO][o.o.p.PluginsService] [opensearch-node2] loaded module [geo] opensearch-node2 | [2021-08-16T12:20:44,391][INFO][o.o.p.PluginsService] [opensearch-node2] loaded module [ingest-common] opensearch-node2 | [2021-08-16T12:20:44,391][INFO][o.o.p.PluginsService] [opensearch-node2] loaded module [ingest-geoip] opensearch-node2 | [2021-08-16T12:20:44,391][INFO][o.o.p.PluginsService] [opensearch-node2] loaded module [ingest-user-agent] opensearch-node2 | [2021-08-16T12:20:44,391][INFO][o.o.p.PluginsService] [opensearch-node2] loaded module [lang-expression] opensearch-node2 | [2021-08-16T12:20:44,391][INFO][o.o.p.PluginsService] [opensearch-node2] loaded module [lang-mustache] opensearch-node2 | [2021-08-16T12:20:44,391][INFO][o.o.p.PluginsService] [opensearch-node2] loaded module [lang-painless] opensearch-node2 | [2021-08-16T12:20:44,391][INFO][o.o.p.PluginsService] [opensearch-node2] loaded module [mapper-extras] opensearch-node2 | [2021-08-16T12:20:44,391][INFO][o.o.p.PluginsService] [opensearch-node2] loaded module [opensearch-dashboards] opensearch-node2 | [2021-08-16T12:20:44,391][INFO][o.o.p.PluginsService] [opensearch-node2] loaded module [parent-join] opensearch-node2 | [2021-08-16T12:20:44,391][INFO][o.o.p.PluginsService] [opensearch-node2] loaded module [percolator] opensearch-node2 | [2021-08-16T12:20:44,391][INFO][o.o.p.PluginsService] [opensearch-node2] loaded module [rank-eval] opensearch-node2 | [2021-08-16T12:20:44,391][INFO][o.o.p.PluginsService] [opensearch-node2] loaded module [reindex] opensearch-node2 | [2021-08-16T12:20:44,391][INFO][o.o.p.PluginsService] [opensearch-node2] loaded module [repository-url] opensearch-node2 | [2021-08-16T12:20:44,391][INFO][o.o.p.PluginsService] [opensearch-node2] loaded module [transport-netty4] opensearch-node2 | [2021-08-16T12:20:44,392][INFO][o.o.p.PluginsService] [opensearch-node2] loaded plugin [opensearch-alerting] opensearch-node2 | [2021-08-16T12:20:44,392][INFO][o.o.p.PluginsService] [opensearch-node2] loaded plugin [opensearch-anomaly-detection] opensearch-node2 | [2021-08-16T12:20:44,392][INFO][o.o.p.PluginsService] [opensearch-node2] loaded plugin [opensearch-asynchronous-search] opensearch-node2 | [2021-08-16T12:20:44,392][INFO][o.o.p.PluginsService] [opensearch-node2] loaded plugin [opensearch-index-management] opensearch-node2 | [2021-08-16T12:20:44,392][INFO][o.o.p.PluginsService] [opensearch-node2] loaded plugin [opensearch-job-scheduler] opensearch-node2 | [2021-08-16T12:20:44,392][INFO][o.o.p.PluginsService] [opensearch-node2] loaded plugin [opensearch-knn] opensearch-node2 | [2021-08-16T12:20:44,392][INFO][o.o.p.PluginsService] [opensearch-node2] loaded plugin [opensearch-notebooks] opensearch-node2 | [2021-08-16T12:20:44,392][INFO][o.o.p.PluginsService] [opensearch-node2] loaded plugin [opensearch-performance-analyzer] opensearch-node2 | [2021-08-16T12:20:44,392][INFO][o.o.p.PluginsService] [opensearch-node2] loaded plugin [opensearch-reports-scheduler] opensearch-node2 | [2021-08-16T12:20:44,393][INFO][o.o.p.PluginsService] [opensearch-node2] loaded plugin [opensearch-security] opensearch-node2 | [2021-08-16T12:20:44,393][INFO][o.o.p.PluginsService] [opensearch-node2] loaded plugin [opensearch-sql] opensearch-node2 | [2021-08-16T12:20:44,410][INFO][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml opensearch-node2 | [2021-08-16T12:20:44,421][DEPRECATION][o.o.d.c.s.Settings] [opensearch-node2] [node.max_local_storage_nodes] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version. opensearch-node2 | [2021-08-16T12:20:44,429][INFO][o.o.e.NodeEnvironment] [opensearch-node2] using [1] data paths, mounts [[/usr/share/opensearch/data (/dev/sda4)]], net usable_space [214.5gb], net total_space [230.1gb], types [ext4] opensearch-node2 | [2021-08-16T12:20:44,429][INFO][o.o.e.NodeEnvironment] [opensearch-node2] heap size [512mb], compressed ordinary object pointers [true] opensearch-node2 | [2021-08-16T12:20:44,522][INFO][o.o.n.Node] [opensearch-node2] node name [opensearch-node2], node ID [S5foaUI5QoSdAsTPCgB1yw], cluster name [opensearch-cluster-sul], roles [master, remote_cluster_client, data, ingest] opensearch-node2 | [2021-08-16T12:20:46,275][DEPRECATION][o.o.d.c.s.Settings] [opensearch-node2] [node.max_local_storage_nodes] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version. opensearch-node2 | [2021-08-16T12:20:47,269][WARN][o.o.s.c.Salt] [opensearch-node2] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes opensearch-node2 | [2021-08-16T12:20:47,289][INFO][o.o.s.a.i.AuditLogImpl] [opensearch-node2] Message routing enabled: true opensearch-node2 | [2021-08-16T12:20:47,331][INFO][o.o.s.f.SecurityFilter] [opensearch-node2] indices are made immutable. opensearch-node2 | [2021-08-16T12:20:47,526][INFO][o.o.a.b.ADCircuitBreakerService] [opensearch-node2] Registered memory breaker. opensearch-node2 | [2021-08-16T12:20:47,851][INFO][o.o.t.NettyAllocator] [opensearch-node2] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}] opensearch-node2 | [2021-08-16T12:20:47,930][INFO][o.o.d.DiscoveryModule] [opensearch-node2] using discovery type [zen] and seed hosts providers [settings] opensearch-node2 | [2021-08-16T12:20:48,267][WARN][o.o.g.DanglingIndicesState] [opensearch-node2] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually opensearch-node2 | [2021-08-16T12:20:48,553][INFO][o.o.p.h.c.PerformanceAnalyzerConfigAction] [opensearch-node2] PerformanceAnalyzer Enabled: false opensearch-node2 | [2021-08-16T12:20:48,603][INFO][o.o.n.Node] [opensearch-node2] initialized opensearch-node2 | [2021-08-16T12:20:48,603][INFO][o.o.n.Node] [opensearch-node2] starting ... opensearch-node2 | [2021-08-16T12:20:48,708][INFO][o.o.t.TransportService] [opensearch-node2] publish_address {172.21.0.2:9300}, bound_addresses {0.0.0.0:9300} opensearch-node2 | [2021-08-16T12:20:48,890][INFO][o.o.b.BootstrapChecks] [opensearch-node2] bound or publishing to a non-loopback address, enforcing bootstrap checks opensearch-node2 | [2021-08-16T12:20:48,892][INFO][o.o.c.c.Coordinator] [opensearch-node2] cluster UUID [TjAai79fSieTTfXpNh9cqA] opensearch-node2 | [2021-08-16T12:20:49,078][INFO][o.o.c.s.MasterService] [opensearch-node2] elected-as-master ([1] nodes joined)[{opensearch-node2}{S5foaUI5QoSdAsTPCgB1yw}{u9nsB6HmSA61eaH1GHst-Q}{172.21.0.2}{172.21.0.2:9300}{dimr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 38, version: 314, delta: master node changed {previous [], current [{opensearch-node2}{S5foaUI5QoSdAsTPCgB1yw}{u9nsB6HmSA61eaH1GHst-Q}{172.21.0.2}{172.21.0.2:9300}{dimr}]} opensearch-node2 | [2021-08-16T12:20:49,125][INFO][o.o.c.s.ClusterApplierService] [opensearch-node2] master node changed {previous [], current [{opensearch-node2}{S5foaUI5QoSdAsTPCgB1yw}{u9nsB6HmSA61eaH1GHst-Q}{172.21.0.2}{172.21.0.2:9300}{dimr}]}, term: 38, version: 314, reason: Publication{term=38, version=314} opensearch-node2 | [2021-08-16T12:20:49,144][WARN][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [opensearch-node2] Config override setting update called with empty string. Ignoring. opensearch-node2 | [2021-08-16T12:20:49,145][INFO][o.o.a.c.ADClusterEventListener] [opensearch-node2] CLuster is not recovered yet. opensearch-node2 | [2021-08-16T12:20:49,160][INFO][o.o.h.AbstractHttpServerTransport] [opensearch-node2] publish_address {172.21.0.2:9200}, bound_addresses {0.0.0.0:9200} opensearch-node2 | [2021-08-16T12:20:49,160][INFO][o.o.n.Node] [opensearch-node2] started opensearch-node2 | [2021-08-16T12:20:49,160][INFO][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] Node started opensearch-node2 | [2021-08-16T12:20:49,161][INFO][o.o.s.c.ConfigurationRepository] [opensearch-node2] Will attempt to create index .opendistro_security and default configs if they are absent opensearch-node2 | [2021-08-16T12:20:49,162][INFO][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] 0 OpenSearch Security modules loaded so far: [] opensearch-node2 | [2021-08-16T12:20:49,162][INFO][o.o.s.c.ConfigurationRepository] [opensearch-node2] Background init thread started. Install default config?: true opensearch-node2 | [2021-08-16T12:20:49,264][INFO][o.o.g.GatewayService] [opensearch-node2] recovered [8] indices into cluster_state opensearch-node2 | [2021-08-16T12:20:49,268][INFO][o.o.s.c.ConfigurationRepository] [opensearch-node2] Index .opendistro_security already exists opensearch-node2 | [2021-08-16T12:20:49,268][INFO][o.o.s.c.ConfigurationRepository] [opensearch-node2] Node started, try to initialize it. Wait for at least yellow cluster state.... opensearch-node2 | [2021-08-16T12:20:49,293][INFO][o.o.c.r.a.a.BalancedShardsAllocator] [opensearch-node2] Cannot move any shard in the cluster as there is no node on which shards can be allocated. Skipping shard iteration opensearch-node2 | [2021-08-16T12:20:49,755][INFO][o.o.s.s.ConfigHelper] [opensearch-node2] Will update 'config' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false opensearch-node2 | [2021-08-16T12:20:49,786][INFO][o.o.c.r.a.a.BalancedShardsAllocator] [opensearch-node2] Cannot move any shard in the cluster as there is no node on which shards can be allocated. Skipping shard iteration opensearch-node2 | [2021-08-16T12:20:49,834][INFO][o.o.s.s.ConfigHelper] [opensearch-node2] Index .opendistro_security already contains doc with id config, skipping update. opensearch-node2 | [2021-08-16T12:20:49,834][INFO][o.o.s.s.ConfigHelper] [opensearch-node2] Will update 'roles' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false opensearch-node2 | [2021-08-16T12:20:49,839][INFO][o.o.s.s.ConfigHelper] [opensearch-node2] Index .opendistro_security already contains doc with id roles, skipping update. opensearch-node2 | [2021-08-16T12:20:49,840][INFO][o.o.s.s.ConfigHelper] [opensearch-node2] Will update 'rolesmapping' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false opensearch-node2 | [2021-08-16T12:20:49,845][INFO][o.o.s.s.ConfigHelper] [opensearch-node2] Index .opendistro_security already contains doc with id rolesmapping, skipping update. opensearch-node2 | [2021-08-16T12:20:49,845][INFO][o.o.s.s.ConfigHelper] [opensearch-node2] Will update 'internalusers' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false opensearch-node2 | [2021-08-16T12:20:49,852][INFO][o.o.s.s.ConfigHelper] [opensearch-node2] Index .opendistro_security already contains doc with id internalusers, skipping update. opensearch-node2 | [2021-08-16T12:20:49,852][INFO][o.o.s.s.ConfigHelper] [opensearch-node2] Will update 'actiongroups' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false opensearch-node2 | [2021-08-16T12:20:49,856][INFO][o.o.s.s.ConfigHelper] [opensearch-node2] Index .opendistro_security already contains doc with id actiongroups, skipping update. opensearch-node2 | [2021-08-16T12:20:49,856][INFO][o.o.s.s.ConfigHelper] [opensearch-node2] Will update 'tenants' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/tenants.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false opensearch-node2 | [2021-08-16T12:20:49,859][INFO][o.o.s.s.ConfigHelper] [opensearch-node2] Index .opendistro_security already contains doc with id tenants, skipping update. opensearch-node2 | [2021-08-16T12:20:49,859][INFO][o.o.s.s.ConfigHelper] [opensearch-node2] Will update 'nodesdn' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/nodes_dn.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true opensearch-node2 | [2021-08-16T12:20:49,860][INFO][o.o.s.s.ConfigHelper] [opensearch-node2] Index .opendistro_security already contains doc with id nodesdn, skipping update. opensearch-node2 | [2021-08-16T12:20:49,861][INFO][o.o.s.s.ConfigHelper] [opensearch-node2] Will update 'whitelist' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/whitelist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true opensearch-node2 | [2021-08-16T12:20:49,862][INFO][o.o.s.s.ConfigHelper] [opensearch-node2] Index .opendistro_security already contains doc with id whitelist, skipping update. opensearch-node2 | [2021-08-16T12:20:49,863][INFO][o.o.s.s.ConfigHelper] [opensearch-node2] Will update 'audit' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/audit.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false opensearch-node2 | [2021-08-16T12:20:49,893][INFO][o.o.s.s.ConfigHelper] [opensearch-node2] Index .opendistro_security already contains doc with id audit, skipping update. opensearch-node2 | [2021-08-16T12:20:49,994][INFO][o.o.c.r.a.AllocationService] [opensearch-node2] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[security-auditlog-2021.07.30][0]]]). opensearch-node2 | [2021-08-16T12:20:50,059][INFO][stdout] [opensearch-node2] [FINE] No subscribers registered for event class org.opensearch.security.securityconf.DynamicConfigFactory$NodesDnModelImpl opensearch-node2 | [2021-08-16T12:20:50,059][INFO][stdout] [opensearch-node2] [FINE] No subscribers registered for event class org.greenrobot.eventbus.NoSubscriberEvent opensearch-node2 | [2021-08-16T12:20:50,060][INFO][o.o.s.a.i.AuditLogImpl] [opensearch-node2] Auditing on REST API is enabled. opensearch-node2 | [2021-08-16T12:20:50,060][INFO][o.o.s.a.i.AuditLogImpl] [opensearch-node2] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing. opensearch-node2 | [2021-08-16T12:20:50,060][INFO][o.o.s.a.i.AuditLogImpl] [opensearch-node2] Auditing on Transport API is enabled. opensearch-node2 | [2021-08-16T12:20:50,060][INFO][o.o.s.a.i.AuditLogImpl] [opensearch-node2] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing. opensearch-node2 | [2021-08-16T12:20:50,060][INFO][o.o.s.a.i.AuditLogImpl] [opensearch-node2] Auditing of request body is enabled. opensearch-node2 | [2021-08-16T12:20:50,060][INFO][o.o.s.a.i.AuditLogImpl] [opensearch-node2] Bulk requests resolution is disabled during request auditing. opensearch-node2 | [2021-08-16T12:20:50,060][INFO][o.o.s.a.i.AuditLogImpl] [opensearch-node2] Index resolution is enabled during request auditing. opensearch-node2 | [2021-08-16T12:20:50,060][INFO][o.o.s.a.i.AuditLogImpl] [opensearch-node2] Sensitive headers auditing is enabled. opensearch-node2 | [2021-08-16T12:20:50,060][INFO][o.o.s.a.i.AuditLogImpl] [opensearch-node2] Auditing requests from kibanaserver users is disabled. opensearch-node2 | [2021-08-16T12:20:50,063][WARN][o.o.s.a.r.AuditMessageRouter] [opensearch-node2] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTERNAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint opensearch-node2 | [2021-08-16T12:20:50,063][INFO][o.o.s.a.i.AuditLogImpl] [opensearch-node2] Auditing of external configuration is disabled. opensearch-node2 | [2021-08-16T12:20:50,063][INFO][o.o.s.a.i.AuditLogImpl] [opensearch-node2] Auditing of internal configuration is enabled. opensearch-node2 | [2021-08-16T12:20:50,063][INFO][o.o.s.a.i.AuditLogImpl] [opensearch-node2] Auditing only metadata information for read request is enabled. opensearch-node2 | [2021-08-16T12:20:50,063][INFO][o.o.s.a.i.AuditLogImpl] [opensearch-node2] Auditing will watch {} for read requests. opensearch-node2 | [2021-08-16T12:20:50,063][INFO][o.o.s.a.i.AuditLogImpl] [opensearch-node2] Auditing read operation requests from kibanaserver users is disabled. opensearch-node2 | [2021-08-16T12:20:50,063][INFO][o.o.s.a.i.AuditLogImpl] [opensearch-node2] Auditing only metadata information for write request is enabled. opensearch-node2 | [2021-08-16T12:20:50,063][INFO][o.o.s.a.i.AuditLogImpl] [opensearch-node2] Auditing diffs for write requests is disabled. opensearch-node2 | [2021-08-16T12:20:50,063][INFO][o.o.s.a.i.AuditLogImpl] [opensearch-node2] Auditing write operation requests from kibanaserver users is disabled. opensearch-node2 | [2021-08-16T12:20:50,063][INFO][o.o.s.a.i.AuditLogImpl] [opensearch-node2] Auditing will watch for write requests. opensearch-node2 | [2021-08-16T12:20:50,063][INFO][o.o.s.a.i.AuditLogImpl] [opensearch-node2] .opendistro_security is used as internal security index. opensearch-node2 | [2021-08-16T12:20:50,063][INFO][o.o.s.a.i.AuditLogImpl] [opensearch-node2] Internal index used for posting audit logs is null opensearch-node2 | [2021-08-16T12:20:50,064][INFO][o.o.s.c.ConfigurationRepository] [opensearch-node2] Hot-reloading of audit configuration is enabled opensearch-node2 | [2021-08-16T12:20:50,064][INFO][o.o.s.c.ConfigurationRepository] [opensearch-node2] Node 'opensearch-node2' initialized ``` Normally i would expect an error which states, that node2 could not reach node1. But as far as i can see the logs, there is no such entry. So in general the connection is available and could be reached from both sides. If you need any other information, just ask. I will provide them. Regards ---
**Author:** ![thefear](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/thefear/32/1927_2.png) [@thefear](https://forum.opensearch.org/u/thefear)\ **Post date:** [August 16, 2021, 3:26pm UTC](https://forum.opensearch.org/t/opensearch-with-multiple-nodes-on-different-servers-not-working/6801/2 "2021-08-16T15:26:26Z")
Same issue here but with TGZ installation. Ask this question on their github page. Mine is [[BUG] Cant set up cluster if security is on · Issue #1401 · opensearch-project/security · GitHub](https://github.com/opensearch-project/security/issues/1401) ---
**Author:** ![searchymcsearchface](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/searchymcsearchface/32/868_2.png) [@searchymcsearchface](https://forum.opensearch.org/u/searchymcsearchface)\ **Post date:** [August 16, 2021, 4:14pm UTC](https://forum.opensearch.org/t/opensearch-with-multiple-nodes-on-different-servers-not-working/6801/3 "2021-08-16T16:14:15Z")
Maye this was a copy/paste issue but on `Node2`, is `networks` actually nested on under the port? ---
**Author:** ![thsul](https://avatars.discourse-cdn.com/v4/letter/t/eb8c5e/32.png) [@thsul](https://forum.opensearch.org/u/thsul)\ **Post date:** [August 17, 2021, 6:01am UTC](https://forum.opensearch.org/t/opensearch-with-multiple-nodes-on-different-servers-not-working/6801/4 "2021-08-17T06:01:44Z")
> [@searchymcsearchface](#): > > is `networks` actually nested on under the port? This is a copy/paste issue. the two network lines are commented via “#”, so i missed to delete them. The container is starting normaly but the opensearch-node2 (container name) is not connecting to the master node. Is there any special configuration needed if the containers run on different servers and not within a shared server (single docker environment on one host system) ? I installed another VM and used the sample docker-compose file from the documentation [Redirecting…](https://opensearch.org/docs/opensearch/install/docker/) with 2x opensearch 1x opensearch-dashboards containers and the cluster is running fine. So i think this is somesthing related to security like @thefear mentioned. I also tried disabling "- “plugins.security.disabled:true” within the environment section in the docker-compose file, but this has not changed anything. ---
**Author:** ![searchymcsearchface](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/searchymcsearchface/32/868_2.png) [@searchymcsearchface](https://forum.opensearch.org/u/searchymcsearchface)\ **Post date:** [August 17, 2021, 7:10pm UTC](https://forum.opensearch.org/t/opensearch-with-multiple-nodes-on-different-servers-not-working/6801/5 "2021-08-17T19:10:52Z")
My gut is telling me here that you’re nodes can’t properly see each other. Can you get into the docker and communicate between the two of them? ---
**Author:** ![AxelMonroyX](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/axelmonroyx/32/393_2.png) [@AxelMonroyX](https://forum.opensearch.org/u/AxelMonroyX)\ **Post date:** [August 17, 2021, 8:01pm UTC](https://forum.opensearch.org/t/opensearch-with-multiple-nodes-on-different-servers-not-working/6801/6 "2021-08-17T20:01:50Z")
If is different machines try to telnet from one machine to another to corroborate communication ```auto telnet ipaddress 9200 ``` in the same machine If you want your containers to join a pre-existing network, use the external option: ```auto services: # ... networks: opensearch-net: external: true name: opensearch-net ``` > **[Networking in Compose](https://docs.docker.com/compose/networking/)** > > How Compose sets up networking between containers ---
**Author:** ![thsul](https://avatars.discourse-cdn.com/v4/letter/t/eb8c5e/32.png) [@thsul](https://forum.opensearch.org/u/thsul)\ **Post date:** [August 18, 2021, 9:22am UTC](https://forum.opensearch.org/t/opensearch-with-multiple-nodes-on-different-servers-not-working/6801/7 "2021-08-18T09:22:58Z")
Thanks for the hint with the “external: true” I tried this without any change in behaviour. Take a look in the screenshot below. A test connection from Docker-Node1 (debian vm) to the the container opensearch-node2 is possible. Also the other way around from Docker-Node2 to container opensearch-node1 via port tcp/9200. Both opensearch-nodes are reachable via browser. So i think there is no connection problem in general. ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/c/cbb0e1ca01bb34ebc0dff2ebd621107e2a169972.png) The logs on opensearch-node1 container are quiet empty while starting the opensearch-node2 container. I can not see any connection at all. The logs on opensearch-node2 are a little bit more informational: ```auto opensearch-node2 | [2021-08-18T09:14:10,287][INFO][o.o.c.c.ClusterBootstrapService] [opensearch-node2] skipping cluster bootstrapping as local node does not match bootstrap requirements: [xx.xx.xx.xx] opensearch-node2 | [2021-08-18T09:14:20,293][WARN][o.o.c.c.ClusterFormationFailureHelper] [opensearch-node2] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [xx.xx.xx.xx] to bootstrap a cluster: have discovered [{opensearch-node2}{LqENYT8cR5mcjNeV7nEsvA}{IRYocN8cTCavmAlkRWuTHg}{172.21.0.2}{172.21.0.2:9300}{dimr}]; discovery will continue using [xx.xx.xx.xx:9300] from hosts providers and [{opensearch-node2}{LqENYT8cR5mcjNeV7nEsvA}{IRYocN8cTCavmAlkRWuTHg}{172.21.0.2}{172.21.0.2:9300}{dimr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0 ``` I took some research which base image is used for Opensearch Docker image. Ubuntu 20.04 should be the base and the telnet command should be available within the container but it is not. ```auto root@xx:/opt/docker/opensearch-dashboards# docker exec -it opensearch-node2 /bin/bash [opensearch@3b596262355b ~]$ telnet %ip-of-node1% 9200 bash: telnet: command not found [opensearch@3b596262355b ~]$ nc bash: nc: command not found ``` I guess that the issue is with two docker instances on two virtual machines, that the docker networks are not aware of each other. The strange thing is, that i can reach the ports via browser and other systems outside of the proxmox system. So in general it should work. ---
**Author:** ![searchymcsearchface](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/searchymcsearchface/32/868_2.png) [@searchymcsearchface](https://forum.opensearch.org/u/searchymcsearchface)\ **Post date:** [August 18, 2021, 1:47pm UTC](https://forum.opensearch.org/t/opensearch-with-multiple-nodes-on-different-servers-not-working/6801/8 "2021-08-18T13:47:15Z")
Docker is supposed to make your life easier and yet… 🙄 I’m sure there is some arcane configuration to get this setup to work. I agree - it _should_ work. ---
**Author:** ![robcowart](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/robcowart/32/1012_2.png) [@robcowart](https://forum.opensearch.org/u/robcowart)\ **Post date:** [August 20, 2021, 9:23am UTC](https://forum.opensearch.org/t/opensearch-with-multiple-nodes-on-different-servers-not-working/6801/9 "2021-08-20T09:23:48Z")
The key to understanding your problem is this log: ```auto opensearch-node2 | [2021-08-16T12:20:49,160][INFO][o.o.h.AbstractHttpServerTransport] [opensearch-node2] publish_address {172.21.0.2:9200}, bound_addresses {0.0.0.0:9200} ``` In particular… `publish_address {172.21.0.2:9200}, bound_addresses {0.0.0.0:9200}` The publish address is the address that an OpenSearch node will tell other members of the cluster to use when communicating with it. This especially important when running in a container. When running in the default network mode of `bridge` the container has an IP address on the container-side of the Docker bridge, i.e. behind a layer of NAT (network address translation). In this case it is `172.21.0.2`. Applications on the host-side of the NAT (i.e. apps running on other hosts/VMs) know nothing about this IP. In fact they cannot reach it directly. They must communicate with the host’s IP, which is then NAT’d through to the container. The OpenSearch node’s service discovery mechanism will publish the IP that must be used to connect to it. By default this will be the local IP the node “sees”, which in this case is the IP of the container. However since the other node is running outside of the local container network, you need to override the default and publish the host’s IP instead. This is done by setting `network.publish_host`. Add the following to the respective nodes: node1: ```auto network.bind_host: 0.0.0.0 network.publish_host: ``` node2: ```auto network.bind_host: 0.0.0.0 network.publish_host: ``` `network.bind_host: 0.0.0.0` ensures that the node will listen on the IP of the container, whatever it may be. And setting `network.publish_host` to the IP of the host/VM, tells nodes running on other hosts the IP to use to reach it. ---
**Author:** ![thsul](https://avatars.discourse-cdn.com/v4/letter/t/eb8c5e/32.png) [@thsul](https://forum.opensearch.org/u/thsul)\ **Post date:** [August 20, 2021, 12:53pm UTC](https://forum.opensearch.org/t/opensearch-with-multiple-nodes-on-different-servers-not-working/6801/10 "2021-08-20T12:53:05Z")
Thank you @robcowart, this is the same direction where i also suspected the fault. I was not aware of the “network.publish\_host” parameter. I adjusted the docker-compose file and here are the outputs: Node1: ```auto opensearch-node1 | [2021-08-20T12:26:43,423][INFO][o.o.n.Node] [opensearch-node1] initialized opensearch-node1 | [2021-08-20T12:26:43,423][INFO][o.o.n.Node] [opensearch-node1] starting ... opensearch-node1 | [2021-08-20T12:26:43,523][INFO][o.o.t.TransportService] [opensearch-node1] publish_address {1.2.3.4:9300}, bound_addresses {0.0.0.0:9300} opensearch-node1 | [2021-08-20T12:26:43,792][INFO][o.o.b.BootstrapChecks] [opensearch-node1] bound or publishing to a non-loopback address, enforcing bootstrap checks opensearch-node1 | [2021-08-20T12:26:43,794][INFO][o.o.c.c.Coordinator] [opensearch-node1] cluster UUID [vwpAdTh2QZWq_zASDcAQgQ] opensearch-node1 | [2021-08-20T12:26:43,893][INFO][o.o.c.s.MasterService] [opensearch-node1] elected-as-master ([1] nodes joined)[{opensearch-node1}{w3E-oRn-THyX-ZnQdlj3Rg}{4UC8aw3dR6-TA1ZDN5E4pA}{1.2.3.4}{1.2.3.4:9300}{dimr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 43, version: 3153, delta: master node changed {previous [], current [{opensearch-node1}{w3E-oRn-THyX-ZnQdlj3Rg}{4UC8aw3dR6-TA1ZDN5E4pA}{1.2.3.4}{1.2.3.4:9300}{dimr}]} opensearch-node1 | [2021-08-20T12:26:44,000][INFO][o.o.c.s.ClusterApplierService] [opensearch-node1] master node changed {previous [], current [{opensearch-node1}{w3E-oRn-THyX-ZnQdlj3Rg}{4UC8aw3dR6-TA1ZDN5E4pA}{1.2.3.4}{1.2.3.4:9300}{dimr}]}, term: 43, version: 3153, reason: Publication{term=43, version=3153} opensearch-node1 | [2021-08-20T12:26:44,016][WARN][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [opensearch-node1] Config override setting update called with empty string. Ignoring. opensearch-node1 | [2021-08-20T12:26:44,017][INFO][o.o.a.c.ADClusterEventListener] [opensearch-node1] CLuster is not recovered yet. opensearch-node1 | [2021-08-20T12:26:44,039][INFO][o.o.h.AbstractHttpServerTransport] [opensearch-node1] publish_address {1.2.3.4:9200}, bound_addresses {0.0.0.0:9200} opensearch-node1 | [2021-08-20T12:26:44,040][INFO][o.o.n.Node] [opensearch-node1] started opensearch-node1 | [2021-08-20T12:26:44,042][INFO][o.o.s.OpenSearchSecurityPlugin] [opensearch-node1] Node started opensearch-node1 | [2021-08-20T12:26:44,043][INFO][o.o.s.c.ConfigurationRepository] [opensearch-node1] Will attempt to create index .opendistro_security and default configs if they are absent opensearch-node1 | [2021-08-20T12:26:44,051][INFO][o.o.s.OpenSearchSecurityPlugin] [opensearch-node1] 0 OpenSearch Security modules loaded so far: [] opensearch-node1 | [2021-08-20T12:26:44,052][INFO][o.o.s.c.ConfigurationRepository] [opensearch-node1] Background init thread started. Install default config?: true ``` Node2: ```auto opensearch-node2 | [2021-08-20T12:39:05,430][INFO][o.o.p.h.c.PerformanceAnalyzerConfigAction] [opensearch-node2] PerformanceAnalyzer Enabled: false opensearch-node2 | [2021-08-20T12:39:05,477][INFO][o.o.n.Node] [opensearch-node2] initialized opensearch-node2 | [2021-08-20T12:39:05,477][INFO][o.o.n.Node] [opensearch-node2] starting ... opensearch-node2 | [2021-08-20T12:39:05,579][INFO][o.o.t.TransportService] [opensearch-node2] publish_address {4.3.2.1:9300}, bound_addresses {0.0.0.0:9300} opensearch-node2 | [2021-08-20T12:39:05,704][INFO][o.o.b.BootstrapChecks] [opensearch-node2] bound or publishing to a non-loopback address, enforcing bootstrap checks opensearch-node2 | [2021-08-20T12:39:15,719][WARN][o.o.c.c.ClusterFormationFailureHelper] [opensearch-node2] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [172.27.11.25, 4.3.2.1] to bootstrap a cluster: have discovered [{opensearch-node2}{skuGM7P0SReEkie6Fm-2zA}{amj27ZtdQYW3ensKuo4NyQ}{4.3.2.1}{4.3.2.1:9300}{dimr}]; discovery will continue using [172.27.11.25:9300] from hosts providers and [{opensearch-node2}{skuGM7P0SReEkie6Fm-2zA}{amj27ZtdQYW3ensKuo4NyQ}{4.3.2.1}{4.3.2.1:9300}{dimr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0 opensearch-node2 | [2021-08-20T12:39:25,723][WARN][o.o.c.c.ClusterFormationFailureHelper] [opensearch-node2] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [172.27.11.25, 4.3.2.1] to bootstrap a cluster: have discovered [{opensearch-node2}{skuGM7P0SReEkie6Fm-2zA}{amj27ZtdQYW3ensKuo4NyQ}{4.3.2.1}{4.3.2.1:9300}{dimr}]; discovery will continue using [172.27.11.25:9300] from hosts providers and [{opensearch-node2}{skuGM7P0SReEkie6Fm-2zA}{amj27ZtdQYW3ensKuo4NyQ}{4.3.2.1}{4.3.2.1:9300}{dimr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0 opensearch-node2 | [2021-08-20T12:39:35,721][WARN][o.o.n.Node] [opensearch-node2] timed out while waiting for initial discovery state - timeout: 30s opensearch-node2 | [2021-08-20T12:39:35,727][WARN][o.o.c.c.ClusterFormationFailureHelper] [opensearch-node2] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [172.27.11.25, 4.3.2.1] to bootstrap a cluster: have discovered [{opensearch-node2}{skuGM7P0SReEkie6Fm-2zA}{amj27ZtdQYW3ensKuo4NyQ}{4.3.2.1}{4.3.2.1:9300}{dimr}]; discovery will continue using [172.27.11.25:9300] from hosts providers and [{opensearch-node2}{skuGM7P0SReEkie6Fm-2zA}{amj27ZtdQYW3ensKuo4NyQ}{4.3.2.1}{4.3.2.1:9300}{dimr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0 opensearch-node2 | [2021-08-20T12:39:35,739][INFO][o.o.h.AbstractHttpServerTransport] [opensearch-node2] publish_address {4.3.2.1:9200}, bound_addresses {0.0.0.0:9200} opensearch-node2 | [2021-08-20T12:39:35,739][INFO][o.o.n.Node] [opensearch-node2] started opensearch-node2 | [2021-08-20T12:39:35,740][INFO][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] Node started opensearch-node2 | [2021-08-20T12:39:35,741][INFO][o.o.s.c.ConfigurationRepository] [opensearch-node2] Will attempt to create index .opendistro_security and default configs if they are absent opensearch-node2 | [2021-08-20T12:39:35,742][INFO][o.o.s.c.ConfigurationRepository] [opensearch-node2] Background init thread started. Install default config?: true opensearch-node2 | [2021-08-20T12:39:35,742][INFO][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] 0 OpenSearch Security modules loaded so far: [] opensearch-node2 | [2021-08-20T12:39:45,731][WARN][o.o.c.c.ClusterFormationFailureHelper] [opensearch-node2] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [172.27.11.25, 4.3.2.1] to bootstrap a cluster: have discovered [{opensearch-node2}{skuGM7P0SReEkie6Fm-2zA}{amj27ZtdQYW3ensKuo4NyQ}{4.3.2.1}{4.3.2.1:9300}{dimr}]; discovery will continue using [172.27.11.25:9300] from hosts providers and [{opensearch-node2}{skuGM7P0SReEkie6Fm-2zA}{amj27ZtdQYW3ensKuo4NyQ}{4.3.2.1}{4.3.2.1:9300}{dimr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0 opensearch-node2 | [2021-08-20T12:54:02,567][WARN][o.o.c.c.ClusterFormationFailureHelper] [opensearch-node2] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [1.2.3.4, 4.3.2.1] to bootstrap a cluster: have discovered [{opensearch-node2}{skuGM7P0SReEkie6Fm-2zA}{wuUkPIvpRPWOoR5xWtllpw}{4.3.2.1}{4.3.2.1:9300}{dimr}]; discovery will continue using [1.2.3.4:9300] from hosts providers and [{opensearch-node2}{skuGM7P0SReEkie6Fm-2zA}{wuUkPIvpRPWOoR5xWtllpw}{4.3.2.1}{4.3.2.1:9300}{dimr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0 opensearch-node2 | [2021-08-20T12:54:02,610][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node2] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security) opensearch-node2 | org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized]; opensearch-node2 | at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:203) ~[opensearch-1.0.0.jar:1.0.0] opensearch-node2 | at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:189) ~[opensearch-1.0.0.jar:1.0.0] opensearch-node2 | at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:72) ~[opensearch-1.0.0.jar:1.0.0] opensearch-node2 | at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:53) ~[opensearch-1.0.0.jar:1.0.0] opensearch-node2 | at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.0.0.jar:1.0.0] opensearch-node2 | at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:141) [opensearch-index-management-1.0.0.0.jar:1.0.0.0] opensearch-node2 | at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:190) [opensearch-1.0.0.jar:1.0.0] opensearch-node2 | at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:99) [opensearch-performance-analyzer-1.0.0.0.jar:1.0.0.0] opensearch-node2 | at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:190) [opensearch-1.0.0.jar:1.0.0] opensearch-node2 | at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:234) [opensearch-security-1.0.0.0.jar:1.0.0.0] opensearch-node2 | at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:154) [opensearch-security-1.0.0.0.jar:1.0.0.0] opensearch-node2 | at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:190) [opensearch-1.0.0.jar:1.0.0] opensearch-node2 | at org.opensearch.action.support.TransportAction.execute(TransportAction.java:168) [opensearch-1.0.0.jar:1.0.0] opensearch-node2 | at org.opensearch.action.support.TransportAction.execute(TransportAction.java:96) [opensearch-1.0.0.jar:1.0.0] opensearch-node2 | at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:99) [opensearch-1.0.0.jar:1.0.0] opensearch-node2 | at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:88) [opensearch-1.0.0.jar:1.0.0] opensearch-node2 | at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:428) [opensearch-1.0.0.jar:1.0.0] opensearch-node2 | at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:546) [opensearch-1.0.0.jar:1.0.0] opensearch-node2 | at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:211) [opensearch-security-1.0.0.0.jar:1.0.0.0] opensearch-node2 | at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:102) [opensearch-security-1.0.0.0.jar:1.0.0.0] opensearch-node2 | at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:375) [opensearch-security-1.0.0.0.jar:1.0.0.0] opensearch-node2 | at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:321) [opensearch-security-1.0.0.0.jar:1.0.0.0] opensearch-node2 | at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:306) [opensearch-security-1.0.0.0.jar:1.0.0.0] opensearch-node2 | at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:166) [opensearch-security-1.0.0.0.jar:1.0.0.0] opensearch-node2 | at java.lang.Thread.run(Thread.java:832) [?:?] opensearch-node2 | [2021-08-20T12:54:10,612][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node2] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security) ``` While starting the second opensearch-node2 node, there are no logs on the first node opensearch-node1. This indicats, that the connection is not established between the both nodes. Another thing in the logs is the message on node2, the second node tries to connect to itself (4.3.2.1) instead of 1.2.3.4 … Tried it also with just the IP of node1 in the configuration of the opensearch-node2. There are connection tries in the log of node2 but not on node1. There is no difference if i use “external: true” for the network adapter. I think there is still some issue with the docker layers between the two systems. ---
**Author:** ![thsul](https://avatars.discourse-cdn.com/v4/letter/t/eb8c5e/32.png) [@thsul](https://forum.opensearch.org/u/thsul)\ **Post date:** [October 27, 2021, 5:43pm UTC](https://forum.opensearch.org/t/opensearch-with-multiple-nodes-on-different-servers-not-working/6801/11 "2021-10-27T17:43:37Z")
Solved: Thanks to an external consultant we found the issue. Port 9300 was not published! This should be fixed in the documentation. ---
**Author:** ![searchymcsearchface](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/searchymcsearchface/32/868_2.png) [@searchymcsearchface](https://forum.opensearch.org/u/searchymcsearchface)\ **Post date:** [October 28, 2021, 2:43pm UTC](https://forum.opensearch.org/t/opensearch-with-multiple-nodes-on-different-servers-not-working/6801/12 "2021-10-28T14:43:54Z")
Agreed. Would you mind filing a bug report in our documentation repo? > **[Build software better, together](https://github.com/opensearch-project/documentation-website/issues/new?assignees=&labels=bug%2C%2Buntriaged&template=BUG_TEMPLATE.md&title=%5BBUG%5D)** > > GitHub is where people build software. More than 83 million people use GitHub to discover, fork, and contribute to over 200 million projects. ---
**Author:** ![beilei](https://avatars.discourse-cdn.com/v4/letter/b/779978/32.png) [@beilei](https://forum.opensearch.org/u/beilei)\ **Post date:** [March 3, 2022, 3:59pm UTC](https://forum.opensearch.org/t/opensearch-with-multiple-nodes-on-different-servers-not-working/6801/13 "2022-03-03T15:59:46Z")
Hello thsul, How do you fix this issue? I have the same problem. ---
**Author:** ![new\_OSUser](https://avatars.discourse-cdn.com/v4/letter/n/258eb7/32.png) [@new\_OSUser](https://forum.opensearch.org/u/new_OSUser)\ **Post date:** [May 3, 2022, 1:31pm UTC](https://forum.opensearch.org/t/opensearch-with-multiple-nodes-on-different-servers-not-working/6801/14 "2022-05-03T13:31:47Z")
I am trying to do the same setup. Were you able to get this to work? Thanks so much! ---
**Author:** ![thsul](https://avatars.discourse-cdn.com/v4/letter/t/eb8c5e/32.png) [@thsul](https://forum.opensearch.org/u/thsul)\ **Post date:** [May 4, 2022, 6:53am UTC](https://forum.opensearch.org/t/opensearch-with-multiple-nodes-on-different-servers-not-working/6801/15 "2022-05-04T06:53:23Z")
``` ports: - 9200:9200 **- 9300:9300** - 9600:9600 # required for Performance Analyzer ``` As i mentioned before, the port 9300 needs to be published/configured in the docker compose file. ---
**Author:** ![trey0five](https://avatars.discourse-cdn.com/v4/letter/t/a6a055/32.png) [@trey0five](https://forum.opensearch.org/u/trey0five)\ **Post date:** [May 16, 2023, 2:59pm UTC](https://forum.opensearch.org/t/opensearch-with-multiple-nodes-on-different-servers-not-working/6801/16 "2023-05-16T14:59:17Z")
Hey, I copied your entire setup and for some reason I can’t get it to show anything for :9200, which is also causing the dashboard not to work (unable to connect). Any idea what could be causing that? ---
**Author:** ![Govind12](https://avatars.discourse-cdn.com/v4/letter/g/858c86/32.png) [@Govind12](https://forum.opensearch.org/u/Govind12)\ **Post date:** [September 3, 2024, 12:58pm UTC](https://forum.opensearch.org/t/opensearch-with-multiple-nodes-on-different-servers-not-working/6801/17 "2024-09-03T12:58:22Z")
Hi @AxelMonroyX what to do i want to setup using multiple machines , suppose i have 3 machines in same network each having different ip , since they are in same network they are able to communicate each other , now I want to setup the opensearch cluster using them , i.e 1 master node , 2 data nodes , 1 client node in each machine and a single dashboard , this should make a cluster which is having 3 master nodes 6 data nodes and 3 client nodes . please suggest me the right configurations ; here is the docker compose file which I’m using , i have attached 1 file only the same file which naming named for other 2 machines 🙂 version: ‘3’ services: opensearch-master1: image: opensearchproject/opensearch:2.16.0 container\_name: opensearch-master1 environment: - cluster.name=opensearch-cluster - node.name=opensearch-master1 - node.master=true - node.data=false - discovery.seed\_hosts=172.20.20.243:9300,172.20.20.197:9300,172.20.20.208:9300 ``` - cluster.initial_cluster_manager_nodes=172.20.20.243,172.20.20.197,172.20.20.208 # - discovery.seed_hosts=opensearch-master1,opensearch-master2,opensearch-master3,172.18.0.3,172.18.0.4,172.18.0.5 # - cluster.initial_master_nodes=opensearch-master1,opensearch-master2,opensearch-master3 - bootstrap.memory_lock=true - "OPENSEARCH_JAVA_OPTS=-Xms2g -Xmx2g" - "DISABLE_INSTALL_DEMO_CONFIG=true" - "DISABLE_SECURITY_PLUGIN=true" - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD} ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 hard: 65536 volumes: - opensearch-master-data:/usr/share/opensearch/data ports: - 9200:9200 - 9600:9600 networks: - opensearch-net ``` opensearch-data1: image: opensearchproject/opensearch:2.16.0 container\_name: opensearch-data1 environment: - cluster.name=opensearch-cluster - node.name=opensearch-data1 - node.master=false - node.data=true ``` - discovery.seed_hosts=172.20.20.243:9300,172.20.20.197:9300,172.20.20.208:9300 - cluster.initial_cluster_manager_nodes=172.20.20.243,172.20.20.197,172.20.20.208 # - discovery.seed_hosts=opensearch-master1,opensearch-master2,opensearch-master3,172.18.0.3,172.18.0.4,172.18.0.5 # - cluster.initial_cluster_manager_nodes=opensearch-master1,opensearch-master2,opensearch-master3 - bootstrap.memory_lock=true - "OPENSEARCH_JAVA_OPTS=-Xms2g -Xmx2g" - "DISABLE_INSTALL_DEMO_CONFIG=true" - "DISABLE_SECURITY_PLUGIN=true" - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD} ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 hard: 65536 volumes: - opensearch-data:/usr/share/opensearch/data ports: - 9201:9200 - 9601:9600 networks: - opensearch-net ``` opensearch-client1: image: opensearchproject/opensearch:2.16.0 container\_name: opensearch-client1 environment: - cluster.name=opensearch-cluster - node.name=opensearch-client1 - node.master=false - node.data=false - node.ingest=true - discovery.seed\_hosts=172.20.20.243:9300,172.20.20.197:9300,172.20.20.208:9300 - cluster.initial\_cluster\_manager\_nodes=172.20.20.243,172.20.20.197,172.20.20.208 ``` # - discovery.seed_hosts=opensearch-master1,opensearch-master2,opensearch-master3,172.18.0.3,172.18.0.4,172.18.0.5 # - cluster.initial_cluster_manager_nodes=opensearch-master1,opensearch-master2,opensearch-master3 - bootstrap.memory_lock=true - "OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g" - "DISABLE_INSTALL_DEMO_CONFIG=true" - "DISABLE_SECURITY_PLUGIN=true" - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD} ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 hard: 65536 ports: - 9202:9200 - 9602:9600 networks: - opensearch-net ``` opensearch-dashboards: image: opensearchproject/opensearch-dashboards:2.16.0 container\_name: opensearch-dashboards environment: # - discovery.seed\_hosts=172.20.20.243,172.20.20.197,172.20.20.208 ``` - OPENSEARCH_HOSTS=["http://172.20.20.243:9200","http://172.20.20.197:9204","http://172.20.20.208:9207"] # - OPENSEARCH_HOSTS=http://opensearch-master1:9200,http://opensearch-master2:9200,http://opensearch-master3:9200 ports: - 5601:5601 volumes: - ./opensearch-dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch-dashboards.yml networks: - opensearch-net ``` volumes: opensearch-master-data: opensearch-data: networks: opensearch-net: external: true