Opensearch with multiple nodes on different servers not working

OpenSearch
1

Hello everyone,

i have setup an development environment with one physical system which is running on proxmox. On the KVM system there should be running three virtual machines in the future. On each virtual machine (VM) a single node of an OpenSearch cluster should be running. Each VM is running on Debian 10 and installed the docker as well as the docker-compose components. There are multiple reasons for this structure and not building everything on a single host.

I followed the instructions from Docker - OpenSearch documentation

The first VM was installed successfully with Opensearch v1.0.0 and is running fine. So the the first Opensearch (Elasticsearch) and the Opensearch-Dashboard (Kibana) is available. The next thing was, to setup the single VM as an second (elasticsearch) node. My problem is, that even if i have the same cluster.name configured, the secnd node is not joining the master. There are now visible errors in the logfiles or even tries.

Node1: docker-compose.yml

version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:1.0.0
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster-sul
      - node.name=opensearch-node1
      - discovery.seed_hosts=opensearch-node1,XX.XX.XX.XX
      - cluster.initial_master_nodes=opensearch-node1,XX.XX.XX.XX
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      - "JAVA_HOME=/usr/share/opensearch/jdk"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:1.0.0
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://XX.XX.XX.XX"]' # must be a string with no spaces when specified as an environment variable
    networks:
      - opensearch-net

volumes:
  opensearch-data1:

networks:
  opensearch-net:

Node2: docker-compose.yml

version: '3'
services:
  opensearch-node2:
    image: opensearchproject/opensearch:1.0.0
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster-sul
      - node.name=opensearch-node2
      - discovery.seed_hosts=XX.XX.XX.XX,YY.YY.YY.YY
      - cluster.initial_master_nodes=XX.XX.XX.XX,YY.YY.YY.YY
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      - "JAVA_HOME=/usr/share/opensearch/jdk"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data2:/usr/share/opensearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
            networks:
                  - opensearch-net


volumes:
  opensearch-data2:

networks:
  opensearch-net:

Output of “GET /_nodes” within the Dev Tools from Node1:

{
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "opensearch-cluster-sul",

There should be two nodes running in one cluster. It is possible to access both elasticsearch instances from external via browser and seeing the status after authentication (basic auth).

Node1 Output of elasticsearch (browser):

// 20210816141518
// https://XX.XX.XX.XX:9200/

{
  "name": "opensearch-node1",
  "cluster_name": "opensearch-cluster-sul",
  "cluster_uuid": "vwpAdTh2QZWq_zASDcAQgQ",
  "version": {
    "distribution": "opensearch",
    "number": "1.0.0",
    "build_type": "tar",
    "build_hash": "34550c5b17124ddc59458ef774f6b43a086522e3",
    "build_date": "2021-07-02T23:22:21.383695Z",
    "build_snapshot": false,
    "lucene_version": "8.8.2",
    "minimum_wire_compatibility_version": "6.8.0",
    "minimum_index_compatibility_version": "6.0.0-beta1"
  },
  "tagline": "The OpenSearch Project: https://opensearch.org/"
}

Node2 elasticsearch output (browser):

/ 20210816141642
// https://YY.YY.YY.YY:9200/

{
  "name": "opensearch-node2",
  "cluster_name": "opensearch-cluster-sul",
  "cluster_uuid": "TjAai79fSieTTfXpNh9cqA",
  "version": {
    "distribution": "opensearch",
    "number": "1.0.0",
    "build_type": "tar",
    "build_hash": "34550c5b17124ddc59458ef774f6b43a086522e3",
    "build_date": "2021-07-02T23:22:21.383695Z",
    "build_snapshot": false,
    "lucene_version": "8.8.2",
    "minimum_wire_compatibility_version": "6.8.0",
    "minimum_index_compatibility_version": "6.0.0-beta1"
  },
  "tagline": "The OpenSearch Project: https://opensearch.org/"
}

Log from starting docker-compose on the second node:

docker-compose up
WARNING: Some networks were defined but are not used by any service: opensearch-net
Starting opensearch-node2 ... done
Attaching to opensearch-node2
opensearch-node2    | [2021-08-16T12:20:41,565][INFO ][o.o.n.Node               ] [opensearch-node2] version[1.0.0], pid[11], build[tar/34550c5b17124ddc59458ef774f6b43a086522e3/2021-07-02T23:22:21.383695Z], OS[Linux/4.19.0-17-amd64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
opensearch-node2    | [2021-08-16T12:20:41,568][INFO ][o.o.n.Node               ] [opensearch-node2] JVM home [/usr/share/opensearch/jdk], using bundled JDK [true]
opensearch-node2    | [2021-08-16T12:20:41,568][INFO ][o.o.n.Node               ] [opensearch-node2] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-11897297000385487018, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/plugins/opensearch-performance-analyzer/pa_config/opensearch_security.policy, -Dopensearch.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -XX:MaxDirectMemorySize=268435456, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
opensearch-node2    | [2021-08-16T12:20:42,674][INFO ][o.o.s.s.t.SSLConfig      ] [opensearch-node2] SSL dual mode is disabled
opensearch-node2    | [2021-08-16T12:20:42,675][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] OpenSearch Config path is /usr/share/opensearch/config
opensearch-node2    | [2021-08-16T12:20:42,916][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-node2] JVM supports TLSv1.3
opensearch-node2    | [2021-08-16T12:20:42,918][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-node2] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively
opensearch-node2    | [2021-08-16T12:20:43,481][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-node2] TLS Transport Client Provider : JDK
opensearch-node2    | [2021-08-16T12:20:43,481][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-node2] TLS Transport Server Provider : JDK
opensearch-node2    | [2021-08-16T12:20:43,481][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-node2] TLS HTTP Provider             : JDK
opensearch-node2    | [2021-08-16T12:20:43,481][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-node2] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2, TLSv1.1]
opensearch-node2    | [2021-08-16T12:20:43,481][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-node2] Enabled TLS protocols for HTTP layer      : [TLSv1.3, TLSv1.2, TLSv1.1]
opensearch-node2    | [2021-08-16T12:20:43,708][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] Clustername: opensearch-cluster-sul
opensearch-node2    | [2021-08-16T12:20:43,715][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] Directory /usr/share/opensearch/config has insecure file permissions (should be 0700)
opensearch-node2    | [2021-08-16T12:20:43,716][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] File /usr/share/opensearch/config/opensearch.yml has insecure file permissions (should be 0600)
opensearch-node2    | [2021-08-16T12:20:43,716][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] File /usr/share/opensearch/config/log4j2.properties has insecure file permissions (should be 0600)
opensearch-node2    | [2021-08-16T12:20:43,717][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] File /usr/share/opensearch/config/esnode.pem has insecure file permissions (should be 0600)
opensearch-node2    | [2021-08-16T12:20:43,717][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] File /usr/share/opensearch/config/esnode-key.pem has insecure file permissions (should be 0600)
opensearch-node2    | [2021-08-16T12:20:43,717][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] File /usr/share/opensearch/config/kirk.pem has insecure file permissions (should be 0600)
opensearch-node2    | [2021-08-16T12:20:43,717][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] File /usr/share/opensearch/config/kirk-key.pem has insecure file permissions (should be 0600)
opensearch-node2    | [2021-08-16T12:20:43,717][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] File /usr/share/opensearch/config/root-ca.pem has insecure file permissions (should be 0600)
opensearch-node2    | [2021-08-16T12:20:43,820][INFO ][o.o.p.c.PluginSettings   ] [opensearch-node2] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
opensearch-node2    | [2021-08-16T12:20:43,951][INFO ][o.o.p.PerformanceAnalyzerPlugin] [opensearch-node2] Shard IndexingPressure not present in this OpenSearch version. Skipping ShardIndexingPressureMetricsCollector
opensearch-node2    | [2021-08-16T12:20:44,337][INFO ][o.o.i.r.ReindexPlugin    ] [opensearch-node2] ReindexPlugin reloadSPI called
opensearch-node2    | [2021-08-16T12:20:44,338][INFO ][o.o.i.r.ReindexPlugin    ] [opensearch-node2] Unable to find any implementation for RemoteReindexExtension
opensearch-node2    | [2021-08-16T12:20:44,351][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-node2] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
opensearch-node2    | [2021-08-16T12:20:44,359][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-node2] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
opensearch-node2    | [2021-08-16T12:20:44,387][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-node2] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
opensearch-node2    | [2021-08-16T12:20:44,390][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded module [aggs-matrix-stats]
opensearch-node2    | [2021-08-16T12:20:44,390][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded module [analysis-common]
opensearch-node2    | [2021-08-16T12:20:44,390][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded module [geo]
opensearch-node2    | [2021-08-16T12:20:44,391][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded module [ingest-common]
opensearch-node2    | [2021-08-16T12:20:44,391][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded module [ingest-geoip]
opensearch-node2    | [2021-08-16T12:20:44,391][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded module [ingest-user-agent]
opensearch-node2    | [2021-08-16T12:20:44,391][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded module [lang-expression]
opensearch-node2    | [2021-08-16T12:20:44,391][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded module [lang-mustache]
opensearch-node2    | [2021-08-16T12:20:44,391][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded module [lang-painless]
opensearch-node2    | [2021-08-16T12:20:44,391][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded module [mapper-extras]
opensearch-node2    | [2021-08-16T12:20:44,391][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded module [opensearch-dashboards]
opensearch-node2    | [2021-08-16T12:20:44,391][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded module [parent-join]
opensearch-node2    | [2021-08-16T12:20:44,391][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded module [percolator]
opensearch-node2    | [2021-08-16T12:20:44,391][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded module [rank-eval]
opensearch-node2    | [2021-08-16T12:20:44,391][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded module [reindex]
opensearch-node2    | [2021-08-16T12:20:44,391][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded module [repository-url]
opensearch-node2    | [2021-08-16T12:20:44,391][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded module [transport-netty4]
opensearch-node2    | [2021-08-16T12:20:44,392][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded plugin [opensearch-alerting]
opensearch-node2    | [2021-08-16T12:20:44,392][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded plugin [opensearch-anomaly-detection]
opensearch-node2    | [2021-08-16T12:20:44,392][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded plugin [opensearch-asynchronous-search]
opensearch-node2    | [2021-08-16T12:20:44,392][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded plugin [opensearch-index-management]
opensearch-node2    | [2021-08-16T12:20:44,392][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded plugin [opensearch-job-scheduler]
opensearch-node2    | [2021-08-16T12:20:44,392][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded plugin [opensearch-knn]
opensearch-node2    | [2021-08-16T12:20:44,392][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded plugin [opensearch-notebooks]
opensearch-node2    | [2021-08-16T12:20:44,392][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded plugin [opensearch-performance-analyzer]
opensearch-node2    | [2021-08-16T12:20:44,392][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded plugin [opensearch-reports-scheduler]
opensearch-node2    | [2021-08-16T12:20:44,393][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded plugin [opensearch-security]
opensearch-node2    | [2021-08-16T12:20:44,393][INFO ][o.o.p.PluginsService     ] [opensearch-node2] loaded plugin [opensearch-sql]
opensearch-node2    | [2021-08-16T12:20:44,410][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
opensearch-node2    | [2021-08-16T12:20:44,421][DEPRECATION][o.o.d.c.s.Settings       ] [opensearch-node2] [node.max_local_storage_nodes] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
opensearch-node2    | [2021-08-16T12:20:44,429][INFO ][o.o.e.NodeEnvironment    ] [opensearch-node2] using [1] data paths, mounts [[/usr/share/opensearch/data (/dev/sda4)]], net usable_space [214.5gb], net total_space [230.1gb], types [ext4]
opensearch-node2    | [2021-08-16T12:20:44,429][INFO ][o.o.e.NodeEnvironment    ] [opensearch-node2] heap size [512mb], compressed ordinary object pointers [true]
opensearch-node2    | [2021-08-16T12:20:44,522][INFO ][o.o.n.Node               ] [opensearch-node2] node name [opensearch-node2], node ID [S5foaUI5QoSdAsTPCgB1yw], cluster name [opensearch-cluster-sul], roles [master, remote_cluster_client, data, ingest]
opensearch-node2    | [2021-08-16T12:20:46,275][DEPRECATION][o.o.d.c.s.Settings       ] [opensearch-node2] [node.max_local_storage_nodes] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
opensearch-node2    | [2021-08-16T12:20:47,269][WARN ][o.o.s.c.Salt             ] [opensearch-node2] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
opensearch-node2    | [2021-08-16T12:20:47,289][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-node2] Message routing enabled: true
opensearch-node2    | [2021-08-16T12:20:47,331][INFO ][o.o.s.f.SecurityFilter   ] [opensearch-node2] <NONE> indices are made immutable.
opensearch-node2    | [2021-08-16T12:20:47,526][INFO ][o.o.a.b.ADCircuitBreakerService] [opensearch-node2] Registered memory breaker.
opensearch-node2    | [2021-08-16T12:20:47,851][INFO ][o.o.t.NettyAllocator     ] [opensearch-node2] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}]
opensearch-node2    | [2021-08-16T12:20:47,930][INFO ][o.o.d.DiscoveryModule    ] [opensearch-node2] using discovery type [zen] and seed hosts providers [settings]
opensearch-node2    | [2021-08-16T12:20:48,267][WARN ][o.o.g.DanglingIndicesState] [opensearch-node2] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
opensearch-node2    | [2021-08-16T12:20:48,553][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [opensearch-node2] PerformanceAnalyzer Enabled: false
opensearch-node2    | [2021-08-16T12:20:48,603][INFO ][o.o.n.Node               ] [opensearch-node2] initialized
opensearch-node2    | [2021-08-16T12:20:48,603][INFO ][o.o.n.Node               ] [opensearch-node2] starting ...
opensearch-node2    | [2021-08-16T12:20:48,708][INFO ][o.o.t.TransportService   ] [opensearch-node2] publish_address {172.21.0.2:9300}, bound_addresses {0.0.0.0:9300}
opensearch-node2    | [2021-08-16T12:20:48,890][INFO ][o.o.b.BootstrapChecks    ] [opensearch-node2] bound or publishing to a non-loopback address, enforcing bootstrap checks
opensearch-node2    | [2021-08-16T12:20:48,892][INFO ][o.o.c.c.Coordinator      ] [opensearch-node2] cluster UUID [TjAai79fSieTTfXpNh9cqA]
opensearch-node2    | [2021-08-16T12:20:49,078][INFO ][o.o.c.s.MasterService    ] [opensearch-node2] elected-as-master ([1] nodes joined)[{opensearch-node2}{S5foaUI5QoSdAsTPCgB1yw}{u9nsB6HmSA61eaH1GHst-Q}{172.21.0.2}{172.21.0.2:9300}{dimr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 38, version: 314, delta: master node changed {previous [], current [{opensearch-node2}{S5foaUI5QoSdAsTPCgB1yw}{u9nsB6HmSA61eaH1GHst-Q}{172.21.0.2}{172.21.0.2:9300}{dimr}]}
opensearch-node2    | [2021-08-16T12:20:49,125][INFO ][o.o.c.s.ClusterApplierService] [opensearch-node2] master node changed {previous [], current [{opensearch-node2}{S5foaUI5QoSdAsTPCgB1yw}{u9nsB6HmSA61eaH1GHst-Q}{172.21.0.2}{172.21.0.2:9300}{dimr}]}, term: 38, version: 314, reason: Publication{term=38, version=314}
opensearch-node2    | [2021-08-16T12:20:49,144][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [opensearch-node2] Config override setting update called with empty string. Ignoring.
opensearch-node2    | [2021-08-16T12:20:49,145][INFO ][o.o.a.c.ADClusterEventListener] [opensearch-node2] CLuster is not recovered yet.
opensearch-node2    | [2021-08-16T12:20:49,160][INFO ][o.o.h.AbstractHttpServerTransport] [opensearch-node2] publish_address {172.21.0.2:9200}, bound_addresses {0.0.0.0:9200}
opensearch-node2    | [2021-08-16T12:20:49,160][INFO ][o.o.n.Node               ] [opensearch-node2] started
opensearch-node2    | [2021-08-16T12:20:49,160][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] Node started
opensearch-node2    | [2021-08-16T12:20:49,161][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-node2] Will attempt to create index .opendistro_security and default configs if they are absent
opensearch-node2    | [2021-08-16T12:20:49,162][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] 0 OpenSearch Security modules loaded so far: []
opensearch-node2    | [2021-08-16T12:20:49,162][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-node2] Background init thread started. Install default config?: true
opensearch-node2    | [2021-08-16T12:20:49,264][INFO ][o.o.g.GatewayService     ] [opensearch-node2] recovered [8] indices into cluster_state
opensearch-node2    | [2021-08-16T12:20:49,268][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-node2] Index .opendistro_security already exists
opensearch-node2    | [2021-08-16T12:20:49,268][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-node2] Node started, try to initialize it. Wait for at least yellow cluster state....
opensearch-node2    | [2021-08-16T12:20:49,293][INFO ][o.o.c.r.a.a.BalancedShardsAllocator] [opensearch-node2] Cannot move any shard in the cluster as there is no node on which shards can be allocated. Skipping shard iteration
opensearch-node2    | [2021-08-16T12:20:49,755][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-node2] Will update 'config' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
opensearch-node2    | [2021-08-16T12:20:49,786][INFO ][o.o.c.r.a.a.BalancedShardsAllocator] [opensearch-node2] Cannot move any shard in the cluster as there is no node on which shards can be allocated. Skipping shard iteration
opensearch-node2    | [2021-08-16T12:20:49,834][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-node2] Index .opendistro_security already contains doc with id config, skipping update.
opensearch-node2    | [2021-08-16T12:20:49,834][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-node2] Will update 'roles' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
opensearch-node2    | [2021-08-16T12:20:49,839][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-node2] Index .opendistro_security already contains doc with id roles, skipping update.
opensearch-node2    | [2021-08-16T12:20:49,840][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-node2] Will update 'rolesmapping' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
opensearch-node2    | [2021-08-16T12:20:49,845][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-node2] Index .opendistro_security already contains doc with id rolesmapping, skipping update.
opensearch-node2    | [2021-08-16T12:20:49,845][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-node2] Will update 'internalusers' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
opensearch-node2    | [2021-08-16T12:20:49,852][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-node2] Index .opendistro_security already contains doc with id internalusers, skipping update.
opensearch-node2    | [2021-08-16T12:20:49,852][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-node2] Will update 'actiongroups' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
opensearch-node2    | [2021-08-16T12:20:49,856][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-node2] Index .opendistro_security already contains doc with id actiongroups, skipping update.
opensearch-node2    | [2021-08-16T12:20:49,856][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-node2] Will update 'tenants' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/tenants.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
opensearch-node2    | [2021-08-16T12:20:49,859][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-node2] Index .opendistro_security already contains doc with id tenants, skipping update.
opensearch-node2    | [2021-08-16T12:20:49,859][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-node2] Will update 'nodesdn' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/nodes_dn.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
opensearch-node2    | [2021-08-16T12:20:49,860][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-node2] Index .opendistro_security already contains doc with id nodesdn, skipping update.
opensearch-node2    | [2021-08-16T12:20:49,861][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-node2] Will update 'whitelist' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/whitelist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
opensearch-node2    | [2021-08-16T12:20:49,862][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-node2] Index .opendistro_security already contains doc with id whitelist, skipping update.
opensearch-node2    | [2021-08-16T12:20:49,863][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-node2] Will update 'audit' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/audit.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
opensearch-node2    | [2021-08-16T12:20:49,893][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-node2] Index .opendistro_security already contains doc with id audit, skipping update.
opensearch-node2    | [2021-08-16T12:20:49,994][INFO ][o.o.c.r.a.AllocationService] [opensearch-node2] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[security-auditlog-2021.07.30][0]]]).
opensearch-node2    | [2021-08-16T12:20:50,059][INFO ][stdout                   ] [opensearch-node2] [FINE] No subscribers registered for event class org.opensearch.security.securityconf.DynamicConfigFactory$NodesDnModelImpl
opensearch-node2    | [2021-08-16T12:20:50,059][INFO ][stdout                   ] [opensearch-node2] [FINE] No subscribers registered for event class org.greenrobot.eventbus.NoSubscriberEvent
opensearch-node2    | [2021-08-16T12:20:50,060][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-node2] Auditing on REST API is enabled.
opensearch-node2    | [2021-08-16T12:20:50,060][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-node2] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing.
opensearch-node2    | [2021-08-16T12:20:50,060][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-node2] Auditing on Transport API is enabled.
opensearch-node2    | [2021-08-16T12:20:50,060][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-node2] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing.
opensearch-node2    | [2021-08-16T12:20:50,060][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-node2] Auditing of request body is enabled.
opensearch-node2    | [2021-08-16T12:20:50,060][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-node2] Bulk requests resolution is disabled during request auditing.
opensearch-node2    | [2021-08-16T12:20:50,060][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-node2] Index resolution is enabled during request auditing.
opensearch-node2    | [2021-08-16T12:20:50,060][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-node2] Sensitive headers auditing is enabled.
opensearch-node2    | [2021-08-16T12:20:50,060][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-node2] Auditing requests from kibanaserver users is disabled.
opensearch-node2    | [2021-08-16T12:20:50,063][WARN ][o.o.s.a.r.AuditMessageRouter] [opensearch-node2] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTERNAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint
opensearch-node2    | [2021-08-16T12:20:50,063][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-node2] Auditing of external configuration is disabled.
opensearch-node2    | [2021-08-16T12:20:50,063][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-node2] Auditing of internal configuration is enabled.
opensearch-node2    | [2021-08-16T12:20:50,063][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-node2] Auditing only metadata information for read request is enabled.
opensearch-node2    | [2021-08-16T12:20:50,063][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-node2] Auditing will watch {} for read requests.
opensearch-node2    | [2021-08-16T12:20:50,063][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-node2] Auditing read operation requests from kibanaserver users is disabled.
opensearch-node2    | [2021-08-16T12:20:50,063][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-node2] Auditing only metadata information for write request is enabled.
opensearch-node2    | [2021-08-16T12:20:50,063][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-node2] Auditing diffs for write requests is disabled.
opensearch-node2    | [2021-08-16T12:20:50,063][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-node2] Auditing write operation requests from kibanaserver users is disabled.
opensearch-node2    | [2021-08-16T12:20:50,063][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-node2] Auditing will watch <NONE> for write requests.
opensearch-node2    | [2021-08-16T12:20:50,063][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-node2] .opendistro_security is used as internal security index.
opensearch-node2    | [2021-08-16T12:20:50,063][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-node2] Internal index used for posting audit logs is null
opensearch-node2    | [2021-08-16T12:20:50,064][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-node2] Hot-reloading of audit configuration is enabled
opensearch-node2    | [2021-08-16T12:20:50,064][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-node2] Node 'opensearch-node2' initialized

Normally i would expect an error which states, that node2 could not reach node1. But as far as i can see the logs, there is no such entry.

So in general the connection is available and could be reached from both sides.
If you need any other information, just ask. I will provide them.

Regards

2

Same issue here but with TGZ installation.

Ask this question on their github page. Mine is [BUG] Cant set up cluster if security is on ¡ Issue #1401 ¡ opensearch-project/security ¡ GitHub

3

Maye this was a copy/paste issue but on Node2, is networks actually nested on under the port?

4

This is a copy/paste issue. the two network lines are commented via “#”, so i missed to delete them. The container is starting normaly but the opensearch-node2 (container name) is not connecting to the master node.

Is there any special configuration needed if the containers run on different servers and not within a shared server (single docker environment on one host system) ?

I installed another VM and used the sample docker-compose file from the documentation Redirecting… with 2x opensearch 1x opensearch-dashboards containers and the cluster is running fine. So i think this is somesthing related to security like @thefear mentioned. I also tried disabling "- “plugins.security.disabled:true” within the environment section in the docker-compose file, but this has not changed anything.

5

My gut is telling me here that you’re nodes can’t properly see each other. Can you get into the docker and communicate between the two of them?

6

If is different machines try to telnet from one machine to another to corroborate communication

telnet ipaddress 9200

in the same machine If you want your containers to join a pre-existing network, use the external option:

services:
  # ...
networks:
  opensearch-net:
    external: true
    name: opensearch-net
2 Likes
7

Thanks for the hint with the “external: true” I tried this without any change in behaviour.

Take a look in the screenshot below. A test connection from Docker-Node1 (debian vm) to the the container opensearch-node2 is possible. Also the other way around from Docker-Node2 to container opensearch-node1 via port tcp/9200. Both opensearch-nodes are reachable via browser. So i think there is no connection problem in general.

image
image1451×827 243 KB

The logs on opensearch-node1 container are quiet empty while starting the opensearch-node2 container. I can not see any connection at all.

The logs on opensearch-node2 are a little bit more informational:

opensearch-node2    | [2021-08-18T09:14:10,287][INFO ][o.o.c.c.ClusterBootstrapService] [opensearch-node2] skipping cluster bootstrapping as local node does not match bootstrap requirements: [xx.xx.xx.xx]
opensearch-node2    | [2021-08-18T09:14:20,293][WARN ][o.o.c.c.ClusterFormationFailureHelper] [opensearch-node2] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [xx.xx.xx.xx] to bootstrap a cluster: have discovered [{opensearch-node2}{LqENYT8cR5mcjNeV7nEsvA}{IRYocN8cTCavmAlkRWuTHg}{172.21.0.2}{172.21.0.2:9300}{dimr}]; discovery will continue using [xx.xx.xx.xx:9300] from hosts providers and [{opensearch-node2}{LqENYT8cR5mcjNeV7nEsvA}{IRYocN8cTCavmAlkRWuTHg}{172.21.0.2}{172.21.0.2:9300}{dimr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0

I took some research which base image is used for Opensearch Docker image. Ubuntu 20.04 should be the base and the telnet command should be available within the container but it is not.

root@xx:/opt/docker/opensearch-dashboards# docker exec -it opensearch-node2 /bin/bash
[opensearch@3b596262355b ~]$ telnet %ip-of-node1% 9200
bash: telnet: command not found
[opensearch@3b596262355b ~]$ nc
bash: nc: command not found

I guess that the issue is with two docker instances on two virtual machines, that the docker networks are not aware of each other. The strange thing is, that i can reach the ports via browser and other systems outside of the proxmox system. So in general it should work.

8

Docker is supposed to make your life easier and yet… :roll_eyes:

I’m sure there is some arcane configuration to get this setup to work. I agree - it should work.

9

The key to understanding your problem is this log:

opensearch-node2    | [2021-08-16T12:20:49,160][INFO ][o.o.h.AbstractHttpServerTransport] [opensearch-node2] publish_address {172.21.0.2:9200}, bound_addresses {0.0.0.0:9200}

In particular… publish_address {172.21.0.2:9200}, bound_addresses {0.0.0.0:9200}

The publish address is the address that an OpenSearch node will tell other members of the cluster to use when communicating with it. This especially important when running in a container.

When running in the default network mode of bridge the container has an IP address on the container-side of the Docker bridge, i.e. behind a layer of NAT (network address translation). In this case it is 172.21.0.2. Applications on the host-side of the NAT (i.e. apps running on other hosts/VMs) know nothing about this IP. In fact they cannot reach it directly. They must communicate with the host’s IP, which is then NAT’d through to the container.

The OpenSearch node’s service discovery mechanism will publish the IP that must be used to connect to it. By default this will be the local IP the node “sees”, which in this case is the IP of the container. However since the other node is running outside of the local container network, you need to override the default and publish the host’s IP instead. This is done by setting network.publish_host.

Add the following to the respective nodes:

node1:

network.bind_host: 0.0.0.0
network.publish_host: <IP_OF_HOST/VM_WHERE_NODE1_IS_RUNNING>

node2:

network.bind_host: 0.0.0.0
network.publish_host: <IP_OF_HOST/VM_WHERE_NODE2_IS_RUNNING>

network.bind_host: 0.0.0.0 ensures that the node will listen on the IP of the container, whatever it may be. And setting network.publish_host to the IP of the host/VM, tells nodes running on other hosts the IP to use to reach it.

3 Likes
10

Thank you @robcowart, this is the same direction where i also suspected the fault. I was not aware of the “network.publish_host” parameter. I adjusted the docker-compose file and here are the outputs:

Node1:

opensearch-node1         | [2021-08-20T12:26:43,423][INFO ][o.o.n.Node               ] [opensearch-node1] initialized
opensearch-node1         | [2021-08-20T12:26:43,423][INFO ][o.o.n.Node               ] [opensearch-node1] starting ...
opensearch-node1         | [2021-08-20T12:26:43,523][INFO ][o.o.t.TransportService   ] [opensearch-node1] publish_address {1.2.3.4:9300}, bound_addresses {0.0.0.0:9300}
opensearch-node1         | [2021-08-20T12:26:43,792][INFO ][o.o.b.BootstrapChecks    ] [opensearch-node1] bound or publishing to a non-loopback address, enforcing bootstrap checks
opensearch-node1         | [2021-08-20T12:26:43,794][INFO ][o.o.c.c.Coordinator      ] [opensearch-node1] cluster UUID [vwpAdTh2QZWq_zASDcAQgQ]
opensearch-node1         | [2021-08-20T12:26:43,893][INFO ][o.o.c.s.MasterService    ] [opensearch-node1] elected-as-master ([1] nodes joined)[{opensearch-node1}{w3E-oRn-THyX-ZnQdlj3Rg}{4UC8aw3dR6-TA1ZDN5E4pA}{1.2.3.4}{1.2.3.4:9300}{dimr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 43, version: 3153, delta: master node changed {previous [], current [{opensearch-node1}{w3E-oRn-THyX-ZnQdlj3Rg}{4UC8aw3dR6-TA1ZDN5E4pA}{1.2.3.4}{1.2.3.4:9300}{dimr}]}
opensearch-node1         | [2021-08-20T12:26:44,000][INFO ][o.o.c.s.ClusterApplierService] [opensearch-node1] master node changed {previous [], current [{opensearch-node1}{w3E-oRn-THyX-ZnQdlj3Rg}{4UC8aw3dR6-TA1ZDN5E4pA}{1.2.3.4}{1.2.3.4:9300}{dimr}]}, term: 43, version: 3153, reason: Publication{term=43, version=3153}
opensearch-node1         | [2021-08-20T12:26:44,016][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [opensearch-node1] Config override setting update called with empty string. Ignoring.
opensearch-node1         | [2021-08-20T12:26:44,017][INFO ][o.o.a.c.ADClusterEventListener] [opensearch-node1] CLuster is not recovered yet.
opensearch-node1         | [2021-08-20T12:26:44,039][INFO ][o.o.h.AbstractHttpServerTransport] [opensearch-node1] publish_address {1.2.3.4:9200}, bound_addresses {0.0.0.0:9200}
opensearch-node1         | [2021-08-20T12:26:44,040][INFO ][o.o.n.Node               ] [opensearch-node1] started
opensearch-node1         | [2021-08-20T12:26:44,042][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node1] Node started
opensearch-node1         | [2021-08-20T12:26:44,043][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-node1] Will attempt to create index .opendistro_security and default configs if they are absent
opensearch-node1         | [2021-08-20T12:26:44,051][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node1] 0 OpenSearch Security modules loaded so far: []
opensearch-node1         | [2021-08-20T12:26:44,052][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-node1] Background init thread started. Install default config?: true

Node2:

opensearch-node2    | [2021-08-20T12:39:05,430][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [opensearch-node2] PerformanceAnalyzer Enabled: false
opensearch-node2    | [2021-08-20T12:39:05,477][INFO ][o.o.n.Node               ] [opensearch-node2] initialized
opensearch-node2    | [2021-08-20T12:39:05,477][INFO ][o.o.n.Node               ] [opensearch-node2] starting ...
opensearch-node2    | [2021-08-20T12:39:05,579][INFO ][o.o.t.TransportService   ] [opensearch-node2] publish_address {4.3.2.1:9300}, bound_addresses {0.0.0.0:9300}
opensearch-node2    | [2021-08-20T12:39:05,704][INFO ][o.o.b.BootstrapChecks    ] [opensearch-node2] bound or publishing to a non-loopback address, enforcing bootstrap checks
opensearch-node2    | [2021-08-20T12:39:15,719][WARN ][o.o.c.c.ClusterFormationFailureHelper] [opensearch-node2] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [172.27.11.25, 4.3.2.1] to bootstrap a cluster: have discovered [{opensearch-node2}{skuGM7P0SReEkie6Fm-2zA}{amj27ZtdQYW3ensKuo4NyQ}{4.3.2.1}{4.3.2.1:9300}{dimr}]; discovery will continue using [172.27.11.25:9300] from hosts providers and [{opensearch-node2}{skuGM7P0SReEkie6Fm-2zA}{amj27ZtdQYW3ensKuo4NyQ}{4.3.2.1}{4.3.2.1:9300}{dimr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
opensearch-node2    | [2021-08-20T12:39:25,723][WARN ][o.o.c.c.ClusterFormationFailureHelper] [opensearch-node2] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [172.27.11.25, 4.3.2.1] to bootstrap a cluster: have discovered [{opensearch-node2}{skuGM7P0SReEkie6Fm-2zA}{amj27ZtdQYW3ensKuo4NyQ}{4.3.2.1}{4.3.2.1:9300}{dimr}]; discovery will continue using [172.27.11.25:9300] from hosts providers and [{opensearch-node2}{skuGM7P0SReEkie6Fm-2zA}{amj27ZtdQYW3ensKuo4NyQ}{4.3.2.1}{4.3.2.1:9300}{dimr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
opensearch-node2    | [2021-08-20T12:39:35,721][WARN ][o.o.n.Node               ] [opensearch-node2] timed out while waiting for initial discovery state - timeout: 30s
opensearch-node2    | [2021-08-20T12:39:35,727][WARN ][o.o.c.c.ClusterFormationFailureHelper] [opensearch-node2] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [172.27.11.25, 4.3.2.1] to bootstrap a cluster: have discovered [{opensearch-node2}{skuGM7P0SReEkie6Fm-2zA}{amj27ZtdQYW3ensKuo4NyQ}{4.3.2.1}{4.3.2.1:9300}{dimr}]; discovery will continue using [172.27.11.25:9300] from hosts providers and [{opensearch-node2}{skuGM7P0SReEkie6Fm-2zA}{amj27ZtdQYW3ensKuo4NyQ}{4.3.2.1}{4.3.2.1:9300}{dimr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
opensearch-node2    | [2021-08-20T12:39:35,739][INFO ][o.o.h.AbstractHttpServerTransport] [opensearch-node2] publish_address {4.3.2.1:9200}, bound_addresses {0.0.0.0:9200}
opensearch-node2    | [2021-08-20T12:39:35,739][INFO ][o.o.n.Node               ] [opensearch-node2] started
opensearch-node2    | [2021-08-20T12:39:35,740][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] Node started
opensearch-node2    | [2021-08-20T12:39:35,741][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-node2] Will attempt to create index .opendistro_security and default configs if they are absent
opensearch-node2    | [2021-08-20T12:39:35,742][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-node2] Background init thread started. Install default config?: true
opensearch-node2    | [2021-08-20T12:39:35,742][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] 0 OpenSearch Security modules loaded so far: []
opensearch-node2    | [2021-08-20T12:39:45,731][WARN ][o.o.c.c.ClusterFormationFailureHelper] [opensearch-node2] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [172.27.11.25, 4.3.2.1] to bootstrap a cluster: have discovered [{opensearch-node2}{skuGM7P0SReEkie6Fm-2zA}{amj27ZtdQYW3ensKuo4NyQ}{4.3.2.1}{4.3.2.1:9300}{dimr}]; discovery will continue using [172.27.11.25:9300] from hosts providers and [{opensearch-node2}{skuGM7P0SReEkie6Fm-2zA}{amj27ZtdQYW3ensKuo4NyQ}{4.3.2.1}{4.3.2.1:9300}{dimr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
opensearch-node2    | [2021-08-20T12:54:02,567][WARN ][o.o.c.c.ClusterFormationFailureHelper] [opensearch-node2] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [1.2.3.4, 4.3.2.1] to bootstrap a cluster: have discovered [{opensearch-node2}{skuGM7P0SReEkie6Fm-2zA}{wuUkPIvpRPWOoR5xWtllpw}{4.3.2.1}{4.3.2.1:9300}{dimr}]; discovery will continue using [1.2.3.4:9300] from hosts providers and [{opensearch-node2}{skuGM7P0SReEkie6Fm-2zA}{wuUkPIvpRPWOoR5xWtllpw}{4.3.2.1}{4.3.2.1:9300}{dimr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
opensearch-node2    | [2021-08-20T12:54:02,610][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node2] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
opensearch-node2    | org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
opensearch-node2    | 	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:203) ~[opensearch-1.0.0.jar:1.0.0]
opensearch-node2    | 	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:189) ~[opensearch-1.0.0.jar:1.0.0]
opensearch-node2    | 	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:72) ~[opensearch-1.0.0.jar:1.0.0]
opensearch-node2    | 	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:53) ~[opensearch-1.0.0.jar:1.0.0]
opensearch-node2    | 	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.0.0.jar:1.0.0]
opensearch-node2    | 	at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:141) [opensearch-index-management-1.0.0.0.jar:1.0.0.0]
opensearch-node2    | 	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:190) [opensearch-1.0.0.jar:1.0.0]
opensearch-node2    | 	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:99) [opensearch-performance-analyzer-1.0.0.0.jar:1.0.0.0]
opensearch-node2    | 	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:190) [opensearch-1.0.0.jar:1.0.0]
opensearch-node2    | 	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:234) [opensearch-security-1.0.0.0.jar:1.0.0.0]
opensearch-node2    | 	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:154) [opensearch-security-1.0.0.0.jar:1.0.0.0]
opensearch-node2    | 	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:190) [opensearch-1.0.0.jar:1.0.0]
opensearch-node2    | 	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:168) [opensearch-1.0.0.jar:1.0.0]
opensearch-node2    | 	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:96) [opensearch-1.0.0.jar:1.0.0]
opensearch-node2    | 	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:99) [opensearch-1.0.0.jar:1.0.0]
opensearch-node2    | 	at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:88) [opensearch-1.0.0.jar:1.0.0]
opensearch-node2    | 	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:428) [opensearch-1.0.0.jar:1.0.0]
opensearch-node2    | 	at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:546) [opensearch-1.0.0.jar:1.0.0]
opensearch-node2    | 	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:211) [opensearch-security-1.0.0.0.jar:1.0.0.0]
opensearch-node2    | 	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:102) [opensearch-security-1.0.0.0.jar:1.0.0.0]
opensearch-node2    | 	at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:375) [opensearch-security-1.0.0.0.jar:1.0.0.0]
opensearch-node2    | 	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:321) [opensearch-security-1.0.0.0.jar:1.0.0.0]
opensearch-node2    | 	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:306) [opensearch-security-1.0.0.0.jar:1.0.0.0]
opensearch-node2    | 	at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:166) [opensearch-security-1.0.0.0.jar:1.0.0.0]
opensearch-node2    | 	at java.lang.Thread.run(Thread.java:832) [?:?]
opensearch-node2    | [2021-08-20T12:54:10,612][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node2] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)

While starting the second opensearch-node2 node, there are no logs on the first node opensearch-node1. This indicats, that the connection is not established between the both nodes. Another thing in the logs is the message on node2, the second node tries to connect to itself (4.3.2.1) instead of 1.2.3.4 …

Tried it also with just the IP of node1 in the configuration of the opensearch-node2. There are connection tries in the log of node2 but not on node1.

There is no difference if i use “external: true” for the network adapter.

I think there is still some issue with the docker layers between the two systems.

11

Solved: Thanks to an external consultant we found the issue. Port 9300 was not published! This should be fixed in the documentation.

1 Like
12

Agreed. Would you mind filing a bug report in our documentation repo?

13

Hello thsul, How do you fix this issue? I have the same problem.

14

I am trying to do the same setup. Were you able to get this to work? Thanks so much!

15 
ports:
  - 9200:9200
  **- 9300:9300**
  - 9600:9600 # required for Performance Analyzer

As i mentioned before, the port 9300 needs to be published/configured in the docker compose file.

16

Hey, I copied your entire setup and for some reason I can’t get it to show anything for :9200, which is also causing the dashboard not to work (unable to connect). Any idea what could be causing that?

17

Hi @AxelMonroyX what to do i want to setup using multiple machines ,
suppose i have 3 machines in same network each having different ip , since they are in same network they are able to communicate each other ,
now I want to setup the opensearch cluster using them , i.e 1 master node , 2 data nodes , 1 client node in each machine and a single dashboard , this should make a cluster which is having 3 master nodes 6 data nodes and 3 client nodes .

please suggest me the right configurations ;

here is the docker compose file which I’m using , i have attached 1 file only the same file which naming named for other 2 machines :slight_smile:

version: ‘3’
services:
opensearch-master1:
image: opensearchproject/opensearch:2.16.0
container_name: opensearch-master1
environment:
- cluster.name=opensearch-cluster
- node.name=opensearch-master1
- node.master=true
- node.data=false
- discovery.seed_hosts=172.20.20.243:9300,172.20.20.197:9300,172.20.20.208:9300

  - cluster.initial_cluster_manager_nodes=172.20.20.243,172.20.20.197,172.20.20.208

    #  - discovery.seed_hosts=opensearch-master1,opensearch-master2,opensearch-master3,172.18.0.3,172.18.0.4,172.18.0.5
    # - cluster.initial_master_nodes=opensearch-master1,opensearch-master2,opensearch-master3
  - bootstrap.memory_lock=true
  - "OPENSEARCH_JAVA_OPTS=-Xms2g -Xmx2g"
  - "DISABLE_INSTALL_DEMO_CONFIG=true"
  - "DISABLE_SECURITY_PLUGIN=true"
  - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
ulimits:
  memlock:
    soft: -1
    hard: -1
  nofile:
    soft: 65536
    hard: 65536
volumes:
  - opensearch-master-data:/usr/share/opensearch/data
ports:
  - 9200:9200
  - 9600:9600
networks:
  - opensearch-net

opensearch-data1:
image: opensearchproject/opensearch:2.16.0
container_name: opensearch-data1
environment:
- cluster.name=opensearch-cluster
- node.name=opensearch-data1
- node.master=false
- node.data=true

  - discovery.seed_hosts=172.20.20.243:9300,172.20.20.197:9300,172.20.20.208:9300
  - cluster.initial_cluster_manager_nodes=172.20.20.243,172.20.20.197,172.20.20.208

    # - discovery.seed_hosts=opensearch-master1,opensearch-master2,opensearch-master3,172.18.0.3,172.18.0.4,172.18.0.5
    # - cluster.initial_cluster_manager_nodes=opensearch-master1,opensearch-master2,opensearch-master3
  - bootstrap.memory_lock=true
  - "OPENSEARCH_JAVA_OPTS=-Xms2g -Xmx2g"
  - "DISABLE_INSTALL_DEMO_CONFIG=true"
  - "DISABLE_SECURITY_PLUGIN=true"
  - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
ulimits:
  memlock:
    soft: -1
    hard: -1
  nofile:
    soft: 65536
    hard: 65536
volumes:
  - opensearch-data:/usr/share/opensearch/data
ports:
  - 9201:9200
  - 9601:9600
networks:
  - opensearch-net

opensearch-client1:
image: opensearchproject/opensearch:2.16.0
container_name: opensearch-client1
environment:
- cluster.name=opensearch-cluster
- node.name=opensearch-client1
- node.master=false
- node.data=false
- node.ingest=true
- discovery.seed_hosts=172.20.20.243:9300,172.20.20.197:9300,172.20.20.208:9300
- cluster.initial_cluster_manager_nodes=172.20.20.243,172.20.20.197,172.20.20.208

    #    - discovery.seed_hosts=opensearch-master1,opensearch-master2,opensearch-master3,172.18.0.3,172.18.0.4,172.18.0.5
    # - cluster.initial_cluster_manager_nodes=opensearch-master1,opensearch-master2,opensearch-master3
  - bootstrap.memory_lock=true
  - "OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g"
  - "DISABLE_INSTALL_DEMO_CONFIG=true"
  - "DISABLE_SECURITY_PLUGIN=true"
  - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
ulimits:
  memlock:
    soft: -1
    hard: -1
  nofile:
    soft: 65536
    hard: 65536
ports:
  - 9202:9200
  - 9602:9600
networks:
  - opensearch-net

opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:2.16.0
container_name: opensearch-dashboards
environment:
# - discovery.seed_hosts=172.20.20.243,172.20.20.197,172.20.20.208

  - OPENSEARCH_HOSTS=["http://172.20.20.243:9200","http://172.20.20.197:9204","http://172.20.20.208:9207"]

  # - OPENSEARCH_HOSTS=http://opensearch-master1:9200,http://opensearch-master2:9200,http://opensearch-master3:9200
ports:
  - 5601:5601
volumes:
  - ./opensearch-dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch-dashboards.yml
networks:
  - opensearch-net

volumes:
opensearch-master-data:
opensearch-data:

networks:
opensearch-net:
external: true