About setting up high availability Elasticsearch cluster

Hi @varma.
Just make sure few points:

  1. cluster.name: RV-opendistro-dev is same for all nodes in cluster?
  2. Can you ping other nodes? I mean is it possible to reach them you can open related ports if required.
  3. Make sure node.name: node-1 is unique for each node
  4. Try to delete content of /var/lib/elasticsearch/nodes on all nodes.
  5. discovery.seed_hosts: [“10.147.106.132”, “10.147.106.139”, “10.147.106.96”] if you are setting this make sure all your nodes are master eligible. i,e You must not have entry like master = false in yml file.
  6. Make sure you have at least one data node in cluster. as Open Distro initializes index with internal settings. This index should be placed at least one of data node.

Apart from that let me share commands you asked for:
firewall-cmd --state to check status of firewall
sudo systemctl stop firewalld to Stop firewall (Its not recommended)

1 Like

Hi @pranali4796 ,can you please have a look on my logs and suggest me if you identify anything.

  1. in two nodes i can start ES with root user only, but in master i am unable to do this
    2.we should not use root user at all?
  2. and can you please how to uninstall open distro completely in one vm, i tried to delete repo and folders and uninstalled ES, but still its reflecting as installed already.
    so can you pls any reference link for this

[2021-05-28T08:04:41,257][INFO ][o.e.n.Node ] [odfe-c1] version[7.10.2], pid[116427], build[oss/rpm/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/3.10.0-1160.25.1.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2021-05-28T08:04:41,305][INFO ][o.e.n.Node ] [odfe-c1] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2021-05-28T08:04:41,305][INFO ][o.e.n.Node ] [odfe-c1] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-17150697033745127386, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///usr/share/elasticsearch/plugins/opendistro-performance-analyzer/pa_config/es_security.policy, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=oss, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2021-05-28T08:04:42,455][INFO ][c.a.o.e.p.c.PluginSettings] [odfe-c1] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2021-05-28T08:04:42,794][INFO ][c.a.o.s.s.t.OpenDistroSSLConfig] [odfe-c1] SSL dual mode is disabled
[2021-05-28T08:04:42,795][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [odfe-c1] ES Config path is /etc/elasticsearch
[2021-05-28T08:04:43,020][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [odfe-c1] JVM supports TLSv1.3
[2021-05-28T08:04:43,022][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [odfe-c1] Config directory is /etc/elasticsearch/, from there the key- and truststore files are resolved relatively
[2021-05-28T08:04:43,587][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [odfe-c1] TLS Transport Client Provider : JDK
[2021-05-28T08:04:43,587][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [odfe-c1] TLS Transport Server Provider : JDK
[2021-05-28T08:04:43,588][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [odfe-c1] TLS HTTP Provider : JDK
[2021-05-28T08:04:43,588][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [odfe-c1] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2, TLSv1.1]
[2021-05-28T08:04:43,588][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [odfe-c1] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2, TLSv1.1]
[2021-05-28T08:04:43,781][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [odfe-c1] Clustername: RV-opendistro-dev
“RV-opendistro-dev.log” 168L, 15024C

Let me answer your doubts one by one…

  1. I believe that you have installed ODFE using root privilege on two nodes. while on another you have that from user account. That’s not a problem.
  2. Yes you can use it using root user privilege’s depends on how you have installed it.
  3. For uninstalling ODFE I can’t find any document.
    You can simply remove related rpms(rpm -e name_of_rpm) and clean related directories manually like /etc/elasticsearch, /var/log/elasticsearch, /var/lib/elasticsearch, /usr/share/elasticsearch etc.
    @varma Cant identify anything from logs. Could you just let me know exact problem. Do you have all nodes in you cluster now?

Hi @pranali4796
below is the error i am getting now,can u please suggest.
if it is certificate related issue, then could you pls provide any to link or document to follow the same
note: i have ssl certificates already

[root@tstsdcapp41695 elasticsearch]# curl -XGET https://10.147.106.131:9200/_cat/nodes?v -u ‘admin:admin’ --insecure
Open Distro Security not initialized.
[root@tstsdcapp41695 elasticsearch]# curl -XGET https://master-ip:93sec^Ce/nodes?v -u ‘admin:admin’ --insecu
[root@tstsdcapp41695 elasticsearch]# curl -XGET https://master-ip:9300/_cat/nodes?v -u ‘admin:admin’ --insecure
curl: (58) NSS: client certificate not found (nickname not specified)

Hello Varma

It means that you have not completed installation and configuration. The error means that you have not run the securirty.sh, which is necessary to complete the authentication and authorization configuration.

Read the documentation and ensure to have all the steps completed successfully.