Hi All,
I tried deploying opendistro helm chart in my IPv6 k8s cluster and I am getting the below responses in pod logs
pod status were showing as running
[root@k8s-rmp-master-0 opendistro-es]$ kubectl get pods -w -n elastic
NAME READY STATUS RESTARTS AGE
elasticsearch-opendistro-es-client-7fbc9b877-h8jjx 1/1 Running 0 8m18s
elasticsearch-opendistro-es-data-0 1/1 Running 0 8m18s
elasticsearch-opendistro-es-kibana-5c454cb6bc-k6j4t 1/1 Running 0 8m17s
elasticsearch-opendistro-es-master-0 1/1 Running 0 8m18s
below is the logs,
data pod logs
[2021-09-30T07:00:19,318][WARN ][o.e.c.c.ClusterFormationFailureHelper] [elasticsearch-opendistro-es-data-0] master not discovered yet: have discovered [{elasticsearch-opendistro-es-data-0}{g9vpjnGhQQSZXm7TlzqHdA}{UNIUsbx3SEaXPduXxEXdIw}{127.0.0.1}{127.0.0.1:9300}{dr}]; discovery will continue using [[fd74:ca9b:3a09:868c:172:18:0:4fce]:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-30T07:00:19,550][WARN ][o.e.d.HandshakingTransportAddressConnector] [elasticsearch-opendistro-es-data-0] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:4fce]:9300]] completed handshake with [{elasticsearch-opendistro-es-master-0}{OqYEghRrTByIBtH3cdIulQ}{2P7FkFCdRbiAKWKVRlResw}{127.0.0.1}{127.0.0.1:9300}{mr}] but followup connection failed
org.elasticsearch.transport.ConnectTransportException: [elasticsearch-opendistro-es-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {elasticsearch-opendistro-es-data-0}{g9vpjnGhQQSZXm7TlzqHdA}{UNIUsbx3SEaXPduXxEXdIw}{127.0.0.1}{127.0.0.1:9300}{dr}
at org.elasticsearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:389) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.action.ActionListener$4.onResponse(ActionListener.java:157) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.TransportService$5.onResponse(TransportService.java:476) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.TransportService$5.onResponse(TransportService.java:466) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:54) [elasticsearch-7.10.2.jar:7.10.2]
at com.amazon.opendistroforelasticsearch.security.transport.OpenDistroSecurityInterceptor$RestoringTransportResponseHandler.handleResponse(OpenDistroSecurityInterceptor.java:278) [opendistro_security-1.13.1.0.jar:1.13.1.0]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1171) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:253) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:247) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:684) [elasticsearch-7.10.2.jar:7.10.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-30T07:00:19,565][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [elasticsearch-opendistro-es-data-0] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:369) ~[?:?]
master pod logs,
[2021-09-30T06:49:44,180][INFO ][o.e.h.AbstractHttpServerTransport] [elasticsearch-opendistro-es-master-0] publish_address {127.0.0.1:9200}, bound_addresses {[::]:9200}
[2021-09-30T06:49:44,180][INFO ][o.e.n.Node ] [elasticsearch-opendistro-es-master-0] started
[2021-09-30T06:49:44,181][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-opendistro-es-master-0] Node started
[2021-09-30T06:49:44,182][INFO ][c.a.o.s.c.ConfigurationRepository] [elasticsearch-opendistro-es-master-0] Will attempt to create index .opendistro_security and default configs if they are absent
[2021-09-30T06:49:44,182][INFO ][c.a.o.s.c.ConfigurationRepository] [elasticsearch-opendistro-es-master-0] Background init thread started. Install default config?: true
[2021-09-30T06:49:44,183][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-opendistro-es-master-0] 0 Open Distro Security modules loaded so far: []
[2021-09-30T06:49:44,211][INFO ][o.e.g.GatewayService ] [elasticsearch-opendistro-es-master-0] recovered [0] indices into cluster_state
[2021-09-30T06:49:44,353][INFO ][o.e.c.m.MetadataCreateIndexService] [elasticsearch-opendistro-es-master-0] [.opendistro_security] creating index, cause [api], templates [], shards [1]/[1]
[2021-09-30T06:49:44,371][INFO ][o.e.c.r.a.AllocationService] [elasticsearch-opendistro-es-master-0] Cluster health status changed from [YELLOW] to [RED] (reason: [index [.opendistro_security] created]).
[2021-09-30T06:50:14,471][INFO ][c.a.o.s.c.ConfigurationRepository] [elasticsearch-opendistro-es-master-0] Index .opendistro_security created?: true
[2021-09-30T06:50:14,481][INFO ][c.a.o.s.s.ConfigHelper ] [elasticsearch-opendistro-es-master-0] Will update 'config' with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/config.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2021-09-30T06:54:43,588][INFO ][c.a.o.j.s.JobSweeper ] [elasticsearch-opendistro-es-master-0] Running full sweep
[2021-09-30T06:59:43,592][INFO ][c.a.o.j.s.JobSweeper ] [elasticsearch-opendistro-es-master-0] Running full sweep
client pod logs
[2021-09-30T06:49:36,120][DEPRECATION][o.e.d.c.s.Settings ] [elasticsearch-opendistro-es-client-7fbc9b877-h8jjx] [node.master] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2021-09-30T06:49:36,422][INFO ][o.e.b.BootstrapChecks ] [elasticsearch-opendistro-es-client-7fbc9b877-h8jjx] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2021-09-30T06:49:44,911][WARN ][o.e.d.HandshakingTransportAddressConnector] [elasticsearch-opendistro-es-client-7fbc9b877-h8jjx] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:4fce]:9300]] completed handshake with [{elasticsearch-opendistro-es-master-0}{OqYEghRrTByIBtH3cdIulQ}{2P7FkFCdRbiAKWKVRlResw}{127.0.0.1}{127.0.0.1:9300}{mr}] but followup connection failed
org.elasticsearch.transport.ConnectTransportException: [elasticsearch-opendistro-es-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {elasticsearch-opendistro-es-client-7fbc9b877-h8jjx}{FJuEhVeiQ7OrP9re1Zrx5A}{A_lQTVXVSQWFQkQOsfj22A}{127.0.0.1}{127.0.0.1:9300}{ir}
at org.elasticsearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:389) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.action.ActionListener$4.onResponse(ActionListener.java:157) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.TransportService$5.onResponse(TransportService.java:476) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.TransportService$5.onResponse(TransportService.java:466) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:54) [elasticsearch-7.10.2.jar:7.10.2]
at com.amazon.opendistroforelasticsearch.security.transport.OpenDistroSecurityInterceptor$RestoringTransportResponseHandler.handleResponse(OpenDistroSecurityInterceptor.java:278) [opendistro_security-1.13.1.0.jar:1.13.1.0]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1171) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:253) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:247) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:684) [elasticsearch-7.10.2.jar:7.10.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-30T06:49:45,002][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [elasticsearch-opendistro-es-client-7fbc9b877-h8jjx] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:369) ~[?:?]
checked the cluster status inside the master pod and below is the response I got,
[root@elasticsearch-opendistro-es-master-0 elasticsearch]# curl -k -u admin:admin https://elasticsearch-opendistro-es-client-service:9200
Open Distro Security not initialized.[root@elasticsearch-opendistro-es-master-0 elasticsearch]#
[root@elasticsearch-opendistro-es-master-0 elasticsearch]#
[root@elasticsearch-opendistro-es-master-0 elasticsearch]#
[root@elasticsearch-opendistro-es-master-0 elasticsearch]# curl -k -u admin:admin https://elasticsearch-opendistro-es-client-service:9200/_cluster/health?pretty=true
Open Distro Security not initialized.
Attaching the opendistro-es helm which I tried in my IPv6 environment and the same was working in IPv4 k8s cluster.
Does opendistro elasticsearch helm chart not supported in IPv6 k8s cluster?
Please share your thoughts.
Thanks,
Ganeshbabu R