Hello,
I am getting errors in OpenDistro Elasticsearch in pod log.
All pods are running and Kibana pod is running, but still cant get to Kibana UI or find fix for the OD Elasticsearch error for es-master-0 pod.
Here is information on the issue:
Here are the pod listings:
$ kubectl get pods -n monitoring-tools
NAME READY STATUS RESTARTS AGE
es-client-86c6895b85-jfh75 1/1 Running 0 171m
es-client-86c6895b85-s4vdd 1/1 Running 0 171m
es-data-0 1/1 Running 0 170m
es-data-1 1/1 Running 0 169m
es-data-2 1/1 Running 0 168m
es-master-0 1/1 Running 0 172m
es-master-1 1/1 Running 0 172m
es-master-2 1/1 Running 0 172m
kibana-75f647cf75-g45sl 1/1 Running 0 166m
Here are the services listing:
$ kubectl get svc -n monitoring-tools
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
elasticsearch LoadBalancer 172.20.93.88 af62b9de1e11144f681231a6cee3a97a-3e9a7fb246023e05.elb.us-east-1.amazonaws.com 443:30054/TCP,9300:31492/TCP,9600:31568/TCP 3h9m
elasticsearch-data ClusterIP None 9300/TCP,9200/TCP,9600/TCP 3h7m
elasticsearch-discovery ClusterIP None 9300/TCP 3h11m
elasticsearch-master ClusterIP None 9300/TCP,9200/TCP,9600/TCP 3h9m
kibana LoadBalancer 172.20.15.40 a1948df9af93740cbadb15d068e1edf5-319eb37de0b06c5b.elb.us-east-1.amazonaws.com 443:30754/TCP 177m
Here is the describe information on the elasticsearch “es-mastor-0” pod:
$ kubectl describe pod/es-master-0 -n monitoring-tools
Name: es-master-0
Namespace: monitoring-tools
Priority: 0
Node: ip-10-133-64-141.ec2.internal/10.133.64.141
Start Time: Fri, 25 Sep 2020 09:50:20 -0700
Labels: component=elasticsearch
controller-revision-hash=es-master-854987689
role=master
statefulset.kubernetes.io/pod-name=es-master-0
Annotations: iam.amazonaws.com/role: arn:aws:iam::992376627226:role/eks-monitoring-tools-es-pod-role
kubernetes.io/psp: eks.privileged
Status: Running
IP: 100.64.19.199
IPs:
IP: 100.64.19.199
Controlled By: StatefulSet/es-master
Init Containers:
init-sysctl:
Container ID: docker://710ae08237671be5b01eac439aeb0fd0c3bbf4739b18d866de63ea0f1af453cb
Image: artifactory.tfs.toyota.com/dnecloud-docker-dev-local/busybox:1.30
Image ID: docker-pullable://artifactory.tfs.toyota.com/dnecloud-docker-dev-local/busybox@sha256:4fe8827f51a5e11bb83afa8227cbccb402df840d32c6b633b7ad079bc8144100
Port:
Host Port:
Command:
sysctl
-w
vm.max_map_count=262144
State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 25 Sep 2020 09:50:22 -0700
Finished: Fri, 25 Sep 2020 09:50:22 -0700
Ready: True
Restart Count: 0
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-6x2fv (ro)
Containers:
elasticsearch:
Container ID: docker://82183f558cc2769c387025818891a47375f743ce4926f04bd8c253e4dd108230
Image: artifactory.tfs.toyota.com/dnecloud-docker-dev-local/opendistro-for-elasticsearch-custom:1.9.0
Image ID: docker-pullable://artifactory.tfs.toyota.com/dnecloud-docker-dev-local/opendistro-for-elasticsearch-custom@sha256:003da704bac3b61d7fce6f65f03cc41355577ff01a83f670c40ed52b6cbec935
Ports: 9300/TCP, 9200/TCP, 9600/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
State: Running
Started: Fri, 25 Sep 2020 09:50:23 -0700
Ready: True
Restart Count: 0
Limits:
cpu: 1
Requests:
cpu: 1
Liveness: tcp-socket :transport delay=60s timeout=1s period=10s #success=1 #failure=3
Environment:
CLUSTER_NAME: logs
cluster.initial_master_nodes: es-master-0,es-master-1,es-master-2
NUMBER_OF_MASTERS: 3
NODE_MASTER: true
NODE_INGEST: false
NODE_DATA: false
NETWORK_HOST: 0.0.0.0
NODE_NAME: es-master-0 (v1:metadata.name)
DISCOVERY_SERVICE: elasticsearch-discovery
KUBERNETES_NAMESPACE: monitoring-tools (v1:metadata.namespace)
PROCESSORS: 1 (limits.cpu)
ES_JAVA_OPTS: -Xms1g -Xmx1g
Mounts:
/usr/share/elasticsearch/config/admin-crt.pem from certs (ro,path=“admin-crt.pem”)
/usr/share/elasticsearch/config/admin-key.pem from certs (ro,path=“admin-key.pem”)
/usr/share/elasticsearch/config/admin-root-ca.pem from certs (ro,path=“admin-root-ca.pem”)
/usr/share/elasticsearch/config/elasticsearch.yml from config (rw,path=“elasticsearch.yml”)
/usr/share/elasticsearch/config/elk-crt.pem from certs (ro,path=“elk-crt.pem”)
/usr/share/elasticsearch/config/elk-key.pem from certs (ro,path=“elk-key.pem”)
/usr/share/elasticsearch/config/elk-root-ca.pem from certs (ro,path=“elk-root-ca.pem”)
/usr/share/elasticsearch/config/logging.yml from config (rw,path=“logging.yml”)
/usr/share/elasticsearch/config/tfs-root-truststore.jks from certs (ro,path=“tfs-root-truststore.jks”)
/usr/share/elasticsearch/plugins/opendistro_security/securityconfig/config.yml from config (rw,path=“config.yml”)
/usr/share/elasticsearch/plugins/opendistro_security/securityconfig/internal_users.yml from config (rw,path=“internal_users.yml”)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-6x2fv (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: elasticsearch
Optional: false
certs:
Type: Secret (a volume populated by a Secret)
SecretName: elasticsearch-tls-data
Optional: false
default-token-6x2fv:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-6x2fv
Optional: false
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Here is log errors for my Elasticsearch es-master-0 pod:
[2020-09-25T20:06:00,077][WARN ][o.e.d.HandshakingTransportAddressConnector] [es-master-0] handshake failed for [connectToRemoteMasterNode[100.64.67.33:9300]]
org.elasticsearch.transport.RemoteTransportException: [es-master-1][100.64.67.33:9300][internal:transport/handshake]
Caused by: org.elasticsearch.ElasticsearchException: Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/
at com.amazon.opendistroforelasticsearch.security.ssl.util.ExceptionUtils.createBadHeaderException(ExceptionUtils.java:57) ~[?:?]
[2020-09-25T20:06:00,143][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:00,296][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:00,485][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:00,512][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:00,622][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:00,773][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:01,047][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:01,075][WARN ][o.e.d.HandshakingTransportAddressConnector] [es-master-0] handshake failed for [connectToRemoteMasterNode[100.64.129.228:9300]]
org.elasticsearch.transport.RemoteTransportException: [es-master-2][100.64.129.228:9300][internal:transport/handshake]
Caused by: org.elasticsearch.ElasticsearchException: Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/
at com.amazon.opendistroforelasticsearch.security.ssl.util.ExceptionUtils.createBadHeaderException(ExceptionUtils.java:57) ~[?:?]
[2020-09-25T20:06:01,144][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:01,298][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:01,488][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:01,515][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:01,629][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:01,772][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:02,039][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:02,073][WARN ][o.e.d.HandshakingTransportAddressConnector] [es-master-0] handshake failed for [connectToRemoteMasterNode[100.64.67.33:9300]]
org.elasticsearch.transport.RemoteTransportException: [es-master-1][100.64.67.33:9300][internal:transport/handshake]
Caused by: org.elasticsearch.ElasticsearchException: Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/
[2020-09-25T20:06:02,075][WARN ][o.e.d.HandshakingTransportAddressConnector] [es-master-0] handshake failed for [connectToRemoteMasterNode[100.64.129.228:9300]]
org.elasticsearch.transport.RemoteTransportException: [es-master-2][100.64.129.228:9300][internal:transport/handshake]
Caused by: org.elasticsearch.ElasticsearchException: Illegal parameter in http or transport request found. This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/
[2020-09-25T20:06:02,139][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:02,300][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:02,489][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:02,516][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:02,624][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:02,772][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:03,045][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:03,072][WARN ][o.e.d.HandshakingTransportAddressConnector] [es-master-0] handshake failed for [connectToRemoteMasterNode[100.64.129.228:9300]]
org.elasticsearch.transport.RemoteTransportException: [es-master-2][100.64.129.228:9300][internal:transport/handshake]
Caused by: org.elasticsearch.ElasticsearchException: Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/
[2020-09-25T20:06:03,139][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:03,295][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:03,492][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:03,519][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:03,623][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:03,775][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:04,045][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [es-master-0] ElasticsearchException[Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
[2020-09-25T20:06:04,075][WARN ][o.e.d.HandshakingTransportAddressConnector] [es-master-0] handshake failed for [connectToRemoteMasterNode[100.64.129.228:9300]]
org.elasticsearch.transport.RemoteTransportException: [es-master-2][100.64.129.228:9300][internal:transport/handshake]
Caused by: org.elasticsearch.ElasticsearchException: Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/
Any help would be greatly appreciated.
Freddie2020