*I’m not good at English. sorry.
I have installed metricbeat-oss-7.11.1-linux-x86_64 and am suffering from an error not being able to connect to Elasticsearch.
My situation-------------
- My Elasticsearch is in a VPC.
- Metricbeat and Elasticsearch/Kibana are on the same instance.
- I’m still a beginner so I’m not sure if it’s correct, AES VPC security group contains Metricbeat instance IP. (Port 443)
Error message is
2021-03-08T10:15:21.461Z ERROR [esclientleg] eslegclient/connection.go:261 error connecting to Elasticsearch at https://vpc-My-Domain.ap-northeast-1.es.amazonaws.com:443: Get "https://vpc-My-Domain.ap-northeast-1.es.amazonaws.com:443": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2021-03-08T10:15:21.461Z ERROR instance/beat.go:971 Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at https://vpc-My-Domain.ap-northeast-1.es.amazonaws.com:443: Get "https://vpc-My-Domain.ap-northeast-1.es.amazonaws.com:443": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)]
Please give me advice.
1 Like
Hey @huwarihuwari ,
did you check if you can access odfe from/within your metricbeat vm? i.e via
curl -XGET https://vpc-My-Domain.ap-northeast-1.es.amazonaws.com -u 'admin:your_admin_password' --insecure
with the assumption that you had your own certificates (and of course port forwarding a default port of es:9200 to 443) and not using demo certificate as you can’t validate that DNS name given by your vpc.
So, first, check if you can access your elasticsearch cluster within your VM where metricbeat is running, and then afterwords you have to also enable ssl there as described here: Troubleshoot - Open Distro Documentation
b.t.w stick to the same version of the beats as ODFE is also bundled (Version History - Open Distro Documentation) i.e. ODFE 1.13.0 comes with es/kibana 7.10.2.
Feel free to also share more info i.e. your metricbeat yml file configuration so that we can better assess it.
Best,
1 Like
Thanks for your advice.
By connecting the instance and the subnet.
curl -XGET https://vpc-My-Domain.ap-northeast-1.es.amazonaws.com -u' admin: your_admin_password' --insecure
I was able to receive the response.
But I got a new error.
ERROR instance/beat.go:951 Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at https://vpc-My-Domain.ap-northeast-1.es.amazonaws.com:443: 401 Unauthorized: Unauthorized]
Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at https://vpc-My-Domain.ap-northeast-1.es.amazonaws.com:443: 401 Unauthorized: Unauthorized]
I set my username and password in metricbeat.yml with Elasticesearch master user access information(aes-siem.KibanaAdmin/aes-siem.KibanaPassword ), but it doesn’t work.
What should I enter here?
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
# Array of hosts to connect to.
#hosts: ["localhost:9200"]
hosts: ["https://vpc-My-Domain.ap-northeast-1.es.amazonaws.com:443"]
# Protocol - either `http` (default) or `https`.
protocol: "https"
# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
username: "Elasticsearch master user username"
password: "Elasticsearch master user password"
setup.ilm.enabled: false
setup.ilm.check_exists: false
Or is there another reason?
Hey @huwarihuwari ,
the error says 401 Unauthorized: Unauthorized – which indicates that the request has not been applied because it lacks valid authentication credentials for the target resource.
In order to test it, you should try to connect to your es instance using the same credentials you are using on the metricbeat.yml conf:
curl -XGET https://vpc-My-Domain.ap-northeast-1.es.amazonaws.com -u 'Elasticsearch master user username': change_me' -k
b.t.w what are you referring to "Elasticsearch master user username"? the admin user on ODFE? or you did set up another internal user which you are using? Also, does that user has permission to write to metricbeat-* as the default index name for metricbeat is [metricbeat-]YYYY.MM.DD.
If you can connect to ES via that user then the other option you may be missing is the ssl.certificate_authorities configurations as mentioned here: Troubleshoot - Open Distro Documentation
Try to apply these changes and see if that resolves the issue.
Thank you for your advice.
I was able to connect with the following command.
curl -XGET https://vpc-My-Domain.ap-northeast-1.es.amazonaws.com -u 'my admin user: my admin pw' -k
"Elasticsearch master user username" is Open Distro Kibana Admin User.
Sorry for the incomprehensible.
I check ssl.certificate_authorities.
Thanks for your cooperation.
I’ve tried setting ssl.certificate_authorities., but I don’t know where to find the information to enter.
(I can’t find SSL file on Open Distro Elasticserch)
I’m probably assuming it’s different in each environment, but what do generally enter?
Indeed. It is different on each deployed mode i.e. docker vs package manager.
Let’s say you did install it via Debian Package - Open Distro Documentation. The config folder which contains demo-certificates (root-ca.pem, node.pem, node-key.pem) is located at /etc/elasticsearch/ for (RPM and DEB) on each node: Debian Package - Open Distro Documentation which then you can use the same certificates (if you do not want to have a dedicated certificate for beats only).
Change the path on :
ssl.certificate_authorities:
- /full/path/to/root-ca.pem
ssl.certificate: "/full/path/to/client.pem"
ssl.key: "/full/path/to/client-key.pem"
from the ones there to your/odfe certificates.
Thank you for your reply.
I’m using Amazon Elasticserch serivice + Kibana.
Do you know where it is in this case?
Thank you for your kindness.
You are most welcome.
Reg. your question about Amazon Elasticsearch Service – it is out of my domain ;), maybe someone who is from AWS or even who has experience on that on AWS will be able to help you. I suppose they also use the package manager or k8s, but I can’t tell much there.