Recently we switched to Opensearch from ELK of elasticsearch.
I have created a docker-compose.yml for bringing up all the three containers (Opensearch, opensearch-dashboard, logstash-output-plugin) from opensearchproject registry.
Here is my docker-compose.yml file
version: '3'
services:
opensearch-node1:
image: opensearchproject/opensearch:latest
container_name: opensearch-node1
environment:
- cluster.name=opensearch-cluster
- node.name=opensearch-node1
- discovery.seed_hosts=opensearch-node1,opensearch-node2
- cluster.initial_master_nodes=opensearch-node1,opensearch-node2
- bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
hard: 65536
volumes:
- opensearch-data1:/usr/share/opensearch/data
ports:
- 0.0.0.0:9200:9200
- 0.0.0.0:9600:9600 # required for Performance Analyzer
networks:
- opensearch-net
opensearch-node2:
image: opensearchproject/opensearch:latest
container_name: opensearch-node2
environment:
- cluster.name=opensearch-cluster
- node.name=opensearch-node2
- discovery.seed_hosts=opensearch-node1,opensearch-node2
- cluster.initial_master_nodes=opensearch-node1,opensearch-node2
- bootstrap.memory_lock=true
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
volumes:
- opensearch-data2:/usr/share/opensearch/data
networks:
- opensearch-net
logstash:
image: opensearchproject/logstash-oss-with-opensearch-output-plugin:latest
container_name: logstash
ports:
- 0.0.0.0:5044:5044/udp
volumes:
- ./logstash.conf/:/logstash_dir/logstash.conf
networks:
- opensearch-net
opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:latest
container_name: opensearch-dashboards
ports:
- 0.0.0.0:5601:5601
expose:
- "5601"
environment:
OPENSEARCH_HOSTS: '["http://opensearch:9200","https://opensearch:9200"]'
networks:
- opensearch-net
volumes:
opensearch-data1:
opensearch-data2:
networks:
opensearch-net:
For now I am able to access the dashboard using 5601 port.
Now my question is how can I ship/send the data from different sources using filebeat?
Kindly assist me configuring filebeat with opensearch, so that I can send the data to opensearch and it will be visible for analysis on dashboard.
I have 3 servers where logstash is running as docker container and all three has
pipeline.
When I check on opensearch-dashboard. I can see data is coming from one host only not form all the three, however data is coming from all three servers/client.
I’m not sure I have the best answer here. One thing that I know has caused problems for others is when you run in a container and try to enrich with the hostname it normally pulls in the container ID. Which host does it show they are coming from?
@dtaivpp yes, it is displaying a container id as host in opensearch-dashboard.
I am running three logstash containers on a separate servers and sending output to single opensearch host. However, the data is coming/displaying from one logstash container. Expectation is here I should get data from three different pipeline/container on to opensearch-dashboard along with their container id.
Ah I think I understand. Have you checked for logs on the individual logstash containers to see if they are reporting any errors connecting to the cluster?
No errors are reporting. All the logstash container are connecting to opensearch successfully…
Three pipeline are working together from different different sources. Hope that should not be an issue.
@dtaivpp The question here is can we run 3 logstash container on 3 different server and send the output to opensearch. If yes, then I have three different logstash container on three servers and I am able to recieve the data form one server only, however I am expecting to receive the data from all three hosts…
We have available filed called “host” under opensearch-dashboard which should display all three logstash container id…
So yes you should be able to send from all 3 logstash hosts to OpenSearch without issues. If the logstash hosts are not reporting errors then it likely means they have connected to the cluster fine.
What I would check and verify at this point is the following:
Try and confirm that the Logstash containers are actually getting logs that should be shipped (you can add an output to std.out or a file to confirm).
Log into the OpenSearch container that you are shipping the logs to and confirm they are being received and indexed without errors.
@dtaivpp I have tried sending events/data to output file and it completely capturing the data. No issues while sending the data to file or output opensearch.
Today also i am working on it to get the expected results. However, opensearch-dashboard is not reporting all three agents together in host field.
Previous I used ELK stack with Filebeat and was getting proper output in host with 3 connect client/agents. To get that information i used following process in filebeat.
processors:
- add_cloud_metadata: ~
Here i think the working mechanism is different than filebeat.
We are not getting the all three hosts together with the data… Can you please let us know the functionality is the same as filebeat or there are any difference in between filebeat and logstash configuration.
I am not sure what is missing here or what is causing that all the is not displayed together. Need your assistance Please.
I have been trying to fix my issue since long and not able to find the solutions so far.
The issue which I am facing is I have 3 logstash client/agents running on 3 separate server and opensearch and opensearch-dashboard is running on another server. All (opensearch, opensearch-dashboard and logstash-oss-with-ouput-plugin) are running as a container using docker-compose.
I am able to send the data from all the logstash client to “opensearch” successfully. I can all the logs/data in opensearch-dashboard.
Now the big challenge here is I am not getting all three logstash client details on opensearch dashboard under the “host” filed. It should display all three container id of logstash client, from where I am getting the data. Sometime it shows two container id’s, sometime one and sometime three.
Just to compare the things Kibana, I was using ELK with filebeat and was able to see all three container were available under “host” filed. I am expecting the same behavior on opensearch-dashboard.
We have also tried to increase the No of Rows value to 9999 from the advance settings. Default value is 500. However, we are facing the same issue that not getting 3 host values together in the dashabord.