Logstash with opensearch docker container

has anyone run logstash in docker container to connect opensearch as per docker-compose file

the above starts network with config_opensearch-net(file has opensearch-net)
below starts but unable to connect

also seen ssl_certificate_verification is required for opensearch and wont work with false on other forums and seen it my last run without logstash container

but documentation has below reference with( ssl_certificate_verification => false)

docker run -it --rm --name logstash --net config_opensearch-net opensearchproject/logstash-oss-with-opensearch-output-plugin:7.16.2 -e ‘input { stdin { } } output {
opensearch {
hosts => [“https://localhost:9200”]
index =^C"opensearch-logstash-docker-%{+YYYY.MM.dd}"
user => “osadmin”
password => “xxxx!”
ssl => true
ssl_certificate_verification => false

[2022-06-21T16:42:49,608][WARN ][logstash.runner ] SIGINT received. Shutting down.
[2022-06-21T16:42:51,447][WARN ][logstash.outputs.opensearch][main] Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>“https://osadmin:xxxxxx@localhost:9200/”, :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=>“OpenSearch Unreachable: [https://osadmin:xxxxxx@localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/] failed: Connection refused (Connection refused)”}

seen below on othe places to make it work ,
can someone confirm if logstash container needs these steps to update cert to make it connect

echo -n | openssl s_client -connect localhost:9200 | sed -ne ‘/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p’ > ./ca_logstash.cer
keytool -import -alias saelk -file ca_logstash.cer -keystore /usr/lib/jvm/jre-1.8.0-openjdk-
changeit (pass)

@infodata The mentioned OpenSearch docker-compose has a security plugin and demo certs disabled. As a result, the OpenSearch cluster runs on HTTP and is not secured.

That’s why your logstash docker might not to work as it tries to authenticate against HTTPS.

To run a secured cluster you either set the below options to false or comment them out.


Also, avoid using localhost in the host value. Use either FQDN/IP of the docker host or container name of the OpenSearch.
The network of the logstash container has to be exactly the same as OpenSearch container.

my current opensearch dockercompose cluster is working on https
to connect to it https works but not http
the logstash is looking for cert need to import it ?

Edit:updating localhost to ip worked and it is https://
updating hosts =>[“https://10.1.1.x”]

which file should have this

@infodata You don’t need to define the cert if you use ssl_certificate_verification => false.

I’ve just tested your example and it worked in my lab. But I used FQDN of my docker host instead of localhost as localhost failed to work.

1 Like

@infodata The one you mentioned in your first post.

docker exec -it 216 bash

bash-4.2$ id
uid=1000(logstash) gid=1000(logstash) groups=1000(logstash)
bash-4.2$ pwd
bash-4.2$ cd config
bash-4.2$ ls
jvm.options log4j2.properties logstash-opensearch-sample.conf logstash-sample.conf logstash.yml pipelines.yml startup.options

bash-4.2$ pwd
bash-4.2$ cd pipeline/
bash-4.2$ ls

for the logstash container which file is used as default conf or all files with *.conf in this folder are used ?

is there a working reference docker compose for logstash

@infodata This is a pure Logstash config question. I suggest following the documentation.