Logstash with opensearch docker container

Security
, ,
1

hello
has anyone run logstash in docker container to connect opensearch as per docker-compose file

the above starts network with config_opensearch-net(file has opensearch-net)
below starts but unable to connect

also seen ssl_certificate_verification is required for opensearch and wont work with false on other forums and seen it my last run without logstash container

but documentation has below reference with( ssl_certificate_verification => false)

docker run -it --rm --name logstash --net config_opensearch-net opensearchproject/logstash-oss-with-opensearch-output-plugin:7.16.2 -e ‘input { stdin { } } output {
opensearch {
hosts => [“https://localhost:9200”]
index =^C"opensearch-logstash-docker-%{+YYYY.MM.dd}"
user => “osadmin”
password => “xxxx!”
ssl => true
ssl_certificate_verification => false
}
}’

[2022-06-21T16:42:49,608][WARN ][logstash.runner ] SIGINT received. Shutting down.
[2022-06-21T16:42:51,447][WARN ][logstash.outputs.opensearch][main] Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>“https://osadmin:xxxxxx@localhost:9200/”, :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=>“OpenSearch Unreachable: [https://osadmin:xxxxxx@localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)”}

seen below on othe places to make it work ,
can someone confirm if logstash container needs these steps to update cert to make it connect

echo -n | openssl s_client -connect localhost:9200 | sed -ne ‘/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p’ > ./ca_logstash.cer
keytool -import -alias saelk -file ca_logstash.cer -keystore /usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.65-3.b17.el7.x86_64/lib/security/cacerts
changeit (pass)

2

@infodata The mentioned OpenSearch docker-compose has a security plugin and demo certs disabled. As a result, the OpenSearch cluster runs on HTTP and is not secured.

That’s why your logstash docker might not to work as it tries to authenticate against HTTPS.

To run a secured cluster you either set the below options to false or comment them out.

"DISABLE_INSTALL_DEMO_CONFIG=true"
"DISABLE_SECURITY_PLUGIN=true"

Also, avoid using localhost in the host value. Use either FQDN/IP of the docker host or container name of the OpenSearch.
The network of the logstash container has to be exactly the same as OpenSearch container.

3

@pablo
my current opensearch dockercompose cluster is working on https
to connect to it https works but not http
the logstash is looking for cert need to import it ?

Edit:updating localhost to ip worked and it is https://
updating hosts =>[“https://10.1.1.x”]

which file should have this
“DISABLE_INSTALL_DEMO_CONFIG=true”
“DISABLE_SECURITY_PLUGIN=true”

4

@infodata You don’t need to define the cert if you use ssl_certificate_verification => false.

I’ve just tested your example and it worked in my lab. But I used FQDN of my docker host instead of localhost as localhost failed to work.

1 Like
5

@infodata The one you mentioned in your first post.

6

docker exec -it 216 bash

bash-4.2$ id
uid=1000(logstash) gid=1000(logstash) groups=1000(logstash)
bash-4.2$ pwd
/usr/share/logstash
bash-4.2$ cd config
bash-4.2$ ls
jvm.options log4j2.properties logstash-opensearch-sample.conf logstash-sample.conf logstash.yml pipelines.yml startup.options

bash-4.2$ pwd
/usr/share/logstash
bash-4.2$ cd pipeline/
bash-4.2$ ls
logstash.conf

for the logstash container which file is used as default conf or all files with *.conf in this folder are used ?

is there a working reference docker compose for logstash

7

@infodata This is a pure Logstash config question. I suggest following the documentation.