Docker run logstash as container

OpenSearch
1

I have opensearch /dashboards running on net config_opensearch-net
have a logstash.conf and a csv file in local nfs folder /usr/share/opensearch/logstash
trying to run below to use local folder volume

docker run -it --rm --name logstash -v /usr/share/opensearch/logstash:/usr/share/logstash/pipeline --net config_opensearch-net opensearchproject/logstash-oss-with-opensearch-output-plugin:7.16.2
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2022-06-21T20:23:11,037][INFO ][logstash.runner ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
[2022-06-21T20:23:11,085][INFO ][logstash.runner ] Starting Logstash {“logstash.version”=>“7.16.2”, “jruby.version”=>“jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]”}
[2022-06-21T20:23:11,193][INFO ][logstash.settings ] Creating directory {:setting=>“path.queue”, :path=>“/usr/share/logstash/data/queue”}
[2022-06-21T20:23:11,218][INFO ][logstash.settings ] Creating directory {:setting=>“path.dead_letter_queue”, :path=>“/usr/share/logstash/data/dead_letter_queue”}
[2022-06-21T20:23:11,948][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>“ea5d5b82-fa6b-48b6-af35-89022ab7cc5f”, :path=>“/usr/share/logstash/data/uuid”}
[2022-06-21T20:23:12,903][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-06-21T20:23:13,197][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>“LogStash::ConfigurationError”, :message=>“Expected one of [ \t\r\n], "#", "input", "filter", "output" at line 1, column 1 (byte 1)”, :backtrace=>[“/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:187:in initialize’”, “org/logstash/execution/JavaBasePipelineExt.java:72:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in initialize’”, “/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:383:in block in converge_state’”]}
[2022-06-21T20:23:13,297][INFO ][logstash.runner ] Logstash shut down.
[2022-06-21T20:23:13,307][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
at usr.share.logstash.lib.bootstrap.environment.(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]

2

I was able to get it working by using specific custom logstash.conf file (-f logstash.conf)

docker run -it --rm --name logstash -v /usr/share/opensearch/logstash/:/usr/share/logstash/pipeline/ --net config_opensearch-net opensearchproject/logstash-oss-with-opensearch-output-plugin:7.16.2 -f /usr/share/logstash/pipeline/logstash.conf

the default conf files have # comment and from reading others these could be issue
https://stackoverflow.com/questions/59639877/logstash-configurationerror-message-expected-one-of-t-r-n-i

bash-4.2$ vi logstash-opensearch-sample.conf
bash-4.2$ grep “#” logstash-opensearch-sample.conf logstash-sample.conf
logstash-opensearch-sample.conf:# Sample Logstash configuration for creating a simple
logstash-opensearch-sample.conf:# Beats → Logstash → OpenSearch pipeline.
logstash-opensearch-sample.conf: #user => “admin”
logstash-opensearch-sample.conf: #password => “admin”
logstash-sample.conf:# Sample Logstash configuration for creating a simple
logstash-sample.conf:# Beats → Logstash → Elasticsearch pipeline.
logstash-sample.conf: #user => “elastic”
logstash-sample.conf: #password => “changeme”
bash-4.2$ pwd
/usr/share/logstash/config

3

Here’s a docker compose that I use, if you want

version: '3'
services:
  logstash:
    image: opensearchproject/logstash-oss-with-opensearch-output-plugin
    container_name: logstash
    restart: unless-stopped
    ports:
      - 5044:5044
    volumes:
      - /docker/app/logstash/pipeline/:/usr/share/logstash/pipeline/

- /docker/app/logstash/pipeline/ change this to where ever you want to store the pipelines