I try ingest the data from s3 bucket to opensearch dashboard using logstash

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

-bash-4.2# /opt/logstash-8.13.4/bin/logstash -f /opt/logstash-8.13.4/config/pipelines.yml
Using bundled JDK: /opt/logstash-8.13.4/jdk
/opt/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_int
/opt/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_f
Sending Logstash logs to /opt/logstash-8.13.4/logs which is now configured via log4j2.properties
[2024-05-15T15:12:10,417][INFO ][logstash.runner          ] Log4j configuration path used is: /opt/logstash-8.13.4/config/log4j2.properties
[2024-05-15T15:12:10,419][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.13.4", "jruby.version"=>"jruby 9.4.5.0 (3.1.4) 2023-11-02 1abae2700f OpenJDK 64-Bit Server VM 17.0.11+9 on 17.0.11+9 +indy +jit [x86_64-linux]"}
[2024-05-15T15:12:10,422][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
[2024-05-15T15:12:10,424][INFO ][logstash.runner          ] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000`
[2024-05-15T15:12:10,424][INFO ][logstash.runner          ] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000`
[2024-05-15T15:12:10,626][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2024-05-15T15:12:10,635][FATAL][logstash.runner          ] Logstash could not be started because there is already another instance using the configured data directory.  If you wish to run multiple instances, you must change the "path.data" setting.
[2024-05-15T15:12:10,639][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:808) ~[jruby.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:767) ~[jruby.jar:?]
        at opt.logstash_minus_8_dot_13_dot_4.lib.bootstrap.environment.<main>(/opt/logstash-8.13.4/lib/bootstrap/environment.rb:90) ~[?:?]
-bash-4.2#

Describe the issue:

Configuration:

Relevant Logs or Screenshots:

Hi @rathiga ,

Please check if the logstash service is already running on your server. To do that, please execute the following command:

systemctl status logstash

-bash-4.2# systemctl status -l logstash
● logstash.service - Logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2024-05-15 14:56:22 UTC; 2 weeks 1 days ago
     Docs: https://www.elastic.co/guide/en/logstash/current/index.html
 Main PID: 14311 (java)
   CGroup: /system.slice/logstash.service
           └─14311 /opt/logstash-8.13.4/jdk/bin/java -Xms1g -Xmx1g -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Dlog4j2.isThreadContextMapInheritable=true -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000 -Dlogstash.jackson.stream-read-constraints.max-number-length=10000 -Djruby.regexp.interruptible=true -Djdk.io.File.enableADS=true --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-opens=java.base/java.security=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio.channels=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED -Dio.netty.allocator.maxOrder=11 -cp /opt/logstash-8.13.4/vendor/jruby/lib/jruby.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/checker-qual-3.33.0.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/commons-codec-1.15.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/commons-compiler-3.1.0.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/commons-logging-1.2.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/error_prone_annotations-2.18.0.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/failureaccess-1.0.1.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/google-java-format-1.15.0.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/guava-32.1.2-jre.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/httpclient-4.5.13.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/httpcore-4.4.14.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/jackson-annotations-2.15.3.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/jackson-core-2.15.3.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/jackson-databind-2.15.3.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/jackson-dataformat-cbor-2.15.3.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/janino-3.1.0.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/javassist-3.29.0-GA.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/jsr305-3.0.2.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/jvm-options-parser-8.13.4.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/log4j-1.2-api-2.17.1.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/log4j-api-2.17.1.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/log4j-core-2.17.1.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/log4j-jcl-2.17.1.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/log4j-slf4j-impl-2.17.1.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/logstash-core.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/reflections-0.10.2.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/slf4j-api-1.7.32.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/snakeyaml-2.2.jar org.logstash.Logstash -f /opt/logstash-8.13.4/config/pipelines.yml

May 30 14:56:33 ip-172-31-4-236.us-west-2.compute.internal logstash[14311]: /opt/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/logstash-integration-aws-7.1.6-java/lib/logstash/inputs/s3.rb:403:in `download_remote_file'
May 30 14:56:33 ip-172-31-4-236.us-west-2.compute.internal logstash[14311]: /opt/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/logstash-integration-aws-7.1.6-java/lib/logstash/inputs/s3.rb:378:in `process_log'
May 30 14:56:33 ip-172-31-4-236.us-west-2.compute.internal logstash[14311]: /opt/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/logstash-integration-aws-7.1.6-java/lib/logstash/inputs/s3.rb:189:in `block in process_files'
May 30 14:56:33 ip-172-31-4-236.us-west-2.compute.internal logstash[14311]: org/jruby/RubyArray.java:1989:in `each'
May 30 14:56:33 ip-172-31-4-236.us-west-2.compute.internal logstash[14311]: /opt/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/logstash-integration-aws-7.1.6-java/lib/logstash/inputs/s3.rb:185:in `process_files'
May 30 14:56:33 ip-172-31-4-236.us-west-2.compute.internal logstash[14311]: /opt/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/logstash-integration-aws-7.1.6-java/lib/logstash/inputs/s3.rb:130:in `block in run'
May 30 14:56:33 ip-172-31-4-236.us-west-2.compute.internal logstash[14311]: /opt/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/interval.rb:20:in `interval'
May 30 14:56:33 ip-172-31-4-236.us-west-2.compute.internal logstash[14311]: /opt/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/logstash-integration-aws-7.1.6-java/lib/logstash/inputs/s3.rb:129:in `run'
May 30 14:56:33 ip-172-31-4-236.us-west-2.compute.internal logstash[14311]: /opt/logstash-8.13.4/logstash-core/lib/logstash/java_pipeline.rb:414:in `inputworker'
May 30 14:56:33 ip-172-31-4-236.us-west-2.compute.internal logstash[14311]: /opt/logstash-8.13.4/logstash-core/lib/logstash/java_pipeline.rb:405:in `block in start_input'
-bash-4.2#

hi @all
i create the input, filter, output stages in pipelines.yml but it not ingest the data

Hi @rathiga,

You cannot start the Logstash instance because there is already a Logstash service running.

Please execute the following command:

systemctl stop logstash

Could you also share your Logstash configurations?

-bash-4.2# ./logstash -f /config/pipelines.yml
Using bundled JDK: /opt/logstash-8.9.0/jdk
Sending Logstash logs to /opt/logstash-8.9.0/logs which is now configured via log4j2.properties
[2024-06-05T16:03:20,112][INFO ][logstash.runner ] Log4j configuration path used is: /opt/logstash-8.9.0/config/log4j2.properties
[2024-06-05T16:03:20,114][INFO ][logstash.runner ] Starting Logstash {“logstash.version”=>“8.9.0”, “jruby.version”=>“jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 17.0.7+7 on 17.0.7+7 +indy +jit [x86_64-linux]”}
[2024-06-05T16:03:20,148][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2024-06-05T16:03:21,754][WARN ][logstash.config.source.multilocal] Ignoring the ‘pipelines.yml’ file because modules or command line options are specified
/opt/logstash-8.9.0/vendor/bundle/jruby/2.6.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:938: warning: constant Tilt::Cache is deprecated
[2024-06-05T16:03:23,610][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>“/config/pipelines.yml”}
[2024-06-05T16:03:23,640][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2024-06-05T16:03:24,706][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2024-06-05T16:03:24,741][INFO ][logstash.runner ] Logstash shut down.
[2024-06-05T16:03:24,795][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
at opt.logstash_minus_8_dot_9_dot_0.lib.bootstrap.environment.(/opt/logstash-8.9.0/lib/bootstrap/environment.rb:91) ~[?:?]

  1. vi /opt/logstash-8.9.0/config/pipelines.yml

pipeline.id: main
pipeline.workers: 1
pipeline.batch.size: 500
queue.type: persisted
queue.max_bytes: 1gb # Default = 1024mb
queue.drain: true
pipeline.output.workers: 1
path.config: “/opt/logstash-8.9.0/logstash-sample.conf”

cd /opt/logstash-8.9.0/config/logstash-sample.conf

input {
s3 {
access_key_id => “AKIAUNWJFISALFXWOPVQ”
secret_access_key => “1LeN8fzeq+AC1zs2SsyZtHsy+avPva5UgcVWvb8u”
bucket => “serverlogs-01backup”
region => “us-west-2”
type => “s3”
prefix => “appserver/rpmappserver-01/ribbideo_logs/”
interval => 60
}
}

output {
opensearch {
hosts => [“http://35.91.61.103:9200”]
index => “rpmappserver-01ribbideo_logs”
user => “admin”
password => “admin”
ssl_certificate_verification => false
}
}

-bash-4.2# sudo systemctl status -l logstash.service
● logstash.service - Logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2024-06-05 16:19:32 UTC; 24s ago
Docs: Logstash Reference [8.14] | Elastic
Main PID: 540 (java)
CGroup: /system.slice/logstash.service
└─540 /opt/logstash-8.9.0/jdk/bin/java -Xms1g -Xmx1g -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Dlog4j2.isThreadContextMapInheritable=true -Djruby.regexp.interruptible=true -Djdk.io.File.enableADS=true --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-opens=java.base/java.security=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio.channels=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED -cp /opt/logstash-8.9.0/vendor/jruby/lib/jruby.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/checker-qual-3.12.0.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/commons-codec-1.15.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/commons-compiler-3.1.0.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/commons-logging-1.2.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/error_prone_annotations-2.11.0.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/failureaccess-1.0.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/google-java-format-1.15.0.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/guava-31.1-jre.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/httpclient-4.5.13.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/httpcore-4.4.14.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/j2objc-annotations-1.3.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jackson-annotations-2.14.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jackson-core-2.14.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jackson-databind-2.14.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jackson-dataformat-cbor-2.14.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jackson-dataformat-yaml-2.14.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/janino-3.1.0.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/javassist-3.29.0-GA.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jsr305-3.0.2.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jvm-options-parser-8.9.0.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/log4j-1.2-api-2.17.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/log4j-api-2.17.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/log4j-core-2.17.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/log4j-jcl-2.17.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/log4j-slf4j-impl-2.17.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/logstash-core.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/reflections-0.10.2.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/slf4j-api-1.7.32.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/snakeyaml-1.33.jar org.logstash.Logstash -f /opt/logstash-8.9.0/config/pipelines.yml

Jun 05 16:19:32 ip-172-31-4-236.us-west-2.compute.internal systemd[1]: Started Logstash.
Jun 05 16:19:32 ip-172-31-4-236.us-west-2.compute.internal logstash[540]: Using bundled JDK: /opt/logstash-8.9.0/jdk
-bash-4.2#

Could you please execute the following command after stopping logstash?

ps -ef | grep logstash

i use one ec2 server for all the three opensearch,
opensearch dashboard & logstash

It’s possible to install it using a single working node.

yes ,now I got the date from s3 bucket but the output is not reflected in opensearch

in logstash
bin/Logstash -f /opt/logstash-8.9.0/config/conf.d/pipe.conf

[logstash.outputs.opensearch][main] Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>“http://admin:xxxxxx@35.91.61.103:9200/”, :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=>“OpenSearch Unreachable: [http://admin:xxxxxx@35.91.61.103:9200/][Manticore::SocketException] Connect to 35.91.61.103:9200 [/35.91.61.103] failed: Connection refused”}

Please change here from http to https

changed & tried
-bash-4.2# bin/logstash -f /opt/logstash-8.9.0/config/conf.d/pipe.conf
Using bundled JDK: /opt/logstash-8.9.0/jdk
Sending Logstash logs to /opt/logstash-8.9.0/logs which is now configured via log4j2.properties
[2024-06-07T15:35:15,943][INFO ][logstash.runner ] Log4j configuration path used is: /opt/logstash-8.9.0/config/log4j2.properties
[2024-06-07T15:35:15,945][INFO ][logstash.runner ] Starting Logstash {“logstash.version”=>“8.9.0”, “jruby.version”=>“jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 17.0.7+7 on 17.0.7+7 +indy +jit [x86_64-linux]”}
[2024-06-07T15:35:15,949][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2024-06-07T15:35:16,162][WARN ][logstash.config.source.multilocal] Ignoring the ‘pipelines.yml’ file because modules or command line options are specified
/opt/logstash-8.9.0/vendor/bundle/jruby/2.6.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:938: warning: constant Tilt::Cache is deprecated
[2024-06-07T15:35:16,757][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2024-06-07T15:35:17,363][INFO ][org.reflections.Reflections] Reflections took 283 ms to scan 1 urls, producing 132 keys and 464 values
[2024-06-07T15:35:19,743][INFO ][logstash.javapipeline ] Pipeline main is configured with pipeline.ecs_compatibility: v8 setting. All plugins in this pipeline will default to ecs_compatibility => v8 unless explicitly configured otherwise.
[2024-06-07T15:35:19,762][INFO ][logstash.outputs.opensearch][main] New OpenSearch output {:class=>“LogStash::Outputs::OpenSearch”, :hosts=>[“https://35.91.61.103:9200”]}
[2024-06-07T15:35:19,764][WARN ][logstash.outputs.opensearch][main] ** WARNING ** Detected UNSAFE options in opensearch output configuration!
** WARNING ** You have enabled encryption but DISABLED certificate verification.
** WARNING ** To make sure your data is secure change :ssl_certificate_verification to true
[2024-06-07T15:35:19,904][INFO ][logstash.outputs.opensearch][main] OpenSearch pool URLs updated {:changes=>{:removed=>, :added=>[https://admin:xxxxxx@35.91.61.103:9200/]}}
[2024-06-07T15:35:20,128][WARN ][logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=>“https://admin:xxxxxx@35.91.61.103:9200/”}
[2024-06-07T15:35:20,166][INFO ][logstash.outputs.opensearch][main] Cluster version determined (2.12.0) {:version=>2}
[2024-06-07T15:35:20,205][INFO ][logstash.outputs.opensearch][main] Using a default mapping template {:version=>2, :ecs_compatibility=>:v8}
[2024-06-07T15:35:20,208][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>“main”, “pipeline.workers”=>4, “pipeline.batch.size”=>125, “pipeline.batch.delay”=>50, “pipeline.max_inflight”=>500, “pipeline.sources”=>[“/opt/logstash-8.9.0/config/conf.d/pipe.conf”], :thread=>“#<Thread:0x1e129dc9@/opt/logstash-8.9.0/logstash-core/lib/logstash/java_pipeline.rb:134 run>”}
[2024-06-07T15:35:20,280][INFO ][logstash.outputs.opensearch][main] Installing OpenSearch template {:name=>“ecs-logstash”}
[2024-06-07T15:35:21,071][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {“seconds”=>0.86}
[2024-06-07T15:35:21,078][INFO ][logstash.inputs.s3 ][main] Registering {:bucket=>“serverlogs-01backup”, :region=>“us-west-2”}
[2024-06-07T15:35:21,272][INFO ][logstash.javapipeline ][main] Pipeline started {“pipeline.id”=>“main”}
[2024-06-07T15:35:21,285][INFO ][logstash.inputs.s3 ][main][1f690fb617f7cda8ffddcf4fa7b46f3bd5acc41298602ce361bb52244feb0bc3] Using default generated file for the sincedb {:filename=>“/opt/logstash-8.9.0/data/plugins/inputs/s3/sincedb_86f0e9b9f3fbe965c79680c4ad84fb16”}
[2024-06-07T15:35:21,293][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
/opt/logstash-8.9.0/vendor/bundle/jruby/2.6.0/gems/manticore-0.9.1-java/lib/manticore/client.rb:284: warning: already initialized constant Manticore::Client::HttpPost

Hi @rathiga,

According to the example here, you also need to add ssl => true to the output in the logstash-sample.conf.

Please let me know whether it helped you.