# I try ingest the data from s3 bucket to opensearch dashboard using logstash

**URL:** <https://forum.opensearch.org/t/i-try-ingest-the-data-from-s3-bucket-to-opensearch-dashboard-using-logstash/19388>\
**Category:** OpenSearch\
**Created:** [May 15, 2024, 3:19pm UTC](https://forum.opensearch.org/t/i-try-ingest-the-data-from-s3-bucket-to-opensearch-dashboard-using-logstash/19388 "2024-05-15T15:19:25Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![rathiga](https://avatars.discourse-cdn.com/v4/letter/r/8edcca/32.png) [@rathiga](https://forum.opensearch.org/u/rathiga)\
**Post date:** [May 15, 2024, 3:19pm UTC](https://forum.opensearch.org/t/i-try-ingest-the-data-from-s3-bucket-to-opensearch-dashboard-using-logstash/19388/1 "2024-05-15T15:19:26Z")

</div>

**Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser):

```auto
-bash-4.2# /opt/logstash-8.13.4/bin/logstash -f /opt/logstash-8.13.4/config/pipelines.yml
Using bundled JDK: /opt/logstash-8.13.4/jdk
/opt/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_int
/opt/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_f
Sending Logstash logs to /opt/logstash-8.13.4/logs which is now configured via log4j2.properties
[2024-05-15T15:12:10,417][INFO][logstash.runner] Log4j configuration path used is: /opt/logstash-8.13.4/config/log4j2.properties
[2024-05-15T15:12:10,419][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"8.13.4", "jruby.version"=>"jruby 9.4.5.0 (3.1.4) 2023-11-02 1abae2700f OpenJDK 64-Bit Server VM 17.0.11+9 on 17.0.11+9 +indy +jit [x86_64-linux]"}
[2024-05-15T15:12:10,422][INFO][logstash.runner] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
[2024-05-15T15:12:10,424][INFO][logstash.runner] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000`
[2024-05-15T15:12:10,424][INFO][logstash.runner] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000`
[2024-05-15T15:12:10,626][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2024-05-15T15:12:10,635][FATAL][logstash.runner] Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.
[2024-05-15T15:12:10,639][FATAL][org.logstash.Logstash] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:808) ~[jruby.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:767) ~[jruby.jar:?]
        at opt.logstash_minus_8_dot_13_dot_4.lib.bootstrap.environment.<main>(/opt/logstash-8.13.4/lib/bootstrap/environment.rb:90) ~[?:?]
-bash-4.2#

```

**Describe the issue** :

**Configuration** :

**Relevant Logs or Screenshots** :

---

<div class="post-metadata">

**Author:** ![Eugene7](https://avatars.discourse-cdn.com/v4/letter/e/bbce88/32.png) [@Eugene7](https://forum.opensearch.org/u/Eugene7)\
**Post date:** [May 16, 2024, 12:09pm UTC](https://forum.opensearch.org/t/i-try-ingest-the-data-from-s3-bucket-to-opensearch-dashboard-using-logstash/19388/2 "2024-05-16T12:09:15Z")

</div>

Hi @rathiga ,

Please check if the logstash service is already running on your server. To do that, please execute the following command:

`systemctl status logstash`

---

<div class="post-metadata">

**Author:** ![rathiga](https://avatars.discourse-cdn.com/v4/letter/r/8edcca/32.png) [@rathiga](https://forum.opensearch.org/u/rathiga)\
**Post date:** [May 30, 2024, 3:04pm UTC](https://forum.opensearch.org/t/i-try-ingest-the-data-from-s3-bucket-to-opensearch-dashboard-using-logstash/19388/3 "2024-05-30T15:04:13Z")

</div>

```auto
-bash-4.2# systemctl status -l logstash
● logstash.service - Logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2024-05-15 14:56:22 UTC; 2 weeks 1 days ago
     Docs: https://www.elastic.co/guide/en/logstash/current/index.html
 Main PID: 14311 (java)
   CGroup: /system.slice/logstash.service
           └─14311 /opt/logstash-8.13.4/jdk/bin/java -Xms1g -Xmx1g -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Dlog4j2.isThreadContextMapInheritable=true -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000 -Dlogstash.jackson.stream-read-constraints.max-number-length=10000 -Djruby.regexp.interruptible=true -Djdk.io.File.enableADS=true --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-opens=java.base/java.security=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio.channels=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED -Dio.netty.allocator.maxOrder=11 -cp /opt/logstash-8.13.4/vendor/jruby/lib/jruby.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/checker-qual-3.33.0.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/commons-codec-1.15.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/commons-compiler-3.1.0.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/commons-logging-1.2.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/error_prone_annotations-2.18.0.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/failureaccess-1.0.1.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/google-java-format-1.15.0.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/guava-32.1.2-jre.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/httpclient-4.5.13.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/httpcore-4.4.14.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/jackson-annotations-2.15.3.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/jackson-core-2.15.3.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/jackson-databind-2.15.3.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/jackson-dataformat-cbor-2.15.3.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/janino-3.1.0.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/javassist-3.29.0-GA.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/jsr305-3.0.2.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/jvm-options-parser-8.13.4.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/log4j-1.2-api-2.17.1.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/log4j-api-2.17.1.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/log4j-core-2.17.1.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/log4j-jcl-2.17.1.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/log4j-slf4j-impl-2.17.1.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/logstash-core.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/reflections-0.10.2.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/slf4j-api-1.7.32.jar:/opt/logstash-8.13.4/logstash-core/lib/jars/snakeyaml-2.2.jar org.logstash.Logstash -f /opt/logstash-8.13.4/config/pipelines.yml

May 30 14:56:33 ip-172-31-4-236.us-west-2.compute.internal logstash[14311]: /opt/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/logstash-integration-aws-7.1.6-java/lib/logstash/inputs/s3.rb:403:in `download_remote_file'
May 30 14:56:33 ip-172-31-4-236.us-west-2.compute.internal logstash[14311]: /opt/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/logstash-integration-aws-7.1.6-java/lib/logstash/inputs/s3.rb:378:in `process_log'
May 30 14:56:33 ip-172-31-4-236.us-west-2.compute.internal logstash[14311]: /opt/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/logstash-integration-aws-7.1.6-java/lib/logstash/inputs/s3.rb:189:in `block in process_files'
May 30 14:56:33 ip-172-31-4-236.us-west-2.compute.internal logstash[14311]: org/jruby/RubyArray.java:1989:in `each'
May 30 14:56:33 ip-172-31-4-236.us-west-2.compute.internal logstash[14311]: /opt/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/logstash-integration-aws-7.1.6-java/lib/logstash/inputs/s3.rb:185:in `process_files'
May 30 14:56:33 ip-172-31-4-236.us-west-2.compute.internal logstash[14311]: /opt/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/logstash-integration-aws-7.1.6-java/lib/logstash/inputs/s3.rb:130:in `block in run'
May 30 14:56:33 ip-172-31-4-236.us-west-2.compute.internal logstash[14311]: /opt/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/interval.rb:20:in `interval'
May 30 14:56:33 ip-172-31-4-236.us-west-2.compute.internal logstash[14311]: /opt/logstash-8.13.4/vendor/bundle/jruby/3.1.0/gems/logstash-integration-aws-7.1.6-java/lib/logstash/inputs/s3.rb:129:in `run'
May 30 14:56:33 ip-172-31-4-236.us-west-2.compute.internal logstash[14311]: /opt/logstash-8.13.4/logstash-core/lib/logstash/java_pipeline.rb:414:in `inputworker'
May 30 14:56:33 ip-172-31-4-236.us-west-2.compute.internal logstash[14311]: /opt/logstash-8.13.4/logstash-core/lib/logstash/java_pipeline.rb:405:in `block in start_input'
-bash-4.2#
```

---

<div class="post-metadata">

**Author:** ![rathiga](https://avatars.discourse-cdn.com/v4/letter/r/8edcca/32.png) [@rathiga](https://forum.opensearch.org/u/rathiga)\
**Post date:** [June 3, 2024, 3:30pm UTC](https://forum.opensearch.org/t/i-try-ingest-the-data-from-s3-bucket-to-opensearch-dashboard-using-logstash/19388/4 "2024-06-03T15:30:14Z")

</div>

hi @all  
i create the input, filter, output stages in pipelines.yml but it not ingest the data

---

<div class="post-metadata">

**Author:** ![Eugene7](https://avatars.discourse-cdn.com/v4/letter/e/bbce88/32.png) [@Eugene7](https://forum.opensearch.org/u/Eugene7)\
**Post date:** [June 4, 2024, 5:11pm UTC](https://forum.opensearch.org/t/i-try-ingest-the-data-from-s3-bucket-to-opensearch-dashboard-using-logstash/19388/5 "2024-06-04T17:11:15Z")

</div>

Hi @rathiga,

You cannot start the Logstash instance because there is already a Logstash service running.

Please execute the following command:

```auto
systemctl stop logstash

```

---

<div class="post-metadata">

**Author:** ![Eugene7](https://avatars.discourse-cdn.com/v4/letter/e/bbce88/32.png) [@Eugene7](https://forum.opensearch.org/u/Eugene7)\
**Post date:** [June 4, 2024, 5:12pm UTC](https://forum.opensearch.org/t/i-try-ingest-the-data-from-s3-bucket-to-opensearch-dashboard-using-logstash/19388/6 "2024-06-04T17:12:08Z")

</div>

Could you also share your Logstash configurations?

---

<div class="post-metadata">

**Author:** ![rathiga](https://avatars.discourse-cdn.com/v4/letter/r/8edcca/32.png) [@rathiga](https://forum.opensearch.org/u/rathiga)\
**Post date:** [June 5, 2024, 4:16pm UTC](https://forum.opensearch.org/t/i-try-ingest-the-data-from-s3-bucket-to-opensearch-dashboard-using-logstash/19388/7 "2024-06-05T16:16:48Z")

</div>

-bash-4.2# ./logstash -f /config/pipelines.yml  
Using bundled JDK: /opt/logstash-8.9.0/jdk  
Sending Logstash logs to /opt/logstash-8.9.0/logs which is now configured via log4j2.properties  
[2024-06-05T16:03:20,112][INFO][logstash.runner] Log4j configuration path used is: /opt/logstash-8.9.0/config/log4j2.properties  
[2024-06-05T16:03:20,114][INFO][logstash.runner] Starting Logstash {“logstash.version”=\>“8.9.0”, “jruby.version”=\>“jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 17.0.7+7 on 17.0.7+7 +indy +jit [x86\_64-linux]”}  
[2024-06-05T16:03:20,148][INFO][logstash.runner] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]  
[2024-06-05T16:03:21,754][WARN][logstash.config.source.multilocal] Ignoring the ‘pipelines.yml’ file because modules or command line options are specified  
/opt/logstash-8.9.0/vendor/bundle/jruby/2.6.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:938: warning: constant Tilt::Cache is deprecated  
[2024-06-05T16:03:23,610][INFO][logstash.config.source.local.configpathloader] No config files found in path {:path=\>“/config/pipelines.yml”}  
[2024-06-05T16:03:23,640][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.  
[2024-06-05T16:03:24,706][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600, :ssl\_enabled=\>false}  
[2024-06-05T16:03:24,741][INFO][logstash.runner] Logstash shut down.  
[2024-06-05T16:03:24,795][FATAL][org.logstash.Logstash] Logstash stopped processing because of an error: (SystemExit) exit  
org.jruby.exceptions.SystemExit: (SystemExit) exit  
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]  
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]  
at opt.logstash\_minus\_8\_dot\_9\_dot\_0.lib.bootstrap.environment.(/opt/logstash-8.9.0/lib/bootstrap/environment.rb:91) ~[?:?]

---

<div class="post-metadata">

**Author:** ![rathiga](https://avatars.discourse-cdn.com/v4/letter/r/8edcca/32.png) [@rathiga](https://forum.opensearch.org/u/rathiga)\
**Post date:** [June 5, 2024, 4:17pm UTC](https://forum.opensearch.org/t/i-try-ingest-the-data-from-s3-bucket-to-opensearch-dashboard-using-logstash/19388/8 "2024-06-05T16:17:42Z")

</div>

1. vi /opt/logstash-8.9.0/config/pipelines.yml

[pipeline.id](http://pipeline.id/): main  
pipeline.workers: 1  
pipeline.batch.size: 500  
queue.type: persisted  
queue.max\_bytes: 1gb # Default = 1024mb  
queue.drain: true  
pipeline.output.workers: 1  
path.config: “/opt/logstash-8.9.0/logstash-sample.conf”

---

<div class="post-metadata">

**Author:** ![rathiga](https://avatars.discourse-cdn.com/v4/letter/r/8edcca/32.png) [@rathiga](https://forum.opensearch.org/u/rathiga)\
**Post date:** [June 5, 2024, 4:18pm UTC](https://forum.opensearch.org/t/i-try-ingest-the-data-from-s3-bucket-to-opensearch-dashboard-using-logstash/19388/9 "2024-06-05T16:18:08Z")

</div>

cd /opt/logstash-8.9.0/config/logstash-sample.conf

input {  
s3 {  
access\_key\_id =\> “AKIAUNWJFISALFXWOPVQ”  
secret\_access\_key =\> “1LeN8fzeq+AC1zs2SsyZtHsy+avPva5UgcVWvb8u”  
bucket =\> “serverlogs-01backup”  
region =\> “us-west-2”  
type =\> “s3”  
prefix =\> “appserver/rpmappserver-01/ribbideo\_logs/”  
interval =\> 60  
}  
}

output {  
opensearch {  
hosts =\> [“[http://35.91.61.103:9200](http://35.91.61.103:9200)”]  
index =\> “rpmappserver-01ribbideo\_logs”  
user =\> “admin”  
password =\> “admin”  
ssl\_certificate\_verification =\> false  
}  
}

---

<div class="post-metadata">

**Author:** ![rathiga](https://avatars.discourse-cdn.com/v4/letter/r/8edcca/32.png) [@rathiga](https://forum.opensearch.org/u/rathiga)\
**Post date:** [June 5, 2024, 4:21pm UTC](https://forum.opensearch.org/t/i-try-ingest-the-data-from-s3-bucket-to-opensearch-dashboard-using-logstash/19388/10 "2024-06-05T16:21:57Z")

</div>

-bash-4.2# sudo systemctl status -l logstash.service  
● logstash.service - Logstash  
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)  
Active: active (running) since Wed 2024-06-05 16:19:32 UTC; 24s ago  
Docs: [Logstash Reference [8.14] | Elastic](https://www.elastic.co/guide/en/logstash/current/index.html)  
Main PID: 540 (java)  
CGroup: /system.slice/logstash.service  
└─540 /opt/logstash-8.9.0/jdk/bin/java -Xms1g -Xmx1g -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Dlog4j2.isThreadContextMapInheritable=true -Djruby.regexp.interruptible=true -Djdk.io.File.enableADS=true --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-opens=java.base/java.security=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio.channels=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED -cp /opt/logstash-8.9.0/vendor/jruby/lib/jruby.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/checker-qual-3.12.0.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/commons-codec-1.15.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/commons-compiler-3.1.0.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/commons-logging-1.2.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/error\_prone\_annotations-2.11.0.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/failureaccess-1.0.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/google-java-format-1.15.0.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/guava-31.1-jre.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/httpclient-4.5.13.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/httpcore-4.4.14.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/j2objc-annotations-1.3.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jackson-annotations-2.14.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jackson-core-2.14.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jackson-databind-2.14.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jackson-dataformat-cbor-2.14.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jackson-dataformat-yaml-2.14.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/janino-3.1.0.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/javassist-3.29.0-GA.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jsr305-3.0.2.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/jvm-options-parser-8.9.0.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/log4j-1.2-api-2.17.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/log4j-api-2.17.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/log4j-core-2.17.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/log4j-jcl-2.17.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/log4j-slf4j-impl-2.17.1.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/logstash-core.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/reflections-0.10.2.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/slf4j-api-1.7.32.jar:/opt/logstash-8.9.0/logstash-core/lib/jars/snakeyaml-1.33.jar org.logstash.Logstash -f /opt/logstash-8.9.0/config/pipelines.yml

Jun 05 16:19:32 ip-172-31-4-236.us-west-2.compute.internal systemd[1]: Started Logstash.  
Jun 05 16:19:32 ip-172-31-4-236.us-west-2.compute.internal logstash[540]: Using bundled JDK: /opt/logstash-8.9.0/jdk  
-bash-4.2#

---

<div class="post-metadata">

**Author:** ![Eugene7](https://avatars.discourse-cdn.com/v4/letter/e/bbce88/32.png) [@Eugene7](https://forum.opensearch.org/u/Eugene7)\
**Post date:** [June 7, 2024, 2:03pm UTC](https://forum.opensearch.org/t/i-try-ingest-the-data-from-s3-bucket-to-opensearch-dashboard-using-logstash/19388/11 "2024-06-07T14:03:51Z")

</div>

Could you please execute the following command after stopping logstash?

```auto
ps -ef | grep logstash

```

---

<div class="post-metadata">

**Author:** ![rathiga](https://avatars.discourse-cdn.com/v4/letter/r/8edcca/32.png) [@rathiga](https://forum.opensearch.org/u/rathiga)\
**Post date:** [June 7, 2024, 2:06pm UTC](https://forum.opensearch.org/t/i-try-ingest-the-data-from-s3-bucket-to-opensearch-dashboard-using-logstash/19388/12 "2024-06-07T14:06:33Z")

</div>

i use one ec2 server for all the three opensearch,  
opensearch dashboard & logstash

---

<div class="post-metadata">

**Author:** ![Eugene7](https://avatars.discourse-cdn.com/v4/letter/e/bbce88/32.png) [@Eugene7](https://forum.opensearch.org/u/Eugene7)\
**Post date:** [June 7, 2024, 3:22pm UTC](https://forum.opensearch.org/t/i-try-ingest-the-data-from-s3-bucket-to-opensearch-dashboard-using-logstash/19388/13 "2024-06-07T15:22:12Z")

</div>

It’s possible to install it using a single working node.

---

<div class="post-metadata">

**Author:** ![rathiga](https://avatars.discourse-cdn.com/v4/letter/r/8edcca/32.png) [@rathiga](https://forum.opensearch.org/u/rathiga)\
**Post date:** [June 7, 2024, 3:25pm UTC](https://forum.opensearch.org/t/i-try-ingest-the-data-from-s3-bucket-to-opensearch-dashboard-using-logstash/19388/14 "2024-06-07T15:25:14Z")

</div>

yes ,now I got the date from s3 bucket but the output is not reflected in opensearch

in logstash  
bin/Logstash -f /opt/logstash-8.9.0/config/conf.d/pipe.conf

[logstash.outputs.opensearch][main] Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=\>“[http://admin:xxxxxx@35.91.61.103:9200/](http://admin:xxxxxx@35.91.61.103:9200/)”, :exception=\>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=\>“OpenSearch Unreachable: [[http://admin:xxxxxx@35.91.61.103:9200/][Manticore::SocketException]](http://admin:xxxxxx@35.91.61.103:9200/%5D%5BManticore::SocketException%5D) Connect to 35.91.61.103:9200 [/35.91.61.103] failed: Connection refused”}

---

<div class="post-metadata">

**Author:** ![Eugene7](https://avatars.discourse-cdn.com/v4/letter/e/bbce88/32.png) [@Eugene7](https://forum.opensearch.org/u/Eugene7)\
**Post date:** [June 7, 2024, 3:29pm UTC](https://forum.opensearch.org/t/i-try-ingest-the-data-from-s3-bucket-to-opensearch-dashboard-using-logstash/19388/15 "2024-06-07T15:29:14Z")

</div>

> [@rathiga](#):
>
> hosts =\> [“[http://35.91.61.103:9200](http://35.91.61.103:9200)”]

Please change here from `http` to `https`

---

<div class="post-metadata">

**Author:** ![rathiga](https://avatars.discourse-cdn.com/v4/letter/r/8edcca/32.png) [@rathiga](https://forum.opensearch.org/u/rathiga)\
**Post date:** [June 7, 2024, 3:38pm UTC](https://forum.opensearch.org/t/i-try-ingest-the-data-from-s3-bucket-to-opensearch-dashboard-using-logstash/19388/16 "2024-06-07T15:38:25Z")

</div>

changed & tried  
-bash-4.2# bin/logstash -f /opt/logstash-8.9.0/config/conf.d/pipe.conf  
Using bundled JDK: /opt/logstash-8.9.0/jdk  
Sending Logstash logs to /opt/logstash-8.9.0/logs which is now configured via log4j2.properties  
[2024-06-07T15:35:15,943][INFO][logstash.runner] Log4j configuration path used is: /opt/logstash-8.9.0/config/log4j2.properties  
[2024-06-07T15:35:15,945][INFO][logstash.runner] Starting Logstash {“logstash.version”=\>“8.9.0”, “jruby.version”=\>“jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 17.0.7+7 on 17.0.7+7 +indy +jit [x86\_64-linux]”}  
[2024-06-07T15:35:15,949][INFO][logstash.runner] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]  
[2024-06-07T15:35:16,162][WARN][logstash.config.source.multilocal] Ignoring the ‘pipelines.yml’ file because modules or command line options are specified  
/opt/logstash-8.9.0/vendor/bundle/jruby/2.6.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:938: warning: constant Tilt::Cache is deprecated  
[2024-06-07T15:35:16,757][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600, :ssl\_enabled=\>false}  
[2024-06-07T15:35:17,363][INFO][org.reflections.Reflections] Reflections took 283 ms to scan 1 urls, producing 132 keys and 464 values  
[2024-06-07T15:35:19,743][INFO][logstash.javapipeline] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.  
[2024-06-07T15:35:19,762][INFO][logstash.outputs.opensearch][main] New OpenSearch output {:class=\>“LogStash::Outputs::OpenSearch”, :hosts=\>[“[https://35.91.61.103:9200](https://35.91.61.103:9200)”]}  
[2024-06-07T15:35:19,764][WARN][logstash.outputs.opensearch][main] \*\* WARNING \*\* Detected UNSAFE options in opensearch output configuration!  
\*\* WARNING \*\* You have enabled encryption but DISABLED certificate verification.  
\*\* WARNING \*\* To make sure your data is secure change :ssl\_certificate\_verification to true  
[2024-06-07T15:35:19,904][INFO][logstash.outputs.opensearch][main] OpenSearch pool URLs updated {:changes=\>{:removed=\>, :added=\>[[https://admin:xxxxxx@35.91.61.103:9200/](https://admin:xxxxxx@35.91.61.103:9200/)]}}  
[2024-06-07T15:35:20,128][WARN][logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=\>“[https://admin:xxxxxx@35.91.61.103:9200/](https://admin:xxxxxx@35.91.61.103:9200/)”}  
[2024-06-07T15:35:20,166][INFO][logstash.outputs.opensearch][main] Cluster version determined (2.12.0) {:version=\>2}  
[2024-06-07T15:35:20,205][INFO][logstash.outputs.opensearch][main] Using a default mapping template {:version=\>2, :ecs\_compatibility=\>:v8}  
[2024-06-07T15:35:20,208][INFO][logstash.javapipeline][main] Starting pipeline {:pipeline\_id=\>“main”, “pipeline.workers”=\>4, “pipeline.batch.size”=\>125, “pipeline.batch.delay”=\>50, “pipeline.max\_inflight”=\>500, “pipeline.sources”=\>[“/opt/logstash-8.9.0/config/conf.d/pipe.conf”], :thread=\>“#\<Thread:0x1e129dc9@/opt/logstash-8.9.0/logstash-core/lib/logstash/java\_pipeline.rb:134 run\>”}  
[2024-06-07T15:35:20,280][INFO][logstash.outputs.opensearch][main] Installing OpenSearch template {:name=\>“ecs-logstash”}  
[2024-06-07T15:35:21,071][INFO][logstash.javapipeline][main] Pipeline Java execution initialization time {“seconds”=\>0.86}  
[2024-06-07T15:35:21,078][INFO][logstash.inputs.s3][main] Registering {:bucket=\>“serverlogs-01backup”, :region=\>“us-west-2”}  
[2024-06-07T15:35:21,272][INFO][logstash.javapipeline][main] Pipeline started {“pipeline.id”=\>“main”}  
[2024-06-07T15:35:21,285][INFO][logstash.inputs.s3][main][1f690fb617f7cda8ffddcf4fa7b46f3bd5acc41298602ce361bb52244feb0bc3] Using default generated file for the sincedb {:filename=\>“/opt/logstash-8.9.0/data/plugins/inputs/s3/sincedb\_86f0e9b9f3fbe965c79680c4ad84fb16”}  
[2024-06-07T15:35:21,293][INFO][logstash.agent] Pipelines running {:count=\>1, :running\_pipelines=\>[:main], :non\_running\_pipelines=\>}  
/opt/logstash-8.9.0/vendor/bundle/jruby/2.6.0/gems/manticore-0.9.1-java/lib/manticore/client.rb:284: warning: already initialized constant Manticore::Client::HttpPost

---

<div class="post-metadata">

**Author:** ![Eugene7](https://avatars.discourse-cdn.com/v4/letter/e/bbce88/32.png) [@Eugene7](https://forum.opensearch.org/u/Eugene7)\
**Post date:** [June 24, 2024, 12:17pm UTC](https://forum.opensearch.org/t/i-try-ingest-the-data-from-s3-bucket-to-opensearch-dashboard-using-logstash/19388/17 "2024-06-24T12:17:55Z")

</div>

Hi @rathiga,

According to the example [here](https://opensearch.org/docs/latest/tools/logstash/index/), you also need to add `ssl => true` to the output in the `logstash-sample.conf`.

Please let me know whether it helped you.