How to ingest logs using logstash in EC2 server to Opensearch EC2 server dashboard

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Hi,
I have an AWS EC2 server in which we have installed Opensearch Dashboard and Openserach.
We got OpenSearch Dashboard in browser.

I have an another EC2 server in which we have logstash to ingest logs to above Opensearch Dashboard but cant succedd it when I ran below command.

./bin/logstash -f /root/logstash-8.4.0/config/pipelines.conf
Using bundled JDK: /root/logstash-8.4.0/jdk
Sending Logstash logs to /root/logstash-8.4.0/logs which is now configured via log4j2.properties
[2024-02-21T13:53:56,534][INFO ][logstash.runner ] Log4j configuration path used is: /root/logstash-8.4.0/config/log4j2.properties
[2024-02-21T13:53:56,544][INFO ][logstash.runner ] Starting Logstash {“logstash.version”=>“8.4.0”, “jruby.version”=>“jruby 9.3.6.0 (2.6.8) 2022-06-27 7a2cbcd376 OpenJDK 64-Bit Server VM 17.0.4+8 on 17.0.4+8 +indy +jit [x86_64-linux]”}
[2024-02-21T13:53:56,553][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
Your settings are invalid. Reason: Setting “xpack.management.enabled” doesn’t exist. Please check if you haven’t made a typo.
[2024-02-21T13:53:56,588][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
at root.logstash_minus_8_dot_4_dot_0.lib.bootstrap.environment.(/root/logstash-8.4.0/lib/bootstrap/environment.rb:91) ~[?:?]

So can you guide me how can we get logs from one server to Opensearch Dashboard Indices.

Regards,
MM

Describe the issue:

Configuration:

Relevant Logs or Screenshots:

2

@MMuralidharan

Your settings are invalid. Reason: Setting “xpack.management.enabled” doesn’t exist.

This option is reserved for Elasticsearch.

Did you follow this OpenSearch documentation and use the OpenSearch output plugin with Logstash?

3

How can ingest logs from local EC2 server to Open-search dashboard? I have installed filebeat by using dpkg -i filebeat-8.12.2-amd64.deb but facing below error when check # filebeat test output command.

parse url… OK
connection…
parse host… OK
dns lookup… OK
addresses: 10.254.148.150
dial up… OK
TLS…
security: server’s certificate chain verification is enabled
handshake… ERROR x509: certificate signed by unknown authority

Regards,
MM

4

This might be caused by the self-signed certificate of Logstash or OpenSearch (depends which service you’re pointing to in the filebeat configuration).

Try disabling certificate verification in the filebeat config (ssl.verification_mode: none) or provide rootCA of the target service ( certificate_authorities).