System: redhat 7,
installation: On Prem
Hello Community,
So a few days ago i had to investigate regarding Opensearch because its open source.
(still using Elasticsearch)
Now i need help regarding how to import or load logfile/logstash and important part about the security,
The Documentation on the website is not that good maybe u guys can help me out.
@amagonawin Could you describe your logs ingest pipeline?
You can use Logstash with the OpenSearch output plugin or Data Prepper to ingest data to OpenSearch.
How did you deploy your test OpenSearch cluster?
Hey, so i have many questions regarding logstash:
first: if i try to use
bin/logstash-plugin install logstash-output-opensearch
like in the Docu, nothing happens
BUT if i
bin/logstash -e "input { stdin { } } output { stdout { } }
it starts and works… even if i never “installed” it like in the Docu… dont know
I use the logstash plugin.
My pipeline.yml looks like this
input {
file {
path => "/var/log/opensearch/*"
}
}
filter {
json {
source => "message"
}
}
output {
opensearch {
hosts => ["//localhost:9200"]
index => "logstash-log-%(+YYYY.MM.dd)"
}
}
How can i use the logstash at my opensearch dashboard? It doesnt show up?
@amagonawin Did you use Logstash OSS with the OpenSearch output plugin.
The output part of your pipeline.yml is missing few things.
Check my example below.
opensearch {
index => "logstash-%{+YYYY.MM.dd}"
hosts => ["https://docker1.pablo.net:9200"]
user => admin
password => Eliatra123
ssl => true
ssl_certificate_verification => false
action => "create"
}
If you didn’t disable security plugin or HTTPS on the 9200 endpoint than you need to enable SSL in the OpenSearch output plugin.
@pablo
Thanks for the fast reply
Hello, i Downloaded : logstash-oss-with-opensearch-output-plugin-8.9.0-linux-x64.tar.gz
Ok, i will give it a shot.
Thanks
@pablo
still dont see any new logfile in my opensearch dashboard.
i startet it with bin/logstash -e “input { stdin { } } output { stdout { } }”
typed hello world, closed it, but i did not got any log entry…
@pablo
that is not the problem
but now i see, somehow he is not taking the changes i made in the yml file…
i also get that but i want to have the log file in my Dashboard, so that i can visuallize it, if you understand what i mean 
@amagonawin I understood that you didn’t get any output after typing “hello” in the input.
In that case this is expected. The command line is only for stdin and stdout and it overwrites the pipeline.yml config file. There will be no logs sent to OpenSearch.
To send the logs to OpenSearch you need to use opensearch output as I’ve shared before.
Also you need to run Logstash using one of the below commands.
./bin/logstash
./bin/logstash -f /path/to/your/conf.file
@pablo
Thanks
so my conf file was also in conf/logstash.conf
i put the same stuff inside as u showed above.
BUT… still getting errors like:
[WARN ][logstash.outputs.opensearch][main] Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"https://admin:xxxxxx@localhost:9200/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting OpenSearch at URL 'https://localhost:9200/'"}
Ok i see, when i created all that i used a " ! " in my password, somehow hes having issues with that, can i somehow change that?
@amagonawin You can change the password with OpenSearch Dashboards security plugin or with securityadmin.sh and internal_users.yml file.
If you used demo configuration then logstash:logstash should work.
@pablo
Hey, so its running and all, but now i have another question,
in my dashboard, it is shown, but i dont see the “message” that i tipped in?
just to be clear, if i want to load the logfiles into my dashboard i use:
→ ./bin/logstash -f /path/to/your/conf.file → do i type the message inside the comand line?
or
→ bin/logstash -e "input { stdin { } } output { stdout { } } or after i used this command?
This is how it looks like at my dashboards:
Im a bit confused at the moment, thanks
@amagonawin If you want to send stdin{} to OpenSearch then you need a different output.
This should do the trick.
config.file
input {
stdin{}
}
output {
opensearch {
index => "logstash-%{+YYYY.MM.dd}"
hosts => ["https://docker1.pablo.net:9200"]
user => admin
password => Eliatra123
ssl => true
ssl_certificate_verification => false
action => "create"
}
}
Also, you can run it as an oneliner.
./bin/logstash -e 'input { stdin {} } output { opensearch { index => "logstash-%{+YYYY.MM.dd}" hosts => ["https://docker3.pablo.net:9200"] user => "admin" password => "Eliatra123" ssl => true ssl_certificate_verification => false action => "create" } }'