Logstash pipeline with Auditbeat and Filebeat

[gray653]

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

OpenSearch 2.12
Logstash 8.9.0
Ubuntu 20.04
Firefox

Describe the issue:

I have installed and managed to use Opensearch and Opensearch Dashboard, as well as able to use Logstash to input logs to Opensearch Dashboard. But now i want to create a pipeline including Auditbeat, Filebeat to push data to Logstash and then to Opensearch. I have searched for a lot of docs but there are still no docs that have clear guide about that. Can someone suggest me docs or briefly explain to me what i should do?

Thanks
Gray

[jasonrojas]

I think this doc from elastic covers it fairly well -

Configure the Logstash output | Auditbeat Reference [7.17] | Elastic

You’ll just need to change the output to opensearch and make sure logstash has the opensearch output plugin.

If you are running a logstash version that doesnt support the opensearch plugin you would need an intermediary sink (output) like redis or similar and then an input on the logstash side to pull from redis ie: (auditbeat → redis → logstash)

[gray653]

I’ve managed to install auditbeat and now trying to install both filebeat and auditbeat to push files to logstash. Thank you so much for your help!

Gray

[gray653]

Im having another problem, this is the pipelines.yml after i use both Auditbeat and Filebeat as the input of Logstash

and when i try running the Logstash, it always shows this error:

saying that I lack something at line 121, but i don’t really know since i have checked my pipelines.yml many times.

Maybe the structure of my pipelines.yml is wrong since output must be seperated from filter?

[jasonrojas]

You’re mixing outputs inside of the filter block, logstash configs are

Input{}
Filter {}
Output {}

[gray653]

Oh right thanks i also thought that my structure seems wrong

Thanks so much for your help.