Logstash config for Multiple pipelines usage

[gray653]

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

OpenSearch 2.12
Logstash 8.9.0
Ubuntu 20.04
Firefox

Describe the issue:

Thing is i am trying to set up a Logstash pipeline with Auditbeat and Filebeat as inputs, and this is my pipelines.yml:

Then when i try running it, it shows no error but there are no indices in my Opensearch Dashboard. Im considering using multiple pipelines right now.

Can anyone explain to me why i see no index although there are no errors when trying to run that pipelines.yml? And can you show me how to config Logstash to use multiple pipelines?

Thanks for you help,
Gray

[gaobinlong]

You can add console output to your logstash config to debug, like this:

output { stdout { codec => rubydebug } }

, after tried that, I found in logstash 8.7.0, the beat field you used doesn’t exist, but the field agent can be used to get the beat type, and another metadata field %{[@metadata][beat]} can also be used to get the beat type, you can have a try, here is my config:

input {
  beats {
    port => 5044
  }
}

output { stdout { codec => rubydebug } }

output {
  opensearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}"
  }
}

, the index and data in OpenSearch are as below:

[gray653]

Thanks for answering my question.

I understood what u meant, and try to correct the pipelines as you said:

I tried both [agent][type] and [agent.type] and [@metadata][beat] but i doesnt seem to change, the logs show no errors but there are no indices pushed to Opensearch Dashboard. I tried to push Auditbeat and Filebeat seperately and it works fine, and the field agent.type is there in both indices:

So I’m really confused now.

[gaobinlong]

The only difference between auditbeat and filebeat in the output is the index name, right? If so how about trying this and remove the if condition:
index => "%{[@metadata][beat]}-%{YYYY.MM.dd}"

[gray653]

I tried what you suggested and it only pushes 1 index only:

so i dont think that solves the problem well

[gaobinlong]

It seems that @ is missing before metadata.

[gray653]

In the pipelines.yml it still has it: