# Logstash config for Multiple pipelines usage **URL:** \ **Category:** Open Source Elasticsearch and Kibana\ **Tags:** configure, install\ **Created:** [March 29, 2024, 9:40am UTC](https://forum.opensearch.org/t/logstash-config-for-multiple-pipelines-usage/18623 "2024-03-29T09:40:03Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![gray653](https://avatars.discourse-cdn.com/v4/letter/g/5f9b8f/32.png) [@gray653](https://forum.opensearch.org/u/gray653)\ **Post date:** [March 29, 2024, 9:40am UTC](https://forum.opensearch.org/t/logstash-config-for-multiple-pipelines-usage/18623/1 "2024-03-29T09:40:03Z")
**Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser): OpenSearch 2.12 Logstash 8.9.0 Ubuntu 20.04 Firefox **Describe the issue** : Thing is i am trying to set up a Logstash pipeline with Auditbeat and Filebeat as inputs, and this is my pipelines.yml: ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/d/d645329f6d51b5187f40708593cf1ab61eb91257.png) Then when i try running it, it shows no error but there are no indices in my Opensearch Dashboard. Im considering using multiple pipelines right now. Can anyone explain to me why i see no index although there are no errors when trying to run that pipelines.yml? And can you show me how to config Logstash to use multiple pipelines? Thanks for you help, Gray ---
**Author:** ![gaobinlong](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/gaobinlong/32/5242_2.png) [@gaobinlong](https://forum.opensearch.org/u/gaobinlong)\ **Post date:** [April 1, 2024, 6:49am UTC](https://forum.opensearch.org/t/logstash-config-for-multiple-pipelines-usage/18623/2 "2024-04-01T06:49:44Z")
You can add console output to your logstash config to debug, like this: ```auto output { stdout { codec => rubydebug } } ``` , after tried that, I found in logstash 8.7.0, the `beat` field you used doesn’t exist, but the field `agent` can be used to get the beat type, and another metadata field `%{[@metadata][beat]}` can also be used to get the beat type, you can have a try, here is my config: ```auto input { beats { port => 5044 } } output { stdout { codec => rubydebug } } output { opensearch { hosts => ["http://localhost:9200"] index => "%{[@metadata][beat]}-%{[@metadata][version]}" } } ``` , the index and data in OpenSearch are as below: ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/7/7d3261fe323e61495dcf744d3e77f968b805f990.png) ---
**Author:** ![gray653](https://avatars.discourse-cdn.com/v4/letter/g/5f9b8f/32.png) [@gray653](https://forum.opensearch.org/u/gray653)\ **Post date:** [April 1, 2024, 9:08am UTC](https://forum.opensearch.org/t/logstash-config-for-multiple-pipelines-usage/18623/3 "2024-04-01T09:08:29Z")
Thanks for answering my question. I understood what u meant, and try to correct the pipelines as you said: ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/9/9dc3a12fd9ccd553b7b565ee1bea8d3866a3d948.png) I tried both [agent][type] and [agent.type] and [@metadata][beat] but i doesnt seem to change, the logs show no errors but there are no indices pushed to Opensearch Dashboard. I tried to push Auditbeat and Filebeat seperately and it works fine, and the field agent.type is there in both indices: ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/1/1274b26400d40f62fcfa25e12c5245eb2deb9379.png) So I’m really confused now. ---
**Author:** ![gaobinlong](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/gaobinlong/32/5242_2.png) [@gaobinlong](https://forum.opensearch.org/u/gaobinlong)\ **Post date:** [April 1, 2024, 9:50am UTC](https://forum.opensearch.org/t/logstash-config-for-multiple-pipelines-usage/18623/4 "2024-04-01T09:50:24Z")
The only difference between auditbeat and filebeat in the output is the index name, right? If so how about trying this and remove the if condition: `index => "%{[@metadata][beat]}-%{YYYY.MM.dd}"` ---
**Author:** ![gray653](https://avatars.discourse-cdn.com/v4/letter/g/5f9b8f/32.png) [@gray653](https://forum.opensearch.org/u/gray653)\ **Post date:** [April 1, 2024, 10:06am UTC](https://forum.opensearch.org/t/logstash-config-for-multiple-pipelines-usage/18623/5 "2024-04-01T10:06:10Z")
I tried what you suggested and it only pushes 1 index only: ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/397ea549b5d57b3a1ad760ec856206116d35a603.png) so i dont think that solves the problem well ---
**Author:** ![gaobinlong](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/gaobinlong/32/5242_2.png) [@gaobinlong](https://forum.opensearch.org/u/gaobinlong)\ **Post date:** [April 3, 2024, 5:32am UTC](https://forum.opensearch.org/t/logstash-config-for-multiple-pipelines-usage/18623/6 "2024-04-03T05:32:51Z")
It seems that `@` is missing before `metadata`. ---
**Author:** ![gray653](https://avatars.discourse-cdn.com/v4/letter/g/5f9b8f/32.png) [@gray653](https://forum.opensearch.org/u/gray653)\ **Post date:** [April 3, 2024, 8:29am UTC](https://forum.opensearch.org/t/logstash-config-for-multiple-pipelines-usage/18623/7 "2024-04-03T08:29:02Z")
In the pipelines.yml it still has it: ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/8/822d7fd15e7c73128b038d5b6a7f0e5b594be76b.png) ---
**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/33547ea01a5b12dcca2958411d3edd97ae2ea8c1.png) [@system](https://forum.opensearch.org/u/system)\ **Post date:** [June 2, 2024, 8:29am UTC](https://forum.opensearch.org/t/logstash-config-for-multiple-pipelines-usage/18623/8 "2024-06-02T08:29:45Z")
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.