# Help with configuring filebeat and logstash for multiple input **URL:** \ **Category:** General Feedback\ **Tags:** configure\ **Created:** [February 24, 2025, 3:37pm UTC](https://forum.opensearch.org/t/help-with-configuring-filebeat-and-logstash-for-multiple-input/23549 "2025-02-24T15:37:34Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![apt](https://avatars.discourse-cdn.com/v4/letter/a/858c86/32.png) [@apt](https://forum.opensearch.org/u/apt)\ **Post date:** [February 24, 2025, 3:37pm UTC](https://forum.opensearch.org/t/help-with-configuring-filebeat-and-logstash-for-multiple-input/23549/1 "2025-02-24T15:37:34Z")
Hello, I have two servers with multiple application that log to different locations. Log differ in structure so I need to do at least a few filters in logstash. On each server I have Filebeat installed. On the third server I have Logstash and Opensearch installed. What I want to achieve is to build a dashboard that will show errors per server, application type etc. How to configure Filebeat/Logstash to separate logs so they will be send to different indices in Opensearch? What would be the best approach? Could you please give me an example? Regards! ---
**Author:** ![nateynate](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/nateynate/32/4913_2.png) [@nateynate](https://forum.opensearch.org/u/nateynate)\ **Post date:** [February 24, 2025, 7:51pm UTC](https://forum.opensearch.org/t/help-with-configuring-filebeat-and-logstash-for-multiple-input/23549/2 "2025-02-24T19:51:08Z")
> [@apt](#): > > How to configure Filebeat/Logstash to separate logs so they will be send to different indices in Opensearch? Hi @apt ! I’ll see if I can’t write you some specific examples. Here’s how I’ve accomplished this. I can’t say if this is best practice or not. 1. Add a `filter{}` section and use the `grok` filter to compare the message contents to a particular pattern. If so, add some tags to them. 2. In your `output {}` sections, use more `if {}` conditionals to check for those tags, and based on their content, provide an `index` statement to specify a target index. So suppose you’ve got an input from a file and you’re using the `grok` filter to check for a particular message layout (this may or may not be syntactically correct): ```auto input { file { path => "/var/log/some.log" } } filter { grok { match => { your match pattern, see documentation } add_tag => "some_tag" } } # Specify a destination based on the tag you added in filter {} output { if [tag] => "some_tag" { opensearch { opensearch plugin option (port, username, index name, etc., see documentation) } } else if [tag] => "some_other_tag" { opensearch { opensearch { opensearch plugin options with different index name } } } } # end of output section ``` Youd’ likely have an `input {}` section that specifies the `beats` input plugin. Logstash is technically an Elastic product, so the specifics stanzas and incantations are available from their documentation. But, this oughtta get you started. Let me know if there’s some specific challenge you’re trying to overcome. Or if you’re just trying to get started, that’s fine too. Don’t forget to share your solution! ---
**Author:** ![apt](https://avatars.discourse-cdn.com/v4/letter/a/858c86/32.png) [@apt](https://forum.opensearch.org/u/apt)\ **Post date:** [February 25, 2025, 12:46pm UTC](https://forum.opensearch.org/t/help-with-configuring-filebeat-and-logstash-for-multiple-input/23549/3 "2025-02-25T12:46:06Z")
@nateynate thanks for your reply! This is my logstash.conf: ```auto input { beats { port => 5044 type => app_logfile } } filter { if [type] == "app_logfile" { if [message] !~ / ERROR / { drop {} } else { grok { match => { 'message' => "%{DATA:app_timestamp} \[%{DATA:worker_number}\] %{LOGLEVEL:log_level} %{DATA:method}: %{GREEDYDATA:log} " } } date { match => ["app_timestamp", "yyyy-MM-dd HH:mm:ss.SSS"] target => "app_timestamp" } }#end else }#end if [type] }#end filter output { opensearch { hosts => ["https://opensearchIP:9200"] user => "user" password => "pass" index => "index-host-app-%{+yyyy-MM-dd}" ssl_certificate_verification => false } }#end output ``` I was hoping I could separate this configuration per path that is configured in filebeat. So if line is read from file1 filebeat will send some tag to the logstash input, if from file2 some other tag. Then I will have separate input{} filter{} output{} for each path in filebeat filebeat.yml: ```auto paths: - app1/logs/*.log - tag this somehow as app1 - app2/logs/*.log - tag this somehow as app2 ``` logstash.config: ```auto input{ for app1 } input{ for app2 } filter { for app1 } filter { for app2 } output { for app1 } output { for app2 } ``` Is this even possible? If not what would be the alternative solution? Like I wrote logs differ in structure and it might not be possible to add appropriate tagging the way it is in your example. ---
**Author:** ![nateynate](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/nateynate/32/4913_2.png) [@nateynate](https://forum.opensearch.org/u/nateynate)\ **Post date:** [February 25, 2025, 5:23pm UTC](https://forum.opensearch.org/t/help-with-configuring-filebeat-and-logstash-for-multiple-input/23549/4 "2025-02-25T17:23:43Z")
> [@apt](#): > > Is this even possible? If not what would be the alternative solution? So I did a few web searches, as I’m not entirely familiar with what the raw events look like from filebeats. If I’m reading right, it looks like the events from filebeats contain an entry `[log][file][path]` that provides the filename that beats got the event from. You could definitely match on that and then route as necessary. Do you have an example of a raw message from beats? There might be other options upon which we could match and route. Nate ---
**Author:** ![apt](https://avatars.discourse-cdn.com/v4/letter/a/858c86/32.png) [@apt](https://forum.opensearch.org/u/apt)\ **Post date:** [February 26, 2025, 7:02am UTC](https://forum.opensearch.org/t/help-with-configuring-filebeat-and-logstash-for-multiple-input/23549/5 "2025-02-26T07:02:54Z")
@nateynate thanks again for your time. You are right this information is passed to the index: ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/5/5dff0e6476670c250c9fa6f2aa03867116557c1b.png) Still am not sure how to separate flow for two application using match and add\_tags. Could you please add this to your proposed example? Anyway I did something like this: In filebeat.yml ```auto - type: filestream enabled: true id: ID1 paths: - "/app1/logs/*xml.log" fields: app_log: "app1" tags: ["app1"] - type: filestream enabled: true id: ID2 paths: - "/app2/logs/*xml.log" fields: app_log: "app2" tags: ["app2"] ``` So there are a few possibilities to differ logs per application. In logstash.conf: ```auto input { beats { port => 5044 } } filter { if "app1" in [tags] { if [message] !~ / ERROR / { drop {} } else { grok { match => { 'message' => "%{DATA:app_timestamp} \[%{DATA:worker_number}\] %{LOGLEVEL:log_level} %{DATA:method}: %{GREEDYDATA:log} " } } date { match => ["app_timestamp", "yyyy-MM-dd HH:mm:ss.SSS"] target => "app_timestamp" } }#end else }#end if else if "app2" in [tags] { if [message] !~ / ERROR / { drop {} } else { grok { match => { 'message' => "%{DATA:app_timestamp} \[%{DATA:worker_number}\] %{LOGLEVEL:log_level} %{DATA:method}: %{GREEDYDATA:log} " } } date { match => ["app_timestamp", "yyyy-MM-dd HH:mm:ss.SSS"] target => "app_timestamp" } }#end else }#end else if }#end filter output { if "app1" in [tags] { opensearch { hosts => ["https://opensearchIP:9200"] user => "user" password => "pass" index => "app1-%{+yyyy-MM-dd}" ssl_certificate_verification => false } } if "app2" in [tags] { opensearch { hosts => ["https://opensearchIP:9200"] user => "user" password => "pass" index => "app2-%{+yyyy-MM-dd}" ssl_certificate_verification => false } } }#end output ``` I’m dropping anything but ERROR but this can be done in filebeat so I will test that too: ```auto include_lines: ['^ERR'] ``` ---
**Author:** ![nateynate](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/nateynate/32/4913_2.png) [@nateynate](https://forum.opensearch.org/u/nateynate)\ **Post date:** [February 26, 2025, 5:07pm UTC](https://forum.opensearch.org/t/help-with-configuring-filebeat-and-logstash-for-multiple-input/23549/6 "2025-02-26T17:07:09Z")
> [@apt](#): > > Could you please add this to your proposed example? I think it all would end up in the `output` section - ```auto output { if [log][file][path] == "/full/file/path" { opensearch { hosts => ["localhost:9200"] index => "someindex" ... } } } ```