Help with configuring filebeat and logstash

General Feedback
1

Hello,

I have two servers with multiple application that log to different locations.
Log differ in structure so I need to do at least a few filters in logstash.
On each server I have Filebeat installed.
On the third server I have Logstash and Opensearch installed.

What I want to achieve is to build a dashboard that will show errors per server, application type etc.

How to configure Filebeat/Logstash to separate logs so they will be send to different indices in Opensearch?

What would be the best approach? Could you please give me an example?

Regards!

2

Hi @apt !

I’ll see if I can’t write you some specific examples. Here’s how I’ve accomplished this. I can’t say if this is best practice or not.

  1. Add a filter{} section and use the grok filter to compare the message contents to a particular pattern. If so, add some tags to them.
  2. In your output {} sections, use more if {} conditionals to check for those tags, and based on their content, provide an index statement to specify a target index.

So suppose you’ve got an input from a file and you’re using the grok filter to check for a particular message layout (this may or may not be syntactically correct):

input {
  file {
    path => "/var/log/some.log"
  }
}

filter {
  grok {
    match => { your match pattern, see documentation }
    add_tag => "some_tag"
 }
}

# Specify a destination based on the tag you added in filter {}

output {
  if [tag] => "some_tag" {
    opensearch {
      opensearch plugin option (port, username, index name, etc., see documentation)
    }
  }
  else if [tag] => "some_other_tag" {
    opensearch {
      opensearch {
        opensearch plugin options with different index name
      }
    }
  }
} # end of output section

Youd’ likely have an input {} section that specifies the beats input plugin. Logstash is technically an Elastic product, so the specifics stanzas and incantations are available from their documentation. But, this oughtta get you started.

Let me know if there’s some specific challenge you’re trying to overcome. Or if you’re just trying to get started, that’s fine too. Don’t forget to share your solution!