I have two servers with multiple application that log to different locations.
Log differ in structure so I need to do at least a few filters in logstash.
On each server I have Filebeat installed.
On the third server I have Logstash and Opensearch installed.
What I want to achieve is to build a dashboard that will show errors per server, application type etc.
How to configure Filebeat/Logstash to separate logs so they will be send to different indices in Opensearch?
What would be the best approach? Could you please give me an example?
I’ll see if I can’t write you some specific examples. Here’s how I’ve accomplished this. I can’t say if this is best practice or not.
Add a filter{} section and use the grok filter to compare the message contents to a particular pattern. If so, add some tags to them.
In your output {} sections, use more if {} conditionals to check for those tags, and based on their content, provide an index statement to specify a target index.
So suppose you’ve got an input from a file and you’re using the grok filter to check for a particular message layout (this may or may not be syntactically correct):
input {
file {
path => "/var/log/some.log"
}
}
filter {
grok {
match => { your match pattern, see documentation }
add_tag => "some_tag"
}
}
# Specify a destination based on the tag you added in filter {}
output {
if [tag] => "some_tag" {
opensearch {
opensearch plugin option (port, username, index name, etc., see documentation)
}
}
else if [tag] => "some_other_tag" {
opensearch {
opensearch {
opensearch plugin options with different index name
}
}
}
} # end of output section
Youd’ likely have an input {} section that specifies the beats input plugin. Logstash is technically an Elastic product, so the specifics stanzas and incantations are available from their documentation. But, this oughtta get you started.
Let me know if there’s some specific challenge you’re trying to overcome. Or if you’re just trying to get started, that’s fine too. Don’t forget to share your solution!
input {
beats {
port => 5044
type => app_logfile
}
}
filter {
if [type] == "app_logfile" {
if [message] !~ / ERROR / {
drop {}
}
else {
grok {
match => { 'message' => "%{DATA:app_timestamp} \[%{DATA:worker_number}\] %{LOGLEVEL:log_level} %{DATA:method}: %{GREEDYDATA:log} " }
}
date {
match => [ "app_timestamp", "yyyy-MM-dd HH:mm:ss.SSS"]
target => "app_timestamp"
}
}#end else
}#end if [type]
}#end filter
output {
opensearch {
hosts => ["https://opensearchIP:9200"]
user => "user"
password => "pass"
index => "index-host-app-%{+yyyy-MM-dd}"
ssl_certificate_verification => false
}
}#end output
I was hoping I could separate this configuration per path that is configured in filebeat.
So if line is read from file1 filebeat will send some tag to the logstash input, if from file2 some other tag. Then I will have separate input{} filter{} output{} for each path in filebeat
filebeat.yml:
paths:
- app1/logs/*.log - tag this somehow as app1
- app2/logs/*.log - tag this somehow as app2
logstash.config:
input{ for app1 }
input{ for app2 }
filter { for app1 }
filter { for app2 }
output { for app1 }
output { for app2 }
Is this even possible? If not what would be the alternative solution? Like I wrote logs differ in structure and it might not be possible to add appropriate tagging the way it is in your example.
So I did a few web searches, as I’m not entirely familiar with what the raw events look like from filebeats. If I’m reading right, it looks like the events from filebeats contain an entry [log][file][path] that provides the filename that beats got the event from.
You could definitely match on that and then route as necessary.
Do you have an example of a raw message from beats? There might be other options upon which we could match and route.
