Multiple Filebeat to logstash config help

OpenSearch
1

Hi Team,

I have almost 40 servers which I installed filebeat. I will send logs to logstash 5044 port. What is the best way to configure logstash for these multiple filebeat? Each logs will have different output.

I guess there are two way. 1-Configuring one logstash config file which have all config for each filebeat ( I will use tag or field to seperate the logs)
2-Create conf file for each server and add them to pipeline.yml. I do not understand what is the difference or advantages?

Thanks,

Emre

2

Hey @a.emrekaraman

Use one port i.e., 5044 not sure you need 40 + configuration files. You can sort out the different devices through fields that get created.

If that doesnt work and you need each 40 + devices to have a tag for sorting, then each one for those filebeat configuration files you can use…

processors:
  - add_tags:
      tags: [node-001]
4

Hi @Gsmitt

Thanks for suggestion. After adding tags for each filebeat config what you share, what about logstash config? Should I create one logstash.conf file and put input,filter,output for each server/tag with “if” operation.

5

Het @a.emrekaraman

Here is an example of my lab Logstash config.

root@ansible:/opt/logstash-8.6.1/config# cat logstash.conf
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
  beats {
    port => 5044
    tags => [ 'beat' ]
      }
}
input {
  udp {
    port => 5144
    tags => ['syslog']
  }
}



filter {

if "syslog" in [tags] {

grok {
      match => ["message", "%{SYSLOG5424PRI}%{GREEDYDATA:message}"]
      overwrite => [ "message" ]
        }


    kv {
       source => "message"
       value_split => "="
    }

   }
  }


filter {

if "syslog" in [tags] {
 mutate {
        remove_field => [ "addr","appcat","craction","crlevel","crscore","devtype","dstdevtype","dstosname","dstserver","dstserver","fazlograte","freediskstorage","interface","log.syslog.priority","masterdstmac","mastersrcmac","osname","policytype","poluuid","setuprate","srchwvendor","srcserver","total","totalsession","used","user","vd"]
  }
 }
}

output {
if "beat" in [tags] {
  opensearch {
    hosts => ["https://my_domain.com:9200"]
    auth_type => {
              type => 'basic'
              user => 'admin'
              password => 'changeit'
            }
    ecs_compatibility => disabled
    ssl => true
    #ssl_certificate_verification => false
    cacert => "/opt/logstash-8.6.1/root-ca.pem"
    #index => "winlogbeat-%{+YYYY.MM.dd}"
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
     }
  }
if "syslog" in [tags] {
          opensearch {
             hosts => ["https://my_doamin.com:9200"]
                        auth_type => {
                            type => 'basic'
                            user => 'admin'
                            password => 'changeit'
                          }
                        ecs_compatibility => disabled
                        ssl => true
                        #ssl_certificate_verification => false
                        cacert => "/opt/logstash-8.6.1/root-ca.pem"
                        index => "firewall-%{+YYYY.MM.dd}"
        }
    }
}

root@ansible:/opt/logstash-8.6.1/config#

I created two ports [5044, 5144] One is for my beat (i.e., Metricbeat,filebeat,winlogbeat) the other is for my firewall. Since the firewall comes in a block I used GROk to separate the log into fields.

All my beat are configured the same, you can either place a tag on individual instances or use
logstash to do in for you.

EDIT: it really depends on what you want to do. You could even use this example that comes with logstash.

root@ansible:/opt/logstash-8.6.1/config# cat logstash-sample.conf
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}
root@ansible:/opt/logstash-8.6.1/config#
6

Hi @Gsmitt

It really helps me . Thanks for explanation and examples. will use same kind of logic for my envirement :slight_smile:

Thanks

1 Like
7

Hey @a.emrekaraman

Glad to help :+1: