Logstash to Logstash configuration

OpenSearch
1

Hi Guys,

Can you please suggest a best way to configure logstash to logstash communication with SSL/TLS encryption.
There are some of the articles in logstash input and output plugin
lumberjack plugin but it did not work for me
some of them are deprecated.

Please advice me on this
Thank you

2

Hey @raj1209

Simple configuration for Logstash using Beats input

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  beats {
    port => 5044
  }
}

output {
  opensearch {
    hosts => ["https://opensearch.domain.com:9200"]
    auth_type => {
              type => 'basic'
              user => 'admin'
              password => 'changeit'
            }
    ecs_compatibility => disabled
    ssl => true   
    cacert => "/opt/logstash-8.6.1/root-ca.pem"
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"    
    }
}

Hope that helps

3

Hi @Gsmitt Thanks for the lightning response , appreciated
In my scenario, It requires the logstash to logstash communication and not filebeat to logstash

Recently we had a merger between two companies and they are using logstash as well and we are trying to receive logs from their logstash ( A ) to our logstash ( B ) and have a secure communication between these two and send this logs to Opensearch

4

Hey

I havent seen that done yet. We use Logstash send directly to Opensearch. I guess you could

Here is another example of an input you could use.

input {
  tcp {
    host       => "0.0.0.0"
    mode       => "server"
    port       => 5144
    ssl_enable => true
    ssl_cert   => "/etc/ssl/logstash.crt"
    ssl_key    => "/etc/ssl/ogstash.key"
    ssl_cacert => "/etc/ssl/certs/my_ca_cert.pem"
    ssl_verify => false
    type       => "syslog"
  }
}

Sorry Havent sent logs from Logstash to logstash.

5

One of the routes I would suggest investigating is shipping logs to an intermediary buffer like redis, kafka etc, then have the other logstash pull from there.
This avoids any issues if the last logstash in your chain is offline for whatever reason and should help prevent the shipping logstash from OOM’ing.

ie:

logstash_shipper → Buffer(redis etc) → Logstash_receiver → wherever.

6

Thank you for replying @jasonrojas
Suppose If I use tcp output plugin in logstash A and tcp input plugin in logstash B does it have any impact of data loss if one of the logstash chain goes down ?

7

You would have to test that to be certain. I think logstash will buffer to a point however those internal buffers will be limited to heap size and system memory etc.

8

Thank you for the suggestion